librelist archives

« back to archive

resque-scheduler recurring jobs

resque-scheduler recurring jobs

From:
Karl Baum
Date:
2010-02-18 @ 16:45
I would like to have a job regularly running every minute or so.  I  
think the cron like Recurring scheduled job in resque-scheduler might  
be the right way to go, but I would like to guarantee that no more  
than one instance of my job is running at a time.  I know this can be  
a problem with plain *nix cron.  Does resque-scheduler account for this?

thx.

-karl

Re: [resque] resque-scheduler recurring jobs

From:
Philippe Lafoucrière
Date:
2010-02-18 @ 20:17
On Thu, Feb 18, 2010 at 5:45 PM, Karl Baum <karl.baum@gmail.com> wrote:
> I would like to have a job regularly running every minute or so.  I
> think the cron like Recurring scheduled job in resque-scheduler might
> be the right way to go, but I would like to guarantee that no more
> than one instance of my job is running at a time.  I know this can be
> a problem with plain *nix cron.  Does resque-scheduler account for this?

Hi,

we had this discussion with bvandenbos a few days ago. There should be
only one instance of the scheduler, since this one will push on top of
the queue the scheduled job.
You may check this discussion :
http://github.com/bvandenbos/resque-scheduler/issues/closed
for the complete details and explanation from bvandendos.

Regards

Re: [resque] resque-scheduler recurring jobs

From:
Karl Baum
Date:
2010-02-18 @ 20:36
Which closed issue is it?  That link just brought me to the list of  
closed issues.

Thanks.

-karl

On Feb 18, 2010, at 3:17 PM, Philippe Lafoucrière wrote:

> On Thu, Feb 18, 2010 at 5:45 PM, Karl Baum <karl.baum@gmail.com>  
> wrote:
>> I would like to have a job regularly running every minute or so.  I
>> think the cron like Recurring scheduled job in resque-scheduler might
>> be the right way to go, but I would like to guarantee that no more
>> than one instance of my job is running at a time.  I know this can be
>> a problem with plain *nix cron.  Does resque-scheduler account for  
>> this?
>
> Hi,
>
> we had this discussion with bvandenbos a few days ago. There should be
> only one instance of the scheduler, since this one will push on top of
> the queue the scheduled job.
> You may check this discussion :
> http://github.com/bvandenbos/resque-scheduler/issues/closed
> for the complete details and explanation from bvandendos.
>
> Regards

Re: [resque] resque-scheduler recurring jobs

From:
Philippe Lafoucrière
Date:
2010-02-18 @ 20:39
Sorry, Ajax got me :)
http://github.com/bvandenbos/resque-scheduler/issues/closed/#issue/7

It's issue 7!
Cheers

Sent from my iPhone

On 18 févr. 2010, at 21:36, Karl Baum <karl.baum@gmail.com> wrote:

> Which closed issue is it?  That link just brought me to the list of
> closed issues.
>
> Thanks.
>
> -karl
>
> On Feb 18, 2010, at 3:17 PM, Philippe Lafoucrière wrote:
>
>> On Thu, Feb 18, 2010 at 5:45 PM, Karl Baum <karl.baum@gmail.com>
>> wrote:
>>> I would like to have a job regularly running every minute or so.  I
>>> think the cron like Recurring scheduled job in resque-scheduler  
>>> might
>>> be the right way to go, but I would like to guarantee that no more
>>> than one instance of my job is running at a time.  I know this can  
>>> be
>>> a problem with plain *nix cron.  Does resque-scheduler account for
>>> this?
>>
>> Hi,
>>
>> we had this discussion with bvandenbos a few days ago. There should  
>> be
>> only one instance of the scheduler, since this one will push on top  
>> of
>> the queue the scheduled job.
>> You may check this discussion :
>> http://github.com/bvandenbos/resque-scheduler/issues/closed
>> for the complete details and explanation from bvandendos.
>>
>> Regards
>

Re: [resque] resque-scheduler recurring jobs

From:
Karl Baum
Date:
2010-02-18 @ 20:46
I am probably missing it, but I can't seem to figure out where this  
post goes into whether the scheduler will not allow more than two jobs  
to run at the same time.

Thanks again!

-karl
On Feb 18, 2010, at 3:39 PM, Philippe Lafoucrière wrote:

> Sorry, Ajax got me :)
> http://github.com/bvandenbos/resque-scheduler/issues/closed/#issue/7
>
> It's issue 7!
> Cheers
>
> Sent from my iPhone
>
> On 18 févr. 2010, at 21:36, Karl Baum <karl.baum@gmail.com> wrote:
>
>> Which closed issue is it?  That link just brought me to the list of
>> closed issues.
>>
>> Thanks.
>>
>> -karl
>>
>> On Feb 18, 2010, at 3:17 PM, Philippe Lafoucrière wrote:
>>
>>> On Thu, Feb 18, 2010 at 5:45 PM, Karl Baum <karl.baum@gmail.com>
>>> wrote:
>>>> I would like to have a job regularly running every minute or so.  I
>>>> think the cron like Recurring scheduled job in resque-scheduler
>>>> might
>>>> be the right way to go, but I would like to guarantee that no more
>>>> than one instance of my job is running at a time.  I know this can
>>>> be
>>>> a problem with plain *nix cron.  Does resque-scheduler account for
>>>> this?
>>>
>>> Hi,
>>>
>>> we had this discussion with bvandenbos a few days ago. There should
>>> be
>>> only one instance of the scheduler, since this one will push on top
>>> of
>>> the queue the scheduled job.
>>> You may check this discussion :
>>> http://github.com/bvandenbos/resque-scheduler/issues/closed
>>> for the complete details and explanation from bvandendos.
>>>
>>> Regards
>>

Re: [resque] resque-scheduler recurring jobs

From:
Michael Russo
Date:
2010-02-18 @ 21:13
The linked issue isn't related to the original poster's question.

Karl, I just took a look at the source for the scheduler and it simply fires
jobs into resque on a timer.  If you have a recurring job 'X' every minute
and no workers running (against the job's particular queue), then when you
add a single worker, you'll have a backlog of 'X's to process.  If you turn
on more than one worker, then X will likely be processed in parallel by two
different workers.

-Michael

On Thu, Feb 18, 2010 at 3:46 PM, Karl Baum <karl.baum@gmail.com> wrote:

> I am probably missing it, but I can't seem to figure out where this
> post goes into whether the scheduler will not allow more than two jobs
> to run at the same time.
>
> Thanks again!
>
> -karl
> On Feb 18, 2010, at 3:39 PM, Philippe Lafoucrière wrote:
>
> > Sorry, Ajax got me :)
> > http://github.com/bvandenbos/resque-scheduler/issues/closed/#issue/7
> >
> > It's issue 7!
> > Cheers
> >
> > Sent from my iPhone
> >
> > On 18 févr. 2010, at 21:36, Karl Baum <karl.baum@gmail.com> wrote:
> >
> >> Which closed issue is it?  That link just brought me to the list of
> >> closed issues.
> >>
> >> Thanks.
> >>
> >> -karl
> >>
> >> On Feb 18, 2010, at 3:17 PM, Philippe Lafoucrière wrote:
> >>
> >>> On Thu, Feb 18, 2010 at 5:45 PM, Karl Baum <karl.baum@gmail.com>
> >>> wrote:
> >>>> I would like to have a job regularly running every minute or so.  I
> >>>> think the cron like Recurring scheduled job in resque-scheduler
> >>>> might
> >>>> be the right way to go, but I would like to guarantee that no more
> >>>> than one instance of my job is running at a time.  I know this can
> >>>> be
> >>>> a problem with plain *nix cron.  Does resque-scheduler account for
> >>>> this?
> >>>
> >>> Hi,
> >>>
> >>> we had this discussion with bvandenbos a few days ago. There should
> >>> be
> >>> only one instance of the scheduler, since this one will push on top
> >>> of
> >>> the queue the scheduled job.
> >>> You may check this discussion :
> >>> http://github.com/bvandenbos/resque-scheduler/issues/closed
> >>> for the complete details and explanation from bvandendos.
> >>>
> >>> Regards
> >>
>
>

Re: [resque] resque-scheduler recurring jobs

From:
Karl Baum
Date:
2010-02-18 @ 21:28
Makes sense.  In my case, I want only one job to run again 1 minute  
after it completes.  I am guessing I should use the Delayed jobs  
option.  Something like:

Resque.enqueue_at(1.minute_from_now, self)

How are other people getting this done?

thx.

-karl

On Feb 18, 2010, at 4:13 PM, Michael Russo wrote:

> The linked issue isn't related to the original poster's question.
>
> Karl, I just took a look at the source for the scheduler and it  
> simply fires jobs into resque on a timer.  If you have a recurring  
> job 'X' every minute and no workers running (against the job's  
> particular queue), then when you add a single worker, you'll have a  
> backlog of 'X's to process.  If you turn on more than one worker,  
> then X will likely be processed in parallel by two different workers.
>
> -Michael
>
> On Thu, Feb 18, 2010 at 3:46 PM, Karl Baum <karl.baum@gmail.com>  
> wrote:
> I am probably missing it, but I can't seem to figure out where this
> post goes into whether the scheduler will not allow more than two jobs
> to run at the same time.
>
> Thanks again!
>
> -karl
> On Feb 18, 2010, at 3:39 PM, Philippe Lafoucrière wrote:
>
> > Sorry, Ajax got me :)
> > http://github.com/bvandenbos/resque-scheduler/issues/closed/#issue/7
> >
> > It's issue 7!
> > Cheers
> >
> > Sent from my iPhone
> >
> > On 18 févr. 2010, at 21:36, Karl Baum <karl.baum@gmail.com> wrote:
> >
> >> Which closed issue is it?  That link just brought me to the list of
> >> closed issues.
> >>
> >> Thanks.
> >>
> >> -karl
> >>
> >> On Feb 18, 2010, at 3:17 PM, Philippe Lafoucrière wrote:
> >>
> >>> On Thu, Feb 18, 2010 at 5:45 PM, Karl Baum <karl.baum@gmail.com>
> >>> wrote:
> >>>> I would like to have a job regularly running every minute or  
> so.  I
> >>>> think the cron like Recurring scheduled job in resque-scheduler
> >>>> might
> >>>> be the right way to go, but I would like to guarantee that no  
> more
> >>>> than one instance of my job is running at a time.  I know this  
> can
> >>>> be
> >>>> a problem with plain *nix cron.  Does resque-scheduler account  
> for
> >>>> this?
> >>>
> >>> Hi,
> >>>
> >>> we had this discussion with bvandenbos a few days ago. There  
> should
> >>> be
> >>> only one instance of the scheduler, since this one will push on  
> top
> >>> of
> >>> the queue the scheduled job.
> >>> You may check this discussion :
> >>> http://github.com/bvandenbos/resque-scheduler/issues/closed
> >>> for the complete details and explanation from bvandendos.
> >>>
> >>> Regards
> >>
>
>

Re: [resque] resque-scheduler recurring jobs

From:
Philippe Lafoucrière
Date:
2010-02-18 @ 22:06
You can take a look at :
http://gist.github.com/299623

Sent from my iPhone

On 18 févr. 2010, at 22:28, Karl Baum <karl.baum@gmail.com> wrote:

> Makes sense.  In my case, I want only one job to run again 1 minute  
> after it completes.  I am guessing I should use the Delayed jobs  
> option.  Something like:
>
> Resque.enqueue_at(1.minute_from_now, self)
>
> How are other people getting this done?
>
> thx.
>
> -karl
>
> On Feb 18, 2010, at 4:13 PM, Michael Russo wrote:
>
>> The linked issue isn't related to the original poster's question.
>>
>> Karl, I just took a look at the source for the scheduler and it  
>> simply fires jobs into resque on a timer.  If you have a recurring  
>> job 'X' every minute and no workers running (against the job's  
>> particular queue), then when you add a single worker, you'll have a  
>> backlog of 'X's to process.  If you turn on more than one worker,  
>> then X will likely be processed in parallel by two different workers.
>>
>> -Michael
>>
>> On Thu, Feb 18, 2010 at 3:46 PM, Karl Baum <karl.baum@gmail.com>  
>> wrote:
>> I am probably missing it, but I can't seem to figure out where this
>> post goes into whether the scheduler will not allow more than two  
>> jobs
>> to run at the same time.
>>
>> Thanks again!
>>
>> -karl
>> On Feb 18, 2010, at 3:39 PM, Philippe Lafoucrière wrote:
>>
>> > Sorry, Ajax got me :)
>> > http://github.com/bvandenbos/resque-scheduler/issues/closed/ 
>> #issue/7
>> >
>> > It's issue 7!
>> > Cheers
>> >
>> > Sent from my iPhone
>> >
>> > On 18 févr. 2010, at 21:36, Karl Baum <karl.baum@gmail.com> wrote:
>> >
>> >> Which closed issue is it?  That link just brought me to the list  
>> of
>> >> closed issues.
>> >>
>> >> Thanks.
>> >>
>> >> -karl
>> >>
>> >> On Feb 18, 2010, at 3:17 PM, Philippe Lafoucrière wrote:
>> >>
>> >>> On Thu, Feb 18, 2010 at 5:45 PM, Karl Baum <karl.baum@gmail.com>
>> >>> wrote:
>> >>>> I would like to have a job regularly running every minute or  
>> so.  I
>> >>>> think the cron like Recurring scheduled job in resque-scheduler
>> >>>> might
>> >>>> be the right way to go, but I would like to guarantee that no  
>> more
>> >>>> than one instance of my job is running at a time.  I know this  
>> can
>> >>>> be
>> >>>> a problem with plain *nix cron.  Does resque-scheduler account  
>> for
>> >>>> this?
>> >>>
>> >>> Hi,
>> >>>
>> >>> we had this discussion with bvandenbos a few days ago. There  
>> should
>> >>> be
>> >>> only one instance of the scheduler, since this one will push on  
>> top
>> >>> of
>> >>> the queue the scheduled job.
>> >>> You may check this discussion :
>> >>> http://github.com/bvandenbos/resque-scheduler/issues/closed
>> >>> for the complete details and explanation from bvandendos.
>> >>>
>> >>> Regards
>> >>
>>
>>
>

Re: [resque] resque-scheduler recurring jobs

From:
Karl Baum
Date:
2010-02-18 @ 22:32
I noticed that you have retry logic within your job.  Does resque not  
have retry logic baked in?

Thanks!

-karl


On Feb 18, 2010, at 5:06 PM, Philippe Lafoucrière wrote:

> You can take a look at :
> http://gist.github.com/299623
>
> Sent from my iPhone
>
> On 18 févr. 2010, at 22:28, Karl Baum <karl.baum@gmail.com> wrote:
>
>> Makes sense.  In my case, I want only one job to run again 1 minute  
>> after it completes.  I am guessing I should use the Delayed jobs  
>> option.  Something like:
>>
>> Resque.enqueue_at(1.minute_from_now, self)
>>
>> How are other people getting this done?
>>
>> thx.
>>
>> -karl
>>
>> On Feb 18, 2010, at 4:13 PM, Michael Russo wrote:
>>
>>> The linked issue isn't related to the original poster's question.
>>>
>>> Karl, I just took a look at the source for the scheduler and it  
>>> simply fires jobs into resque on a timer.  If you have a recurring  
>>> job 'X' every minute and no workers running (against the job's  
>>> particular queue), then when you add a single worker, you'll have  
>>> a backlog of 'X's to process.  If you turn on more than one  
>>> worker, then X will likely be processed in parallel by two  
>>> different workers.
>>>
>>> -Michael
>>>
>>> On Thu, Feb 18, 2010 at 3:46 PM, Karl Baum <karl.baum@gmail.com>  
>>> wrote:
>>> I am probably missing it, but I can't seem to figure out where this
>>> post goes into whether the scheduler will not allow more than two  
>>> jobs
>>> to run at the same time.
>>>
>>> Thanks again!
>>>
>>> -karl
>>> On Feb 18, 2010, at 3:39 PM, Philippe Lafoucrière wrote:
>>>
>>> > Sorry, Ajax got me :)
>>> > http://github.com/bvandenbos/resque-scheduler/issues/closed/#issue/ 
>>> 7
>>> >
>>> > It's issue 7!
>>> > Cheers
>>> >
>>> > Sent from my iPhone
>>> >
>>> > On 18 févr. 2010, at 21:36, Karl Baum <karl.baum@gmail.com> wrote:
>>> >
>>> >> Which closed issue is it?  That link just brought me to the  
>>> list of
>>> >> closed issues.
>>> >>
>>> >> Thanks.
>>> >>
>>> >> -karl
>>> >>
>>> >> On Feb 18, 2010, at 3:17 PM, Philippe Lafoucrière wrote:
>>> >>
>>> >>> On Thu, Feb 18, 2010 at 5:45 PM, Karl Baum <karl.baum@gmail.com>
>>> >>> wrote:
>>> >>>> I would like to have a job regularly running every minute or  
>>> so.  I
>>> >>>> think the cron like Recurring scheduled job in resque-scheduler
>>> >>>> might
>>> >>>> be the right way to go, but I would like to guarantee that no  
>>> more
>>> >>>> than one instance of my job is running at a time.  I know  
>>> this can
>>> >>>> be
>>> >>>> a problem with plain *nix cron.  Does resque-scheduler  
>>> account for
>>> >>>> this?
>>> >>>
>>> >>> Hi,
>>> >>>
>>> >>> we had this discussion with bvandenbos a few days ago. There  
>>> should
>>> >>> be
>>> >>> only one instance of the scheduler, since this one will push  
>>> on top
>>> >>> of
>>> >>> the queue the scheduled job.
>>> >>> You may check this discussion :
>>> >>> http://github.com/bvandenbos/resque-scheduler/issues/closed
>>> >>> for the complete details and explanation from bvandendos.
>>> >>>
>>> >>> Regards
>>> >>
>>>
>>>
>>

Re: [resque] resque-scheduler recurring jobs

From:
Philippe Lafoucrière
Date:
2010-02-19 @ 10:30
On Thu, Feb 18, 2010 at 11:32 PM, Karl Baum <karl.baum@gmail.com> wrote:
> I noticed that you have retry logic within your job.  Does resque not have
> retry logic baked in?
> Thanks!

Nope, and retry makes only sense with resque-scheduler, otherwise you
will loop instantly (almost)

Re: [resque] resque-scheduler recurring jobs

From:
Philippe Lafoucrière
Date:
2010-02-18 @ 22:05

Sent from my iPhone

On 18 févr. 2010, at 22:13, Michael Russo <mjrusso@gmail.com> wrote:

> The linked issue isn't related to the original poster's question.
>

You should read all the comments for the issue :
Basically, no more than 1 scheduler must run, otherwise jobs are  
queued more than once. For this particular case, yes, i think dj is  
more simple since scheduled jobs are handled by the "regular" workers,  
like any other Job.


> Karl, I just took a look at the source for the scheduler and it  
> simply fires jobs into resque on a timer.  If you have a recurring  
> job 'X' every minute and no workers running (against the job's  
> particular queue), then when you add a single worker, you'll have a  
> backlog of 'X's to process.  If you turn on more than one worker,  
> then X will likely be processed in parallel by two different workers.
>
> -Michael
>
> On Thu, Feb 18, 2010 at 3:46 PM, Karl Baum <karl.baum@gmail.com>  
> wrote:
> I am probably missing it, but I can't seem to figure out where this
> post goes into whether the scheduler will not allow more than two jobs
> to run at the same time.
>
> Thanks again!
>
> -karl
> On Feb 18, 2010, at 3:39 PM, Philippe Lafoucrière wrote:
>
> > Sorry, Ajax got me :)
> > http://github.com/bvandenbos/resque-scheduler/issues/closed/#issue/7
> >
> > It's issue 7!
> > Cheers
> >
> > Sent from my iPhone
> >
> > On 18 févr. 2010, at 21:36, Karl Baum <karl.baum@gmail.com> wrote:
> >
> >> Which closed issue is it?  That link just brought me to the list of
> >> closed issues.
> >>
> >> Thanks.
> >>
> >> -karl
> >>
> >> On Feb 18, 2010, at 3:17 PM, Philippe Lafoucrière wrote:
> >>
> >>> On Thu, Feb 18, 2010 at 5:45 PM, Karl Baum <karl.baum@gmail.com>
> >>> wrote:
> >>>> I would like to have a job regularly running every minute or  
> so.  I
> >>>> think the cron like Recurring scheduled job in resque-scheduler
> >>>> might
> >>>> be the right way to go, but I would like to guarantee that no  
> more
> >>>> than one instance of my job is running at a time.  I know this  
> can
> >>>> be
> >>>> a problem with plain *nix cron.  Does resque-scheduler account  
> for
> >>>> this?
> >>>
> >>> Hi,
> >>>
> >>> we had this discussion with bvandenbos a few days ago. There  
> should
> >>> be
> >>> only one instance of the scheduler, since this one will push on  
> top
> >>> of
> >>> the queue the scheduled job.
> >>> You may check this discussion :
> >>> http://github.com/bvandenbos/resque-scheduler/issues/closed
> >>> for the complete details and explanation from bvandendos.
> >>>
> >>> Regards
> >>
>
>

Re: [resque] resque-scheduler recurring jobs

From:
Michael Russo
Date:
2010-02-18 @ 23:29
I read the comments again and they still aren't relevant.

It's like me answering Karl's original question by stating: "Don't fork a
couple of times in the resque worker's child process and you should be good
to go!"

There are multiple ways for the same type of job to be executed at the same
time, even if you exclude the really, really bad ideas that could make this
happen -- like forking in the perform method or running multiple schedulers
in tandem.

Using enqueue_at could help, but without other measures in place it isn't
bulletproof.

-Michael

2010/2/18 Philippe Lafoucrière <philippe.lafoucriere@gmail.com>

>
>
> You should read all the comments for the issue :
> Basically, no more than 1 scheduler must run, otherwise jobs are queued
> more than once. For this particular case, yes, i think dj is more simple
> since scheduled jobs are handled by the "regular" workers, like any other
> Job.
>
>
>>
>

Re: [resque] resque-scheduler recurring jobs

From:
Philippe Lafoucrière
Date:
2010-02-19 @ 08:55
On Fri, Feb 19, 2010 at 12:29 AM, Michael Russo <mjrusso@gmail.com> wrote:
> I read the comments again and they still aren't relevant.
> It's like me answering Karl's original question by stating: "Don't fork a
> couple of times in the resque worker's child process and you should be good
> to go!"

Yes, is doesn't say anything more than that:
"Since you wouldn't want to run the scheduler on all worker machines
(and having >1 scheduler would be bad)"
=> for scheduled tasks (not delayed), 2 schedulers means 2 tasks.
Sorry, that's the way I understand the question.