librelist archives

« back to archive

Exit when queue is empty or reaches threshold

Exit when queue is empty or reaches threshold

From:
Adam Tucker
Date:
2010-10-19 @ 02:12
I will look into the feasibility of implementing this in a few months when
current projects quiet down, but I wanted to toss out the idea in case
anyone has already done something similar.

We sometimes run into situations where a queue gets hit a lot harder than
normal. The spike is transient, and I usually just add another worker or
two. Once the storm passes I manually go in and `kill -QUIT` the extraneous
workers. I love the way Resque uses signal traps, and would like to see a
way to tell the worker to process jobs until a certain threshold, then quit.
I think it would be easiest to implement an "exit when queue is empty"
function, but "exit when queued_jobs <= 500" would be more practical for
sites whose queues rarely hit zero.

-Adam

Re: [resque] Exit when queue is empty or reaches threshold

From:
Luke Antins
Date:
2010-10-19 @ 02:31
I looked at this plugin the other day: 
http://github.com/iwarshak/resque-batchworker
Seems to do exactly what you need? fire up one or more workers, have 
them exit when the queue is empty.

FWIW:
I've recently started using a version of this plugin: 
http://github.com/staugaard/resque-multi-job-forks
Forking before every job is a significant overhead for two of my queues 
(tiny/small jobs).
I've not had these queues backup since!
Infact its awesomely speedy at chewing through jobs now! For this 
situation, for me, anyhow...

=)

Luke

> I will look into the feasibility of implementing this in a few months 
> when current projects quiet down, but I wanted to toss out the idea in 
> case anyone has already done something similar.
>
> We sometimes run into situations where a queue gets hit a lot harder 
> than normal. The spike is transient, and I usually just add another 
> worker or two. Once the storm passes I manually go in and `kill -QUIT` 
> the extraneous workers. I love the way Resque uses signal traps, and 
> would like to see a way to tell the worker to process jobs until a 
> certain threshold, then quit. I think it would be easiest to implement 
> an "exit when queue is empty" function, but "exit when queued_jobs <= 
> 500" would be more practical for sites whose queues rarely hit zero.
>
> -Adam

Re: [resque] Exit when queue is empty or reaches threshold

From:
Kunal Shah
Date:
2010-10-19 @ 03:16
fwiw multijobs per fork caused immediate problems for me. It may have  
beem how im setting up my pg connection though.

I've used the daemons library to control my resque workers to great  
success (55 mm jobs and counting).

To increase queue velocity we found the best thing to do was decrease  
the polling interval and have no more workers than cpu's for a very  
active queue. On an ec2 c1.xlarge instance this means 8 workers.

side note -We ran into a special problem where we needed to monitor  
the db's lock count and though we ultimately did not need to we  
considered using the prefork hook to check that count and sleep if it  
was too high.

K


On Oct 18, 2010, at 10:31 PM, Luke Antins <luke@lividpenguin.com> wrote:

> I looked at this plugin the other day:
> http://github.com/iwarshak/resque-batchworker
> Seems to do exactly what you need? fire up one or more workers, have
> them exit when the queue is empty.
>
> FWIW:
> I've recently started using a version of this plugin:
> http://github.com/staugaard/resque-multi-job-forks
> Forking before every job is a significant overhead for two of my  
> queues
> (tiny/small jobs).
> I've not had these queues backup since!
> Infact its awesomely speedy at chewing through jobs now! For this
> situation, for me, anyhow...
>
> =)
>
> Luke
>
>> I will look into the feasibility of implementing this in a few months
>> when current projects quiet down, but I wanted to toss out the idea  
>> in
>> case anyone has already done something similar.
>>
>> We sometimes run into situations where a queue gets hit a lot harder
>> than normal. The spike is transient, and I usually just add another
>> worker or two. Once the storm passes I manually go in and `kill - 
>> QUIT`
>> the extraneous workers. I love the way Resque uses signal traps, and
>> would like to see a way to tell the worker to process jobs until a
>> certain threshold, then quit. I think it would be easiest to  
>> implement
>> an "exit when queue is empty" function, but "exit when queued_jobs <=
>> 500" would be more practical for sites whose queues rarely hit zero.
>>
>> -Adam

Re: [resque] Exit when queue is empty or reaches threshold

From:
Adam Tucker
Date:
2010-10-19 @ 03:22
I looked at multijobs a few weeks ago, but it required an older version of
Resque, and based on the last commit I assumed it was no longer under active
development. Batchworker looks almost ideal for my situation, thanks for the
link!

-A

On Mon, Oct 18, 2010 at 11:16 PM, Kunal Shah <kunal@dropio.com> wrote:

> fwiw multijobs per fork caused immediate problems for me. It may have
> beem how im setting up my pg connection though.
>
> I've used the daemons library to control my resque workers to great
> success (55 mm jobs and counting).
>
> To increase queue velocity we found the best thing to do was decrease
> the polling interval and have no more workers than cpu's for a very
> active queue. On an ec2 c1.xlarge instance this means 8 workers.
>
> side note -We ran into a special problem where we needed to monitor
> the db's lock count and though we ultimately did not need to we
> considered using the prefork hook to check that count and sleep if it
> was too high.
>
> K
>
>
> On Oct 18, 2010, at 10:31 PM, Luke Antins <luke@lividpenguin.com> wrote:
>
> > I looked at this plugin the other day:
> > http://github.com/iwarshak/resque-batchworker
> > Seems to do exactly what you need? fire up one or more workers, have
> > them exit when the queue is empty.
> >
> > FWIW:
> > I've recently started using a version of this plugin:
> > http://github.com/staugaard/resque-multi-job-forks
> > Forking before every job is a significant overhead for two of my
> > queues
> > (tiny/small jobs).
> > I've not had these queues backup since!
> > Infact its awesomely speedy at chewing through jobs now! For this
> > situation, for me, anyhow...
> >
> > =)
> >
> > Luke
> >
> >> I will look into the feasibility of implementing this in a few months
> >> when current projects quiet down, but I wanted to toss out the idea
> >> in
> >> case anyone has already done something similar.
> >>
> >> We sometimes run into situations where a queue gets hit a lot harder
> >> than normal. The spike is transient, and I usually just add another
> >> worker or two. Once the storm passes I manually go in and `kill -
> >> QUIT`
> >> the extraneous workers. I love the way Resque uses signal traps, and
> >> would like to see a way to tell the worker to process jobs until a
> >> certain threshold, then quit. I think it would be easiest to
> >> implement
> >> an "exit when queue is empty" function, but "exit when queued_jobs <=
> >> 500" would be more practical for sites whose queues rarely hit zero.
> >>
> >> -Adam
>

Re: [resque] Exit when queue is empty or reaches threshold

From:
Luke Antins
Date:
2010-10-19 @ 04:12
Several forks of resque-multi-job-forks work on later versions of resque.
I ended up forking it myself and updating it for v1.9.10 upwards: 
http://github.com/lantins/resque-multi-job-forks/tree/personal_gem

Luke
> I looked at multijobs a few weeks ago, but it required an older 
> version of Resque, and based on the last commit I assumed it was no 
> longer under active development. Batchworker looks almost ideal for my 
> situation, thanks for the link!
>
> -A
>
> On Mon, Oct 18, 2010 at 11:16 PM, Kunal Shah <kunal@dropio.com 
> <mailto:kunal@dropio.com>> wrote:
>
>     fwiw multijobs per fork caused immediate problems for me. It may have
>     beem how im setting up my pg connection though.
>
>     I've used the daemons library to control my resque workers to great
>     success (55 mm jobs and counting).
>
>     To increase queue velocity we found the best thing to do was decrease
>     the polling interval and have no more workers than cpu's for a very
>     active queue. On an ec2 c1.xlarge instance this means 8 workers.
>
>     side note -We ran into a special problem where we needed to monitor
>     the db's lock count and though we ultimately did not need to we
>     considered using the prefork hook to check that count and sleep if it
>     was too high.
>
>     K
>
>
>     On Oct 18, 2010, at 10:31 PM, Luke Antins <luke@lividpenguin.com
>     <mailto:luke@lividpenguin.com>> wrote:
>
>     > I looked at this plugin the other day:
>     > http://github.com/iwarshak/resque-batchworker
>     > Seems to do exactly what you need? fire up one or more workers, have
>     > them exit when the queue is empty.
>     >
>     > FWIW:
>     > I've recently started using a version of this plugin:
>     > http://github.com/staugaard/resque-multi-job-forks
>     > Forking before every job is a significant overhead for two of my
>     > queues
>     > (tiny/small jobs).
>     > I've not had these queues backup since!
>     > Infact its awesomely speedy at chewing through jobs now! For this
>     > situation, for me, anyhow...
>     >
>     > =)
>     >
>     > Luke
>     >
>     >> I will look into the feasibility of implementing this in a few
>     months
>     >> when current projects quiet down, but I wanted to toss out the idea
>     >> in
>     >> case anyone has already done something similar.
>     >>
>     >> We sometimes run into situations where a queue gets hit a lot
>     harder
>     >> than normal. The spike is transient, and I usually just add another
>     >> worker or two. Once the storm passes I manually go in and `kill -
>     >> QUIT`
>     >> the extraneous workers. I love the way Resque uses signal
>     traps, and
>     >> would like to see a way to tell the worker to process jobs until a
>     >> certain threshold, then quit. I think it would be easiest to
>     >> implement
>     >> an "exit when queue is empty" function, but "exit when
>     queued_jobs <=
>     >> 500" would be more practical for sites whose queues rarely hit
>     zero.
>     >>
>     >> -Adam
>
>

Re: [resque] Exit when queue is empty or reaches threshold

From:
Luke Antins
Date:
2010-10-19 @ 04:05
>
> fwiw multijobs per fork caused immediate problems for me. It may have
> beem how im setting up my pg connection though.
Oh?
> I've used the daemons library to control my resque workers to great
> success (55 mm jobs and counting).
Daemon polls queue lengths, if its above x fire off y workers? or 
something to that effect?
If so... I like it! =)

Luke
> To increase queue velocity we found the best thing to do was decrease
> the polling interval and have no more workers than cpu's for a very
> active queue. On an ec2 c1.xlarge instance this means 8 workers.
> side note -We ran into a special problem where we needed to monitor
> the db's lock count and though we ultimately did not need to we
> considered using the prefork hook to check that count and sleep if it
> was too high.
>
> K
>
>
> On Oct 18, 2010, at 10:31 PM, Luke Antins<luke@lividpenguin.com>  wrote:
>
>    
>> I looked at this plugin the other day:
>> http://github.com/iwarshak/resque-batchworker
>> Seems to do exactly what you need? fire up one or more workers, have
>> them exit when the queue is empty.
>>
>> FWIW:
>> I've recently started using a version of this plugin:
>> http://github.com/staugaard/resque-multi-job-forks
>> Forking before every job is a significant overhead for two of my
>> queues
>> (tiny/small jobs).
>> I've not had these queues backup since!
>> Infact its awesomely speedy at chewing through jobs now! For this
>> situation, for me, anyhow...
>>
>> =)
>>
>> Luke
>>
>>      
>>> I will look into the feasibility of implementing this in a few months
>>> when current projects quiet down, but I wanted to toss out the idea
>>> in
>>> case anyone has already done something similar.
>>>
>>> We sometimes run into situations where a queue gets hit a lot harder
>>> than normal. The spike is transient, and I usually just add another
>>> worker or two. Once the storm passes I manually go in and `kill -
>>> QUIT`
>>> the extraneous workers. I love the way Resque uses signal traps, and
>>> would like to see a way to tell the worker to process jobs until a
>>> certain threshold, then quit. I think it would be easiest to
>>> implement
>>> an "exit when queue is empty" function, but "exit when queued_jobs<=
>>> 500" would be more practical for sites whose queues rarely hit zero.
>>>
>>> -Adam
>>>