librelist archives

« back to archive

What queues do you have?

What queues do you have?

From:
Tony Arcieri
Date:
2010-03-30 @ 06:04
When I started off using Resque I just shoved everything into a "default"
queue since I wasn't quite sure how to split up jobs into different queues.

Now I'm thinking it might be good to separate CPU-bound jobs (where I want 1
worker per CPU) from I/O bound jobs (where I can have many more workers than
I have CPUs)

Is this a common practice?  How do people normally split up their queues?

-- 
Tony Arcieri
Medioh! A Kudelski Brand

Re: [resque] What queues do you have?

From:
Chris Wanstrath
Date:
2010-03-30 @ 06:47
On Mon, Mar 29, 2010 at 11:04 PM, Tony Arcieri <tony@medioh.com> wrote:

> Now I'm thinking it might be good to separate CPU-bound jobs (where I want 1
> worker per CPU) from I/O bound jobs (where I can have many more workers than
> I have CPUs)

That seems like a great use-case for me.

> Is this a common practice?  How do people normally split up their queues?

http://img.skitch.com/20100330-cj9xtq154jepcsh3qnepi8n9ew.png

We use queues mostly for locality (e.g. only this machine or this
class of machine may run this job) or priority (e.g. these jobs are
time sensitive, important, or disposable).

-- 
Chris Wanstrath
http://github.com/defunkt

Re: [resque] What queues do you have?

From:
Tim Haines
Date:
2010-03-30 @ 07:27
I've possibly gone too far in dividing my jobs between queues at the moment,
but I'm only two weeks in, so still getting used to it.

I'm only running one server - but I find it very useful to divide based on
whether the job requires the rails environment or not (lower memory
requirements if you can avoid it).  (In addition to priority, disposability,
show where bottle necks are in a large job split into small etc).

Tim.



On Tue, Mar 30, 2010 at 7:47 PM, Chris Wanstrath <chris@ozmm.org> wrote:

> On Mon, Mar 29, 2010 at 11:04 PM, Tony Arcieri <tony@medioh.com> wrote:
>
> > Now I'm thinking it might be good to separate CPU-bound jobs (where I
> want 1
> > worker per CPU) from I/O bound jobs (where I can have many more workers
> than
> > I have CPUs)
>
> That seems like a great use-case for me.
>
> > Is this a common practice?  How do people normally split up their queues?
>
> http://img.skitch.com/20100330-cj9xtq154jepcsh3qnepi8n9ew.png
>
> We use queues mostly for locality (e.g. only this machine or this
> class of machine may run this job) or priority (e.g. these jobs are
> time sensitive, important, or disposable).
>
> --
> Chris Wanstrath
> http://github.com/defunkt
>

Re: [resque] What queues do you have?

From:
Mason Jones
Date:
2010-03-30 @ 16:16
On Mon, Mar 29, 2010 at 11:47 PM, Chris Wanstrath <chris@ozmm.org> wrote:
> On Mon, Mar 29, 2010 at 11:04 PM, Tony Arcieri <tony@medioh.com> wrote:
>
>> Now I'm thinking it might be good to separate CPU-bound jobs (where I want 1
>> worker per CPU) from I/O bound jobs (where I can have many more workers than
>> I have CPUs)
>
> That seems like a great use-case for me.
>
>> Is this a common practice?  How do people normally split up their queues?
>
> http://img.skitch.com/20100330-cj9xtq154jepcsh3qnepi8n9ew.png
>
> We use queues mostly for locality (e.g. only this machine or this
> class of machine may run this job) or priority (e.g. these jobs are
> time sensitive, important, or disposable).

Yeah, that makes a great deal of sense. You're going to want to run
workers on different types of hardware, for example, as well as to
spread the load out. Especially if you're running in a virtual
environment like EC2, you'll likely have some jobs that want lots of
memory, others that want lots of CPU, and so on. Then in addition, if
you have jobs that are very I/O heavy, you'll want to spread them out
accordingly so you're not overwhelming any individual servers. The
trick there is to be able to measure first and determine how many
workers you want to run on a particular server configuration.