librelist archives

« back to archive

Shuffling queues on *

Shuffling queues on *

From:
Steve Klabnik
Date:
2013-07-15 @ 16:13
Hey everyone!

Originally posted at https://github.com/resque/resque/issues/1075

--------------------------------------
Appologies if there is another issue about this but I couldn't find
one. I found it an odd choice that when * is specified, workers
process queues in alphabetical order. I can't imagine most apps name
queues where the name is both sensical and in the order that they want
the work processed. Seems to be 2 different concerns being conflated
there.

What we've seen in production is that the default workers end up
starving other queues if they hit a queue full of hard work that takes
precedence alphabetically.

I took a look at WorkerQueueList and MultiQueue and it seems like the
best solution would be to have WorkerQueueList to shuffle for the *
queue and for MultiQueue to use a circular list when popping jobs in
the non-blocking interface. If not, I think that if the first queue in
a list has a high volume of quick jobs and the second queue has a low
volume of slow jobs, the slow jobs will get starved.

Thoughts?

Re: [resque] Shuffling queues on *

From:
Grant Austin
Date:
2013-07-15 @ 21:54
I like the idea of having at least one alternative queue selection
strategy. Perhaps providing a default strategy that mimics the current
strategy and one or more additional strategies (round robin, LRU, etc).


On Mon, Jul 15, 2013 at 10:13 AM, Steve Klabnik <steve@steveklabnik.com>wrote:

> Hey everyone!
>
> Originally posted at https://github.com/resque/resque/issues/1075
>
> --------------------------------------
> Appologies if there is another issue about this but I couldn't find
> one. I found it an odd choice that when * is specified, workers
> process queues in alphabetical order. I can't imagine most apps name
> queues where the name is both sensical and in the order that they want
> the work processed. Seems to be 2 different concerns being conflated
> there.
>
> What we've seen in production is that the default workers end up
> starving other queues if they hit a queue full of hard work that takes
> precedence alphabetically.
>
> I took a look at WorkerQueueList and MultiQueue and it seems like the
> best solution would be to have WorkerQueueList to shuffle for the *
> queue and for MultiQueue to use a circular list when popping jobs in
> the non-blocking interface. If not, I think that if the first queue in
> a list has a high volume of quick jobs and the second queue has a low
> volume of slow jobs, the slow jobs will get starved.
>
> Thoughts?
>



-- 
"Chance is always powerful. Let your hook always be cast. In the pool where
you least expect it, there will be fish."

Ovid (43 BC-17 AD)

Re: [resque] Shuffling queues on *

From:
Eoin Coffey
Date:
2013-07-15 @ 16:18
Quoting from your response:

"The idea is that * means you don't care what order they're processed in.
If you do, then you should name them.

That said, I'm not opposed to changing this behavior, that's just why it's
been that way in the past. We do feature discussion on the mailing list,
though, so how about cross-posting this there? resque@librelist.com.
Thanks!"

Sometimes you have to use '*' because you can not statically enumerate the
whole set of queues.


On Mon, Jul 15, 2013 at 10:13 AM, Steve Klabnik <steve@steveklabnik.com>wrote:

> Hey everyone!
>
> Originally posted at https://github.com/resque/resque/issues/1075
>
> --------------------------------------
> Appologies if there is another issue about this but I couldn't find
> one. I found it an odd choice that when * is specified, workers
> process queues in alphabetical order. I can't imagine most apps name
> queues where the name is both sensical and in the order that they want
> the work processed. Seems to be 2 different concerns being conflated
> there.
>
> What we've seen in production is that the default workers end up
> starving other queues if they hit a queue full of hard work that takes
> precedence alphabetically.
>
> I took a look at WorkerQueueList and MultiQueue and it seems like the
> best solution would be to have WorkerQueueList to shuffle for the *
> queue and for MultiQueue to use a circular list when popping jobs in
> the non-blocking interface. If not, I think that if the first queue in
> a list has a high volume of quick jobs and the second queue has a low
> volume of slow jobs, the slow jobs will get starved.
>
> Thoughts?
>