librelist archives

« back to archive

message ordering guarantees

message ordering guarantees

From:
Ian Kallen
Date:
2011-09-08 @ 21:58
Are there message order guarantees with resque? I was assuming it's
backended with a redis list and jobs are added/popped like a FIFO
queue but didn't see that explicitly called out in the goal/design
description for resque. My current use would broken if jobs in a
particular queue are performed in a random order.
thanks!

Re: [resque] message ordering guarantees

From:
Caius Durling
Date:
2011-09-08 @ 22:01
On 8 Sep 2011, at 22:58, Ian Kallen wrote:

> Are there message order guarantees with resque? I was assuming it's
> backended with a redis list and jobs are added/popped like a FIFO
> queue but didn't see that explicitly called out in the goal/design
> description for resque. My current use would broken if jobs in a
> particular queue are performed in a random order.

From what I've seen of the internals it is FIFO. Uses a redis list as you assume.

C

Re: [resque] message ordering guarantees

From:
Francis Hwang
Date:
2011-09-08 @ 22:12
In practice, however, you may have edge cases that mess with the
ordering. For example, if you have a high number of workers that grab
a lot of the jobs at once, there's no guarantee that one won't finish
before the other. In my experience, it makes your life much simpler if
you assume that the FIFO qualities of resque jobs aren't a
hard-and-fast guarantee.

We have a ton of dependencies in our jobs--sometimes one event will
end up firing literally thousands of other jobs, with specific
ordering as to which can be fired before which. What's worked best for
us is to have certain jobs fire other jobs when they finish, as
opposed to enqueueing them all at once in a specific order and then
losing sleep about if the queues come through in the exact order you
want.

FH



On Thu, Sep 8, 2011 at 6:01 PM, Caius Durling <caius@pizzapowered.com> wrote:
>
> On 8 Sep 2011, at 22:58, Ian Kallen wrote:
>
> Are there message order guarantees with resque? I was assuming it's
> backended with a redis list and jobs are added/popped like a FIFO
> queue but didn't see that explicitly called out in the goal/design
> des cription for resque. My current use would broken if jobs in a
> particular queue are performed in a random order.
>
> From what I've seen of the internals it is FIFO. Uses a redis list as you
> assume.
> C

Re: [resque] message ordering guarantees

From:
Ian Kallen
Date:
2011-09-09 @ 14:59
Thanks for the replies. For now, I'm going to serialize all of the
jobs through a single worker, which will be fine for the level of
scale currently needed. When it's time to scale that up I'll do
something else to coordinate the workflow.

On Thu, Sep 8, 2011 at 3:12 PM, Francis Hwang
<francis.hwang@profitably.com> wrote:
> In practice, however, you may have edge cases that mess with the
> ordering. For example, if you have a high number of workers that grab
> a lot of the jobs at once, there's no guarantee that one won't finish
> before the other. In my experience, it makes your life much simpler if
> you assume that the FIFO qualities of resque jobs aren't a
> hard-and-fast guarantee.
>
> We have a ton of dependencies in our jobs--sometimes one event will
> end up firing literally thousands of other jobs, with specific
> ordering as to which can be fired before which. What's worked best for
> us is to have certain jobs fire other jobs when they finish, as
> opposed to enqueueing them all at once in a specific order and then
> losing sleep about if the queues come through in the exact order you
> want.
>
> FH
>
>
>
> On Thu, Sep 8, 2011 at 6:01 PM, Caius Durling <caius@pizzapowered.com> wrote:
>>
>> On 8 Sep 2011, at 22:58, Ian Kallen wrote:
>>
>> Are there message order guarantees with resque? I was assuming it's
>> backended with a redis list and jobs are added/popped like a FIFO
>> queue but didn't see that explicitly called out in the goal/design
>> des cription for resque. My current use would broken if jobs in a
>> particular queue are performed in a random order.
>>
>> From what I've seen of the internals it is FIFO. Uses a redis list as you
>> assume.
>> C
>

Re: [resque] message ordering guarantees

From:
Philippe Lafoucrière
Date:
2011-09-12 @ 10:09
On Fri, Sep 9, 2011 at 4:59 PM, Ian Kallen <spidaman.list@gmail.com> wrote:
> Thanks for the replies. For now, I'm going to serialize all of the
> jobs through a single worker, which will be fine for the level of
> scale currently needed. When it's time to scale that up I'll do
> something else to coordinate the workflow.

Be careful with timeouts, we did something equivalent, and some
default timeouts are 15 minutes. that means you won't unqueue any job
during this hang time ;)