librelist archives

« back to archive

Multiple batches at the same time

Multiple batches at the same time

From:
Jonathan Hyman
Date:
2014-01-15 @ 15:00
Hi,

I have a set of jobs which I enqueue that will in turn enqueue other jobs.
I want to set up a batch notification for when all the “first” jobs finish,
and another batch notification for when all the “second” jobs finish. That
is, imagine the pseudocode:

class Job1
  def perform
    10.times { Job2.perform_async() }
  endend
5.times { Job1.perform_async() }

I want a batch for when all 5 Job1 jobs finish, then another batch when all
50 Job2 jobs finish. It's really easy to configure a batch when all 5 jobs
finish or all 55 jobs finish, but 2 separate batches is a harder case.

The best I can think of right now is to hack it using private APIs. Has
anyone done this in their setup?

Thanks,
Jon

Re: [sidekiq] Multiple batches at the same time

From:
Mike Perham
Date:
2014-01-15 @ 21:01
I can't think of any way to do this cleanly as implemented today.  As you
note, it's very racy because you're editing the batch in parallel as an
"outsider".  That falls into the category of "shared mutable state".


On Wed, Jan 15, 2014 at 7:00 AM, Jonathan Hyman <hyman.jon@gmail.com> wrote:

> Hi,
>
> I have a set of jobs which I enqueue that will in turn enqueue other jobs.
> I want to set up a batch notification for when all the “first” jobs finish,
> and another batch notification for when all the “second” jobs finish. That
> is, imagine the pseudocode:
>
> class Job1
>   def perform
>     10.times { Job2.perform_async() }
>   endend
> 5.times { Job1.perform_async() }
>
> I want a batch for when all 5 Job1 jobs finish, then another batch when
> all 50 Job2 jobs finish. It's really easy to configure a batch when all 5
> jobs finish or all 55 jobs finish, but 2 separate batches is a harder case.
>
> The best I can think of right now is to hack it using private APIs. Has
> anyone done this in their setup?
>
> Thanks,
> Jon
>

Re: Multiple batches at the same time

From:
Jonathan Hyman
Date:
2014-01-15 @ 15:18
To be clear, you can somewhat accomplish this by creating two batches and
passing one of the batch bids into Job1, but there are some tricky race
conditions here which is why my best solution involves private APIs.


On Wed, Jan 15, 2014 at 10:00 AM, Jonathan Hyman <hyman.jon@gmail.com>wrote:

> Hi,
>
> I have a set of jobs which I enqueue that will in turn enqueue other jobs.
> I want to set up a batch notification for when all the “first” jobs finish,
> and another batch notification for when all the “second” jobs finish. That
> is, imagine the pseudocode:
>
> class Job1
>   def perform
>     10.times { Job2.perform_async() }
>   endend
> 5.times { Job1.perform_async() }
>
> I want a batch for when all 5 Job1 jobs finish, then another batch when
> all 50 Job2 jobs finish. It's really easy to configure a batch when all 5
> jobs finish or all 55 jobs finish, but 2 separate batches is a harder case.
>
> The best I can think of right now is to hack it using private APIs. Has
> anyone done this in their setup?
>
> Thanks,
> Jon
>