librelist archives

« back to archive

Rails 4.0 Queue and Sidekiq

Rails 4.0 Queue and Sidekiq

From:
Peter M. Goldstein
Date:
2012-08-13 @ 21:45
Hi,

I'm curious about the status of Sidekiq support for the Rails 4.0 Queue
interface.  From all of the documentation, comments, presentations, etc. it
sounds like Sidekiq is high on the list for support for this interface.
 Specifically, the Rails core developers seem to expect that developers
will be able to use Sidekiq with this interface from day one.

Looking at the two implementations, there seems to be a substantial
(although certainly not insurmountable) gap between the interface expected
by the Rails 4.0 Queue (which is basically just a ::Queue), and the Sidekiq
implementation (which requires inclusion of the Sidekiq::Worker module in
the passed in class and passes in arguments to perform).

Is there support in the core Sidekiq gem for this integration?  If not,
does another gem already exist that provides this integration?  If support
doesn't already exist, either in the core gem or in an adapter, has there
been any thought as to how this might be added?

I'm happy to contribute some development time on this issue if it's an open
problem and I can be helpful.

Thanks.

Best,

Peter

Re: [sidekiq] Rails 4.0 Queue and Sidekiq

From:
Mike Perham
Date:
2012-08-13 @ 22:13
Rails-core and I have certainly taken different approaches.  I
registered my difference of opinion on their mailing list.

Sidekiq's design simplifies the job into the minimal number of simple
data elements necessary to process a job.  These elements are the
arguments to Worker#perform.  This also makes the job easy to marshal
out of process via a simple serialization format like JSON and easy to
inspect with the Web UI or other tools for debugging purposes.

Rails's Queue design is more object-oriented and works well for
in-process implementations.  It however necessitates Marshal for
serialization between processes; the message is a binary blob.  Their
take was that serialization is a tool-specific concern and shouldn't
be implied as part of the API.

The initial implementation is on this branch but I haven't kept it up to date.

https://github.com/mperham/sidekiq/tree/rails4

mike

On Mon, Aug 13, 2012 at 2:45 PM, Peter M. Goldstein
<peter.m.goldstein@gmail.com> wrote:
> Hi,
>
> I'm curious about the status of Sidekiq support for the Rails 4.0 Queue
> interface.  From all of the documentation, comments, presentations, etc. it
> sounds like Sidekiq is high on the list for support for this interface.
> Specifically, the Rails core developers seem to expect that developers will
> be able to use Sidekiq with this interface from day one.
>
> Looking at the two implementations, there seems to be a substantial
> (although certainly not insurmountable) gap between the interface expected
> by the Rails 4.0 Queue (which is basically just a ::Queue), and the Sidekiq
> implementation (which requires inclusion of the Sidekiq::Worker module in
> the passed in class and passes in arguments to perform).
>
> Is there support in the core Sidekiq gem for this integration?  If not, does
> another gem already exist that provides this integration?  If support
> doesn't already exist, either in the core gem or in an adapter, has there
> been any thought as to how this might be added?
>
> I'm happy to contribute some development time on this issue if it's an open
> problem and I can be helpful.
>
> Thanks.
>
> Best,
>
> Peter

Re: [sidekiq] Rails 4.0 Queue and Sidekiq

From:
Peter M. Goldstein
Date:
2012-08-13 @ 22:33
Mike,

Thanks for the quick reply.

I'm somewhat more sympathetic to your view in this difference of opinion.
 In my experience, asynchronous worker tasks tend to be better represented
with a 'type' and a small number of simple, serializable arguments.  It's
somewhat appealing conceptually to view tasks as instances of an object,
with all required data encapsulated, but the realities of interprocess
communication tend to get in the way.  As you note, serialization,
inspection, debugging, logging, etc. are all more of a pain when your job
is an opaque object with a minimal interface.

That all said, I'm assuming that the Rails core team won't be changing
their position, and that some sort of adapter like the one on this branch
will be required eventually for those of us who want to use Sidekiq with
Rails.  Is there any way I can productively assist here?  I'm happy to
bring the code up to date, make sure it's got adequate test coverage, and
package it into a pull request.  Let me know if that would be helpful.

Thanks.

--Peter

On Mon, Aug 13, 2012 at 3:13 PM, Mike Perham <mperham@gmail.com> wrote:

> Rails-core and I have certainly taken different approaches.  I
> registered my difference of opinion on their mailing list.
>
> Sidekiq's design simplifies the job into the minimal number of simple
> data elements necessary to process a job.  These elements are the
> arguments to Worker#perform.  This also makes the job easy to marshal
> out of process via a simple serialization format like JSON and easy to
> inspect with the Web UI or other tools for debugging purposes.
>
> Rails's Queue design is more object-oriented and works well for
> in-process implementations.  It however necessitates Marshal for
> serialization between processes; the message is a binary blob.  Their
> take was that serialization is a tool-specific concern and shouldn't
> be implied as part of the API.
>
> The initial implementation is on this branch but I haven't kept it up to
> date.
>
> https://github.com/mperham/sidekiq/tree/rails4
>
> mike
>
> On Mon, Aug 13, 2012 at 2:45 PM, Peter M. Goldstein
> <peter.m.goldstein@gmail.com> wrote:
> > Hi,
> >
> > I'm curious about the status of Sidekiq support for the Rails 4.0 Queue
> > interface.  From all of the documentation, comments, presentations, etc.
> it
> > sounds like Sidekiq is high on the list for support for this interface.
> > Specifically, the Rails core developers seem to expect that developers
> will
> > be able to use Sidekiq with this interface from day one.
> >
> > Looking at the two implementations, there seems to be a substantial
> > (although certainly not insurmountable) gap between the interface
> expected
> > by the Rails 4.0 Queue (which is basically just a ::Queue), and the
> Sidekiq
> > implementation (which requires inclusion of the Sidekiq::Worker module in
> > the passed in class and passes in arguments to perform).
> >
> > Is there support in the core Sidekiq gem for this integration?  If not,
> does
> > another gem already exist that provides this integration?  If support
> > doesn't already exist, either in the core gem or in an adapter, has there
> > been any thought as to how this might be added?
> >
> > I'm happy to contribute some development time on this issue if it's an
> open
> > problem and I can be helpful.
> >
> > Thanks.
> >
> > Best,
> >
> > Peter
>

Re: [sidekiq] Rails 4.0 Queue and Sidekiq

From:
Mike Perham
Date:
2012-08-13 @ 23:01
I hadn't merged it into Sidekiq proper because Rails 4.0 is still far
away afaik and who knows if the API will change or be removed.  Anyone
know if there is a 4.0 beta/rc plan in the works?  If you want to
merge master back into rails4, ensure everything works and write
tests, I'm open to a PR.  I just don't want to release support for an
API which might change or never see the light of day.

On Mon, Aug 13, 2012 at 3:33 PM, Peter M. Goldstein
<peter.m.goldstein@gmail.com> wrote:
> required eventually for those of us who want to use Sidekiq with Rails.  Is
> there any way I can productively assist here?  I'm happy to bring the code
> up to date, make sure it's got adequate test coverage, and package it into a
> pull request.  Let me know if that would be helpful.
>

Re: [sidekiq] Rails 4.0 Queue and Sidekiq

From:
Peter M. Goldstein
Date:
2012-08-13 @ 23:10
Fair enough.

Let me cue up an updated branch with tests, and I can just rebase/update it
as master evolves.   When Rails 4 starts solidifying and heading to a final
release I can submit a pull request.

--Peter

On Mon, Aug 13, 2012 at 4:01 PM, Mike Perham <mperham@gmail.com> wrote:

> I hadn't merged it into Sidekiq proper because Rails 4.0 is still far
> away afaik and who knows if the API will change or be removed.  Anyone
> know if there is a 4.0 beta/rc plan in the works?  If you want to
> merge master back into rails4, ensure everything works and write
> tests, I'm open to a PR.  I just don't want to release support for an
> API which might change or never see the light of day.
>
> On Mon, Aug 13, 2012 at 3:33 PM, Peter M. Goldstein
> <peter.m.goldstein@gmail.com> wrote:
> > required eventually for those of us who want to use Sidekiq with Rails.
>  Is
> > there any way I can productively assist here?  I'm happy to bring the
> code
> > up to date, make sure it's got adequate test coverage, and package it
> into a
> > pull request.  Let me know if that would be helpful.
> >
>