librelist archives

« back to archive

Resque and BKPOP

Resque and BKPOP

From:
Tony Arcieri
Date:
2010-03-30 @ 04:20
Hello all,

I haven't really researched this issue much so please forgive my ignorance,
but it's my understanding that Resque workers presently poll Redis for
jobs.  Has there been any thought given to using BKPOP to eliminate polling?

http://groups.google.com/group/redis-db/browse_thread/thread/467b6f6291ed4808

I say this completely out of curiousity... Resque is certainly more than
adequate for our needs and I don't need less polling "for speed" yet.

-- 
Tony Arcieri
Medioh! A Kudelski Brand

Re: [resque] Resque and BKPOP

From:
Chris Wanstrath
Date:
2010-03-30 @ 04:54
On Mon, Mar 29, 2010 at 9:20 PM, Tony Arcieri <tony@medioh.com> wrote:

> I haven't really researched this issue much so please forgive my ignorance,
> but it's my understanding that Resque workers presently poll Redis for
> jobs.  Has there been any thought given to using BKPOP to eliminate polling?
>
> http://groups.google.com/group/redis-db/browse_thread/thread/467b6f6291ed4808
>
> I say this completely out of curiousity... Resque is certainly more than
> adequate for our needs and I don't need less polling "for speed" yet.

I would like this to be supported in the 2.0 release of Resque:

http://github.com/defunkt/resque/issues#issue/80

No work has been done on it yet, at least not by me.

-- 
Chris Wanstrath
http://github.com/defunkt

Re: [resque] Resque and BKPOP

From:
Mason Jones
Date:
2010-03-30 @ 05:22
I would think that the new pubsub/channel command that Salvatore's
been looking at this week might be the way to go, actually. Have to
see how it comes out, but it certainly sounds like the right thing for
a queueing system!


On Mon, Mar 29, 2010 at 9:54 PM, Chris Wanstrath <chris@ozmm.org> wrote:
> On Mon, Mar 29, 2010 at 9:20 PM, Tony Arcieri <tony@medioh.com> wrote:
>
>> I haven't really researched this issue much so please forgive my ignorance,
>> but it's my understanding that Resque workers presently poll Redis for
>> jobs.  Has there been any thought given to using BKPOP to eliminate polling?
>>
>> http://groups.google.com/group/redis-db/browse_thread/thread/467b6f6291ed4808
>>
>> I say this completely out of curiousity... Resque is certainly more than
>> adequate for our needs and I don't need less polling "for speed" yet.
>
> I would like this to be supported in the 2.0 release of Resque:
>
> http://github.com/defunkt/resque/issues#issue/80
>
> No work has been done on it yet, at least not by me.
>
> --
> Chris Wanstrath
> http://github.com/defunkt
>

Re: [resque] Resque and BKPOP

From:
Chris Wanstrath
Date:
2010-03-30 @ 05:47
On Mon, Mar 29, 2010 at 10:22 PM, Mason Jones <masonoise@gmail.com> wrote:

> I would think that the new pubsub/channel command that Salvatore's
> been looking at this week might be the way to go, actually. Have to
> see how it comes out, but it certainly sounds like the right thing for
> a queueing system!

Well, pub/sub is for messaging, not queueing. There are quite a few
conceptual differences.

With Resque and similar systems you are saying, "I know exactly what
needs to be done but I can't do it now."

With messaging systems you are saying, "I know exactly what happened
but I don't know how to react."

More or less.

Either way, I'd love to support pub/sub in a nice and clean way if it
makes sense. We'll have to see.

Chris

Re: [resque] Resque and BKPOP

From:
Mason Jones
Date:
2010-03-30 @ 06:08
On Mon, Mar 29, 2010 at 10:47 PM, Chris Wanstrath <chris@ozmm.org> wrote:
> On Mon, Mar 29, 2010 at 10:22 PM, Mason Jones <masonoise@gmail.com> wrote:
>
>> I would think that the new pubsub/channel command that Salvatore's
>> been looking at this week might be the way to go, actually. Have to
>> see how it comes out, but it certainly sounds like the right thing for
>> a queueing system!
>
> Well, pub/sub is for messaging, not queueing. There are quite a few
> conceptual differences.
>
> With Resque and similar systems you are saying, "I know exactly what
> needs to be done but I can't do it now."
>
> With messaging systems you are saying, "I know exactly what happened
> but I don't know how to react."
>
> More or less.
>
> Either way, I'd love to support pub/sub in a nice and clean way if it
> makes sense. We'll have to see.

Yes, definitely; like I said, we'll see how it comes out. The way that
things have been described with the channel notion, it feels like it
would be a potentially interesting alternative; sort of a configurable
behavioral difference. The way Resque is so clean and easy when it
comes to setting up workers, I can imagine some nice benefits from
combining it with the pub/sub ideas. I realized that my original email
sort of implied it as a replacement, which wasn't what I meant to
say...

Re: [resque] Resque and BKPOP

From:
Chris Wanstrath
Date:
2010-03-30 @ 06:50
On Mon, Mar 29, 2010 at 11:08 PM, Mason Jones <masonoise@gmail.com> wrote:

> Yes, definitely; like I said, we'll see how it comes out. The way that
> things have been described with the channel notion, it feels like it
> would be a potentially interesting alternative; sort of a configurable
> behavioral difference. The way Resque is so clean and easy when it
> comes to setting up workers, I can imagine some nice benefits from
> combining it with the pub/sub ideas. I realized that my original email
> sort of implied it as a replacement, which wasn't what I meant to
> say...

Absolutely. I've been thinking about this tonight as well.

For queues:

$ QUEUES=one,two,three rake environment resque:worker

For pub/sub:

$ CLASSES=one,two,three rake environment resque:worker

It gets hairy when you get beyond the basic APIs, though. Resque knows
which job needs to run when enqueueing - publishing is a different
story. pub/sub might need its own library (or plugin).

-- 
Chris Wanstrath
http://github.com/defunkt

Re: [resque] Resque and BKPOP

From:
Tony Arcieri
Date:
2010-03-30 @ 05:35
On Mon, Mar 29, 2010 at 11:22 PM, Mason Jones <masonoise@gmail.com> wrote:

> I would think that the new pubsub/channel command that Salvatore's
> been looking at this week might be the way to go, actually. Have to
> see how it comes out, but it certainly sounds like the right thing for
> a queueing system!
>

How does pubsub benefit what is effectively a round-robin queuing system?
It seems like a blocking pop is more appropriate, especially as it's a
ready-made fit to replace the current polling system.

You could get a lot more elaborate and have workers publish their load
average or some other metric so you can intelligently route jobs to the
least loaded nodes, but blocking pop is a quick fix to eliminate polling and
should be adequate for most people's needs, especially if they're getting by
with a polling-based system.

-- 
Tony Arcieri
Medioh! A Kudelski Brand