librelist archives

« back to archive

resque-pool [WAS: Re: [resque] Re: Converting from gearman]

resque-pool [WAS: Re: [resque] Re: Converting from gearman]

From:
Kevin Menard
Date:
2011-01-15 @ 21:22
I really need to give this a try.  We have a resque management script as
part of rubber [1] that allows faster deploys by forking before loading
the enviroment, but resque-pool looks like a much more complete solution.

I hope you don't change the name though.  Sometimes being too clever can
cause people to no longer understand what it is your project does.
resque-pool is a great name for what it provides.

-- 
Kevin

[1] 
https://github.com/wr0ngway/rubber/blob/master/lib/generators/vulcanize/tem
plates/resque/script/resque_worker_management.rb

On 1/14/11 8:53 PM, "nicholas a. evans" <nick@ekenosen.net> wrote:

>On Fri, Jan 14, 2011 at 5:04 PM, Tim Uckun <timuckun@gmail.com> wrote:
>>> Does every resque worker share the same rails env? Currently I am
>>> loading the rails env on every worker so it would be a big memory boon
>>> for me if they could share the same rails environment.
>>>
>>> No.
>>
>> Is there a way this could be done?  It seems like it would be a big
>> boost if could.
>
>I wrote a simple gem called "resque-pool" which will start up a "pool
>manager" which can preload your app environment and then fork a "pool"
>of workers.  If you are using REE (with its copy-on-write friendly
>GC), it can lead to enormous memory savings.  Also, start up time is
>much faster since it only loads the environment once and then forks; I
>don't have 30 workers all vying for 8 CPUs when it is first started
>up.  I've been using it in my production environment for half a year.
>
>I'm currently testing (in my production environment) a beta version
>that comes with its own daemonizing executable (which manages pidfile
>and logfiles for you), an example init.d script which should work on
>Debian/Ubuntu/Gentoo, an example monitrc, and an example chef recipe
>that should Just Work at Engine Yard; all you need to provide is a
>yaml config specifying how many workers you want running.  I'll
>release a new non-beta version of the gem next week.  And I'm also
>just about done with a resque-web plugin that lets you add and remove
>workers from the web interface.  IMHO, it's the easiest and best way
>to manage a large group of resque workers.  Let me know if it helps
>you out.  Pull requests are appreciated.
>
>I'm thinking about renaming it to "resque-squad" for the next big
>release, to better match the "rescue worker" pun.
>
>http://github.com/nevans/resque-pool
>gem install resque-pool
>
>-- 
>Nick Evans