librelist archives

« back to archive

[ANN] resque-pool: simple resque worker management

[ANN] resque-pool: simple resque worker management

From:
nicholas a. evans
Date:
2010-07-29 @ 22:36
Hey all: I'm looking for feedback (including pull requests) on
resque-pool, which will read in a config file, load your environment
only once (saving CPU, and memory with REE), quickly fork off all of
your workers, restart workers if they die, and has a few nice signals
it'll listen for (e.g. HUP will reread the config file and start/stop
workers to match the new config).

Example config/resque-pool.yml:

   foo: 3
   "bar,foo": 3
   "foo,bar,baz": 6

Example lib/tasks/resque.rake:

   require 'resque/pool/tasks'
   task "resque:setup" => :environment do
     # generic worker setup, e.g. Hoptoad for failed jobs
   end
   task "resque:pool:setup" do
     ActiveRecord::Base.connection.disconnect!
     Resque::Pool.after_prefork do |job|
       ActiveRecord::Base.establish_connection
     end
   end

Then simply run "rake resque:pool", which will start up 12 workers
with the config given above.  If you need to adjust the number of
workers, edit the config file and send a HUP to reload.

http://github.com/nevans/resque-pool
gem install resque-pool

More details are in the README.md.  I'm running this in production
now, but I've got a few TODO items I'd like to cross off before
declaring it "1.0".  Let me know if you find it useful.  Or if you
don't find it useful, why not?

--
Nick

Re: [resque] [ANN] resque-pool: simple resque worker management

From:
Ashley Martens
Date:
2010-07-30 @ 00:04
We would be totally happy with something like this. How about monitoring 
hooks to something like nagios for notification of problems?

On Jul 29, 2010, at 15:36, "nicholas a. evans" <nick@ekenosen.net> wrote:

> Hey all: I'm looking for feedback (including pull requests) on
> resque-pool, which will read in a config file, load your environment
> only once (saving CPU, and memory with REE), quickly fork off all of
> your workers, restart workers if they die, and has a few nice signals
> it'll listen for (e.g. HUP will reread the config file and start/stop
> workers to match the new config).
> 
> Example config/resque-pool.yml:
> 
>    foo: 3
>    "bar,foo": 3
>    "foo,bar,baz": 6
> 
> Example lib/tasks/resque.rake:
> 
>    require 'resque/pool/tasks'
>    task "resque:setup" => :environment do
>      # generic worker setup, e.g. Hoptoad for failed jobs
>    end
>    task "resque:pool:setup" do
>      ActiveRecord::Base.connection.disconnect!
>      Resque::Pool.after_prefork do |job|
>        ActiveRecord::Base.establish_connection
>      end
>    end
> 
> Then simply run "rake resque:pool", which will start up 12 workers
> with the config given above.  If you need to adjust the number of
> workers, edit the config file and send a HUP to reload.
> 
> http://github.com/nevans/resque-pool
> gem install resque-pool
> 
> More details are in the README.md.  I'm running this in production
> now, but I've got a few TODO items I'd like to cross off before
> declaring it "1.0".  Let me know if you find it useful.  Or if you
> don't find it useful, why not?
> 
> --
> Nick

Re: [resque] [ANN] resque-pool: simple resque worker management

From:
nicholas a. evans
Date:
2010-07-30 @ 03:22
On Thu, Jul 29, 2010 at 8:04 PM, Ashley Martens <ashleym1972@gmail.com> wrote:
> We would be totally happy with something like this. How about monitoring
hooks to something like nagios for notification of problems?

If you are talking about adding in some callbacks at key points (like
the current Resque::Pool.after_prefork), then I'll most likely merge
any branches you send my way.  Would that be sufficient?

-- 
Nick

Re: [resque] [ANN] resque-pool: simple resque worker management

From:
Lee Marlow
Date:
2010-07-29 @ 23:25
We have done something similar here, too, but haven't packaged it up.
One thing we added was to allow configuration of a worker's nice
level.  Maybe we'll fork it and try to add some of the things we've
found useful.

Nice job.

On Thu, Jul 29, 2010 at 4:36 PM, nicholas a. evans <nick@ekenosen.net> wrote:
> Hey all: I'm looking for feedback (including pull requests) on
> resque-pool, which will read in a config file, load your environment
> only once (saving CPU, and memory with REE), quickly fork off all of
> your workers, restart workers if they die, and has a few nice signals
> it'll listen for (e.g. HUP will reread the config file and start/stop
> workers to match the new config).
>
> Example config/resque-pool.yml:
>
>    foo: 3
>    "bar,foo": 3
>    "foo,bar,baz": 6
>
> Example lib/tasks/resque.rake:
>
>    require 'resque/pool/tasks'
>    task "resque:setup" => :environment do
>      # generic worker setup, e.g. Hoptoad for failed jobs
>    end
>    task "resque:pool:setup" do
>      ActiveRecord::Base.connection.disconnect!
>      Resque::Pool.after_prefork do |job|
>        ActiveRecord::Base.establish_connection
>      end
>    end
>
> Then simply run "rake resque:pool", which will start up 12 workers
> with the config given above.  If you need to adjust the number of
> workers, edit the config file and send a HUP to reload.
>
> http://github.com/nevans/resque-pool
> gem install resque-pool
>
> More details are in the README.md.  I'm running this in production
> now, but I've got a few TODO items I'd like to cross off before
> declaring it "1.0".  Let me know if you find it useful.  Or if you
> don't find it useful, why not?
>
> --
> Nick
>

Re: [resque] [ANN] resque-pool: simple resque worker management

From:
nicholas a. evans
Date:
2010-07-30 @ 03:14
On Thu, Jul 29, 2010 at 7:25 PM, Lee Marlow <lee.marlow@gmail.com> wrote:
> We have done something similar here, too, but haven't packaged it up.

After I started mine, I noticed a couple of other people had also
tried the same thing.  As I was joining this list, I noticed that only
a week ago someone posted a gist that did much the same thing.

> One thing we added was to allow configuration of a worker's nice
> level.

That could be useful.  My main desire is to preserve a nice config
file format for the simple case (using all default settings and only
mapping the queues to worker counts).  If the config file doesn't
contain a "workers" key, then I'll use the current config format
(queues => counts).  If it does, then I'd use an alternate format.
I'm thinking of something like so:

pidfile: foo.pid
logfile: path/to/logs
nice: 4
workers:
  foo:
    count: 2
    logfile: different/path/to/log
    nice: -9
  "foo,bar": 4
  bar: 1

where the worker settings override the top level config settings which
override the defaults.  If you want further configurability than what
that provides, we can add some ruby callbacks.  What do you think?

> Maybe we'll fork it and try to add some of the things we've found useful.

I look forward to it.

> Nice job.

Thanks.  :-)

-- 
Nick

Re: [resque] [ANN] resque-pool: simple resque worker management

From:
Mason Jones
Date:
2010-07-29 @ 22:52
I like this idea quite a bit, thanks for sharing it. We're actually
looking at exactly this sort of problem. Right now we just have a
handful of workers, which we're starting manually in screen sessions;
obviously not ideal. Going to more robust production, we're going to
have a couple more queues and more workers.

The one quick question that comes to mind is where the workers log to,
and whether it's possible to indicate whether the workers should be
verbose or not? Ideally each worker would have its own log file
somewhere, in order to be able to check if things seem haywire.

I'll grab it from github and check it out shortly, for sure.


On Thu, Jul 29, 2010 at 3:36 PM, nicholas a. evans <nick@ekenosen.net> wrote:
> Hey all: I'm looking for feedback (including pull requests) on
> resque-pool, which will read in a config file, load your environment
> only once (saving CPU, and memory with REE), quickly fork off all of
> your workers, restart workers if they die, and has a few nice signals
> it'll listen for (e.g. HUP will reread the config file and start/stop
> workers to match the new config).
>
> Example config/resque-pool.yml:
>
>    foo: 3
>    "bar,foo": 3
>    "foo,bar,baz": 6
>
> Example lib/tasks/resque.rake:
>
>    require 'resque/pool/tasks'
>    task "resque:setup" => :environment do
>      # generic worker setup, e.g. Hoptoad for failed jobs
>    end
>    task "resque:pool:setup" do
>      ActiveRecord::Base.connection.disconnect!
>      Resque::Pool.after_prefork do |job|
>        ActiveRecord::Base.establish_connection
>      end
>    end
>
> Then simply run "rake resque:pool", which will start up 12 workers
> with the config given above.  If you need to adjust the number of
> workers, edit the config file and send a HUP to reload.
>
> http://github.com/nevans/resque-pool
> gem install resque-pool
>
> More details are in the README.md.  I'm running this in production
> now, but I've got a few TODO items I'd like to cross off before
> declaring it "1.0".  Let me know if you find it useful.  Or if you
> don't find it useful, why not?
>
> --
> Nick
>

Re: [resque] [ANN] resque-pool: simple resque worker management

From:
nicholas a. evans
Date:
2010-07-30 @ 02:19
On Thu, Jul 29, 2010 at 6:52 PM, Mason Jones <masonoise@gmail.com> wrote:
> The one quick question that comes to mind is where the workers log to,
> and whether it's possible to indicate whether the workers should be
> verbose or not? Ideally each worker would have its own log file
> somewhere, in order to be able to check if things seem haywire.

Right now, I'm just using a simple shell script (run from monit) to
background the rake task, redirecting stderr and stdout to a log file,
and capturing the pid into a pidfile.  I'm currently letting all of
the workers share their stderr/stdout, since it's been a while since I
actually found anything interesting in the resque logfiles.  Pulling
all of that functionality into the ruby script (and adding the ability
to split the logs back out per worker, if so desired) is something I
intend to do.  But, until then, I've committed my shell script into
the examples directory.

    http://github.com/nevans/resque-pool/blob/master/examples/startup_script.sh

Hope that helps.

-- 
Nick

Re: [resque] [ANN] resque-pool: simple resque worker management

From:
David Balatero
Date:
2010-07-29 @ 23:32
Hey Mason,

As a side note: I used to run in screen jobs in production, but now I have
resque, resque-web, and redis monitored by God. Here are my God configs for
that: http://gist.github.com/499505

If you use the god cookbook from Chef in conjunction, you can get up and
running fast. Use these config files as your template files, and set up
"god_monitor" directives for each one.

My cap deploy consists of a standard Heroku deploy + pushing new code to our
Resque machines, and running god restart resque && god restart resque-web to
get the workers running new code.

HTH
- David


On Thu, Jul 29, 2010 at 3:52 PM, Mason Jones <masonoise@gmail.com> wrote:

> I like this idea quite a bit, thanks for sharing it. We're actually
> looking at exactly this sort of problem. Right now we just have a
> handful of workers, which we're starting manually in screen sessions;
> obviously not ideal. Going to more robust production, we're going to
> have a couple more queues and more workers.
>
> The one quick question that comes to mind is where the workers log to,
> and whether it's possible to indicate whether the workers should be
> verbose or not? Ideally each worker would have its own log file
> somewhere, in order to be able to check if things seem haywire.
>
> I'll grab it from github and check it out shortly, for sure.
>
>
> On Thu, Jul 29, 2010 at 3:36 PM, nicholas a. evans <nick@ekenosen.net>
> wrote:
> > Hey all: I'm looking for feedback (including pull requests) on
> > resque-pool, which will read in a config file, load your environment
> > only once (saving CPU, and memory with REE), quickly fork off all of
> > your workers, restart workers if they die, and has a few nice signals
> > it'll listen for (e.g. HUP will reread the config file and start/stop
> > workers to match the new config).
> >
> > Example config/resque-pool.yml:
> >
> >    foo: 3
> >    "bar,foo": 3
> >    "foo,bar,baz": 6
> >
> > Example lib/tasks/resque.rake:
> >
> >    require 'resque/pool/tasks'
> >    task "resque:setup" => :environment do
> >      # generic worker setup, e.g. Hoptoad for failed jobs
> >    end
> >    task "resque:pool:setup" do
> >      ActiveRecord::Base.connection.disconnect!
> >      Resque::Pool.after_prefork do |job|
> >        ActiveRecord::Base.establish_connection
> >      end
> >    end
> >
> > Then simply run "rake resque:pool", which will start up 12 workers
> > with the config given above.  If you need to adjust the number of
> > workers, edit the config file and send a HUP to reload.
> >
> > http://github.com/nevans/resque-pool
> > gem install resque-pool
> >
> > More details are in the README.md.  I'm running this in production
> > now, but I've got a few TODO items I'd like to cross off before
> > declaring it "1.0".  Let me know if you find it useful.  Or if you
> > don't find it useful, why not?
> >
> > --
> > Nick
> >
>