librelist archives

« back to archive

How do you keep your workers alive (monit) ?

How do you keep your workers alive (monit) ?

From:
Thibaut Barrère
Date:
2010-08-12 @ 21:51
Hi,

I'm just deploying my first worker on Ubuntu (the previous one was on
Windows, and I kept it alive with FireDaemon).

I'd be tempted to use monit (
http://github.com/defunkt/resque/blob/master/examples/monit/resque.monit).

Is that what you guys are using ?

-- Thibaut

Re: [resque] How do you keep your workers alive (monit) ?

From:
David Balatero
Date:
2010-08-12 @ 22:09
I use God:

http://gist.github.com/499505

On Thu, Aug 12, 2010 at 2:51 PM, Thibaut Barrère
<thibaut.barrere@gmail.com>wrote:

> Hi,
>
> I'm just deploying my first worker on Ubuntu (the previous one was on
> Windows, and I kept it alive with FireDaemon).
>
> I'd be tempted to use monit (
> http://github.com/defunkt/resque/blob/master/examples/monit/resque.monit).
>
> Is that what you guys are using ?
>
> -- Thibaut
>

Re: [resque] How do you keep your workers alive (monit) ?

From:
Adam Tucker
Date:
2010-08-12 @ 22:16
We use Monit. The configs get built during capistrano deployment, and the
section for Resque looks like this:

<% worker_queue_names.each do |worker| %>

#### resque <%= worker %> ####
check process resque_<%= worker.gsub(',','_') %>
  with pidfile <%= current_path %>/tmp/pids/resque.<%= worker.gsub(',','_')
%>.pid
  start program = "<%= current_path %>/script/monitor <%= current_path
%>/script/resque start <%= worker %> <%= rails_env %>"
  stop program = "<%= current_path %>/script/monitor <%= current_path
%>/script/resque stop <%= worker %>"

  group resque

<% end %>

It's set up so that we can specify what queues run on which boxes inside
deploy.rb.Our script/resque file looks like this:

#!/bin/bash

pid_filename="$(echo $2 | tr ',' '_')"

case $1 in
  start)
    echo $$ > tmp/pids/resque.$pid_filename.pid;
    exec 2>&1 env QUEUE=$2 rake environment RAILS_ENV=$3 resque:work
1>tmp/resque.out
    ;;
  stop)
    pid_filename="$(echo $2 | tr ',' '_')"
    kill `cat tmp/pids/resque.$pid_filename.pid` ;;
  *)
    echo "usage: resque {start|stop} {queue1[,queue2,etc]} {environment}" ;;
esac
exit 0

Works nicely for our needs!

On Thu, Aug 12, 2010 at 6:09 PM, David Balatero <dbalatero@gmail.com> wrote:

> I use God:
>
> http://gist.github.com/499505
>
>
> On Thu, Aug 12, 2010 at 2:51 PM, Thibaut Barrère <
> thibaut.barrere@gmail.com> wrote:
>
>> Hi,
>>
>> I'm just deploying my first worker on Ubuntu (the previous one was on
>> Windows, and I kept it alive with FireDaemon).
>>
>> I'd be tempted to use monit (
>> http://github.com/defunkt/resque/blob/master/examples/monit/resque.monit
>> ).
>>
>> Is that what you guys are using ?
>>
>> -- Thibaut
>>
>
>

Re: [resque] How do you keep your workers alive (monit) ?

From:
David Kelso
Date:
2010-08-12 @ 22:17
Is it just me, or is god's syntax a disaster? It uses the same block
to check conditions, and also say what to do when those conditions are
satisfied.
Also, having to repeat the same declarations over and over for every
monitored process doesn't really make sense to me.
Apart from the syntax though, god does solve this issue really well.

On Thu, Aug 12, 2010 at 3:09 PM, David Balatero <dbalatero@gmail.com> wrote:
> I use God:
> http://gist.github.com/499505
>
> On Thu, Aug 12, 2010 at 2:51 PM, Thibaut Barrère <thibaut.barrere@gmail.com>
> wrote:
>>
>> Hi,
>> I'm just deploying my first worker on Ubuntu (the previous one was on
>> Windows, and I kept it alive with FireDaemon).
>> I'd be tempted to use monit
>> (http://github.com/defunkt/resque/blob/master/examples/monit/resque.monit).
>> Is that what you guys are using ?
>> -- Thibaut
>

Re: [resque] How do you keep your workers alive (monit) ?

From:
Thibaut Barrère
Date:
2010-08-12 @ 22:23
@Adam - thanks, nice use of ERB, I do something similar with virtual hosts.

On god: are the memory leaks a thing of the past in your cases ?

-- Thibaut

Re: [resque] How do you keep your workers alive (monit) ?

From:
Matt Duncan
Date:
2010-08-12 @ 22:34
Another option you may want to look at is Bluepill (
http://github.com/arya/bluepill), I've been really happy with it.  I haven't
used God so I can't really comment on the pros/cons between the two, but
they seem pretty similar for the most part.


Here's an excerpt from a Bluepill config that should work for Resque
---

RESQUE_WORKERS.times do |i|
    app.process("resque_#{i}") do |process|
      process.group = "resque"
      process.pid_file = File.join(RAILS_ROOT, "tmp", "pids",
"resque_#{i}.pid")
      process.daemonize = true

      process.working_dir = RAILS_ROOT
      process.start_command = "/usr/bin/env RAILS_ENV=#{ENVIRONMENT}
QUEUE=#{RESQUE_QUEUES} rake environment resque:work"

      process.checks :mem_usage, :every => 10.seconds, :below =>
150.megabytes, :times => [2, 2]
      process.checks :cpu_usage, :every => 10.seconds, :below => 80, :times
=> [9, 10]
      process.checks :flapping, :times => 5, :within => 5.minutes, :retry_in
=> 10.minutes

      process.stdout = "/var/log/resque.log"
      process.stderr = "/var/log/resque.log"
    end
  end

On Thu, Aug 12, 2010 at 6:23 PM, Thibaut Barrère
<thibaut.barrere@gmail.com>wrote:

> @Adam - thanks, nice use of ERB, I do something similar with virtual hosts.
>
> On god: are the memory leaks a thing of the past in your cases ?
>
> -- Thibaut
>

Re: [resque] How do you keep your workers alive (monit) ?

From:
David Balatero
Date:
2010-08-12 @ 22:32
I have not seen God memory leaks. I believe Eric Lindvall did a bunch of
good work on plugging those:
http://groups.google.com/group/god-rb/browse_thread/thread/86bec1664757086f

On Thu, Aug 12, 2010 at 3:23 PM, Thibaut Barrère
<thibaut.barrere@gmail.com>wrote:

> @Adam - thanks, nice use of ERB, I do something similar with virtual hosts.
>
> On god: are the memory leaks a thing of the past in your cases ?
>
> -- Thibaut
>

Re: [resque] How do you keep your workers alive (monit) ?

From:
Tony Arcieri
Date:
2010-08-12 @ 22:30
On Thu, Aug 12, 2010 at 4:17 PM, David Kelso <david@kelso.id.au> wrote:

> Also, having to repeat the same declarations over and over for every
> monitored process doesn't really make sense to me.


God's configuration files are Ruby code. If you're having trouble with
repeating yourself in Ruby code, that's your problem, not God's.

On the contrary, that's a lot bigger frustration with Monit.

-- 
Tony Arcieri
Medioh! A Kudelski Brand

Re: [resque] How do you keep your workers alive (monit) ?

From:
David Kelso
Date:
2010-08-12 @ 22:55
Yeah, that's fair. I retract that statement. I do wish some of those
basic operations were keywords by default though.
The complaint about the same block being used for two things still
stands though :)

Also, I haven't had any problem with god memory leaks. We do restart
it on a weekly basis with deployments though.

On Thu, Aug 12, 2010 at 3:30 PM, Tony Arcieri <tony.arcieri@medioh.com> wrote:
> On Thu, Aug 12, 2010 at 4:17 PM, David Kelso <david@kelso.id.au> wrote:
>>
>> Also, having to repeat the same declarations over and over for every
>> monitored process doesn't really make sense to me.
>
> God's configuration files are Ruby code. If you're having trouble with
> repeating yourself in Ruby code, that's your problem, not God's.
> On the contrary, that's a lot bigger frustration with Monit.
>
> --
> Tony Arcieri
> Medioh! A Kudelski Brand
>

Re: [resque] How do you keep your workers alive (monit) ?

From:
David Balatero
Date:
2010-08-12 @ 23:00
On Thu, Aug 12, 2010 at 3:55 PM, David Kelso <david@kelso.id.au> wrote:

> Yeah, that's fair. I retract that statement. I do wish some of those
> basic operations were keywords by default though.
> The complaint about the same block being used for two things still
> stands though :)
>
> Also, I haven't had any problem with god memory leaks. We do restart
> it on a weekly basis with deployments though.
>

To add my experience -- we rarely restart God, but we do restart our workers
via god restart resque, etc. No memory leaks seen.


>
> On Thu, Aug 12, 2010 at 3:30 PM, Tony Arcieri <tony.arcieri@medioh.com>
> wrote:
> > On Thu, Aug 12, 2010 at 4:17 PM, David Kelso <david@kelso.id.au> wrote:
> >>
> >> Also, having to repeat the same declarations over and over for every
> >> monitored process doesn't really make sense to me.
> >
> > God's configuration files are Ruby code. If you're having trouble with
> > repeating yourself in Ruby code, that's your problem, not God's.
> > On the contrary, that's a lot bigger frustration with Monit.
> >
> > --
> > Tony Arcieri
> > Medioh! A Kudelski Brand
> >
>

Re: [resque] How do you keep your workers alive (monit) ?

From:
Thibaut Barrère
Date:
2010-08-13 @ 07:54
Thanks a lot guys, all that information is very useful (and I will probably
write it back into some blog post).

cheers,

-- Thibaut

Re: [resque] How do you keep your workers alive (monit) ?

From:
nicholas a. evans
Date:
2010-08-13 @ 13:29
On Thu, Aug 12, 2010 at 5:51 PM, Thibaut Barrère
<thibaut.barrere@gmail.com> wrote:
> I'm just deploying my first worker on Ubuntu (the previous one was on
> Windows, and I kept it alive with FireDaemon).
> I'd be tempted to use monit
> (http://github.com/defunkt/resque/blob/master/examples/monit/resque.monit).
> Is that what you guys are using ?

I originally used god, but had a bunch of issues with it (probably
user error, since others are using it without problems) that led me
back to annoying-to-configure monit.  Nowadays, I use my own
resque-pool gem to manage the workers.  The resque-pool itself is
currently started by a simple monit config, but it seems stable enough
that I could probably just start it from a standard init.d script.

http://github.com/nevans/resque-pool

If you try it out, let me know what you think.

-- 
Nick

Re: [resque] How do you keep your workers alive (monit) ?

From:
Thibaut Barrère
Date:
2010-09-07 @ 08:36
Hi guys,

just a follow-up in case that's inspiring to any newcomer: I finally used
god (after trying monit but without much success on my configuration for
some reason, maybe I was just tired or something) and I'm very happy with
it.

I can share cap recipes too (cap deploy will stop the worker, redeploy the
web app then start the worker again).

The worker is rock solid stable so far :)

cheers and thanks for all the useful input, as usual!

-- Thibaut

Re: [resque] How do you keep your workers alive (monit) ?

From:
Rinaldi Fonseca
Date:
2010-09-07 @ 17:06
Hi, Thibaut
*
*
*
*
*Could you post your steps for to configurate God in your application?*
*
*
*Thanks!*
*
*Att

Rinaldi Fonseca
BS  - Computer Science
Pontifical Catholic University of Minas Gerais

Brazil


On Tue, Sep 7, 2010 at 5:36 AM, Thibaut Barrère
<thibaut.barrere@gmail.com>wrote:

> Hi guys,
>
> just a follow-up in case that's inspiring to any newcomer: I finally used
> god (after trying monit but without much success on my configuration for
> some reason, maybe I was just tired or something) and I'm very happy with
> it.
>
> I can share cap recipes too (cap deploy will stop the worker, redeploy the
> web app then start the worker again).
>
> The worker is rock solid stable so far :)
>
> cheers and thanks for all the useful input, as usual!
>
> -- Thibaut
>
>

Re: [resque] How do you keep your workers alive (monit) ?

From:
Thibaut Barrère
Date:
2010-09-07 @ 18:36
Hello Rinaldi,

*> Could you post your steps for to configurate God in your application?*
*
*
I put the files up there: http://gist.github.com/568773

Here are the steps roughly:
* install god (sudo gem install god)
* sudo mkdir /etc/god
then put the modified master.conf into /etc/god and try god out to see if
everything works fine
* make god reboot ready
  - sudo cp god-service /etc/init.d
  - sudo chmod +x /etc/init.d/god-service
  - sudo update-rc god-service defaults
* try out your init.d script by calling /etc/init.d/god-service
start|stop|status
* prepare your app resque.god configuration
* add the god part to your config/deploy.rb

You may want to tweak the deploy.rb to match more closely the expected
behaviour.
Also (as suggested earlier) you can tweak the configuration using ERB to
create n workers instead of one here.

hope this helps!

-- Thibaut