librelist archives

« back to archive

Resque and God?

Resque and God?

From:
Tim Haines
Date:
2010-03-22 @ 22:00
Hey guys,

Has anyone had experience getting god to monitor your resque workers?  I
tried to set it up last night, but hit a brick wall.  I basically took the
example resque.god from github (
http://github.com/defunkt/resque/blob/master/examples/god/resque.god), and
adapted it to my paths and queues etc, but it just won't get started.  I've
attached a debug log of what it's doing below.  Can anyone explain what's
going wrong or see something obvious?

The weird thing is, the same script (with adjusted paths) works locally.
 Both local and production using god 0.8.

Additionally, if I run the command from the start line in the terminal on
production, it works fine too..


: sudo /opt/ruby-enterprise-1.8.6-20090520/bin/god -c
 ~/god/fav_processor.god -D --log-level debug
I [2010-03-22 10:23:41]  INFO: Loading /home/favstar/god/fav_processor.god
I [2010-03-22 10:23:41]  INFO: Using pid file directory: /var/run/god
I [2010-03-22 10:23:41]  INFO: Started on drbunix:///tmp/god.17165.sock
I [2010-03-22 10:23:41]  INFO: favprocessor move 'unmonitored' to 'init'
D [2010-03-22 10:23:41] DEBUG: driver schedule
#<God::Conditions::ProcessRunning:0x17e6a48> in 0 seconds
I [2010-03-22 10:23:41]  INFO: favprocessor moved 'unmonitored' to 'init'
I [2010-03-22 10:23:41]  INFO: favprocessor [trigger] process is not running
(ProcessRunning)
D [2010-03-22 10:23:41] DEBUG: favprocessor ProcessRunning [false]
{false=>:start, true=>:up}
I [2010-03-22 10:23:41]  INFO: favprocessor move 'init' to 'start'
I [2010-03-22 10:23:41]  INFO: favprocessor start: QUEUE=*
 /opt/ruby-enterprise-1.8.6-20090520/bin/rake -f
/var/rails/favstar_worker/current/Rakefile environment resque:work
D [2010-03-22 10:23:41] DEBUG: driver schedule
#<God::Conditions::ProcessRunning:0x17e64f8> in 0 seconds
D [2010-03-22 10:23:41] DEBUG: driver schedule
#<God::Conditions::Tries:0x17e6390> in 0 seconds
I [2010-03-22 10:23:41]  INFO: favprocessor moved 'init' to 'start'
I [2010-03-22 10:23:41]  INFO: favprocessor [trigger] process is running
(ProcessRunning)
D [2010-03-22 10:23:41] DEBUG: favprocessor ProcessRunning [true]
{true=>:up}
I [2010-03-22 10:23:41]  INFO: favprocessor move 'start' to 'up'
D [2010-03-22 10:23:41] DEBUG: driver schedule
#<God::Conditions::ProcessRunning:0x17e5c60> in 0 seconds
I [2010-03-22 10:23:41]  INFO: favprocessor moved 'start' to 'up'
I [2010-03-22 10:23:41]  INFO: favprocessor [ok] process is running
(ProcessRunning)
D [2010-03-22 10:23:41] DEBUG: favprocessor ProcessRunning [false]
{true=>:start}
D [2010-03-22 10:23:41] DEBUG: driver schedule
#<God::Conditions::ProcessRunning:0x17e5c60> in 10 seconds
I [2010-03-22 10:23:51]  INFO: favprocessor [trigger] process is not running
(ProcessRunning)
D [2010-03-22 10:23:51] DEBUG: favprocessor ProcessRunning [true]
{true=>:start}
I [2010-03-22 10:23:51]  INFO: favprocessor move 'up' to 'start'
I [2010-03-22 10:23:51]  INFO: favprocessor start: QUEUE=*
 /opt/ruby-enterprise-1.8.6-20090520/bin/rake -f
/var/rails/favstar_worker/current/Rakefile environment resque:work
D [2010-03-22 10:23:51] DEBUG: driver schedule
#<God::Conditions::ProcessRunning:0x17e64f8> in 0 seconds
D [2010-03-22 10:23:51] DEBUG: driver schedule
#<God::Conditions::Tries:0x17e6390> in 0 seconds
I [2010-03-22 10:23:51]  INFO: favprocessor moved 'up' to 'start'
I [2010-03-22 10:23:51]  INFO: favprocessor [trigger] process is running
(ProcessRunning)
D [2010-03-22 10:23:51] DEBUG: favprocessor ProcessRunning [true]
{true=>:up}
I [2010-03-22 10:23:51]  INFO: favprocessor move 'start' to 'up'
D [2010-03-22 10:23:51] DEBUG: driver schedule
#<God::Conditions::ProcessRunning:0x17e5c60> in 0 seconds
I [2010-03-22 10:23:51]  INFO: favprocessor moved 'start' to 'up'
I [2010-03-22 10:23:51]  INFO: favprocessor [ok] process is running
(ProcessRunning)
D [2010-03-22 10:23:51] DEBUG: favprocessor ProcessRunning [false]
{true=>:start}
D [2010-03-22 10:23:51] DEBUG: driver schedule
#<God::Conditions::ProcessRunning:0x17e5c60> in 10 seconds



I [2010-03-22 10:24:01]  INFO: favprocessor [trigger] process is not running
(ProcessRunning)
D [2010-03-22 10:24:01] DEBUG: favprocessor ProcessRunning [true]
{true=>:start}
I [2010-03-22 10:24:01]  INFO: favprocessor move 'up' to 'start'
I [2010-03-22 10:24:01]  INFO: favprocessor start: QUEUE=*
 /opt/ruby-enterprise-1.8.6-20090520/bin/rake -f
/var/rails/favstar_worker/current/Rakefile environment resque:work
D [2010-03-22 10:24:01] DEBUG: driver schedule
#<God::Conditions::ProcessRunning:0x17e64f8> in 0 seconds
D [2010-03-22 10:24:01] DEBUG: driver schedule
#<God::Conditions::Tries:0x17e6390> in 0 seconds
I [2010-03-22 10:24:01]  INFO: favprocessor moved 'up' to 'start'
I [2010-03-22 10:24:01]  INFO: favprocessor [trigger] process is running
(ProcessRunning)
D [2010-03-22 10:24:01] DEBUG: favprocessor ProcessRunning [true]
{true=>:up}
I [2010-03-22 10:24:01]  INFO: favprocessor move 'start' to 'up'
D [2010-03-22 10:24:01] DEBUG: driver schedule
#<God::Conditions::ProcessRunning:0x17e5c60> in 0 seconds
I [2010-03-22 10:24:01]  INFO: favprocessor moved 'start' to 'up'
I [2010-03-22 10:24:01]  INFO: favprocessor [ok] process is running
(ProcessRunning)
D [2010-03-22 10:24:01] DEBUG: favprocessor ProcessRunning [false]
{true=>:start}
D [2010-03-22 10:24:01] DEBUG: driver schedule
#<God::Conditions::ProcessRunning:0x17e5c60> in 10 seconds
I [2010-03-22 10:24:11]  INFO: favprocessor [trigger] process is not running
(ProcessRunning)
D [2010-03-22 10:24:11] DEBUG: favprocessor ProcessRunning [true]
{true=>:start}
I [2010-03-22 10:24:11]  INFO: favprocessor move 'up' to 'start'
I [2010-03-22 10:24:11]  INFO: favprocessor start: QUEUE=*
 /opt/ruby-enterprise-1.8.6-20090520/bin/rake -f
/var/rails/favstar_worker/current/Rakefile environment resque:work
D [2010-03-22 10:24:11] DEBUG: driver schedule
#<God::Conditions::ProcessRunning:0x17e64f8> in 0 seconds
D [2010-03-22 10:24:11] DEBUG: driver schedule
#<God::Conditions::Tries:0x17e6390> in 0 seconds
I [2010-03-22 10:24:11]  INFO: favprocessor moved 'up' to 'start'
I [2010-03-22 10:24:11]  INFO: favprocessor [trigger] process is running
(ProcessRunning)
D [2010-03-22 10:24:11] DEBUG: favprocessor ProcessRunning [true]
{true=>:up}
I [2010-03-22 10:24:11]  INFO: favprocessor move 'start' to 'up'
D [2010-03-22 10:24:11] DEBUG: driver schedule
#<God::Conditions::ProcessRunning:0x17e5c60> in 0 seconds
I [2010-03-22 10:24:11]  INFO: favprocessor moved 'start' to 'up'
I [2010-03-22 10:24:11]  INFO: favprocessor [ok] process is running
(ProcessRunning)
D [2010-03-22 10:24:11] DEBUG: favprocessor ProcessRunning [false]
{true=>:start}
D [2010-03-22 10:24:11] DEBUG: driver schedule
#<God::Conditions::ProcessRunning:0x17e5c60> in 10 seconds



I [2010-03-22 10:24:21]  INFO: favprocessor [trigger] process is not running
(ProcessRunning)
D [2010-03-22 10:24:21] DEBUG: favprocessor ProcessRunning [true]
{true=>:start}
I [2010-03-22 10:24:21]  INFO: favprocessor move 'up' to 'start'
I [2010-03-22 10:24:21]  INFO: favprocessor start: QUEUE=*
 /opt/ruby-enterprise-1.8.6-20090520/bin/rake -f
/var/rails/favstar_worker/current/Rakefile environment resque:work
D [2010-03-22 10:24:21] DEBUG: driver schedule
#<God::Conditions::ProcessRunning:0x17e64f8> in 0 seconds
D [2010-03-22 10:24:21] DEBUG: driver schedule
#<God::Conditions::Tries:0x17e6390> in 0 seconds
I [2010-03-22 10:24:21]  INFO: favprocessor moved 'up' to 'start'
I [2010-03-22 10:24:21]  INFO: favprocessor [trigger] process is running
(ProcessRunning)
D [2010-03-22 10:24:21] DEBUG: favprocessor ProcessRunning [true]
{true=>:up}
I [2010-03-22 10:24:21]  INFO: favprocessor move 'start' to 'up'
D [2010-03-22 10:24:21] DEBUG: driver schedule
#<God::Conditions::ProcessRunning:0x17e5c60> in 0 seconds
I [2010-03-22 10:24:21]  INFO: favprocessor moved 'start' to 'up'
I [2010-03-22 10:24:21]  INFO: favprocessor [ok] process is running
(ProcessRunning)
D [2010-03-22 10:24:21] DEBUG: favprocessor ProcessRunning [false]
{true=>:start}
D [2010-03-22 10:24:21] DEBUG: driver schedule
#<God::Conditions::ProcessRunning:0x17e5c60> in 10 seconds

Re: [resque] Resque and God?

From:
Andrew Nordman
Date:
2010-03-23 @ 14:34
I had this exact problem while I was initially configuring god to manage
resque, but I can't for the life of me recall exactly how I fixed it.  When
you don't run god without -D, does god run but not load the resque config,
or does god just not start?

- Andrew Nordman

On Mon, Mar 22, 2010 at 5:00 PM, Tim Haines <tmhaines@gmail.com> wrote:

> Hey guys,
>
> Has anyone had experience getting god to monitor your resque workers?  I
> tried to set it up last night, but hit a brick wall.  I basically took the
> example resque.god from github (
> http://github.com/defunkt/resque/blob/master/examples/god/resque.god), and
> adapted it to my paths and queues etc, but it just won't get started.  I've
> attached a debug log of what it's doing below.  Can anyone explain what's
> going wrong or see something obvious?
>
> The weird thing is, the same script (with adjusted paths) works locally.
>  Both local and production using god 0.8.
>
> Additionally, if I run the command from the start line in the terminal on
> production, it works fine too..
>
>
> : sudo /opt/ruby-enterprise-1.8.6-20090520/bin/god -c
>  ~/god/fav_processor.god -D --log-level debug
> I [2010-03-22 10:23:41]  INFO: Loading /home/favstar/god/fav_processor.god
> I [2010-03-22 10:23:41]  INFO: Using pid file directory: /var/run/god
> I [2010-03-22 10:23:41]  INFO: Started on drbunix:///tmp/god.17165.sock
> I [2010-03-22 10:23:41]  INFO: favprocessor move 'unmonitored' to 'init'
> D [2010-03-22 10:23:41] DEBUG: driver schedule
> #<God::Conditions::ProcessRunning:0x17e6a48> in 0 seconds
> I [2010-03-22 10:23:41]  INFO: favprocessor moved 'unmonitored' to 'init'
> I [2010-03-22 10:23:41]  INFO: favprocessor [trigger] process is not
> running (ProcessRunning)
> D [2010-03-22 10:23:41] DEBUG: favprocessor ProcessRunning [false]
> {false=>:start, true=>:up}
> I [2010-03-22 10:23:41]  INFO: favprocessor move 'init' to 'start'
> I [2010-03-22 10:23:41]  INFO: favprocessor start: QUEUE=*
>  /opt/ruby-enterprise-1.8.6-20090520/bin/rake -f
> /var/rails/favstar_worker/current/Rakefile environment resque:work
> D [2010-03-22 10:23:41] DEBUG: driver schedule
> #<God::Conditions::ProcessRunning:0x17e64f8> in 0 seconds
> D [2010-03-22 10:23:41] DEBUG: driver schedule
> #<God::Conditions::Tries:0x17e6390> in 0 seconds
> I [2010-03-22 10:23:41]  INFO: favprocessor moved 'init' to 'start'
> I [2010-03-22 10:23:41]  INFO: favprocessor [trigger] process is running
> (ProcessRunning)
> D [2010-03-22 10:23:41] DEBUG: favprocessor ProcessRunning [true]
> {true=>:up}
> I [2010-03-22 10:23:41]  INFO: favprocessor move 'start' to 'up'
> D [2010-03-22 10:23:41] DEBUG: driver schedule
> #<God::Conditions::ProcessRunning:0x17e5c60> in 0 seconds
> I [2010-03-22 10:23:41]  INFO: favprocessor moved 'start' to 'up'
> I [2010-03-22 10:23:41]  INFO: favprocessor [ok] process is running
> (ProcessRunning)
> D [2010-03-22 10:23:41] DEBUG: favprocessor ProcessRunning [false]
> {true=>:start}
> D [2010-03-22 10:23:41] DEBUG: driver schedule
> #<God::Conditions::ProcessRunning:0x17e5c60> in 10 seconds
> I [2010-03-22 10:23:51]  INFO: favprocessor [trigger] process is not
> running (ProcessRunning)
> D [2010-03-22 10:23:51] DEBUG: favprocessor ProcessRunning [true]
> {true=>:start}
> I [2010-03-22 10:23:51]  INFO: favprocessor move 'up' to 'start'
> I [2010-03-22 10:23:51]  INFO: favprocessor start: QUEUE=*
>  /opt/ruby-enterprise-1.8.6-20090520/bin/rake -f
> /var/rails/favstar_worker/current/Rakefile environment resque:work
> D [2010-03-22 10:23:51] DEBUG: driver schedule
> #<God::Conditions::ProcessRunning:0x17e64f8> in 0 seconds
> D [2010-03-22 10:23:51] DEBUG: driver schedule
> #<God::Conditions::Tries:0x17e6390> in 0 seconds
> I [2010-03-22 10:23:51]  INFO: favprocessor moved 'up' to 'start'
> I [2010-03-22 10:23:51]  INFO: favprocessor [trigger] process is running
> (ProcessRunning)
> D [2010-03-22 10:23:51] DEBUG: favprocessor ProcessRunning [true]
> {true=>:up}
> I [2010-03-22 10:23:51]  INFO: favprocessor move 'start' to 'up'
> D [2010-03-22 10:23:51] DEBUG: driver schedule
> #<God::Conditions::ProcessRunning:0x17e5c60> in 0 seconds
> I [2010-03-22 10:23:51]  INFO: favprocessor moved 'start' to 'up'
> I [2010-03-22 10:23:51]  INFO: favprocessor [ok] process is running
> (ProcessRunning)
> D [2010-03-22 10:23:51] DEBUG: favprocessor ProcessRunning [false]
> {true=>:start}
> D [2010-03-22 10:23:51] DEBUG: driver schedule
> #<God::Conditions::ProcessRunning:0x17e5c60> in 10 seconds
>
>
>
> I [2010-03-22 10:24:01]  INFO: favprocessor [trigger] process is not
> running (ProcessRunning)
> D [2010-03-22 10:24:01] DEBUG: favprocessor ProcessRunning [true]
> {true=>:start}
> I [2010-03-22 10:24:01]  INFO: favprocessor move 'up' to 'start'
> I [2010-03-22 10:24:01]  INFO: favprocessor start: QUEUE=*
>  /opt/ruby-enterprise-1.8.6-20090520/bin/rake -f
> /var/rails/favstar_worker/current/Rakefile environment resque:work
> D [2010-03-22 10:24:01] DEBUG: driver schedule
> #<God::Conditions::ProcessRunning:0x17e64f8> in 0 seconds
> D [2010-03-22 10:24:01] DEBUG: driver schedule
> #<God::Conditions::Tries:0x17e6390> in 0 seconds
> I [2010-03-22 10:24:01]  INFO: favprocessor moved 'up' to 'start'
> I [2010-03-22 10:24:01]  INFO: favprocessor [trigger] process is running
> (ProcessRunning)
> D [2010-03-22 10:24:01] DEBUG: favprocessor ProcessRunning [true]
> {true=>:up}
> I [2010-03-22 10:24:01]  INFO: favprocessor move 'start' to 'up'
> D [2010-03-22 10:24:01] DEBUG: driver schedule
> #<God::Conditions::ProcessRunning:0x17e5c60> in 0 seconds
> I [2010-03-22 10:24:01]  INFO: favprocessor moved 'start' to 'up'
> I [2010-03-22 10:24:01]  INFO: favprocessor [ok] process is running
> (ProcessRunning)
> D [2010-03-22 10:24:01] DEBUG: favprocessor ProcessRunning [false]
> {true=>:start}
> D [2010-03-22 10:24:01] DEBUG: driver schedule
> #<God::Conditions::ProcessRunning:0x17e5c60> in 10 seconds
> I [2010-03-22 10:24:11]  INFO: favprocessor [trigger] process is not
> running (ProcessRunning)
> D [2010-03-22 10:24:11] DEBUG: favprocessor ProcessRunning [true]
> {true=>:start}
> I [2010-03-22 10:24:11]  INFO: favprocessor move 'up' to 'start'
> I [2010-03-22 10:24:11]  INFO: favprocessor start: QUEUE=*
>  /opt/ruby-enterprise-1.8.6-20090520/bin/rake -f
> /var/rails/favstar_worker/current/Rakefile environment resque:work
> D [2010-03-22 10:24:11] DEBUG: driver schedule
> #<God::Conditions::ProcessRunning:0x17e64f8> in 0 seconds
> D [2010-03-22 10:24:11] DEBUG: driver schedule
> #<God::Conditions::Tries:0x17e6390> in 0 seconds
> I [2010-03-22 10:24:11]  INFO: favprocessor moved 'up' to 'start'
> I [2010-03-22 10:24:11]  INFO: favprocessor [trigger] process is running
> (ProcessRunning)
> D [2010-03-22 10:24:11] DEBUG: favprocessor ProcessRunning [true]
> {true=>:up}
> I [2010-03-22 10:24:11]  INFO: favprocessor move 'start' to 'up'
> D [2010-03-22 10:24:11] DEBUG: driver schedule
> #<God::Conditions::ProcessRunning:0x17e5c60> in 0 seconds
> I [2010-03-22 10:24:11]  INFO: favprocessor moved 'start' to 'up'
> I [2010-03-22 10:24:11]  INFO: favprocessor [ok] process is running
> (ProcessRunning)
> D [2010-03-22 10:24:11] DEBUG: favprocessor ProcessRunning [false]
> {true=>:start}
> D [2010-03-22 10:24:11] DEBUG: driver schedule
> #<God::Conditions::ProcessRunning:0x17e5c60> in 10 seconds
>
>
>
> I [2010-03-22 10:24:21]  INFO: favprocessor [trigger] process is not
> running (ProcessRunning)
> D [2010-03-22 10:24:21] DEBUG: favprocessor ProcessRunning [true]
> {true=>:start}
> I [2010-03-22 10:24:21]  INFO: favprocessor move 'up' to 'start'
> I [2010-03-22 10:24:21]  INFO: favprocessor start: QUEUE=*
>  /opt/ruby-enterprise-1.8.6-20090520/bin/rake -f
> /var/rails/favstar_worker/current/Rakefile environment resque:work
> D [2010-03-22 10:24:21] DEBUG: driver schedule
> #<God::Conditions::ProcessRunning:0x17e64f8> in 0 seconds
> D [2010-03-22 10:24:21] DEBUG: driver schedule
> #<God::Conditions::Tries:0x17e6390> in 0 seconds
> I [2010-03-22 10:24:21]  INFO: favprocessor moved 'up' to 'start'
> I [2010-03-22 10:24:21]  INFO: favprocessor [trigger] process is running
> (ProcessRunning)
> D [2010-03-22 10:24:21] DEBUG: favprocessor ProcessRunning [true]
> {true=>:up}
> I [2010-03-22 10:24:21]  INFO: favprocessor move 'start' to 'up'
> D [2010-03-22 10:24:21] DEBUG: driver schedule
> #<God::Conditions::ProcessRunning:0x17e5c60> in 0 seconds
> I [2010-03-22 10:24:21]  INFO: favprocessor moved 'start' to 'up'
> I [2010-03-22 10:24:21]  INFO: favprocessor [ok] process is running
> (ProcessRunning)
> D [2010-03-22 10:24:21] DEBUG: favprocessor ProcessRunning [false]
> {true=>:start}
> D [2010-03-22 10:24:21] DEBUG: driver schedule
> #<God::Conditions::ProcessRunning:0x17e5c60> in 10 seconds
>
>

Re: [resque] Resque and God?

From:
Tim Haines
Date:
2010-03-23 @ 20:04
Hey Andrew,

Thanks for following up.  I discovered 2 things which helped me solve this.
 First off, in god, you can use the undocumented .log property on watches to
set a log file for stdout output to get logged in (otherwise it goes to
/dev/null).  After I learnt that, I resolved a couple of problems, and then
got stuck again when it told me it couldn't find the rake file.  It turns
out that rake while running under god somehow checks permissions
differently, and I had to grant read access on the parent's parent folder.
 For some reason this wasn't a problem when I was simply running the workers
from the terminal.

So this solved the problems - and I now I have my resque workers monitored
by god.

I still have a couple of things I'm not sure about though:

1) What does god do when you run god stop on one of the workers, or god
terminate.  God terminate seems to let the workers continue the job they're
working on.  I'm not quite sure how I should adjust capistrano to handle
rollouts of new worker code etc..

2) The log file seems to only capture the stdout output of the parent
process.  Not the child process, and not output from the workers themselves.
 Ideally I'd like to capture this too  - but perhaps I'll move to a
different logging mechanism once I get the time.

Cheers,

Tim.

On Wed, Mar 24, 2010 at 3:34 AM, Andrew Nordman <cadwallion@gmail.com>wrote:

> I had this exact problem while I was initially configuring god to manage
> resque, but I can't for the life of me recall exactly how I fixed it.  When
> you don't run god without -D, does god run but not load the resque config,
> or does god just not start?
>
> - Andrew Nordman
>
>
> On Mon, Mar 22, 2010 at 5:00 PM, Tim Haines <tmhaines@gmail.com> wrote:
>
>> Hey guys,
>>
>> Has anyone had experience getting god to monitor your resque workers?  I
>> tried to set it up last night, but hit a brick wall.  I basically took the
>> example resque.god from github (
>> http://github.com/defunkt/resque/blob/master/examples/god/resque.god),
>> and adapted it to my paths and queues etc, but it just won't get started.
>>  I've attached a debug log of what it's doing below.  Can anyone explain
>> what's going wrong or see something obvious?
>>
>> The weird thing is, the same script (with adjusted paths) works locally.
>>  Both local and production using god 0.8.
>>
>> Additionally, if I run the command from the start line in the terminal on
>> production, it works fine too..
>>
>>
>> : sudo /opt/ruby-enterprise-1.8.6-20090520/bin/god -c
>>  ~/god/fav_processor.god -D --log-level debug
>> I [2010-03-22 10:23:41]  INFO: Loading /home/favstar/god/fav_processor.god
>> I [2010-03-22 10:23:41]  INFO: Using pid file directory: /var/run/god
>> I [2010-03-22 10:23:41]  INFO: Started on drbunix:///tmp/god.17165.sock
>> I [2010-03-22 10:23:41]  INFO: favprocessor move 'unmonitored' to 'init'
>> D [2010-03-22 10:23:41] DEBUG: driver schedule
>> #<God::Conditions::ProcessRunning:0x17e6a48> in 0 seconds
>>  I [2010-03-22 10:23:41]  INFO: favprocessor moved 'unmonitored' to 'init'
>> I [2010-03-22 10:23:41]  INFO: favprocessor [trigger] process is not
>> running (ProcessRunning)
>> D [2010-03-22 10:23:41] DEBUG: favprocessor ProcessRunning [false]
>> {false=>:start, true=>:up}
>> I [2010-03-22 10:23:41]  INFO: favprocessor move 'init' to 'start'
>> I [2010-03-22 10:23:41]  INFO: favprocessor start: QUEUE=*
>>  /opt/ruby-enterprise-1.8.6-20090520/bin/rake -f
>> /var/rails/favstar_worker/current/Rakefile environment resque:work
>> D [2010-03-22 10:23:41] DEBUG: driver schedule
>> #<God::Conditions::ProcessRunning:0x17e64f8> in 0 seconds
>> D [2010-03-22 10:23:41] DEBUG: driver schedule
>> #<God::Conditions::Tries:0x17e6390> in 0 seconds
>> I [2010-03-22 10:23:41]  INFO: favprocessor moved 'init' to 'start'
>> I [2010-03-22 10:23:41]  INFO: favprocessor [trigger] process is running
>> (ProcessRunning)
>> D [2010-03-22 10:23:41] DEBUG: favprocessor ProcessRunning [true]
>> {true=>:up}
>> I [2010-03-22 10:23:41]  INFO: favprocessor move 'start' to 'up'
>> D [2010-03-22 10:23:41] DEBUG: driver schedule
>> #<God::Conditions::ProcessRunning:0x17e5c60> in 0 seconds
>> I [2010-03-22 10:23:41]  INFO: favprocessor moved 'start' to 'up'
>> I [2010-03-22 10:23:41]  INFO: favprocessor [ok] process is running
>> (ProcessRunning)
>> D [2010-03-22 10:23:41] DEBUG: favprocessor ProcessRunning [false]
>> {true=>:start}
>> D [2010-03-22 10:23:41] DEBUG: driver schedule
>> #<God::Conditions::ProcessRunning:0x17e5c60> in 10 seconds
>> I [2010-03-22 10:23:51]  INFO: favprocessor [trigger] process is not
>> running (ProcessRunning)
>> D [2010-03-22 10:23:51] DEBUG: favprocessor ProcessRunning [true]
>> {true=>:start}
>> I [2010-03-22 10:23:51]  INFO: favprocessor move 'up' to 'start'
>> I [2010-03-22 10:23:51]  INFO: favprocessor start: QUEUE=*
>>  /opt/ruby-enterprise-1.8.6-20090520/bin/rake -f
>> /var/rails/favstar_worker/current/Rakefile environment resque:work
>> D [2010-03-22 10:23:51] DEBUG: driver schedule
>> #<God::Conditions::ProcessRunning:0x17e64f8> in 0 seconds
>> D [2010-03-22 10:23:51] DEBUG: driver schedule
>> #<God::Conditions::Tries:0x17e6390> in 0 seconds
>> I [2010-03-22 10:23:51]  INFO: favprocessor moved 'up' to 'start'
>> I [2010-03-22 10:23:51]  INFO: favprocessor [trigger] process is running
>> (ProcessRunning)
>> D [2010-03-22 10:23:51] DEBUG: favprocessor ProcessRunning [true]
>> {true=>:up}
>> I [2010-03-22 10:23:51]  INFO: favprocessor move 'start' to 'up'
>> D [2010-03-22 10:23:51] DEBUG: driver schedule
>> #<God::Conditions::ProcessRunning:0x17e5c60> in 0 seconds
>> I [2010-03-22 10:23:51]  INFO: favprocessor moved 'start' to 'up'
>> I [2010-03-22 10:23:51]  INFO: favprocessor [ok] process is running
>> (ProcessRunning)
>> D [2010-03-22 10:23:51] DEBUG: favprocessor ProcessRunning [false]
>> {true=>:start}
>> D [2010-03-22 10:23:51] DEBUG: driver schedule
>> #<God::Conditions::ProcessRunning:0x17e5c60> in 10 seconds
>>
>>
>>
>> I [2010-03-22 10:24:01]  INFO: favprocessor [trigger] process is not
>> running (ProcessRunning)
>> D [2010-03-22 10:24:01] DEBUG: favprocessor ProcessRunning [true]
>> {true=>:start}
>> I [2010-03-22 10:24:01]  INFO: favprocessor move 'up' to 'start'
>> I [2010-03-22 10:24:01]  INFO: favprocessor start: QUEUE=*
>>  /opt/ruby-enterprise-1.8.6-20090520/bin/rake -f
>> /var/rails/favstar_worker/current/Rakefile environment resque:work
>> D [2010-03-22 10:24:01] DEBUG: driver schedule
>> #<God::Conditions::ProcessRunning:0x17e64f8> in 0 seconds
>> D [2010-03-22 10:24:01] DEBUG: driver schedule
>> #<God::Conditions::Tries:0x17e6390> in 0 seconds
>> I [2010-03-22 10:24:01]  INFO: favprocessor moved 'up' to 'start'
>> I [2010-03-22 10:24:01]  INFO: favprocessor [trigger] process is running
>> (ProcessRunning)
>> D [2010-03-22 10:24:01] DEBUG: favprocessor ProcessRunning [true]
>> {true=>:up}
>> I [2010-03-22 10:24:01]  INFO: favprocessor move 'start' to 'up'
>> D [2010-03-22 10:24:01] DEBUG: driver schedule
>> #<God::Conditions::ProcessRunning:0x17e5c60> in 0 seconds
>> I [2010-03-22 10:24:01]  INFO: favprocessor moved 'start' to 'up'
>> I [2010-03-22 10:24:01]  INFO: favprocessor [ok] process is running
>> (ProcessRunning)
>> D [2010-03-22 10:24:01] DEBUG: favprocessor ProcessRunning [false]
>> {true=>:start}
>> D [2010-03-22 10:24:01] DEBUG: driver schedule
>> #<God::Conditions::ProcessRunning:0x17e5c60> in 10 seconds
>> I [2010-03-22 10:24:11]  INFO: favprocessor [trigger] process is not
>> running (ProcessRunning)
>> D [2010-03-22 10:24:11] DEBUG: favprocessor ProcessRunning [true]
>> {true=>:start}
>> I [2010-03-22 10:24:11]  INFO: favprocessor move 'up' to 'start'
>> I [2010-03-22 10:24:11]  INFO: favprocessor start: QUEUE=*
>>  /opt/ruby-enterprise-1.8.6-20090520/bin/rake -f
>> /var/rails/favstar_worker/current/Rakefile environment resque:work
>> D [2010-03-22 10:24:11] DEBUG: driver schedule
>> #<God::Conditions::ProcessRunning:0x17e64f8> in 0 seconds
>> D [2010-03-22 10:24:11] DEBUG: driver schedule
>> #<God::Conditions::Tries:0x17e6390> in 0 seconds
>> I [2010-03-22 10:24:11]  INFO: favprocessor moved 'up' to 'start'
>> I [2010-03-22 10:24:11]  INFO: favprocessor [trigger] process is running
>> (ProcessRunning)
>> D [2010-03-22 10:24:11] DEBUG: favprocessor ProcessRunning [true]
>> {true=>:up}
>> I [2010-03-22 10:24:11]  INFO: favprocessor move 'start' to 'up'
>> D [2010-03-22 10:24:11] DEBUG: driver schedule
>> #<God::Conditions::ProcessRunning:0x17e5c60> in 0 seconds
>> I [2010-03-22 10:24:11]  INFO: favprocessor moved 'start' to 'up'
>> I [2010-03-22 10:24:11]  INFO: favprocessor [ok] process is running
>> (ProcessRunning)
>> D [2010-03-22 10:24:11] DEBUG: favprocessor ProcessRunning [false]
>> {true=>:start}
>> D [2010-03-22 10:24:11] DEBUG: driver schedule
>> #<God::Conditions::ProcessRunning:0x17e5c60> in 10 seconds
>>
>>
>>
>> I [2010-03-22 10:24:21]  INFO: favprocessor [trigger] process is not
>> running (ProcessRunning)
>> D [2010-03-22 10:24:21] DEBUG: favprocessor ProcessRunning [true]
>> {true=>:start}
>> I [2010-03-22 10:24:21]  INFO: favprocessor move 'up' to 'start'
>> I [2010-03-22 10:24:21]  INFO: favprocessor start: QUEUE=*
>>  /opt/ruby-enterprise-1.8.6-20090520/bin/rake -f
>> /var/rails/favstar_worker/current/Rakefile environment resque:work
>> D [2010-03-22 10:24:21] DEBUG: driver schedule
>> #<God::Conditions::ProcessRunning:0x17e64f8> in 0 seconds
>> D [2010-03-22 10:24:21] DEBUG: driver schedule
>> #<God::Conditions::Tries:0x17e6390> in 0 seconds
>> I [2010-03-22 10:24:21]  INFO: favprocessor moved 'up' to 'start'
>> I [2010-03-22 10:24:21]  INFO: favprocessor [trigger] process is running
>> (ProcessRunning)
>> D [2010-03-22 10:24:21] DEBUG: favprocessor ProcessRunning [true]
>> {true=>:up}
>> I [2010-03-22 10:24:21]  INFO: favprocessor move 'start' to 'up'
>> D [2010-03-22 10:24:21] DEBUG: driver schedule
>> #<God::Conditions::ProcessRunning:0x17e5c60> in 0 seconds
>> I [2010-03-22 10:24:21]  INFO: favprocessor moved 'start' to 'up'
>> I [2010-03-22 10:24:21]  INFO: favprocessor [ok] process is running
>> (ProcessRunning)
>> D [2010-03-22 10:24:21] DEBUG: favprocessor ProcessRunning [false]
>> {true=>:start}
>> D [2010-03-22 10:24:21] DEBUG: driver schedule
>> #<God::Conditions::ProcessRunning:0x17e5c60> in 10 seconds
>>
>>
>

Re: [resque] Resque and God?

From:
Andrew Nordman
Date:
2010-03-23 @ 20:47
Tim,

Glad to hear you got it up and running.

1) At least how I have it setup, I added a resque:restart task which runs
'god load <configs>' followed by 'god restart resque' to clean my workers
out.  If you want fine grain detail, you can run a 'god signal <sig> resque'
to send a specific signal to all of resque or particular workers.  The
signals that resque responds to is in the README.  If you're worried about
killing a worker mid-job, this may be the route for you.

2) I configure a logger object for each of my workers to either separate
logs per worker or joint logs depending upon the worker, so unfortunately I
don't have much insight into the way god logs child and parent processes
under its watch.

- Andrew

On Tue, Mar 23, 2010 at 3:04 PM, Tim Haines <tmhaines@gmail.com> wrote:

> Hey Andrew,
>
> Thanks for following up.  I discovered 2 things which helped me solve this.
>  First off, in god, you can use the undocumented .log property on watches to
> set a log file for stdout output to get logged in (otherwise it goes to
> /dev/null).  After I learnt that, I resolved a couple of problems, and then
> got stuck again when it told me it couldn't find the rake file.  It turns
> out that rake while running under god somehow checks permissions
> differently, and I had to grant read access on the parent's parent folder.
>  For some reason this wasn't a problem when I was simply running the workers
> from the terminal.
>
> So this solved the problems - and I now I have my resque workers monitored
> by god.
>
> I still have a couple of things I'm not sure about though:
>
> 1) What does god do when you run god stop on one of the workers, or god
> terminate.  God terminate seems to let the workers continue the job they're
> working on.  I'm not quite sure how I should adjust capistrano to handle
> rollouts of new worker code etc..
>
> 2) The log file seems to only capture the stdout output of the parent
> process.  Not the child process, and not output from the workers themselves.
>  Ideally I'd like to capture this too  - but perhaps I'll move to a
> different logging mechanism once I get the time.
>
> Cheers,
>
> Tim.
>
>
> On Wed, Mar 24, 2010 at 3:34 AM, Andrew Nordman <cadwallion@gmail.com>wrote:
>
>> I had this exact problem while I was initially configuring god to manage
>> resque, but I can't for the life of me recall exactly how I fixed it.  When
>> you don't run god without -D, does god run but not load the resque config,
>> or does god just not start?
>>
>> - Andrew Nordman
>>
>>
>> On Mon, Mar 22, 2010 at 5:00 PM, Tim Haines <tmhaines@gmail.com> wrote:
>>
>>> Hey guys,
>>>
>>> Has anyone had experience getting god to monitor your resque workers?  I
>>> tried to set it up last night, but hit a brick wall.  I basically took the
>>> example resque.god from github (
>>> http://github.com/defunkt/resque/blob/master/examples/god/resque.god),
>>> and adapted it to my paths and queues etc, but it just won't get started.
>>>  I've attached a debug log of what it's doing below.  Can anyone explain
>>> what's going wrong or see something obvious?
>>>
>>> The weird thing is, the same script (with adjusted paths) works locally.
>>>  Both local and production using god 0.8.
>>>
>>> Additionally, if I run the command from the start line in the terminal on
>>> production, it works fine too..
>>>
>>>
>>> : sudo /opt/ruby-enterprise-1.8.6-20090520/bin/god -c
>>>  ~/god/fav_processor.god -D --log-level debug
>>> I [2010-03-22 10:23:41]  INFO: Loading
>>> /home/favstar/god/fav_processor.god
>>> I [2010-03-22 10:23:41]  INFO: Using pid file directory: /var/run/god
>>> I [2010-03-22 10:23:41]  INFO: Started on drbunix:///tmp/god.17165.sock
>>> I [2010-03-22 10:23:41]  INFO: favprocessor move 'unmonitored' to 'init'
>>> D [2010-03-22 10:23:41] DEBUG: driver schedule
>>> #<God::Conditions::ProcessRunning:0x17e6a48> in 0 seconds
>>>  I [2010-03-22 10:23:41]  INFO: favprocessor moved 'unmonitored' to
>>> 'init'
>>> I [2010-03-22 10:23:41]  INFO: favprocessor [trigger] process is not
>>> running (ProcessRunning)
>>> D [2010-03-22 10:23:41] DEBUG: favprocessor ProcessRunning [false]
>>> {false=>:start, true=>:up}
>>> I [2010-03-22 10:23:41]  INFO: favprocessor move 'init' to 'start'
>>> I [2010-03-22 10:23:41]  INFO: favprocessor start: QUEUE=*
>>>  /opt/ruby-enterprise-1.8.6-20090520/bin/rake -f
>>> /var/rails/favstar_worker/current/Rakefile environment resque:work
>>> D [2010-03-22 10:23:41] DEBUG: driver schedule
>>> #<God::Conditions::ProcessRunning:0x17e64f8> in 0 seconds
>>> D [2010-03-22 10:23:41] DEBUG: driver schedule
>>> #<God::Conditions::Tries:0x17e6390> in 0 seconds
>>> I [2010-03-22 10:23:41]  INFO: favprocessor moved 'init' to 'start'
>>> I [2010-03-22 10:23:41]  INFO: favprocessor [trigger] process is running
>>> (ProcessRunning)
>>> D [2010-03-22 10:23:41] DEBUG: favprocessor ProcessRunning [true]
>>> {true=>:up}
>>> I [2010-03-22 10:23:41]  INFO: favprocessor move 'start' to 'up'
>>> D [2010-03-22 10:23:41] DEBUG: driver schedule
>>> #<God::Conditions::ProcessRunning:0x17e5c60> in 0 seconds
>>> I [2010-03-22 10:23:41]  INFO: favprocessor moved 'start' to 'up'
>>> I [2010-03-22 10:23:41]  INFO: favprocessor [ok] process is running
>>> (ProcessRunning)
>>> D [2010-03-22 10:23:41] DEBUG: favprocessor ProcessRunning [false]
>>> {true=>:start}
>>> D [2010-03-22 10:23:41] DEBUG: driver schedule
>>> #<God::Conditions::ProcessRunning:0x17e5c60> in 10 seconds
>>> I [2010-03-22 10:23:51]  INFO: favprocessor [trigger] process is not
>>> running (ProcessRunning)
>>> D [2010-03-22 10:23:51] DEBUG: favprocessor ProcessRunning [true]
>>> {true=>:start}
>>> I [2010-03-22 10:23:51]  INFO: favprocessor move 'up' to 'start'
>>> I [2010-03-22 10:23:51]  INFO: favprocessor start: QUEUE=*
>>>  /opt/ruby-enterprise-1.8.6-20090520/bin/rake -f
>>> /var/rails/favstar_worker/current/Rakefile environment resque:work
>>> D [2010-03-22 10:23:51] DEBUG: driver schedule
>>> #<God::Conditions::ProcessRunning:0x17e64f8> in 0 seconds
>>> D [2010-03-22 10:23:51] DEBUG: driver schedule
>>> #<God::Conditions::Tries:0x17e6390> in 0 seconds
>>> I [2010-03-22 10:23:51]  INFO: favprocessor moved 'up' to 'start'
>>> I [2010-03-22 10:23:51]  INFO: favprocessor [trigger] process is running
>>> (ProcessRunning)
>>> D [2010-03-22 10:23:51] DEBUG: favprocessor ProcessRunning [true]
>>> {true=>:up}
>>> I [2010-03-22 10:23:51]  INFO: favprocessor move 'start' to 'up'
>>> D [2010-03-22 10:23:51] DEBUG: driver schedule
>>> #<God::Conditions::ProcessRunning:0x17e5c60> in 0 seconds
>>> I [2010-03-22 10:23:51]  INFO: favprocessor moved 'start' to 'up'
>>> I [2010-03-22 10:23:51]  INFO: favprocessor [ok] process is running
>>> (ProcessRunning)
>>> D [2010-03-22 10:23:51] DEBUG: favprocessor ProcessRunning [false]
>>> {true=>:start}
>>> D [2010-03-22 10:23:51] DEBUG: driver schedule
>>> #<God::Conditions::ProcessRunning:0x17e5c60> in 10 seconds
>>>
>>>
>>>
>>> I [2010-03-22 10:24:01]  INFO: favprocessor [trigger] process is not
>>> running (ProcessRunning)
>>> D [2010-03-22 10:24:01] DEBUG: favprocessor ProcessRunning [true]
>>> {true=>:start}
>>> I [2010-03-22 10:24:01]  INFO: favprocessor move 'up' to 'start'
>>> I [2010-03-22 10:24:01]  INFO: favprocessor start: QUEUE=*
>>>  /opt/ruby-enterprise-1.8.6-20090520/bin/rake -f
>>> /var/rails/favstar_worker/current/Rakefile environment resque:work
>>> D [2010-03-22 10:24:01] DEBUG: driver schedule
>>> #<God::Conditions::ProcessRunning:0x17e64f8> in 0 seconds
>>> D [2010-03-22 10:24:01] DEBUG: driver schedule
>>> #<God::Conditions::Tries:0x17e6390> in 0 seconds
>>> I [2010-03-22 10:24:01]  INFO: favprocessor moved 'up' to 'start'
>>> I [2010-03-22 10:24:01]  INFO: favprocessor [trigger] process is running
>>> (ProcessRunning)
>>> D [2010-03-22 10:24:01] DEBUG: favprocessor ProcessRunning [true]
>>> {true=>:up}
>>> I [2010-03-22 10:24:01]  INFO: favprocessor move 'start' to 'up'
>>> D [2010-03-22 10:24:01] DEBUG: driver schedule
>>> #<God::Conditions::ProcessRunning:0x17e5c60> in 0 seconds
>>> I [2010-03-22 10:24:01]  INFO: favprocessor moved 'start' to 'up'
>>> I [2010-03-22 10:24:01]  INFO: favprocessor [ok] process is running
>>> (ProcessRunning)
>>> D [2010-03-22 10:24:01] DEBUG: favprocessor ProcessRunning [false]
>>> {true=>:start}
>>> D [2010-03-22 10:24:01] DEBUG: driver schedule
>>> #<God::Conditions::ProcessRunning:0x17e5c60> in 10 seconds
>>> I [2010-03-22 10:24:11]  INFO: favprocessor [trigger] process is not
>>> running (ProcessRunning)
>>> D [2010-03-22 10:24:11] DEBUG: favprocessor ProcessRunning [true]
>>> {true=>:start}
>>> I [2010-03-22 10:24:11]  INFO: favprocessor move 'up' to 'start'
>>> I [2010-03-22 10:24:11]  INFO: favprocessor start: QUEUE=*
>>>  /opt/ruby-enterprise-1.8.6-20090520/bin/rake -f
>>> /var/rails/favstar_worker/current/Rakefile environment resque:work
>>> D [2010-03-22 10:24:11] DEBUG: driver schedule
>>> #<God::Conditions::ProcessRunning:0x17e64f8> in 0 seconds
>>> D [2010-03-22 10:24:11] DEBUG: driver schedule
>>> #<God::Conditions::Tries:0x17e6390> in 0 seconds
>>> I [2010-03-22 10:24:11]  INFO: favprocessor moved 'up' to 'start'
>>> I [2010-03-22 10:24:11]  INFO: favprocessor [trigger] process is running
>>> (ProcessRunning)
>>> D [2010-03-22 10:24:11] DEBUG: favprocessor ProcessRunning [true]
>>> {true=>:up}
>>> I [2010-03-22 10:24:11]  INFO: favprocessor move 'start' to 'up'
>>> D [2010-03-22 10:24:11] DEBUG: driver schedule
>>> #<God::Conditions::ProcessRunning:0x17e5c60> in 0 seconds
>>> I [2010-03-22 10:24:11]  INFO: favprocessor moved 'start' to 'up'
>>> I [2010-03-22 10:24:11]  INFO: favprocessor [ok] process is running
>>> (ProcessRunning)
>>> D [2010-03-22 10:24:11] DEBUG: favprocessor ProcessRunning [false]
>>> {true=>:start}
>>> D [2010-03-22 10:24:11] DEBUG: driver schedule
>>> #<God::Conditions::ProcessRunning:0x17e5c60> in 10 seconds
>>>
>>>
>>>
>>> I [2010-03-22 10:24:21]  INFO: favprocessor [trigger] process is not
>>> running (ProcessRunning)
>>> D [2010-03-22 10:24:21] DEBUG: favprocessor ProcessRunning [true]
>>> {true=>:start}
>>> I [2010-03-22 10:24:21]  INFO: favprocessor move 'up' to 'start'
>>> I [2010-03-22 10:24:21]  INFO: favprocessor start: QUEUE=*
>>>  /opt/ruby-enterprise-1.8.6-20090520/bin/rake -f
>>> /var/rails/favstar_worker/current/Rakefile environment resque:work
>>> D [2010-03-22 10:24:21] DEBUG: driver schedule
>>> #<God::Conditions::ProcessRunning:0x17e64f8> in 0 seconds
>>> D [2010-03-22 10:24:21] DEBUG: driver schedule
>>> #<God::Conditions::Tries:0x17e6390> in 0 seconds
>>> I [2010-03-22 10:24:21]  INFO: favprocessor moved 'up' to 'start'
>>> I [2010-03-22 10:24:21]  INFO: favprocessor [trigger] process is running
>>> (ProcessRunning)
>>> D [2010-03-22 10:24:21] DEBUG: favprocessor ProcessRunning [true]
>>> {true=>:up}
>>> I [2010-03-22 10:24:21]  INFO: favprocessor move 'start' to 'up'
>>> D [2010-03-22 10:24:21] DEBUG: driver schedule
>>> #<God::Conditions::ProcessRunning:0x17e5c60> in 0 seconds
>>> I [2010-03-22 10:24:21]  INFO: favprocessor moved 'start' to 'up'
>>> I [2010-03-22 10:24:21]  INFO: favprocessor [ok] process is running
>>> (ProcessRunning)
>>> D [2010-03-22 10:24:21] DEBUG: favprocessor ProcessRunning [false]
>>> {true=>:start}
>>> D [2010-03-22 10:24:21] DEBUG: driver schedule
>>> #<God::Conditions::ProcessRunning:0x17e5c60> in 10 seconds
>>>
>>>
>>
>