librelist archives

« back to archive

Rake task to launch multiple workers in dev/prod + simple management

Rake task to launch multiple workers in dev/prod + simple management

From:
Karel Minařík
Date:
2010-07-22 @ 16:11
Hi all,

on a recent job, we wanted a simple way to launch multiple workers in  
development *and* production, a la `rake resque:workers`.

After couple of refactorings, I came up with this Rake task, which  
allows you to run "groups" of workers, where the first group is  
considered "main", its PID files are saved, cleaned up before shutdown  
etc. Every group tries to clean after itself on clean shutdown. Chris  
told me, that someone may be interested in this.

You can find the heavily annotated source code here:

--> http://gist.github.com/486161

Everything is in the source, really, but just some notes on the  
reasons leading to this:

First, we wanted to have one single Rake task to launch multiple  
workers in production and in development. This was inspired by the  
bundled`rake resque:workers` task.

Second, we wanted operations guys to simply guarantee that this one  
Rake task is running and that's all. We would rather configure and  
tune everything (like the number of workers) inside the task then  
bother them.

Third, we wanted an easy way how to _very_ easily launch bunch of  
workers in production (on sudden spikes in jobs, etc), just by SSH-ing  
to the box and running the "one and only" task, safe that it won't  
interfere with the "main" group, etc.

If you'd think I have done anything stupid and there's better/simpler  
way, I'd be glad to hear it.

Best!,

Karel

--
http://karmi.cz

Re: [resque] Rake task to launch multiple workers in dev/prod + simple management

From:
Mason Jones
Date:
2010-07-22 @ 16:36
This seems really useful, thanks for sharing it. I'm wondering how it
might interact with any monitoring you have? For example, if you're
monitoring the workers and notice that one has died/stopped for some
reason, how would you handle starting a replacement?


On Thu, Jul 22, 2010 at 9:11 AM, Karel Minařík <karel.minarik@gmail.com> wrote:
> Hi all,
>
> on a recent job, we wanted a simple way to launch multiple workers in
> development *and* production, a la `rake resque:workers`.
>
> After couple of refactorings, I came up with this Rake task, which
> allows you to run "groups" of workers, where the first group is
> considered "main", its PID files are saved, cleaned up before shutdown
> etc. Every group tries to clean after itself on clean shutdown. Chris
> told me, that someone may be interested in this.
>
> You can find the heavily annotated source code here:
>
> --> http://gist.github.com/486161
>
> Everything is in the source, really, but just some notes on the
> reasons leading to this:
>
> First, we wanted to have one single Rake task to launch multiple
> workers in production and in development. This was inspired by the
> bundled`rake resque:workers` task.
>
> Second, we wanted operations guys to simply guarantee that this one
> Rake task is running and that's all. We would rather configure and
> tune everything (like the number of workers) inside the task then
> bother them.
>
> Third, we wanted an easy way how to _very_ easily launch bunch of
> workers in production (on sudden spikes in jobs, etc), just by SSH-ing
> to the box and running the "one and only" task, safe that it won't
> interfere with the "main" group, etc.
>
> If you'd think I have done anything stupid and there's better/simpler
> way, I'd be glad to hear it.
>
> Best!,
>
> Karel
>
> --
> http://karmi.cz
>
>

Re: [resque] Rake task to launch multiple workers in dev/prod + simple management

From:
Karel Minařík
Date:
2010-07-22 @ 17:08
Hi,

it's rather crude in this way: we are monitoring only the "master"  
process (the Rake task). When any worker in the "main/monitored group"  
dies, it takes the "master" process down as well.

So Monit (in our case) knows "master" has died, kicks in, and starts  
it again. This clears the workers left behind (with the unfortunate  
effect that they abandon their jobs, AFAIK) and starts new "master"  
group.

Apart from that, anybody can just SSH to the machine and run ad hoc  
(_not_ monitored) groups.

Karel

On 22.Jul, 2010, at 18:36 , Mason Jones wrote:

> This seems really useful, thanks for sharing it. I'm wondering how it
> might interact with any monitoring you have? For example, if you're
> monitoring the workers and notice that one has died/stopped for some
> reason, how would you handle starting a replacement?
>
>
> On Thu, Jul 22, 2010 at 9:11 AM, Karel Minařík <karel.minarik@gmail.c 
> om> wrote:
>> Hi all,
>>
>> on a recent job, we wanted a simple way to launch multiple workers in
>> development *and* production, a la `rake resque:workers`.
>>
>> After couple of refactorings, I came up with this Rake task, which
>> allows you to run "groups" of workers, where the first group is
>> considered "main", its PID files are saved, cleaned up before  
>> shutdown
>> etc. Every group tries to clean after itself on clean shutdown. Chris
>> told me, that someone may be interested in this.
>>
>> You can find the heavily annotated source code here:
>>
>> --> http://gist.github.com/486161
>>
>> Everything is in the source, really, but just some notes on the
>> reasons leading to this:
>>
>> First, we wanted to have one single Rake task to launch multiple
>> workers in production and in development. This was inspired by the
>> bundled`rake resque:workers` task.
>>
>> Second, we wanted operations guys to simply guarantee that this one
>> Rake task is running and that's all. We would rather configure and
>> tune everything (like the number of workers) inside the task then
>> bother them.
>>
>> Third, we wanted an easy way how to _very_ easily launch bunch of
>> workers in production (on sudden spikes in jobs, etc), just by SSH- 
>> ing
>> to the box and running the "one and only" task, safe that it won't
>> interfere with the "main" group, etc.
>>
>> If you'd think I have done anything stupid and there's better/simpler
>> way, I'd be glad to hear it.
>>
>> Best!,
>>
>> Karel
>>
>> --
>> http://karmi.cz
>>
>>