librelist archives

« back to archive

Slow Forking and Startup Time

Slow Forking and Startup Time

From:
David John
Date:
2010-05-31 @ 23:38
Hey All, 

I'm currently moving my application from a Linode setup to EC2.  Redis is 
currently installed on a remote instance with various worker instances 
interacting with the queue.  Thats all going fantastic. 

My problem is with the amount of time it takes for a worker to be 
'instantiated' and slow forking.  Starting a worker will usually take 
between 30 seconds and a minute(from god.rb starting the worker rake task 
and the worker actively starting work on the queue).  I could live with 
that, but I've not experienced such a wait time on my current Linode 
production box so I believe its one of my symptoms to a bigger problem.  
Next issue is that jobs that took a second or less in my previous 
environment now seem to take about 5 to 10 times longer.. 

I'm assuming this must be some sort of issue with my Ubuntu install on 
EC2?  One notable difference is that I'm running REE 1.8.7-2010.01 in my 
new setup, and REE 1.8.6 on the old Linode boxes.  

Anyone else experienced these issues?

Cheers,
David

Re: [resque] Slow Forking and Startup Time

From:
Tony Arcieri
Date:
2010-06-02 @ 04:50
On Mon, May 31, 2010 at 5:38 PM, David John <dj@davidjohn.ca> wrote:

> My problem is with the amount of time it takes for a worker to be
> 'instantiated' and slow forking.  Starting a worker will usually take
> between 30 seconds and a minute(from god.rb starting the worker rake task
> and the worker actively starting work on the queue).


This sounds rather unusual to me.  For our application, starting a worker on
EC2 using the JRuby Server VM which has notoriously poor startup time (but
is well-suited to something like a long-lived background job worker) takes
approximately 20 seconds, and that's also loading all of our models ahead of
time.  We are on a large 64-bit instance, but even so, 30 seconds sounds
unusually long.

-- 
Tony Arcieri
Medioh! A Kudelski Brand

Re: [resque] Slow Forking and Startup Time

From:
Mason Jones
Date:
2010-06-02 @ 05:43
On Tue, Jun 1, 2010 at 9:50 PM, Tony Arcieri <tony.arcieri@medioh.com> wrote:
> On Mon, May 31, 2010 at 5:38 PM, David John <dj@davidjohn.ca> wrote:
>>
>> My problem is with the amount of time it takes for a worker to be
>> 'instantiated' and slow forking.  Starting a worker will usually take
>> between 30 seconds and a minute(from god.rb starting the worker rake task
>> and the worker actively starting work on the queue).
>
> This sounds rather unusual to me.  For our application, starting a worker on
> EC2 using the JRuby Server VM which has notoriously poor startup time (but
> is well-suited to something like a long-lived background job worker) takes
> approximately 20 seconds, and that's also loading all of our models ahead of
> time.  We are on a large 64-bit instance, but even so, 30 seconds sounds
> unusually long.

Likewise, we've got workers running in an m1.large (64-bit) EC2
instance, and they run within a Rails instance. They each take perhaps
10 seconds to start, I'd say, at most. That includes starting the New
Relic gems as well. Your case definitely seems unusual.

Re: [resque] Slow Forking and Startup Time

From:
David John
Date:
2010-06-02 @ 06:37
Hey Guys, 

I've compared rails console startup times on my old Linode box and my EC2 
instances and its vastly slower on EC2.  I suppose I have some 
troubleshooting to do and this definitely doesn't qualify as a Resque 
issue..  That being said,  are you guys running REE 1.8.7-2010-01 on your 
instances?

Thanks,
DJ

On 2010-06-02, at 1:43 AM, Mason Jones wrote:

> On Tue, Jun 1, 2010 at 9:50 PM, Tony Arcieri <tony.arcieri@medioh.com> wrote:
>> On Mon, May 31, 2010 at 5:38 PM, David John <dj@davidjohn.ca> wrote:
>>> 
>>> My problem is with the amount of time it takes for a worker to be
>>> 'instantiated' and slow forking.  Starting a worker will usually take
>>> between 30 seconds and a minute(from god.rb starting the worker rake task
>>> and the worker actively starting work on the queue).
>> 
>> This sounds rather unusual to me.  For our application, starting a worker on
>> EC2 using the JRuby Server VM which has notoriously poor startup time (but
>> is well-suited to something like a long-lived background job worker) takes
>> approximately 20 seconds, and that's also loading all of our models ahead of
>> time.  We are on a large 64-bit instance, but even so, 30 seconds sounds
>> unusually long.
> 
> Likewise, we've got workers running in an m1.large (64-bit) EC2
> instance, and they run within a Rails instance. They each take perhaps
> 10 seconds to start, I'd say, at most. That includes starting the New
> Relic gems as well. Your case definitely seems unusual.

Re: [resque] Slow Forking and Startup Time

From:
Mason Jones
Date:
2010-06-02 @ 06:53
On Tue, Jun 1, 2010 at 11:37 PM, David John <dj@davidjohn.ca> wrote:
> Hey Guys,
>
> I've compared rails console startup times on my old Linode box and my 
EC2 instances and its vastly slower on EC2.  I suppose I have some 
troubleshooting to do and this definitely doesn't qualify as a Resque 
issue..  That being said,  are you guys running REE 1.8.7-2010-01 on your 
instances?

We're actually running plain old 1.8.6 right now, sadly enough, but
I've just installed and started testing 1.8.7. I haven't noticed any
particular difference in startup times (yet).

I also did initial setup on a small instance running 1.8.6 and it was
certainly starting workers faster than 30 seconds.


> On 2010-06-02, at 1:43 AM, Mason Jones wrote:
>
>> On Tue, Jun 1, 2010 at 9:50 PM, Tony Arcieri <tony.arcieri@medioh.com> wrote:
>>> On Mon, May 31, 2010 at 5:38 PM, David John <dj@davidjohn.ca> wrote:
>>>>
>>>> My problem is with the amount of time it takes for a worker to be
>>>> 'instantiated' and slow forking.  Starting a worker will usually take
>>>> between 30 seconds and a minute(from god.rb starting the worker rake task
>>>> and the worker actively starting work on the queue).
>>>
>>> This sounds rather unusual to me.  For our application, starting a worker on
>>> EC2 using the JRuby Server VM which has notoriously poor startup time (but
>>> is well-suited to something like a long-lived background job worker) takes
>>> approximately 20 seconds, and that's also loading all of our models ahead of
>>> time.  We are on a large 64-bit instance, but even so, 30 seconds sounds
>>> unusually long.
>>
>> Likewise, we've got workers running in an m1.large (64-bit) EC2
>> instance, and they run within a Rails instance. They each take perhaps
>> 10 seconds to start, I'd say, at most. That includes starting the New
>> Relic gems as well. Your case definitely seems unusual.
>
>

Re: [resque] Slow Forking and Startup Time

From:
Chris Wanstrath
Date:
2010-06-02 @ 04:20
On Mon, May 31, 2010 at 4:38 PM, David John <dj@davidjohn.ca> wrote:

> My problem is with the amount of time it takes for a worker to be 
'instantiated' and slow forking.  Starting a worker will usually take 
between 30 seconds and a minute(from god.rb starting the worker rake task 
and the worker actively starting work on the queue).

Is this unique to workers, or does your Rails app take 30 seconds to start?