librelist archives

« back to archive

Reducing database churn

Reducing database churn

From:
Damon Cortesi
Date:
2010-06-28 @ 23:25
Hi folks,

I've got a question about using resque in Rails. Been using resque for over
6 months in production now, love it, processed over 150M items with it. I'm
using it in combination with mongodb (via MongoMapper) to process and store
twitter/facebook data.

One of issues I've run in to is that every time a resque worker forks, it
initiates a new connection to my mongodb server (I start resque with syntax
similar to `QUEUES=* COUNT=10 rake resque:workers` and use typical
MongoMapper models/connection setup). Given that most jobs are pretty fast
running, there's a lot of connection churn and this seems inefficient.

Should I just create a worker that pops from the queue until it's empty
instead of forking a new worker for every queue item? Is this more a
MongoMapper issue than a redis issue?

Suggestions welcome, many thanks!

Damon

--
Damon P. Cortesi (@dacort)
Founder, CTO: UntitledStartup.com
www. tweetstats | tweepsearch | rowfeeder .com

Re: [resque] Reducing database churn

From:
Karl Baum
Date:
2010-06-29 @ 01:03
I ran into the same issue with activerecord.  You can try an init hook:

Resque.before_first_fork do
   //initialize mongo_mapper here
end

Or to do less frequent forking, you can use:

http://github.com/samgranieri/resque-jobs-per-fork


On Jun 28, 2010, at 7:25 PM, Damon Cortesi wrote:

> Hi folks,
>
> I've got a question about using resque in Rails. Been using resque  
> for over 6 months in production now, love it, processed over 150M  
> items with it. I'm using it in combination with mongodb (via  
> MongoMapper) to process and store twitter/facebook data.
>
> One of issues I've run in to is that every time a resque worker  
> forks, it initiates a new connection to my mongodb server (I start  
> resque with syntax similar to `QUEUES=* COUNT=10 rake  
> resque:workers` and use typical MongoMapper models/connection  
> setup). Given that most jobs are pretty fast running, there's a lot  
> of connection churn and this seems inefficient.
>
> Should I just create a worker that pops from the queue until it's  
> empty instead of forking a new worker for every queue item? Is this  
> more a MongoMapper issue than a redis issue?
>
> Suggestions welcome, many thanks!
>
> Damon
>
> --
> Damon P. Cortesi (@dacort)
> Founder, CTO: UntitledStartup.com
> www. tweetstats | tweepsearch | rowfeeder .com

Re: [resque] Reducing database churn

From:
Sam Granieri
Date:
2010-06-30 @ 01:09
Thanks for the plug!

Sent from my iPad

On Jun 28, 2010, at 8:03 PM, Karl Baum <karl.baum@gmail.com> wrote:

> I ran into the same issue with activerecord.  You can try an init hook:
> 
> Resque.before_first_fork do
>   //initialize mongo_mapper here
> end
> 
> Or to do less frequent forking, you can use:
> 
> http://github.com/samgranieri/resque-jobs-per-fork
> 
> 
> On Jun 28, 2010, at 7:25 PM, Damon Cortesi wrote:
> 
>> Hi folks,
>> 
>> I've got a question about using resque in Rails. Been using resque  
>> for over 6 months in production now, love it, processed over 150M  
>> items with it. I'm using it in combination with mongodb (via  
>> MongoMapper) to process and store twitter/facebook data.
>> 
>> One of issues I've run in to is that every time a resque worker  
>> forks, it initiates a new connection to my mongodb server (I start  
>> resque with syntax similar to `QUEUES=* COUNT=10 rake  
>> resque:workers` and use typical MongoMapper models/connection  
>> setup). Given that most jobs are pretty fast running, there's a lot  
>> of connection churn and this seems inefficient.
>> 
>> Should I just create a worker that pops from the queue until it's  
>> empty instead of forking a new worker for every queue item? Is this  
>> more a MongoMapper issue than a redis issue?
>> 
>> Suggestions welcome, many thanks!
>> 
>> Damon
>> 
>> --
>> Damon P. Cortesi (@dacort)
>> Founder, CTO: UntitledStartup.com
>> www. tweetstats | tweepsearch | rowfeeder .com
> 

Re: [resque] Reducing database churn

From:
Damon Cortesi
Date:
2010-06-29 @ 06:05
Awesome, I'll look into both of those options.

Thanks, Karl!

On Mon, Jun 28, 2010 at 6:03 PM, Karl Baum <karl.baum@gmail.com> wrote:

> I ran into the same issue with activerecord.  You can try an init hook:
>
> Resque.before_first_fork do
>   //initialize mongo_mapper here
> end
>
> Or to do less frequent forking, you can use:
>
> http://github.com/samgranieri/resque-jobs-per-fork
>
>
> On Jun 28, 2010, at 7:25 PM, Damon Cortesi wrote:
>
> > Hi folks,
> >
> > I've got a question about using resque in Rails. Been using resque
> > for over 6 months in production now, love it, processed over 150M
> > items with it. I'm using it in combination with mongodb (via
> > MongoMapper) to process and store twitter/facebook data.
> >
> > One of issues I've run in to is that every time a resque worker
> > forks, it initiates a new connection to my mongodb server (I start
> > resque with syntax similar to `QUEUES=* COUNT=10 rake
> > resque:workers` and use typical MongoMapper models/connection
> > setup). Given that most jobs are pretty fast running, there's a lot
> > of connection churn and this seems inefficient.
> >
> > Should I just create a worker that pops from the queue until it's
> > empty instead of forking a new worker for every queue item? Is this
> > more a MongoMapper issue than a redis issue?
> >
> > Suggestions welcome, many thanks!
> >
> > Damon
> >
> > --
> > Damon P. Cortesi (@dacort)
> > Founder, CTO: UntitledStartup.com
> > www. tweetstats | tweepsearch | rowfeeder .com
>
>


-- 
--
Damon P. Cortesi (@dacort)
Founder, CTO: UntitledStartup.com
www. tweetstats | tweepsearch | rowfeeder .com

Re: [resque] Reducing database churn

From:
Adam Fields
Date:
2010-06-29 @ 03:31
On Mon, Jun 28, 2010 at 09:03:24PM -0400, Karl Baum wrote:
> I ran into the same issue with activerecord.  You can try an init hook:
> 
> Resque.before_first_fork do
>    //initialize mongo_mapper here
> end

If you do this, watch out for connection timeouts, otherwise you'll
end up with a lot of dead jobs.

-- 
				- Adam
----------
If you liked this email, you might also like:
"Some iPad apps I like" 
-- http://workstuff.tumblr.com/post/680301206
"In praise of the Sous Vide Supreme" 
-- http://www.aquick.org/blog/2010/05/17/in-praise-of-the-sous-vide-supreme/
"Just Married!" 
-- http://www.flickr.com/photos/fields/4721009476/
"fields: It is really annoying that the facebook notification sound is the
same..." 
-- http://twitter.com/fields/statuses/17202859186
----------
** I design intricate-yet-elegant processes for user and machine problems.
** Custom development project broken? Contact me, I can help.
** Some of what I do: http://workstuff.tumblr.com/post/70505118/aboutworkstuff

[ http://www.adamfields.com/resume.html ].. Experience
[ http://www.morningside-analytics.com ] .. Latest Venture
[ http://www.confabb.com ] ................ Founder