librelist archives

« back to archive

Tricks to reduce memory consumption ?

Tricks to reduce memory consumption ?

From:
Thibaut Barrère
Date:
2011-02-13 @ 14:25
Hi list,

after switching one of my servers to 64-bits, I can really see the
difference in terms of memory use (doubled on some process :-).

Would you have tips to reduce memory in general while using resque ?

Things I already thought of:
- load only what's necessary in rails environment for workers (any
specific trick for that, such as a given gem or others ?)
- switching to ree-1.8.7 for the workers (while keeping 1.9.2 on the
front, with rvm)
- compiling 32-bits versions of some processes

Any additional hints you may have used ?

thanks!

-- Thibaut

Re: Tricks to reduce memory consumption ?

From:
Thibaut Barrère
Date:
2011-02-13 @ 17:48
Hi again,

in case that's useful to someone else, I carried on with my first idea
and came up with this (if you use Bundler like in Rails 3):

https://gist.github.com/824880

It only takes one extra environment variable and a custom bundler
group (:worker here).

This saved me around 65mb on my 2-workers setup.

-- Thibaut

Re: [resque] Re: Tricks to reduce memory consumption ?

From:
Jason Amster
Date:
2011-02-13 @ 22:33
For some of our apps we created new environments (e.g. production_workers)
and then removed a bunch of stuff  that gets loaded, and select fewer
plugins and initializers:

config.frameworks *-=* [:action_controller, :action_view, :action_mailer,
:active_resource]
config.plugins = [:acts_as_state_machine, :resque] #query_analyzer for dev
config.initializers = [:workers_only, :resque_pool]

Please notice the -= notation as we're asking the worker environments to
remove the queue to be loaded.   We didn't see the need to load any of the
above frameworks, as our jobs didn't require them.  YMMV.

Be careful to not remove a key library though, we have some dependencies
that were hidden, and got burned a few times.  Keep an eye on the failures.

-Jason Amster
BeenVerified.com


On Sun, Feb 13, 2011 at 12:48 PM, Thibaut Barrère <thibaut.barrere@gmail.com
> wrote:

> Hi again,
>
> in case that's useful to someone else, I carried on with my first idea
> and came up with this (if you use Bundler like in Rails 3):
>
> https://gist.github.com/824880
>
> It only takes one extra environment variable and a custom bundler
> group (:worker here).
>
> This saved me around 65mb on my 2-workers setup.
>
> -- Thibaut
>



-- 
================================
Jason Amster | BeenVerified, Inc.
Chief Technology Officer
404 Park Avenue S., 11th Floor | New York, New York 10016
jamster@beenverified.com | www.beenverified.com