librelist archives

« back to archive

reconnect to redis after fork

reconnect to redis after fork

From:
Victor Borda
Date:
2012-06-07 @ 17:37
Hi,

I have recently taken over a Rails 3.2.3 project running on a Heroku Cedar
stack. The project uses resque for background processing, primarily to
dispatch notifications out to users. It uses an instance of redis that is
made available via the Redis-To-Go heroku add-on.

Just in the last week, I started seeing an error in the logs that:
    Error dispatching notification to user: Tried to use a connection from a
child process without
    reconnecting. You need to reconnect to Redis after forking.

This appears to be due to Redis-To-Go upgrading the version of redis they
use to the most recent version, which includes a fix meant to prevent
multiple uses of the same connection. It would appear that although we
weren't seeing any ill-effects previously from collisions due to a shared
connection, this was just luck due to the fact that we are still in beta and
just have a handful of users.

The solution appears to be changing some of our logic in
/config/initializers/resque.rb

In particular, we had previously had this logic in our startup code:
    # REDISTOGO_URL is provided by Heroku
    uri = URI.parse(ENV["REDISTOGO_URL"] || resque_config[Rails.env])

    $resque_redis_config = { host: uri.host, port: uri.port, password:
uri.password }

    Resque.redis = Redis.new $resque_redis_config


And this logic for after_fork:
    Resque.after_fork = Proc.new { ActiveRecord::Base.retrieve_connection}


Adding the following line to after_fork stopped the reconnect error:
    Resque.after_fork = Proc.new {

      ActiveRecord::Base.retrieve_connection

      Resque.redis = Redis.new $resque_redis_config

    }


However, that has resulted in a new error, apparently from redis:
    2012-06-07T17:17:00+00:00 app[scheduler.1]: 2012-06-07 17:17:00 queueing
NotificationDispatcher (send_notifications)

    2012-06-07T17:17:03+00:00 app[worker.1]: rake aborted!

    2012-06-07T17:17:03+00:00 app[worker.1]: ERR command not allowed when
used memory > 'maxmemory'


Thus, my suspicion is that although the new logic has stopped the redis
connection sharing, it has resulted in some sort of memory runaway, possibly
from too many active redis connections. I plan to upgrade the redis-to-go
plan from the 5MB it gives on the free plan, but obviously, if this is an
uncontrolled memory grab situation, then that's going to only help very
temporarily. 

Any thoughts on:
1. Is the after_fork logic good, or does it need to be adjusted?
2. How to verify whether the connections resque has with redis is justified,
or runaway?
Best,
Victor

Re: [resque] reconnect to redis after fork

From:
Ashley Martens
Date:
2012-06-07 @ 21:34
You have to remember that the parent is not using the connection at the 
same time as the child. 



On Jun 7, 2012, at 10:37, Victor Borda <masterchief@builtonlogic.com> wrote:

> Hi,
> 
> I have recently taken over a Rails 3.2.3 project running on a Heroku 
Cedar stack. The project uses resque for background processing, primarily 
to dispatch notifications out to users. It uses an instance of redis that 
is made available via the Redis-To-Go heroku add-on. 
> 
> Just in the last week, I started seeing an error in the logs that:
>     Error dispatching notification to user: Tried to use a connection 
from a child process without  
>     reconnecting. You need to reconnect to Redis after forking.
> 
> This appears to be due to Redis-To-Go upgrading the version of redis 
they use to the most recent version, which includes a fix meant to prevent
multiple uses of the same connection. It would appear that although we 
weren't seeing any ill-effects previously from collisions due to a shared 
connection, this was just luck due to the fact that we are still in beta 
and just have a handful of users.
> 
> The solution appears to be changing some of our logic in 
/config/initializers/resque.rb
> 
> In particular, w e had previously had this logic in our startup code:
>     # REDISTOGO_URL is provided by Heroku
>     uri = URI.parse(ENV["REDISTOGO_URL"] || resque_config[Rails.env])
>     $resque_redis_config = { host: uri.host, port: uri.port, password: 
uri.password }
>     Resque.redis = Redis.new $resque_redis_config
> 
> And this logic for after_fork:
>     Resque.after_fork = Proc.new { ActiveRecord::Base.retrieve_connection}
> 
> Adding the following line to after_fork stopped the reconnect error:
>     Resque.after_fork = Proc.new {
>       ActiveRecord::Base.retrieve_connection
>       Resque.redis = Redis.new $resque_redis_config
>     }
> 
> However, that has resulted in a new error, apparently from redis:
>     2012-06-07T17:17:00+00:00 app[scheduler.1]: 2012-06-07 17:17:00 
queueing NotificationDispatcher (send_notifications)
>     2012-06-07T17:17:03+00:00 app[worker.1]: rake aborted!
>     2012-06-07T17:17:03+00:00 app[worker.1]: ERR command not allowed 
when used memory > 'maxmemory'
> 
> Thus, my suspicion is that although the new logic has stopped the redis 
connection sharing, it has resulted in some sort of memory runaway, 
possibly from too many active redis connections. I plan to upgrade the 
redis-to-go plan from the 5MB it gives on the free plan, but obviously, if
this is an uncontrolled memory grab situation, then that's going to only 
help very temporarily. 
> 
> Any thoughts on:
> Is the after_fork logic good, or does it need to be adjusted?
> How to verify whether the co nnections resque has with redis is 
justified, or runaway?
> Best,
> Victor

Re: [resque] reconnect to redis after fork

From:
Victor Borda
Date:
2012-06-08 @ 00:16
Even if we could guarantee that to be the case, redis does not want the
parent and child to share a connection. They specifically put in a fix to
prevent it.

Best,
Victor 


From:  Ashley Martens <ashleym1972@gmail.com>
Reply-To:  <resque@librelist.com>
Date:  Thursday, June 7, 2012 2:34 PM
To:  <resque@librelist.com>
Subject:  Re: [resque] reconnect to redis after fork

You have to remember that the parent is not using the connection at the same
time as the child. 



On Jun 7, 2012, at 10:37, Victor Borda <masterchief@builtonlogic.com> wrote:

> Hi,
> 
> I have recently taken over a Rails 3.2.3 project running on a Heroku Cedar
> stack. The project uses resque for background processing, primarily to
> dispatch notifications out to users. It uses an instance of redis that is made
> available via the Redis-To-Go heroku add-on.
> 
> Just in the last week, I started seeing an error in the logs that:
>     Error dispatching notification to user: Tried to  use a connection from a
> child process without
>     reconnecting. You need to reconnect to Redis after forking.
> 
> This appears to be due to Redis-To-Go upgrading the version of redis they use
> to the most recent version, which includes a fix meant to prevent multiple
> uses of the same connection. It would appear that although we weren't seeing
> any ill-effects previously from collisions due to a shared connection, this
> was just luck due to the fact that we are still in beta and just have a
> handful of users.
> 
> The solution appears to be changing some of our logic in
> /config/initializers/resque.rb
> 
> In particular, w e had previously had this logic in our startup code:
>     # REDISTOGO_URL is provided by Heroku
>     uri = URI.parse(ENV["REDISTOGO_URL"] || resque_config[Rails.env])
> 
>     $resque_redis_config = { host: uri.host, port: uri.port, password:
> uri.password }
> 
>     Resque.redis = Redis.new $resque_redis_config
> 
> 
> And this logic for after_fork:
>     Resque.after_fork = Proc.new { ActiveRecord::Base.retrieve_connection}
> 
> 
> Adding the following line to after_fork stopped the reconnect error:
>     Resque.after_fork = Proc.new {
> 
>       ActiveRecord::Base.retrieve_connection
> 
>       Resque.redis = Redis.new $resque_redis_config
> 
>     }
> 
> 
> However, that has resulted in a new error, apparently from redis:
>     2012-06-07T17:17:00+00:00 app[scheduler.1]: 2012-06-07 17:17:00 queueing
> NotificationDispatcher (send_notifications)
> 
>     2012-06-07T17:17:03+00:00 app[worker.1]: rake aborted!
> 
>     2012-06-07T17:17:03+00:00 app[worker.1]: ERR command not allowed when used
> memory > 'maxmemory'
> 
> 
> Thus, my suspicion is that although the new logic has stopped the redis
> connection sharing, it has resulted in some sort of memory runaway, possibly
> from too many active redis connections. I plan to upgrade the redis-to-go plan
> from the 5MB it gives on the free plan, but obviously, if this is an
> uncontrolled memory grab situation, then that's going to only help very
> temporarily. 
> 
> Any thoughts on:
> 1. Is the after_fork logic good, or does it need to be adjusted?
> 2. How to verify whether the co nnections resque has with redis is justified,
> or runaway?
> Best,
> Victor

unsubscribe

From:
Jason Toy
Date:
2012-06-08 @ 00:20

      
        

Re: [resque] unsubscribe

From:
Farid Aliev
Date:
2012-06-08 @ 00:22
Done.

On Thu, Jun 7, 2012 at 5:20 PM, Jason Toy <jasontoy@gmail.com> wrote:

>
>


-- 
Farid Aliev.
cell: (415) 225 7861

Re: [resque] reconnect to redis after fork

From:
James Sanders
Date:
2012-06-11 @ 16:45
Ashley, look at https://github.com/defunkt/resque/issues/587 for a
discussion of how resque's connection sharing can be a problem, despite the
parent and child not using the connection at the same time.

Victor, check out https://github.com/redis/redis-rb/pull/226 for the
discussion of the change in redis, and
https://github.com/defunkt/resque/issues/588 for the still-open pull
request fixing the problem in resque. Note - to my knowledge, current
versions of resque specifically depend on a version of the redis client
that does not have the reconnection error, so I'm not sure if Heroku is
ignoring the dependency checks or what.

I'm not sure why your fix would be resulting in runaway memory usage, but
it is creating new redis clients unnecessarily. A better fix is:

  Resque.after_fork do
    Resque.redis.client.reconnect
  end

I hope that helps.

-James