librelist archives

« back to archive

resque worker hanging on redis stat incr call

resque worker hanging on redis stat incr call

From:
Karl Baum
Date:
2011-01-18 @ 17:48
I am noticing that my redis workers are hanging indefinitely every now and
then on calls to redis.incr when trying to update resque stats on 
completion.  Wanted to know if anyone else has ran into this issue.  I am 
using 

ruby 1.8.7 (2010-04-19 patchlevel 253) [x86_64-linux], MBARI 0x6770, Ruby 
Enterprise Edition 2010.02
redis-rb 2.1.1
Redis server version 2.0.4
Resque 1.10.0

The thread dump of the worker looks like:

------------------------------------------------------------------------------
#<Thread:0xd38268 run> [main] [current] alive=true priority=0
------------------------------------------------------------------------------
    ./lib/thread_dump.rb:9:in `execute_thread_dump'
      \_ ./lib/resque_init.rb:76:in `register_signal_handlers'
      \_ 
/home/ubuntu/.rvm/gems/ree-1.8.7-2010.02/gems/redis-2.1.1/lib/redis/connection.rb:45:in
`call'
      \_ 
/home/ubuntu/.rvm/gems/ree-1.8.7-2010.02/gems/redis-2.1.1/lib/redis/connection.rb:45:in
`write'
      \_ 
/home/ubuntu/.rvm/gems/ree-1.8.7-2010.02/gems/redis-2.1.1/lib/redis/connection.rb:45:in
`write'
      \_ 
/home/ubuntu/.rvm/gems/ree-1.8.7-2010.02/gems/redis-2.1.1/lib/redis/client.rb:60:in
`process'
      \_ 
/home/ubuntu/.rvm/gems/ree-1.8.7-2010.02/gems/redis-2.1.1/lib/redis/client.rb:59:in
`each'
      \_ 
/home/ubuntu/.rvm/gems/ree-1.8.7-2010.02/gems/redis-2.1.1/lib/redis/client.rb:59:in
`process'
      \_ 
/home/ubuntu/.rvm/gems/ree-1.8.7-2010.02/gems/redis-2.1.1/lib/redis/client.rb:154:in
`ensure_connected'
      \_ 
/home/ubuntu/.rvm/gems/ree-1.8.7-2010.02/gems/redis-2.1.1/lib/redis/client.rb:58:in
`process'
      \_ 
/home/ubuntu/.rvm/gems/ree-1.8.7-2010.02/gems/redis-2.1.1/lib/redis/client.rb:118:in
`logging'
      \_ 
/home/ubuntu/.rvm/gems/ree-1.8.7-2010.02/gems/redis-2.1.1/lib/redis/client.rb:57:in
`process'
      \_ 
/home/ubuntu/.rvm/gems/ree-1.8.7-2010.02/gems/redis-2.1.1/lib/redis/client.rb:29:in
`raw_call_command_without_newrelic_trace'
      \_ 
/home/ubuntu/.rvm/gems/ree-1.8.7-2010.02/gems/rpm_contrib-1.0.13/lib/rpm_contrib/instrumentation/redis.rb:22:in
`call'
      \_ 
/home/ubuntu/.rvm/gems/ree-1.8.7-2010.02/gems/newrelic_rpm-2.13.4/lib/new_relic/agent/method_tracer.rb:141:in
`trace_execution_scoped'
      \_ 
/home/ubuntu/.rvm/gems/ree-1.8.7-2010.02/gems/rpm_contrib-1.0.13/lib/rpm_contrib/instrumentation/redis.rb:20:in
`call'
      \_ 
/home/ubuntu/.rvm/gems/ree-1.8.7-2010.02/gems/redis-2.1.1/lib/redis.rb:510:in
`incrby'
      \_ 
/home/ubuntu/.rvm/gems/ree-1.8.7-2010.02/gems/redis-namespace-0.8.0/lib/redis/namespace.rb:188:in
`send'
      \_ 
/home/ubuntu/.rvm/gems/ree-1.8.7-2010.02/gems/redis-namespace-0.8.0/lib/redis/namespace.rb:188:in
`method_missing'
      \_ 
/home/ubuntu/.rvm/gems/ree-1.8.7-2010.02/gems/resque-1.10.0/lib/resque/stat.rb:27:in
`incr'
      \_ 
/home/ubuntu/.rvm/gems/ree-1.8.7-2010.02/gems/resque-1.10.0/lib/resque/stat.rb:32:in
`<<'
      \_ 
/home/ubuntu/.rvm/gems/ree-1.8.7-2010.02/gems/resque-1.10.0/lib/resque/worker.rb:384:in
`processed!'
      \_ 
/home/ubuntu/.rvm/gems/ree-1.8.7-2010.02/gems/resque-1.10.0/lib/resque/worker.rb:373:in
`done_working'
      \_ 
/home/ubuntu/.rvm/gems/ree-1.8.7-2010.02/gems/resque-1.10.0/lib/resque/worker.rb:128:in
`work'
      \_ 
/home/ubuntu/.rvm/gems/ree-1.8.7-2010.02/gems/resque-1.10.0/lib/resque/worker.rb:110:in
`loop'
      \_ 
/home/ubuntu/.rvm/gems/ree-1.8.7-2010.02/gems/resque-1.10.0/lib/resque/worker.rb:110:in
`work'
      \_ 
/home/ubuntu/.rvm/gems/ree-1.8.7-2010.02/gems/resque-1.10.0/lib/resque/tasks.rb:24
      \_ 
/home/ubuntu/.rvm/gems/ree-1.8.7-2010.02/gems/rake-0.8.7/lib/rake.rb:636:in
`call'
      \_ 
/home/ubuntu/.rvm/gems/ree-1.8.7-2010.02/gems/rake-0.8.7/lib/rake.rb:636:in
`execute'
      \_ 
/home/ubuntu/.rvm/gems/ree-1.8.7-2010.02/gems/rake-0.8.7/lib/rake.rb:631:in
`each'
      \_ 
/home/ubuntu/.rvm/gems/ree-1.8.7-2010.02/gems/rake-0.8.7/lib/rake.rb:631:in
`execute'
      \_ 
/home/ubuntu/.rvm/gems/ree-1.8.7-2010.02/gems/rake-0.8.7/lib/rake.rb:597:in
`invoke_with_call_chain'
      \_ 
/home/ubuntu/.rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/monitor.rb:242:in 
`synchronize'
      \_ 
/home/ubuntu/.rvm/gems/ree-1.8.7-2010.02/gems/rake-0.8.7/lib/rake.rb:590:in
`invoke_with_call_chain'
      \_ 
/home/ubuntu/.rvm/gems/ree-1.8.7-2010.02/gems/rake-0.8.7/lib/rake.rb:583:in
`invoke'
      \_ 
/home/ubuntu/.rvm/gems/ree-1.8.7-2010.02/gems/rake-0.8.7/lib/rake.rb:2051:in
`invoke_task'
      \_ 
/home/ubuntu/.rvm/gems/ree-1.8.7-2010.02/gems/rake-0.8.7/lib/rake.rb:2029:in
`top_level'
      \_ 
/home/ubuntu/.rvm/gems/ree-1.8.7-2010.02/gems/rake-0.8.7/lib/rake.rb:2029:in
`each'
      \_ 
/home/ubuntu/.rvm/gems/ree-1.8.7-2010.02/gems/rake-0.8.7/lib/rake.rb:2029:in
`top_level'
      \_ 
/home/ubuntu/.rvm/gems/ree-1.8.7-2010.02/gems/rake-0.8.7/lib/rake.rb:2068:in
`standard_exception_handling'
      \_ 
/home/ubuntu/.rvm/gems/ree-1.8.7-2010.02/gems/rake-0.8.7/lib/rake.rb:2023:in
`top_level'
      \_ 
/home/ubuntu/.rvm/gems/ree-1.8.7-2010.02/gems/rake-0.8.7/lib/rake.rb:2001:in
`run'
      \_ 
/home/ubuntu/.rvm/gems/ree-1.8.7-2010.02/gems/rake-0.8.7/lib/rake.rb:2068:in
`standard_exception_handling'
      \_ 
/home/ubuntu/.rvm/gems/ree-1.8.7-2010.02/gems/rake-0.8.7/lib/rake.rb:1998:in
`run'
      \_ /home/ubuntu/.rvm/gems/ree-1.8.7-2010.02/gems/rake-0.8.7/bin/rake:31

thx.

-karl