librelist archives

« back to archive

Job not re-queued when Sidekiq shut down or lost when Sidekiq restarted?

Job not re-queued when Sidekiq shut down or lost when Sidekiq restarted?

From:
Jack Royal-Gordon
Date:
2013-02-13 @ 01:49
I had a Sidekiq job that was just getting started on a Heroku background 
worker when I uploaded changes, triggering a slug build and a shutdown and
restart of the Sidekiq process.  Sidekiq should have re-queued the job but
did not (or did it re-queue the job and then lose it due to a connection 
timeout immediately after the restart?).  Here is a slice of the Heroku 
log:

<--- Shutdown requested so new slug could be installed
2013-02-12 22:20:09+00:00 heroku sidekiq.1 - - State changed from up to down
2013-02-12 22:20:09+00:00 heroku api    - - Scale to sidekiq=0 by xxx
2013-02-12 22:20:09+00:00 app sidekiq.1 - - done: 17.384 sec
2013-02-12 22:20:13+00:00 heroku sidekiq.1 - - Stopping all processes with SIGTERM
2013-02-12 22:20:14+00:00 app sidekiq.1 - - Shutting down
<--- New request arrives and Sidekiq worker scaled up
2013-02-12 22:20:14+00:00 heroku api    - - Scale to sidekiq=1 by xxx
2013-02-12 22:20:14+00:00 app sidekiq.1 - - Shutting down 10 quiet workers
2013-02-12 22:20:19+00:00 heroku sidekiq.1 - - Process exited with status 0
2013-02-12 22:20:20+00:00 heroku sidekiq.1 - - Starting process with 
command `bundle exec sidekiq -c 10`
2013-02-12 22:20:21+00:00 heroku sidekiq.1 - - State changed from starting to up
2013-02-12 22:20:28+00:00 app sidekiq.1 - - Configuring threadsafe!!!
2013-02-12 22:20:39+00:00 app sidekiq.1 - -         SECURITY WARNING: No 
secret option provided to Rack::Session::Cookie.
2013-02-12 22:20:39+00:00 app sidekiq.1 - -         This poses a security 
threat. It is strongly recommended that you
2013-02-12 22:20:39+00:00 app sidekiq.1 - -         provide a secret to 
prevent exploits that may be possible from crafted
2013-02-12 22:20:39+00:00 app sidekiq.1 - -         cookies. This will not
be supported in future versions of Rack, and
2013-02-12 22:20:39+00:00 app sidekiq.1 - -         future versions will 
even invalidate your existing user cookies.
2013-02-12 22:20:39+00:00 app sidekiq.1 - -
2013-02-12 22:20:39+00:00 app sidekiq.1 - -         Called from: 
/app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.1.3/lib/action_dispatch/middleware/session/abstract_store.rb:28:in
`initialize'.
2013-02-12 22:20:39+00:00 app sidekiq.1 - -
2013-02-12 22:20:45+00:00 app sidekiq.1 - - Booting Sidekiq 2.6.4 with 
Redis at redis://stingfish.redistogo.com:9782/0
2013-02-12 22:20:45+00:00 app sidekiq.1 - - Running in ruby 1.9.2p290 
(2011-07-09 revision 32553) [x86_64-linux]
2013-02-12 22:20:45+00:00 app sidekiq.1 - - See LICENSE and the LGPL-3.0 
for licensing details.
2013-02-12 22:20:45+00:00 app sidekiq.1 - - Starting processing, hit 
Ctrl-C to stop
<--- Was it the restart which lost the job due to a connection timeout?
2013-02-12 23:56:35+00:00 app sidekiq.1 - - Connection timed out
2013-02-12 23:57:10+00:00 app sidekiq.1 - - {:context=>"scheduling poller 
thread died!"}

Re: [sidekiq] Job not re-queued when Sidekiq shut down or lost when Sidekiq restarted?

From:
Kevin Menard
Date:
2013-02-13 @ 02:24
One a job is popped off the queue, it's "gone" unless it gets 
re-enqueued by some middleware (e.g., the retry one).  A feature of 
Sidekiq Pro is reliable queuing, which would do what you're looking for.

-- 
Kevin
> Jack Royal-Gordon <mailto:jackrg@pobox.com>
> Tuesday, February 12, 2013 8:49 PM
> I had a Sidekiq job that was just getting started on a Heroku 
> background worker when I uploaded changes, triggering a slug build and 
> a shutdown and restart of the Sidekiq process. Sidekiq should have 
> re-queued the job but did not (or did it re-queue the job and then 
> lose it due to a connection timeout immediately after the restart?). 
> Here is a slice of the Heroku log:
>
> <--- Shutdown requested so new slug could be installed
> 2013-02-12 22:20:09+00:00 heroku sidekiq.1 - - State changed from up 
> to down
> 2013-02-12 22:20:09+00:00 heroku api - - Scale to sidekiq=0 by xxx
> 2013-02-12 22:20:09+00:00 app sidekiq.1 - - done: 17.384 sec
> 2013-02-12 22:20:13+00:00 heroku sidekiq.1 - - Stopping all processes 
> with SIGTERM
> 2013-02-12 22:20:14+00:00 app sidekiq.1 - - Shutting down
> <--- New request arrives and Sidekiq worker scaled up
> 2013-02-12 22:20:14+00:00 heroku api - - Scale to sidekiq=1 by xxx
> 2013-02-12 22:20:14+00:00 app sidekiq.1 - - Shutting down 10 quiet workers
> 2013-02-12 22:20:19+00:00 heroku sidekiq.1 - - Process exited with 
> status 0
> 2013-02-12 22:20:20+00:00 heroku sidekiq.1 - - Starting process with 
> command `bundle exec sidekiq -c 10`
> 2013-02-12 22:20:21+00:00 heroku sidekiq.1 - - State changed from 
> starting to up
> 2013-02-12 22:20:28+00:00 app sidekiq.1 - - Configuring threadsafe!!!
> 2013-02-12 22:20:39+00:00 app sidekiq.1 - - SECURITY WARNING: No 
> secret option provided to Rack::Session::Cookie.
> 2013-02-12 22:20:39+00:00 app sidekiq.1 - - This poses a security 
> threat. It is strongly recommended that you
> 2013-02-12 22:20:39+00:00 app sidekiq.1 - - provide a secret to 
> prevent exploits that may be possible from crafted
> 2013-02-12 22:20:39+00:00 app sidekiq.1 - - cookies. This will not be 
> supported in future versions of Rack, and
> 2013-02-12 22:20:39+00:00 app sidekiq.1 - - future versions will even 
> invalidate your existing user cookies.
> 2013-02-12 22:20:39+00:00 app sidekiq.1 - -
> 2013-02-12 22:20:39+00:00 app sidekiq.1 - - Called from: 
> 
/app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.1.3/lib/action_dispatch/middleware/session/abstract_store.rb:28:in

> `initialize'.
> 2013-02-12 22:20:39+00:00 app sidekiq.1 - -
> 2013-02-12 22:20:45+00:00 app sidekiq.1 - - Booting Sidekiq 2.6.4 with 
> Redis at redis://stingfish.redistogo.com:9782/0
> 2013-02-12 22:20:45+00:00 app sidekiq.1 - - Running in ruby 1.9.2p290 
> (2011-07-09 revision 32553) [x86_64-linux]
> 2013-02-12 22:20:45+00:00 app sidekiq.1 - - See LICENSE and the 
> LGPL-3.0 for licensing details.
> 2013-02-12 22:20:45+00:00 app sidekiq.1 - - Starting processing, hit 
> Ctrl-C to stop
> <--- Was it the restart which lost the job due to a connection timeout?
> 2013-02-12 23:56:35+00:00 app sidekiq.1 - - Connection timed out
> 2013-02-12 23:57:10+00:00 app sidekiq.1 - - {:context=>"scheduling 
> poller thread died!"}
>

Re: [sidekiq] Job not re-queued when Sidekiq shut down or lost when Sidekiq restarted?

From:
Mike Perham
Date:
2013-02-13 @ 02:37
Not quite true. Sidekiq tracks jobs in progress and tries to push them 
back to Redis if the workers don't complete when shutting down. There was 
a bug in this recently. Check the changelog since 2.6.4. 

On 12 Feb 2013, at 18:24, Kevin Menard <kevin@mogotest.com> wrote:

> One a job is popped off the queue, it's "gone" unless it gets 
re-enqueued by some middleware (e.g., the retry one).  A feature of 
Sidekiq Pro is reliable queuing, which would do what you're looking for.
> 
> -- 
> Kevin
>> Jack Royal-Gordon	Tuesday, February 12, 2013 8:49 PM
>> I had a Sidekiq job that was just getting started on a Heroku 
background worker when I uploaded changes, triggering a slug build and a 
shutdown and restart of the Sidekiq process. Sidekiq should have re-queued
the job but did not (or did it re-queue the job and then lose it due to a 
connection timeout immediately after the restart?). Here is a slice of the
Heroku log:
>> 
>> <--- Shutdown requested so new slug could be installed
>> 2013-02-12 22:20:09+00:00 heroku sidekiq.1 - - State changed from up to down
>> 2013-02-12 22:20:09+00:00 heroku api - - Scale to sidekiq=0 by xxx
>> 2013-02-12 22:20:09+00:00 app sidekiq.1 - - done: 17.384 sec
>> 2013-02-12 22:20:13+00:00 heroku sidekiq.1 - - Stopping all processes 
with SIGTERM
>> 2013-02-12 22:20:14+00:00 app sidekiq.1 - - Shutting down
>> <--- New request arrives and Sidekiq worker scaled up
>> 2013-02-12 22:20:14+00:00 heroku api - - Scale to sidekiq=1 by xxx
>> 2013-02-12 22:20:14+00:00 app sidekiq.1 - - Shutting down 10 quiet workers
>> 2013-02-12 22:20:19+00:00 heroku sidekiq.1 - - Process exited with status 0
>> 2013-02-12 22:20:20+00:00 heroku sidekiq.1 - - Starting process with 
command `bundle exec sidekiq -c 10`
>> 2013-02-12 22:20:21+00:00 heroku sidekiq.1 - - State changed from 
starting to up
>> 2013-02-12 22:20:28+00:00 app sidekiq.1 - - Configuring threadsafe!!!
>> 2013-02-12 22:20:39+00:00 app sidekiq.1 - - SECURITY WARNING: No secret
option provided to Rack::Session::Cookie.
>> 2013-02-12 22:20:39+00:00 app sidekiq.1 - - This poses a security 
threat. It is strongly recommended that you
>> 2013-02-12 22:20:39+00:00 app sidekiq.1 - - provide a secret to prevent
exploits that may be possible from crafted
>> 2013-02-12 22:20:39+00:00 app sidekiq.1 - - cookies. This will not be 
supported in future versions of Rack, and
>> 2013-02-12 22:20:39+00:00 app sidekiq.1 - - future versions will even 
invalidate your existing user cookies.
>> 2013-02-12 22:20:39+00:00 app sidekiq.1 - -
>> 2013-02-12 22:20:39+00:00 app sidekiq.1 - - Called from: 
/app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.1.3/lib/action_dispatch/middleware/session/abstract_store.rb:28:in
`initialize'.
>> 2013-02-12 22:20:39+00:00 app sidekiq.1 - -
>> 2013-02-12 22:20:45+00:00 app sidekiq.1 - - Booting Sidekiq 2.6.4 with 
Redis at redis://stingfish.redistogo.com:9782/0
>> 2013-02-12 22:20:45+00:00 app sidekiq.1 - - Running in ruby 1.9.2p290 
(2011-07-09 revision 32553) [x86_64-linux]
>> 2013-02-12 22:20:45+00:00 app sidekiq.1 - - See LICENSE and the 
LGPL-3.0 for licensing details.
>> 2013-02-12 22:20:45+00:00 app sidekiq.1 - - Starting processing, hit 
Ctrl-C to stop
>> <--- Was it the restart which lost the job due to a connection timeout?
>> 2013-02-12 23:56:35+00:00 app sidekiq.1 - - Connection timed out
>> 2013-02-12 23:57:10+00:00 app sidekiq.1 - - {:context=>"scheduling 
poller thread died!"}

Re: [sidekiq] Job not re-queued when Sidekiq shut down or lost when Sidekiq restarted?

From:
Kevin Menard
Date:
2013-02-13 @ 02:42
Ahh, right.  Sorry, I assumed Heroku was forcibly shutting down the 
worker.  And I've seen things get a little wonky on JRuby due to issues 
with signal handlers.

-- 
Kevin
> Mike Perham <mailto:mperham@gmail.com>
> Tuesday, February 12, 2013 9:37 PM
> Not quite true. Sidekiq tracks jobs in progress and tries to push them 
> back to Redis if the workers don't complete when shutting down. There 
> was a bug in this recently. Check the changelog since 2.6.4.
>
> On 12 Feb 2013, at 18:24, Kevin Menard <kevin@mogotest.com 
> <mailto:kevin@mogotest.com>> wrote:
>
> Kevin Menard <mailto:kevin@mogotest.com>
> Tuesday, February 12, 2013 9:24 PM
> One a job is popped off the queue, it's "gone" unless it gets 
> re-enqueued by some middleware (e.g., the retry one).  A feature of 
> Sidekiq Pro is reliable queuing, which would do what you're looking for.
>
> -- 
> Kevin
> Jack Royal-Gordon <mailto:jackrg@pobox.com>
> Tuesday, February 12, 2013 8:49 PM
> I had a Sidekiq job that was just getting started on a Heroku 
> background worker when I uploaded changes, triggering a slug build and 
> a shutdown and restart of the Sidekiq process. Sidekiq should have 
> re-queued the job but did not (or did it re-queue the job and then 
> lose it due to a connection timeout immediately after the restart?). 
> Here is a slice of the Heroku log:
>
> <--- Shutdown requested so new slug could be installed
> 2013-02-12 22:20:09+00:00 heroku sidekiq.1 - - State changed from up 
> to down
> 2013-02-12 22:20:09+00:00 heroku api - - Scale to sidekiq=0 by xxx
> 2013-02-12 22:20:09+00:00 app sidekiq.1 - - done: 17.384 sec
> 2013-02-12 22:20:13+00:00 heroku sidekiq.1 - - Stopping all processes 
> with SIGTERM
> 2013-02-12 22:20:14+00:00 app sidekiq.1 - - Shutting down
> <--- New request arrives and Sidekiq worker scaled up
> 2013-02-12 22:20:14+00:00 heroku api - - Scale to sidekiq=1 by xxx
> 2013-02-12 22:20:14+00:00 app sidekiq.1 - - Shutting down 10 quiet workers
> 2013-02-12 22:20:19+00:00 heroku sidekiq.1 - - Process exited with 
> status 0
> 2013-02-12 22:20:20+00:00 heroku sidekiq.1 - - Starting process with 
> command `bundle exec sidekiq -c 10`
> 2013-02-12 22:20:21+00:00 heroku sidekiq.1 - - State changed from 
> starting to up
> 2013-02-12 22:20:28+00:00 app sidekiq.1 - - Configuring threadsafe!!!
> 2013-02-12 22:20:39+00:00 app sidekiq.1 - - SECURITY WARNING: No 
> secret option provided to Rack::Session::Cookie.
> 2013-02-12 22:20:39+00:00 app sidekiq.1 - - This poses a security 
> threat. It is strongly recommended that you
> 2013-02-12 22:20:39+00:00 app sidekiq.1 - - provide a secret to 
> prevent exploits that may be possible from crafted
> 2013-02-12 22:20:39+00:00 app sidekiq.1 - - cookies. This will not be 
> supported in future versions of Rack, and
> 2013-02-12 22:20:39+00:00 app sidekiq.1 - - future versions will even 
> invalidate your existing user cookies.
> 2013-02-12 22:20:39+00:00 app sidekiq.1 - -
> 2013-02-12 22:20:39+00:00 app sidekiq.1 - - Called from: 
> 
/app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.1.3/lib/action_dispatch/middleware/session/abstract_store.rb:28:in

> `initialize'.
> 2013-02-12 22:20:39+00:00 app sidekiq.1 - -
> 2013-02-12 22:20:45+00:00 app sidekiq.1 - - Booting Sidekiq 2.6.4 with 
> Redis at redis://stingfish.redistogo.com:9782/0
> 2013-02-12 22:20:45+00:00 app sidekiq.1 - - Running in ruby 1.9.2p290 
> (2011-07-09 revision 32553) [x86_64-linux]
> 2013-02-12 22:20:45+00:00 app sidekiq.1 - - See LICENSE and the 
> LGPL-3.0 for licensing details.
> 2013-02-12 22:20:45+00:00 app sidekiq.1 - - Starting processing, hit 
> Ctrl-C to stop
> <--- Was it the restart which lost the job due to a connection timeout?
> 2013-02-12 23:56:35+00:00 app sidekiq.1 - - Connection timed out
> 2013-02-12 23:57:10+00:00 app sidekiq.1 - - {:context=>"scheduling 
> poller thread died!"}
>
> Jack Royal-Gordon <mailto:jackrg@pobox.com>
> Tuesday, February 12, 2013 8:49 PM
> I had a Sidekiq job that was just getting started on a Heroku 
> background worker when I uploaded changes, triggering a slug build and 
> a shutdown and restart of the Sidekiq process. Sidekiq should have 
> re-queued the job but did not (or did it re-queue the job and then 
> lose it due to a connection timeout immediately after the restart?). 
> Here is a slice of the Heroku log:
>
> <--- Shutdown requested so new slug could be installed
> 2013-02-12 22:20:09+00:00 heroku sidekiq.1 - - State changed from up 
> to down
> 2013-02-12 22:20:09+00:00 heroku api - - Scale to sidekiq=0 by xxx
> 2013-02-12 22:20:09+00:00 app sidekiq.1 - - done: 17.384 sec
> 2013-02-12 22:20:13+00:00 heroku sidekiq.1 - - Stopping all processes 
> with SIGTERM
> 2013-02-12 22:20:14+00:00 app sidekiq.1 - - Shutting down
> <--- New request arrives and Sidekiq worker scaled up
> 2013-02-12 22:20:14+00:00 heroku api - - Scale to sidekiq=1 by xxx
> 2013-02-12 22:20:14+00:00 app sidekiq.1 - - Shutting down 10 quiet workers
> 2013-02-12 22:20:19+00:00 heroku sidekiq.1 - - Process exited with 
> status 0
> 2013-02-12 22:20:20+00:00 heroku sidekiq.1 - - Starting process with 
> command `bundle exec sidekiq -c 10`
> 2013-02-12 22:20:21+00:00 heroku sidekiq.1 - - State changed from 
> starting to up
> 2013-02-12 22:20:28+00:00 app sidekiq.1 - - Configuring threadsafe!!!
> 2013-02-12 22:20:39+00:00 app sidekiq.1 - - SECURITY WARNING: No 
> secret option provided to Rack::Session::Cookie.
> 2013-02-12 22:20:39+00:00 app sidekiq.1 - - This poses a security 
> threat. It is strongly recommended that you
> 2013-02-12 22:20:39+00:00 app sidekiq.1 - - provide a secret to 
> prevent exploits that may be possible from crafted
> 2013-02-12 22:20:39+00:00 app sidekiq.1 - - cookies. This will not be 
> supported in future versions of Rack, and
> 2013-02-12 22:20:39+00:00 app sidekiq.1 - - future versions will even 
> invalidate your existing user cookies.
> 2013-02-12 22:20:39+00:00 app sidekiq.1 - -
> 2013-02-12 22:20:39+00:00 app sidekiq.1 - - Called from: 
> 
/app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.1.3/lib/action_dispatch/middleware/session/abstract_store.rb:28:in

> `initialize'.
> 2013-02-12 22:20:39+00:00 app sidekiq.1 - -
> 2013-02-12 22:20:45+00:00 app sidekiq.1 - - Booting Sidekiq 2.6.4 with 
> Redis at redis://stingfish.redistogo.com:9782/0
> 2013-02-12 22:20:45+00:00 app sidekiq.1 - - Running in ruby 1.9.2p290 
> (2011-07-09 revision 32553) [x86_64-linux]
> 2013-02-12 22:20:45+00:00 app sidekiq.1 - - See LICENSE and the 
> LGPL-3.0 for licensing details.
> 2013-02-12 22:20:45+00:00 app sidekiq.1 - - Starting processing, hit 
> Ctrl-C to stop
> <--- Was it the restart which lost the job due to a connection timeout?
> 2013-02-12 23:56:35+00:00 app sidekiq.1 - - Connection timed out
> 2013-02-12 23:57:10+00:00 app sidekiq.1 - - {:context=>"scheduling 
> poller thread died!"}
>