librelist archives

« back to archive

issues remain with terminating retrying workers

issues remain with terminating retrying workers

From:
Date:
2013-04-09 @ 18:48
I am still having trouble terminating workers which are in retry using 
the sidekiq api.

I am able to run this block of code in the production console, and the 
workers in the retry queue disappear.  But when I execute the same code 
in my rails app, it doesn't make the retrying workers go away.

here's the code:  (in this example, I have a Deployment object, whose 
id is the only argument to the the worker)

   def terminate(current_user)
     @deployment.update_attribute(:terminated_by, current_user.id)
     @deployment.update_attribute(:terminated_at, Time.now)

     if @deployment.terminate!
       do_terminate(@deployment.id)

       send_email do
         
DeploymentsMailer.delay.deployment_terminated(@deployment.mail_recipient, 
@deployment)
       end
       Deployment.log_result(@deployment, "Deployment terminated by 
#{current_user.given_name}")
       send_timing(@deployment.name, "terminated by 
#{current_user.given_name}", "complete")
     end
   end

   def do_terminate(id)
     rm_arr = [id]
     rq = Sidekiq::RetrySet.new
     rq.select do |job|
       rm_arr.include?(job.args.first)
     end.map(&:delete)
   end

In the past, Mike has suggested that there's something different about 
production console vs production code.  I can't figure out what that 
difference could be.

Can anyone suggest any steps to troubleshoot?

Thanks in advance,
Jason

Re: [sidekiq] issues remain with terminating retrying workers

From:
Mike Perham
Date:
2013-04-09 @ 21:19
Your code looks fine.  Show us your config.


On Tue, Apr 9, 2013 at 11:48 AM, <nospam@jason-michael.com> wrote:

> I am still having trouble terminating workers which are in retry using
> the sidekiq api.
>
> I am able to run this block of code in the production console, and the
> workers in the retry queue disappear.  But when I execute the same code
> in my rails app, it doesn't make the retrying workers go away.
>
> here's the code:  (in this example, I have a Deployment object, whose
> id is the only argument to the the worker)
>
>    def terminate(current_user)
>      @deployment.update_attribute(:terminated_by, current_user.id)
>      @deployment.update_attribute(:terminated_at, Time.now)
>
>      if @deployment.terminate!
>        do_terminate(@deployment.id)
>
>        send_email do
>
> DeploymentsMailer.delay.deployment_terminated(@deployment.mail_recipient,
> @deployment)
>        end
>        Deployment.log_result(@deployment, "Deployment terminated by
> #{current_user.given_name}")
>        send_timing(@deployment.name, "terminated by
> #{current_user.given_name}", "complete")
>      end
>    end
>
>    def do_terminate(id)
>      rm_arr = [id]
>      rq = Sidekiq::RetrySet.new
>      rq.select do |job|
>        rm_arr.include?(job.args.first)
>      end.map(&:delete)
>    end
>
> In the past, Mike has suggested that there's something different about
> production console vs production code.  I can't figure out what that
> difference could be.
>
> Can anyone suggest any steps to troubleshoot?
>
> Thanks in advance,
> Jason
>

Re: [sidekiq] issues remain with terminating retrying workers

From:
Date:
2013-04-09 @ 22:00
/config/initializers/sidekiq.rb
if defined?(PhusionPassenger)
   PhusionPassenger.on_event(:starting_worker_process) do |forked|
     Sidekiq.configure_client do |config|
       config.redis = { :size => 1 }
     end if forked
   end
end

/etc/redis.conf

daemonize yes
pidfile /var/run/redis/redis.pid
port 6379
bind 127.0.0.1
timeout 0
loglevel notice
logfile /var/log/redis/redis.log
databases 16
save 900 1
save 300 10
save 60 10000
rdbcompression yes
dbfilename dump.rdb
dir /var/lib/redis/
slave-serve-stale-data yes
appendonly no
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
slowlog-log-slower-than 10000
slowlog-max-len 1024
vm-enabled no
vm-swap-file /tmp/redis.swap
vm-max-memory 0
vm-page-size 32
vm-pages 134217728
vm-max-threads 4
hash-max-zipmap-entries 512
hash-max-zipmap-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
activerehashing yes

is there another config of interest?

On 2013-04-09 16:19, Mike Perham wrote:
> Your code looks fine.  Show us your config.
>
> On Tue, Apr 9, 2013 at 11:48 AM, <nospam@jason-michael.com [4]> 
> wrote:
>
>> I am still having trouble terminating workers which are in retry
>> using
>> the sidekiq api.
>>
>> I am able to run this block of code in the production console, and
>> the
>> workers in the retry queue disappear.  But when I execute the same
>> code
>> in my rails app, it doesn't make the retrying workers go away.
>>
>> here's the code:  (in this example, I have a Deployment object,
>> whose
>> id is the only argument to the the worker)
>>
>>    def terminate(current_user)
>>      @deployment.update_attribute(:terminated_by,
>> current_user.id [1])
>>      @deployment.update_attribute(:terminated_at, Time.now)
>>
>>      if @deployment.terminate!
>>        do_terminate(@deployment.id [2])
>>
>>        send_email do
>>
>>
> 
> DeploymentsMailer.delay.deployment_terminated(@deployment.mail_recipient,
>> @deployment)
>>        end
>>        Deployment.log_result(@deployment, "Deployment
>> terminated by
>> #{current_user.given_name}")
>>        send_timing(@deployment.name [3], "terminated by
>> #{current_user.given_name}", "complete")
>>      end
>>    end
>>
>>    def do_terminate(id)
>>      rm_arr = [id]
>>      rq = Sidekiq::RetrySet.new
>>      rq.select do |job|
>>        rm_arr.include?(job.args.first)
>>      end.map(&:delete)
>>    end
>>
>> In the past, Mike has suggested that there's something different
>> about
>> production console vs production code.  I can't figure out what
>> that
>> difference could be.
>>
>> Can anyone suggest any steps to troubleshoot?
>>
>> Thanks in advance,
>> Jason
>
>
>
> Links:
> ------
> [1] http://current_user.id
> [2] http://deployment.id
> [3] http://deployment.name
> [4] mailto:nospam@jason-michael.com

Re: [sidekiq] issues remain with terminating retrying workers

From:
Mike Perham
Date:
2013-04-09 @ 22:45
Yeah, with 2.9.0 you can remove the Passenger stuff:

Sidekiq.configure_client do |config|
  config.redis = { :size => 1 }
end

then your initializer is about as simple as can be.  If you're ok with 5
redis connections per rails process, you can delete the initializer
altogether.  Less code == less potential for bugs.



On Tue, Apr 9, 2013 at 3:00 PM, <nospam@jason-michael.com> wrote:

> /config/initializers/sidekiq.rb
> if defined?(PhusionPassenger)
>    PhusionPassenger.on_event(:starting_worker_process) do |forked|
>      Sidekiq.configure_client do |config|
>        config.redis = { :size => 1 }
>      end if forked
>    end
> end
>
> /etc/redis.conf
>
> daemonize yes
> pidfile /var/run/redis/redis.pid
> port 6379
> bind 127.0.0.1
> timeout 0
> loglevel notice
> logfile /var/log/redis/redis.log
> databases 16
> save 900 1
> save 300 10
> save 60 10000
> rdbcompression yes
> dbfilename dump.rdb
> dir /var/lib/redis/
> slave-serve-stale-data yes
> appendonly no
> appendfsync everysec
> no-appendfsync-on-rewrite no
> auto-aof-rewrite-percentage 100
> auto-aof-rewrite-min-size 64mb
> slowlog-log-slower-than 10000
> slowlog-max-len 1024
> vm-enabled no
> vm-swap-file /tmp/redis.swap
> vm-max-memory 0
> vm-page-size 32
> vm-pages 134217728
> vm-max-threads 4
> hash-max-zipmap-entries 512
> hash-max-zipmap-value 64
> list-max-ziplist-entries 512
> list-max-ziplist-value 64
> set-max-intset-entries 512
> zset-max-ziplist-entries 128
> zset-max-ziplist-value 64
> activerehashing yes
>
> is there another config of interest?
>
> On 2013-04-09 16:19, Mike Perham wrote:
> > Your code looks fine.  Show us your config.
> >
> > On Tue, Apr 9, 2013 at 11:48 AM, <nospam@jason-michael.com [4]>
> > wrote:
> >
> >> I am still having trouble terminating workers which are in retry
> >> using
> >> the sidekiq api.
> >>
> >> I am able to run this block of code in the production console, and
> >> the
> >> workers in the retry queue disappear.  But when I execute the same
> >> code
> >> in my rails app, it doesn't make the retrying workers go away.
> >>
> >> here's the code:  (in this example, I have a Deployment object,
> >> whose
> >> id is the only argument to the the worker)
> >>
> >>    def terminate(current_user)
> >>      @deployment.update_attribute(:terminated_by,
> >> current_user.id [1])
> >>      @deployment.update_attribute(:terminated_at, Time.now)
> >>
> >>      if @deployment.terminate!
> >>        do_terminate(@deployment.id [2])
> >>
> >>        send_email do
> >>
> >>
> >
> > DeploymentsMailer.delay.deployment_terminated(@deployment.mail_recipient,
> >> @deployment)
> >>        end
> >>        Deployment.log_result(@deployment, "Deployment
> >> terminated by
> >> #{current_user.given_name}")
> >>        send_timing(@deployment.name [3], "terminated by
> >> #{current_user.given_name}", "complete")
> >>      end
> >>    end
> >>
> >>    def do_terminate(id)
> >>      rm_arr = [id]
> >>      rq = Sidekiq::RetrySet.new
> >>      rq.select do |job|
> >>        rm_arr.include?(job.args.first)
> >>      end.map(&:delete)
> >>    end
> >>
> >> In the past, Mike has suggested that there's something different
> >> about
> >> production console vs production code.  I can't figure out what
> >> that
> >> difference could be.
> >>
> >> Can anyone suggest any steps to troubleshoot?
> >>
> >> Thanks in advance,
> >> Jason
> >
> >
> >
> > Links:
> > ------
> > [1] http://current_user.id
> > [2] http://deployment.id
> > [3] http://deployment.name
> > [4] mailto:nospam@jason-michael.com
>

Re: [sidekiq] issues remain with terminating retrying workers

From:
Jason Michael
Date:
2013-04-09 @ 23:02
I'll make these changes and test with and without the specific size.

So why can't I kill retrying workers?


On Tue, Apr 9, 2013 at 5:45 PM, Mike Perham <mperham@gmail.com> wrote:

> Yeah, with 2.9.0 you can remove the Passenger stuff:
>
> Sidekiq.configure_client do |config|
>   config.redis = { :size => 1 }
> end
>
>  then your initializer is about as simple as can be.  If you're ok with 5
> redis connections per rails process, you can delete the initializer
> altogether.  Less code == less potential for bugs.
>
>
>
> On Tue, Apr 9, 2013 at 3:00 PM, <nospam@jason-michael.com> wrote:
>
>> /config/initializers/sidekiq.rb
>> if defined?(PhusionPassenger)
>>    PhusionPassenger.on_event(:starting_worker_process) do |forked|
>>      Sidekiq.configure_client do |config|
>>        config.redis = { :size => 1 }
>>      end if forked
>>    end
>> end
>>
>> /etc/redis.conf
>>
>> daemonize yes
>> pidfile /var/run/redis/redis.pid
>> port 6379
>> bind 127.0.0.1
>> timeout 0
>> loglevel notice
>> logfile /var/log/redis/redis.log
>> databases 16
>> save 900 1
>> save 300 10
>> save 60 10000
>> rdbcompression yes
>> dbfilename dump.rdb
>> dir /var/lib/redis/
>> slave-serve-stale-data yes
>> appendonly no
>> appendfsync everysec
>> no-appendfsync-on-rewrite no
>> auto-aof-rewrite-percentage 100
>> auto-aof-rewrite-min-size 64mb
>> slowlog-log-slower-than 10000
>> slowlog-max-len 1024
>> vm-enabled no
>> vm-swap-file /tmp/redis.swap
>> vm-max-memory 0
>> vm-page-size 32
>> vm-pages 134217728
>> vm-max-threads 4
>> hash-max-zipmap-entries 512
>> hash-max-zipmap-value 64
>> list-max-ziplist-entries 512
>> list-max-ziplist-value 64
>> set-max-intset-entries 512
>> zset-max-ziplist-entries 128
>> zset-max-ziplist-value 64
>> activerehashing yes
>>
>> is there another config of interest?
>>
>> On 2013-04-09 16:19, Mike Perham wrote:
>> > Your code looks fine.  Show us your config.
>> >
>> > On Tue, Apr 9, 2013 at 11:48 AM, <nospam@jason-michael.com [4]>
>> > wrote:
>> >
>> >> I am still having trouble terminating workers which are in retry
>> >> using
>> >> the sidekiq api.
>> >>
>> >> I am able to run this block of code in the production console, and
>> >> the
>> >> workers in the retry queue disappear.  But when I execute the same
>> >> code
>> >> in my rails app, it doesn't make the retrying workers go away.
>> >>
>> >> here's the code:  (in this example, I have a Deployment object,
>> >> whose
>> >> id is the only argument to the the worker)
>> >>
>> >>    def terminate(current_user)
>> >>      @deployment.update_attribute(:terminated_by,
>> >> current_user.id [1])
>> >>      @deployment.update_attribute(:terminated_at, Time.now)
>> >>
>> >>      if @deployment.terminate!
>> >>        do_terminate(@deployment.id [2])
>> >>
>> >>        send_email do
>> >>
>> >>
>> >
>> >
>> DeploymentsMailer.delay.deployment_terminated(@deployment.mail_recipient,
>> >> @deployment)
>> >>        end
>> >>        Deployment.log_result(@deployment, "Deployment
>> >> terminated by
>> >> #{current_user.given_name}")
>> >>        send_timing(@deployment.name [3], "terminated by
>> >> #{current_user.given_name}", "complete")
>> >>      end
>> >>    end
>> >>
>> >>    def do_terminate(id)
>> >>      rm_arr = [id]
>> >>      rq = Sidekiq::RetrySet.new
>> >>      rq.select do |job|
>> >>        rm_arr.include?(job.args.first)
>> >>      end.map(&:delete)
>> >>    end
>> >>
>> >> In the past, Mike has suggested that there's something different
>> >> about
>> >> production console vs production code.  I can't figure out what
>> >> that
>> >> difference could be.
>> >>
>> >> Can anyone suggest any steps to troubleshoot?
>> >>
>> >> Thanks in advance,
>> >> Jason
>> >
>> >
>> >
>> > Links:
>> > ------
>> > [1] http://current_user.id
>> > [2] http://deployment.id
>> > [3] http://deployment.name
>> > [4] mailto:nospam@jason-michael.com
>>
>
>

Re: [sidekiq] issues remain with terminating retrying workers

From:
Mike Perham
Date:
2013-04-09 @ 23:07
I'm a little confused by your terminology.  You are trying match and delete
jobs on the retry queue, yes?  I think of workers as the processor threads
within the Sidekiq server that process jobs.

You absolutely should be able to find and remove retries (jobs on the retry
queue) using the API.  The fact that it works in rails console shows that
it is possible.  I still have no idea why it works in one process but not
in another.


On Tue, Apr 9, 2013 at 4:02 PM, Jason Michael <chewmanfoo@gmail.com> wrote:

> I'll make these changes and test with and without the specific size.
>
> So why can't I kill retrying workers?
>
>
> On Tue, Apr 9, 2013 at 5:45 PM, Mike Perham <mperham@gmail.com> wrote:
>
>> Yeah, with 2.9.0 you can remove the Passenger stuff:
>>
>> Sidekiq.configure_client do |config|
>>   config.redis = { :size => 1 }
>> end
>>
>>  then your initializer is about as simple as can be.  If you're ok with 5
>> redis connections per rails process, you can delete the initializer
>> altogether.  Less code == less potential for bugs.
>>
>>
>>
>> On Tue, Apr 9, 2013 at 3:00 PM, <nospam@jason-michael.com> wrote:
>>
>>> /config/initializers/sidekiq.rb
>>> if defined?(PhusionPassenger)
>>>    PhusionPassenger.on_event(:starting_worker_process) do |forked|
>>>      Sidekiq.configure_client do |config|
>>>        config.redis = { :size => 1 }
>>>      end if forked
>>>    end
>>> end
>>>
>>> /etc/redis.conf
>>>
>>> daemonize yes
>>> pidfile /var/run/redis/redis.pid
>>> port 6379
>>> bind 127.0.0.1
>>> timeout 0
>>> loglevel notice
>>> logfile /var/log/redis/redis.log
>>> databases 16
>>> save 900 1
>>> save 300 10
>>> save 60 10000
>>> rdbcompression yes
>>> dbfilename dump.rdb
>>> dir /var/lib/redis/
>>> slave-serve-stale-data yes
>>> appendonly no
>>> appendfsync everysec
>>> no-appendfsync-on-rewrite no
>>> auto-aof-rewrite-percentage 100
>>> auto-aof-rewrite-min-size 64mb
>>> slowlog-log-slower-than 10000
>>> slowlog-max-len 1024
>>> vm-enabled no
>>> vm-swap-file /tmp/redis.swap
>>> vm-max-memory 0
>>> vm-page-size 32
>>> vm-pages 134217728
>>> vm-max-threads 4
>>> hash-max-zipmap-entries 512
>>> hash-max-zipmap-value 64
>>> list-max-ziplist-entries 512
>>> list-max-ziplist-value 64
>>> set-max-intset-entries 512
>>> zset-max-ziplist-entries 128
>>> zset-max-ziplist-value 64
>>> activerehashing yes
>>>
>>> is there another config of interest?
>>>
>>> On 2013-04-09 16:19, Mike Perham wrote:
>>> > Your code looks fine.  Show us your config.
>>> >
>>> > On Tue, Apr 9, 2013 at 11:48 AM, <nospam@jason-michael.com [4]>
>>> > wrote:
>>> >
>>> >> I am still having trouble terminating workers which are in retry
>>> >> using
>>> >> the sidekiq api.
>>> >>
>>> >> I am able to run this block of code in the production console, and
>>> >> the
>>> >> workers in the retry queue disappear.  But when I execute the same
>>> >> code
>>> >> in my rails app, it doesn't make the retrying workers go away.
>>> >>
>>> >> here's the code:  (in this example, I have a Deployment object,
>>> >> whose
>>> >> id is the only argument to the the worker)
>>> >>
>>> >>    def terminate(current_user)
>>> >>      @deployment.update_attribute(:terminated_by,
>>> >> current_user.id [1])
>>> >>      @deployment.update_attribute(:terminated_at, Time.now)
>>> >>
>>> >>      if @deployment.terminate!
>>> >>        do_terminate(@deployment.id [2])
>>> >>
>>> >>        send_email do
>>> >>
>>> >>
>>> >
>>> >
>>> DeploymentsMailer.delay.deployment_terminated(@deployment.mail_recipient,
>>> >> @deployment)
>>> >>        end
>>> >>        Deployment.log_result(@deployment, "Deployment
>>> >> terminated by
>>> >> #{current_user.given_name}")
>>> >>        send_timing(@deployment.name [3], "terminated by
>>> >> #{current_user.given_name}", "complete")
>>> >>      end
>>> >>    end
>>> >>
>>> >>    def do_terminate(id)
>>> >>      rm_arr = [id]
>>> >>      rq = Sidekiq::RetrySet.new
>>> >>      rq.select do |job|
>>> >>        rm_arr.include?(job.args.first)
>>> >>      end.map(&:delete)
>>> >>    end
>>> >>
>>> >> In the past, Mike has suggested that there's something different
>>> >> about
>>> >> production console vs production code.  I can't figure out what
>>> >> that
>>> >> difference could be.
>>> >>
>>> >> Can anyone suggest any steps to troubleshoot?
>>> >>
>>> >> Thanks in advance,
>>> >> Jason
>>> >
>>> >
>>> >
>>> > Links:
>>> > ------
>>> > [1] http://current_user.id
>>> > [2] http://deployment.id
>>> > [3] http://deployment.name
>>> > [4] mailto:nospam@jason-michael.com
>>>
>>
>>
>