librelist archives

« back to archive

Monkey Patching Sidekiq::Fetcher

Monkey Patching Sidekiq::Fetcher

From:
Jack Royal-Gordon
Date:
2014-06-09 @ 19:45
I’m scheduling website scaling jobs, and one of the websites that I’m 
scraping limits access to one account per IP at a time. As a result, I 
wish to implement a queue blocking feature in Sidekiq (ala Limit-Fetch, 
which is currently not supported with ReliableFetch). First I tried the 
following code:

module Sidekiq
  class Fetcher
    def block_queue(queue)
      @blocked_queues ||= []
      @blocked_queues << queue
    end

    def unblock_queue(queue)
      @blocked_queues.delete(queue)
    end

    private
    
    def queues_cmd_with_blocked_queues
      return queues_cmd_without_blocked_queues - @blocked_queues
    end
    
    alias_method_chain :queues_cmd, :blocked_queues
  end
end

This gave me “undefined method `queues_cmd' for class `Sidekiq::Fetcher'” 
when starting Sidekiq. So then I tried:

module Sidekiq
  class Fetcher
    module FetcherWithBlockedQueues
      def block_queue(queue)
        @blocked_queues ||= []
        @blocked_queues << queue
      end

      def unblock_queue(queue)
        @blocked_queues.delete(queue)
      end

      private

      def queues_cmd
        return super - @blocked_queues
      end
    end

    include FetcherWithBlockedQueues
  end
end

This gave me “uninitialized constant 
Sidekiq::Fetcher::FetcherWithBlockedQueueus” upon startup. 

I know that this is, strictly speaking, a Ruby question and not a Sidekiq 
question, but perhaps someone on the list can point me in the right 
direction for making this subtle but significant change to Sidekiq.

Re: [sidekiq] Monkey Patching Sidekiq::Fetcher

From:
Mike Perham
Date:
2014-06-09 @ 20:14
Fetcher is only valid within the server context.  You should be able to
bootstrap everything like so:

Sidekiq.configure_server do |config|
  require 'sidekiq/fetch'
  require 'my_fetch_extension'
  Sidekiq::Fetch.send(:include, MyExtension)
end if ENV['GIANT_HACK'] == '1'

The bigger picture is that Sidekiq is not and never was designed for rate
limiting.  If you need rate limiting, you should investigate a system which
has it built in.




On Mon, Jun 9, 2014 at 12:45 PM, Jack Royal-Gordon <jackrg@pobox.com> wrote:

> I’m scheduling website scaling jobs, and one of the websites that I’m
> scraping limits access to one account per IP at a time. As a result, I wish
> to implement a queue blocking feature in Sidekiq (ala Limit-Fetch, which is
> currently not supported with ReliableFetch). First I tried the following
> code:
>
> module Sidekiq
>   class Fetcher
>     def block_queue(queue)
>       @blocked_queues ||= []
>       @blocked_queues << queue
>     end
>
>     def unblock_queue(queue)
>       @blocked_queues.delete(queue)
>     end
>
>     private
>
>     def queues_cmd_with_blocked_queues
>       return queues_cmd_without_blocked_queues - @blocked_queues
>     end
>
>     alias_method_chain :queues_cmd, :blocked_queues
>   end
> end
>
> This gave me “undefined method `queues_cmd' for class `Sidekiq::Fetcher'”
> when starting Sidekiq. So then I tried:
>
> module Sidekiq
>   class Fetcher
>     module FetcherWithBlockedQueues
>       def block_queue(queue)
>         @blocked_queues ||= []
>         @blocked_queues << queue
>       end
>
>       def unblock_queue(queue)
>         @blocked_queues.delete(queue)
>       end
>
>       private
>
>       def queues_cmd
>         return super - @blocked_queues
>       end
>     end
>
>     include FetcherWithBlockedQueues
>   end
> end
>
> This gave me “uninitialized constant
> Sidekiq::Fetcher::FetcherWithBlockedQueueus” upon startup.
>
> I know that this is, strictly speaking, a Ruby question and not a Sidekiq
> question, but perhaps someone on the list can point me in the right
> direction for making this subtle but significant change to Sidekiq.
>



-- 
Mike Perham | mike@sidekiq.org