librelist archives

« back to archive

Trying to get started.

Trying to get started.

From:
Tim Uckun
Date:
2011-01-16 @ 02:28
Ok I decided to give resque a go but I can't seem to get my worker to
run. Here is what I have done so far.

install redis from the repos of ubuntu (2.01 I think)
made sure it's running, connected to it via telnet etc..

Set up the gemfile like this.
#for resque
gem 'resque'
gem 'redis'
gem 'redis-namespace'
gem  'yajl-ruby'
gem "resque-scheduler"
gem "SystemTimer"

Set up an initializer like this.

require 'resque'
Resque.redis = AppConfig.redis_server

(the config file has localhost:6379)

Set up a resque.rake in the lib/tasks

require 'resque/tasks'
task "resque:setup" => :environment

Modified one of my existing workers to work with resque I added a
method to it like this.

def self.perform(data)
    #resque calls this..
    keyword_processor = Worker::Processors::Keyword.new :logger => Rails.logger
    logger.debug "starting keyword feeder from resque"
    puts "putting something to stdout.."
  end

Then I ran the rake task trying different ways to get it to go
QUEUE=* rake resque:work
rake resque:work QUEUE=*
QUEUE=* rake environment resque:work

etc..

They all did the same thing rake launched and was presumably ready to
accept the tasks.

*** Starting worker golden:30363:*


Then I fired up a rails console
Resque.enqueue(Worker::Processors::Keyword, '')

The response is
=> 1

Nothing happens here. Nothing in the logs, nothing in the console running rake.

Redis log shows activity it looks like this.

28695] 16 Jan 15:25:17 - DB 0: 10 keys (0 volatile) in 16 slots HT.
[28695] 16 Jan 15:25:17 - 1 clients connected (0 slaves), 559504 bytes in use
[28695] 16 Jan 15:25:20 - Accepted 127.0.0.1:44157
[28695] 16 Jan 15:25:22 * 10 changes in 300 seconds. Saving...
[28695] 16 Jan 15:25:22 * Background saving started by pid 30429
[30429] 16 Jan 15:25:22 * DB saved on disk
[28695] 16 Jan 15:25:22 * Background saving terminated with success
[28695] 16 Jan 15:25:22 - DB 0: 12 keys (0 volatile) in 16 slots HT.
[28695] 16 Jan 15:25:22 - 2 clients connected (0 slaves), 562616 bytes in use
[28695] 16 Jan 15:25:27 - DB 0: 12 keys (0 volatile) in 16 slots HT.
[28695] 16 Jan 15:25:27 - 2 clients connected (0 slaves), 562620 bytes in use
[28695] 16 Jan 15:25:32 - DB 0: 12 keys (0 volatile) in 16 slots HT.
[28695] 16 Jan 15:25:32 - 2 clients connected (0 slaves), 562620 bytes in use
[28695] 16 Jan 15:25:37 - DB 0: 12 keys (0 volatile) in 16 slots HT.
[28695] 16 Jan 15:25:37 - 2 clients connected (0 slaves), 562620 bytes in use
[28695] 16 Jan 15:25:42 - DB 0: 12 keys (0 volatile) in 16 slots HT.
[28695] 16 Jan 15:25:42 - 2 clients connected (0 slaves), 562616 bytes in use
[28695] 16 Jan 15:25:47 - DB 0: 12 keys (0 volatile) in 16 slots HT.
[28695] 16 Jan 15:25:47 - 2 clients connected (0 slaves), 562620 bytes in use
[28695] 16 Jan 15:25:52 - DB 0: 12 keys (0 volatile) in 16 slots HT.
[28695] 16 Jan 15:25:52 - 2 clients connected (0 slaves), 562616 bytes in use
[28695] 16 Jan 15:25:57 - DB 0: 12 keys (0 volatile) in 16 slots HT.
[28695] 16 Jan 15:25:57 - 2 clients connected (0 slaves), 562616 bytes in use


so where did I go wrong?

Thanks.

Re: [resque] Trying to get started.

From:
Karl Baum
Date:
2011-01-16 @ 02:49
Try running resque with VVERBOSE=1

Also, did you define your @queue on your worker?

Starting up resque-web will also help you to understand what's going on.

Just execute resque-web.

Hope that helps.
On Jan 15, 2011, at 9:28 PM, Tim Uckun wrote:

> Ok I decided to give resque a go but I can't seem to get my worker to
> run. Here is what I have done so far.
> 
> install redis from the repos of ubuntu (2.01 I think)
> made sure it's running, connected to it via telnet etc..
> 
> Set up the gemfile like this.
> #for resque
> gem 'resque'
> gem 'redis'
> gem 'redis-namespace'
> gem  'yajl-ruby'
> gem "resque-scheduler"
> gem "SystemTimer"
> 
> Set up an initializer like this.
> 
> require 'resque'
> Resque.redis = AppConfig.redis_server
> 
> (the config file has localhost:6379)
> 
> Set up a resque.rake in the lib/tasks
> 
> require 'resque/tasks'
> task "resque:setup" => :environment
> 
> Modified one of my existing workers to work with resque I added a
> method to it like this.
> 
> def self.perform(data)
>    #resque calls this..
>    keyword_processor = Worker::Processors::Keyword.new :logger => Rails.logger
>    logger.debug "starting keyword feeder from resque"
>    puts "putting something to stdout.."
>  end
> 
> Then I ran the rake task trying different ways to get it to go
> QUEUE=* rake resque:work
> rake resque:work QUEUE=*
> QUEUE=* rake environment resque:work
> 
> etc..
> 
> They all did the same thing rake launched and was presumably ready to
> accept the tasks.
> 
> *** Starting worker golden:30363:*
> 
> 
> Then I fired up a rails console
> Resque.enqueue(Worker::Processors::Keyword, '')
> 
> The response is
> => 1
> 
> Nothing happens here. Nothing in the logs, nothing in the console running rake.
> 
> Redis log shows activity it looks like this.
> 
> 28695] 16 Jan 15:25:17 - DB 0: 10 keys (0 volatile) in 16 slots HT.
> [28695] 16 Jan 15:25:17 - 1 clients connected (0 slaves), 559504 bytes in use
> [28695] 16 Jan 15:25:20 - Accepted 127.0.0.1:44157
> [28695] 16 Jan 15:25:22 * 10 changes in 300 seconds. Saving...
> [28695] 16 Jan 15:25:22 * Background saving started by pid 30429
> [30429] 16 Jan 15:25:22 * DB saved on disk
> [28695] 16 Jan 15:25:22 * Background saving terminated with success
> [28695] 16 Jan 15:25:22 - DB 0: 12 keys (0 volatile) in 16 slots HT.
> [28695] 16 Jan 15:25:22 - 2 clients connected (0 slaves), 562616 bytes in use
> [28695] 16 Jan 15:25:27 - DB 0: 12 keys (0 volatile) in 16 slots HT.
> [28695] 16 Jan 15:25:27 - 2 clients connected (0 slaves), 562620 bytes in use
> [28695] 16 Jan 15:25:32 - DB 0: 12 keys (0 volatile) in 16 slots HT.
> [28695] 16 Jan 15:25:32 - 2 clients connected (0 slaves), 562620 bytes in use
> [28695] 16 Jan 15:25:37 - DB 0: 12 keys (0 volatile) in 16 slots HT.
> [28695] 16 Jan 15:25:37 - 2 clients connected (0 slaves), 562620 bytes in use
> [28695] 16 Jan 15:25:42 - DB 0: 12 keys (0 volatile) in 16 slots HT.
> [28695] 16 Jan 15:25:42 - 2 clients connected (0 slaves), 562616 bytes in use
> [28695] 16 Jan 15:25:47 - DB 0: 12 keys (0 volatile) in 16 slots HT.
> [28695] 16 Jan 15:25:47 - 2 clients connected (0 slaves), 562620 bytes in use
> [28695] 16 Jan 15:25:52 - DB 0: 12 keys (0 volatile) in 16 slots HT.
> [28695] 16 Jan 15:25:52 - 2 clients connected (0 slaves), 562616 bytes in use
> [28695] 16 Jan 15:25:57 - DB 0: 12 keys (0 volatile) in 16 slots HT.
> [28695] 16 Jan 15:25:57 - 2 clients connected (0 slaves), 562616 bytes in use
> 
> 
> so where did I go wrong?
> 
> Thanks.

Re: [resque] Trying to get started.

From:
Tim Uckun
Date:
2011-01-16 @ 09:36
** [10:13:43 2011-01-16] 32430: Sleeping for 5



> Starting up resque-web will also help you to understand what's going on.
>
> Just execute resque-web.
>
> Hope that helps.

Thanks that helped a lot. I did pull up the web site and noticed that
the worker was failing with an exception.

The documentation says

Failure
If a job raises an exception, it is logged and handed off to the
Resque::Failure module. Failures are logged either locally in Redis or
using some different backend.
Keep this in mind when writing your jobs: you may want to throw
exceptions you would not normally throw in order to assist debugging.

In my case there was no logging of the error to the console, the rails
log file or syslog. I presume when it says logged it means to redis
right?

Is there a way I can handle the errors myself?

Re: [resque] Trying to get started.

From:
Karl Baum
Date:
2011-01-16 @ 23:51
Correct, by default it gets put on resque failure queue which can be 
viewed with resque-web.  If you want to handle errors differently, you can
implement your own Resque::Failure backend by extending 
Resque::Failure::Base.  Then set a new backend like:

Resque::Failure.backend = MyCustomFailureBackend

Specifically you might want to take a look at Resque::Failure::Multiple as
it allows for setting multiple failure backends.  I think you might want a
logger backend in addition to redis.


On Jan 16, 2011, at 4:36 AM, Tim Uckun wrote:

> ** [10:13:43 2011-01-16] 32430: Sleeping for 5
> 
> 
> 
>> Starting up resque-web will also help you to understand what's going on.
>> 
>> Just execute resque-web.
>> 
>> Hope that helps.
> 
> Thanks that helped a lot. I did pull up the web site and noticed that
> the worker was failing with an exception.
> 
> The documentation says
> 
> Failure
> If a job raises an exception, it is logged and handed off to the
> Resque::Failure module. Failures are logged either locally in Redis or
> using some different backend.
> Keep this in mind when writing your jobs: you may want to throw
> exceptions you would not normally throw in order to assist debugging.
> 
> In my case there was no logging of the error to the console, the rails
> log file or syslog. I presume when it says logged it means to redis
> right?
> 
> Is there a way I can handle the errors myself?

Re: [resque] Trying to get started.

From:
Thibaut Barrère
Date:
2011-01-17 @ 11:08
>
> Specifically you might want to take a look at Resque::Failure::Multiple as
> it allows for setting multiple failure backends.  I think you might want a
> logger backend in addition to redis.


Here's my working config (tested against resque 1.5.0) to keep errors in
Redis and into a custom backend:

https://gist.github.com/782719

I seem to recall two points (but you may want to check if this is still
true)
- the Multiple backend must be configured last, as the last configured
backend wins
- the order of backends specified into Multiple is important - you may want
to use the most important for you in first position

Take a look at the existings backends (
https://github.com/defunkt/resque/tree/master/lib/resque/failure) to see how
to implement one.

cheers,

-- Thibaut

Re: [resque] Trying to get started.

From:
Tim Uckun
Date:
2011-01-17 @ 11:56
On Tue, Jan 18, 2011 at 12:08 AM, Thibaut Barrère
<thibaut.barrere@gmail.com> wrote:
>> Specifically you might want to take a look at Resque::Failure::Multiple as
>> it allows for setting multiple failure backends.  I think you might want a
>> logger backend in addition to redis.


I'll have to look into that.  I currently have a lot of error handling
code which excludes some errors, logs others, emails others etc.  I
wanted to leverage the existing code (remember I am converting
existing workers to resque) so I first attempted to put a
around_perform like this.

def self.around_perform(*args)
    @logger.debug "#{self.class.to_s} called with #{args}"
    yield  *args
   @logger.debug "End of #{self.class.to_s}"
rescue => e
  handle_error e
end

This didn't work for some strange reason. The code never yielded. I
tried making it an instance method but that didn't work either so I
set it aside and went for the

def self.on_failure (e)
    handle_error e
 end

That worked pretty well so that's where I am now.

I am curious about they the yield didn't work but I don't need it for
now so on to the rest of the coding.