librelist archives

« back to archive

Job ID's

Job ID's

From:
Aaron Quint
Date:
2009-12-29 @ 20:11
Hey All

I've been hacking on a fork of resque that diverges a bit from the
very very simple atomic-list as queue.
http://github.com/quirkey/resque/tree/job-status

Its pretty divergent, but the idea is each job/payload keeps its own
UUID. Even though this requires a little more overhead and storage in
Redis, the potential benefits are large. My need for this has come
from putting LONG tasks in the queue. For these tasks it would be nice
to,

1 - be able to report on the status of the job, from within the job
2 - be able to watch/zoom in on a specific job
3 - be able to re prioritize the specific job
4 - Potentially be able to kill or move the queue of a specific job

The first one is especially handy for my needs. I was going to just
make this into an outside gem, but it required some invasive changes
to the way jobs are stored that I dont think I could have done with
just monkey patching.

Is this useful for other people? Is it worth merging? Should I just
try to maintain my fork?

--AQ


Aaron Quint
http://www.quirkey.com

Re: [resque] Job ID's

From:
Jonathan
Date:
2009-12-29 @ 22:52
Nice...
I too needed to be able to report on the status of jobs, watch/zoom- 
in, kill, etc... And I find your approach interesting.

I actually perform my async (Resque) job-request(s) within a  
"transaction" where I store a reference for each request in my MySQL  
database. I then store the generated "id" (from MySQL) as part of the  
payload to Resque.

Finally, I have a "wrapper" around each request-payload (i.e. a Job- 
Runner class) that is responsible for keeping the two databases (redis  
& mysql) in sync.

So, although I may not replace my solution, I'd be interested in  
peeking at the code.

Cheers,
Jonathan

On Dec 29, 2009, at 12:11 PM, Aaron Quint wrote:

> Hey All
>
> I've been hacking on a fork of resque that diverges a bit from the
> very very simple atomic-list as queue.
> http://github.com/quirkey/resque/tree/job-status
>
> Its pretty divergent, but the idea is each job/payload keeps its own
> UUID. Even though this requires a little more overhead and storage in
> Redis, the potential benefits are large. My need for this has come
> from putting LONG tasks in the queue. For these tasks it would be nice
> to,
>
> 1 - be able to report on the status of the job, from within the job
> 2 - be able to watch/zoom in on a specific job
> 3 - be able to re prioritize the specific job
> 4 - Potentially be able to kill or move the queue of a specific job
>
> The first one is especially handy for my needs. I was going to just
> make this into an outside gem, but it required some invasive changes
> to the way jobs are stored that I dont think I could have done with
> just monkey patching.
>
> Is this useful for other people? Is it worth merging? Should I just
> try to maintain my fork?
>
> --AQ
>
>
> Aaron Quint
> http://www.quirkey.com

Re: [resque] Job ID's

From:
Aaron Quint
Date:
2009-12-30 @ 15:02
We seem to have similar needs, I just figured storing references to
job in my main DB kind of defeated the purpose of using a super fast
KV Store like Redis. However, you're solution sounds pretty solid. My
need for this kind of system comes out of a feature that I'm working
on thats somewhat similar to the way EngineYard Cloud deploys work (if
you've seen them). Basically, you put a (very) long running job in the
queue, and in the browser you can long poll for updates to a sort of
'log' of activity for that job, and potentially even show a progress
meter.

--AQ

Aaron Quint
http://www.quirkey.com



On Tue, Dec 29, 2009 at 5:52 PM, Jonathan <jonathan@singlefeed.com> wrote:
> Nice...
> I too needed to be able to report on the status of jobs, watch/zoom-
> in, kill, etc... And I find your approach interesting.
>
> I actually perform my async (Resque) job-request(s) within a
> "transaction" where I store a reference for each request in my MySQL
> database. I then store the generated "id" (from MySQL) as part of the
> payload to Resque.
>
> Finally, I have a "wrapper" around each request-payload (i.e. a Job-
> Runner class) that is responsible for keeping the two databases (redis
> & mysql) in sync.
>
> So, although I may not replace my solution, I'd be interested in
> peeking at the code.
>
> Cheers,
> Jonathan
>
> On Dec 29, 2009, at 12:11 PM, Aaron Quint wrote:
>
>> Hey All
>>
>> I've been hacking on a fork of resque that diverges a bit from the
>> very very simple atomic-list as queue.
>> http://github.com/quirkey/resque/tree/job-status
>>
>> Its pretty divergent, but the idea is each job/payload keeps its own
>> UUID. Even though this requires a little more overhead and storage in
>> Redis, the potential benefits are large. My need for this has come
>> from putting LONG tasks in the queue. For these tasks it would be nice
>> to,
>>
>> 1 - be able to report on the status of the job, from within the job
>> 2 - be able to watch/zoom in on a specific job
>> 3 - be able to re prioritize the specific job
>> 4 - Potentially be able to kill or move the queue of a specific job
>>
>> The first one is especially handy for my needs. I was going to just
>> make this into an outside gem, but it required some invasive changes
>> to the way jobs are stored that I dont think I could have done with
>> just monkey patching.
>>
>> Is this useful for other people? Is it worth merging? Should I just
>> try to maintain my fork?
>>
>> --AQ
>>
>>
>> Aaron Quint
>> http://www.quirkey.com
>
>