librelist archives

« back to archive

Fail a job

Fail a job

From:
Carlos Eduardo
Date:
2010-11-24 @ 02:38
Hi,

I using resque to convert videos using ffmpeg.

If a conversion fails, how do I send the same conversion to the (re)queue
again?

Is there a way to persist and fail a job he returned to the queue?

How does this work?


Thanks!

Carlos Eduardo

Re: [resque] Fail a job

From:
Mason Jones
Date:
2010-11-24 @ 04:41
Assuming that you have a worker that's doing the conversion, and you
could catch any errors, wouldn't you be able to simply re-enqueue the
job from within the worker, and have another worker pick it up at that
point to re-try? You could even add an additional parameter with the
number of retries, so that you can abandon it after a certain number
of attempts to keep it from trying forever.


On Tue, Nov 23, 2010 at 6:38 PM, Carlos Eduardo
<carloseduardo@egenial.com.br> wrote:
> Hi,
>
> I using resque to convert videos using ffmpeg.
>
> If a conversion fails, how do I send the same conversion to the (re)queue
> again?
>
> Is there a way to persist and fail a job he returned to the queue?
>
> How does this work?
>
>
> Thanks!
>
> Carlos Eduardo
>

Re: [resque] Fail a job

From:
Tony Arcieri
Date:
2010-11-24 @ 05:04
What happens if the worker fails by its hard disk crashing or power supply
failing?

On Tue, Nov 23, 2010 at 9:41 PM, Mason Jones <masonoise@gmail.com> wrote:

> Assuming that you have a worker that's doing the conversion, and you
> could catch any errors, wouldn't you be able to simply re-enqueue the
> job from within the worker, and have another worker pick it up at that
> point to re-try? You could even add an additional parameter with the
> number of retries, so that you can abandon it after a certain number
> of attempts to keep it from trying forever.
>
>
> On Tue, Nov 23, 2010 at 6:38 PM, Carlos Eduardo
> <carloseduardo@egenial.com.br> wrote:
> > Hi,
> >
> > I using resque to convert videos using ffmpeg.
> >
> > If a conversion fails, how do I send the same conversion to the (re)queue
> > again?
> >
> > Is there a way to persist and fail a job he returned to the queue?
> >
> > How does this work?
> >
> >
> > Thanks!
> >
> > Carlos Eduardo
> >
>



-- 
Tony Arcieri
Medioh! A Kudelski Brand

Re: [resque] Fail a job

From:
Mason Jones
Date:
2010-11-24 @ 06:15
In a disaster scenario like that, then you're of course going to have
to do some additional tracking if you want to ensure a retry, yes. I'd
probably look into flagging the state when the worker starts, so you
can periodically check for workers that started but didn't complete.
To be honest, though, if you're looking for a 100% durable queueing
solution, you're likely better off looking at options other than
Resque. Resque is fantastic and easy to use, but as the README itself
says, there are other systems that are better for some situations. On
the other hand, if you do the work to add more such features, we'd all
be grateful, I'm sure!


On Tue, Nov 23, 2010 at 9:04 PM, Tony Arcieri <tony.arcieri@medioh.com> wrote:
> What happens if the worker fails by its hard disk crashing or power supply
> failing?
>
> On Tue, Nov 23, 2010 at 9:41 PM, Mason Jones <masonoise@gmail.com> wrote:
>>
>> Assuming that you have a worker that's doing the conversion, and you
>> could catch any errors, wouldn't you be able to simply re-enqueue the
>> job from within the worker, and have another worker pick it up at that
>> point to re-try? You could even add an additional parameter with the
>> number of retries, so that you can abandon it after a certain number
>> of attempts to keep it from trying forever.
>>
>>
>> On Tue, Nov 23, 2010 at 6:38 PM, Carlos Eduardo
>> <carloseduardo@egenial.com.br> wrote:
>> > Hi,
>> >
>> > I using resque to convert videos using ffmpeg.
>> >
>> > If a conversion fails, how do I send the same conversion to the
>> > (re)queue
>> > again?
>> >
>> > Is there a way to persist and fail a job he returned to the queue?
>> >
>> > How does this work?
>> >
>> >
>> > Thanks!
>> >
>> > Carlos Eduardo
>> >
>
>
>
> --
> Tony Arcieri
> Medioh! A Kudelski Brand
>

Re: [resque] Fail a job

From:
Karl Baum
Date:
2010-11-24 @ 11:53
I think you just need retry capability?  If so, i use resque-retry.

https://github.com/lantins/resque-retry


On Nov 24, 2010, at 1:15 AM, Mason Jones wrote:

> In a disaster scenario like that, then you're of course going to have
> to do some additional tracking if you want to ensure a retry, yes. I'd
> probably look into flagging the state when the worker starts, so you
> can periodically check for workers that started but didn't complete.
> To be honest, though, if you're looking for a 100% durable queueing
> solution, you're likely better off looking at options other than
> Resque. Resque is fantastic and easy to use, but as the README itself
> says, there are other systems that are better for some situations. On
> the other hand, if you do the work to add more such features, we'd all
> be grateful, I'm sure!
> 
> 
> On Tue, Nov 23, 2010 at 9:04 PM, Tony Arcieri <tony.arcieri@medioh.com> wrote:
>> What happens if the worker fails by its hard disk crashing or power supply
>> failing?
>> 
>> On Tue, Nov 23, 2010 at 9:41 PM, Mason Jones <masonoise@gmail.com> wrote:
>>> 
>>> Assuming that you have a worker that's doing the conversion, and you
>>> could catch any errors, wouldn't you be able to simply re-enqueue the
>>> job from within the worker, and have another worker pick it up at that
>>> point to re-try? You could even add an additional parameter with the
>>> number of retries, so that you can abandon it after a certain number
>>> of attempts to keep it from trying forever.
>>> 
>>> 
>>> On Tue, Nov 23, 2010 at 6:38 PM, Carlos Eduardo
>>> <carloseduardo@egenial.com.br> wrote:
>>>> Hi,
>>>> 
>>>> I using resque to convert videos using ffmpeg.
>>>> 
>>>> If a conversion fails, how do I send the same conversion to the
>>>> (re)queue
>>>> again?
>>>> 
>>>> Is there a way to persist and fail a job he returned to the queue?
>>>> 
>>>> How does this work?
>>>> 
>>>> 
>>>> Thanks!
>>>> 
>>>> Carlos Eduardo
>>>> 
>> 
>> 
>> 
>> --
>> Tony Arcieri
>> Medioh! A Kudelski Brand
>> 

Re: [resque] Fail a job

From:
Tony Arcieri
Date:
2010-11-24 @ 03:17
I'm working on a very similar problem with video transcoding where I would
like to re-enque failed videos after troubleshooting has occurred for a
particular problem.

I've been hacking every now and then on a gem to manage complex multistage
workflows with Resque as the backend for executing jobs, but unfortunately I
never got around to finishing it:

https://github.com/tarcieri/resque-workflow

On Tue, Nov 23, 2010 at 7:38 PM, Carlos Eduardo <
carloseduardo@egenial.com.br> wrote:

> Hi,
>
> I using resque to convert videos using ffmpeg.
>
> If a conversion fails, how do I send the same conversion to the (re)queue
> again?
>
> Is there a way to persist and fail a job he returned to the queue?
>
> How does this work?
>
>
> Thanks!
>
> Carlos Eduardo
>



-- 
Tony Arcieri
Medioh! A Kudelski Brand

Re: [resque] Fail a job

From:
Carlos Eduardo G. Franco
Date:
2010-11-24 @ 11:12
Hi Tony,

The resque workflow is working 100% with the latest version of resque?

I plan on using your dsl, seems less interesting and should work fine with
https://github.com/gcaixeta/streamio-ffmpeg ffmpeg wrapper.


Thanks!


Carlos Eduardo