librelist archives

« back to archive

check before enqueue + return success

check before enqueue + return success

From:
Date:
2011-01-26 @ 16:11
hi
is there a possibility for the queue to check its length before
accepting new jobs, so that enqueue would return true or false,
according to if the job really was enqueued or not?
btw, can there be a problem if two or more rails instances are
enqueueing simultaniously?

Re: [resque] check before enqueue + return success

From:
Lee Marlow
Date:
2011-01-26 @ 16:49
You may want to checkout the resque-lock plugin, I believe it does
what you are looking for.

https://github.com/defunkt/resque-lock

On Wed, Jan 26, 2011 at 9:11 AM,  <commb07@googlemail.com> wrote:
> hi
> is there a possibility for the queue to check its length before
> accepting new jobs, so that enqueue would return true or false,
> according to if the job really was enqueued or not?
> btw, can there be a problem if two or more rails instances are
> enqueueing simultaniously?
>

Re: [resque] check before enqueue + return success

From:
Date:
2011-01-26 @ 22:56
i may not see it, but that does not seem to do what i want.
its not important to me how many jobs (even with the same arguments) are
running, but the queue should not get too long. my jobs will typically
take a while (some 10 seconds), so if there are already some in the
queue, there shouldnt be added more, as it would take longer than people
usually want to wait.

> You may want to checkout the resque-lock plugin, I believe it does
> what you are looking for.
>
> https://github.com/defunkt/resque-lock
>
> On Wed, Jan 26, 2011 at 9:11 AM,  <commb07@googlemail.com> wrote:
>> hi
>> is there a possibility for the queue to check its length before
>> accepting new jobs, so that enqueue would return true or false,
>> according to if the job really was enqueued or not?
>> btw, can there be a problem if two or more rails instances are
>> enqueueing simultaniously?
>>

Re: [resque] check before enqueue + return success

From:
Steve Howell
Date:
2011-01-26 @ 23:04
You can call the redis command LLEN to find out the length of the queues,
and check that before you enqueue tasks through resque.

Resque is a pretty light encapsulation on top of redis (for the most part),
and it's worth learning a bit of redis to understand what's going on under
the hood.

Here is some documentation on redis:

http://redis.io/commands

On Wed, Jan 26, 2011 at 2:56 PM, <commb07@googlemail.com> wrote:

> i may not see it, but that does not seem to do what i want.
> its not important to me how many jobs (even with the same arguments) are
> running, but the queue should not get too long. my jobs will typically
> take a while (some 10 seconds), so if there are already some in the
> queue, there shouldnt be added more, as it would take longer than people
> usually want to wait.
>
> > You may want to checkout the resque-lock plugin, I believe it does
> > what you are looking for.
> >
> > https://github.com/defunkt/resque-lock
> >
> > On Wed, Jan 26, 2011 at 9:11 AM,  <commb07@googlemail.com> wrote:
> >> hi
> >> is there a possibility for the queue to check its length before
> >> accepting new jobs, so that enqueue would return true or false,
> >> according to if the job really was enqueued or not?
> >> btw, can there be a problem if two or more rails instances are
> >> enqueueing simultaniously?
> >>
>
>

Re: [resque] check before enqueue + return success

From:
Date:
2011-01-27 @ 02:31
hm, i guess ill have to dive deeper into this to be able to do what i
want. i was hoping there was a plugin or even functionality in resque to
make it easier do this.


> You can call the redis command LLEN to find out the length of the
> queues, and check that before you enqueue tasks through resque.
>
> Resque is a pretty light encapsulation on top of redis (for the most
> part), and it's worth learning a bit of redis to understand what's
> going on under the hood.
>
> Here is some documentation on redis:
>
> http://redis.io/commands
>
> On Wed, Jan 26, 2011 at 2:56 PM, <commb07@googlemail.com
> <mailto:commb07@googlemail.com>> wrote:
>
>     i may not see it, but that does not seem to do what i want.
>     its not important to me how many jobs (even with the same
>     arguments) are
>     running, but the queue should not get too long. my jobs will typically
>     take a while (some 10 seconds), so if there are already some in the
>     queue, there shouldnt be added more, as it would take longer than
>     people
>     usually want to wait.
>
>     > You may want to checkout the resque-lock plugin, I believe it does
>     > what you are looking for.
>     >
>     > https://github.com/defunkt/resque-lock
>     >
>     > On Wed, Jan 26, 2011 at 9:11 AM,  <commb07@googlemail.com
>     <mailto:commb07@googlemail.com>> wrote:
>     >> hi
>     >> is there a possibility for the queue to check its length before
>     >> accepting new jobs, so that enqueue would return true or false,
>     >> according to if the job really was enqueued or not?
>     >> btw, can there be a problem if two or more rails instances are
>     >> enqueueing simultaniously?
>     >>
>
>

Re: [resque] check before enqueue + return success

From:
Steve Howell
Date:
2011-01-27 @ 03:56
What you're trying to do should probably involve only about 5 to 15 lines of
code in your application, if you organize it correctly.   The reason that
nobody's written a plugin might have to do with the fact that a lightweight
solution is well within reach without requiring a plugin.

Resque exposes the redis instance to you, so your application code could
have something like this (untested):

  klass = ...
  queue = Resque.queue_from_class(klass)
  if Resque.redis.llen(queue) < 500
    self.enqueue(...)
  else
    log("Resque queue is too long, so we are bailing")
  end

At work we say that resque was our gateway drug to getting hooked on redis.
Resque itself is wonderful, but once we read resque's code, which is fairly
straightforward, we found that we were soon doing lots of powerful things
with redis itself, using resque as our inspiration and guide.

Here is a little peek beneath the covers of resque:

https://github.com/defunkt/resque/blob/master/lib/resque.rb

If you look at line 142 there, you'll see that Resque actually expose a
size() method to you, so you actually don't even need to get down to the
redis layer for your needs:

  # Returns an integer representing the size of a queue.
  # Queue name should be a string.
  def size(queue)
    redis.llen("queue:#{queue}").to_i
  end

To call any of the redis commands in the link I gave you earlier, you just
follow the documentation and lowercase the command name for the Ruby
library.

Good luck with the deep dive.

Cheers,

Steve

On Wed, Jan 26, 2011 at 6:31 PM, <commb07@googlemail.com> wrote:

> hm, i guess ill have to dive deeper into this to be able to do what i
> want. i was hoping there was a plugin or even functionality in resque to
> make it easier do this.
>
>
> > You can call the redis command LLEN to find out the length of the
> > queues, and check that before you enqueue tasks through resque.
> >
> > Resque is a pretty light encapsulation on top of redis (for the most
> > part), and it's worth learning a bit of redis to understand what's
> > going on under the hood.
> >
> > Here is some documentation on redis:
> >
> > http://redis.io/commands
> >
> > On Wed, Jan 26, 2011 at 2:56 PM, <commb07@googlemail.com
> > <mailto:commb07@googlemail.com>> wrote:
> >
> >     i may not see it, but that does not seem to do what i want.
> >     its not important to me how many jobs (even with the same
> >     arguments) are
> >     running, but the queue should not get too long. my jobs will
> typically
> >     take a while (some 10 seconds), so if there are already some in the
> >     queue, there shouldnt be added more, as it would take longer than
> >     people
> >     usually want to wait.
> >
> >     > You may want to checkout the resque-lock plugin, I believe it does
> >     > what you are looking for.
> >     >
> >     > https://github.com/defunkt/resque-lock
> >     >
> >     > On Wed, Jan 26, 2011 at 9:11 AM,  <commb07@googlemail.com
> >     <mailto:commb07@googlemail.com>> wrote:
> >     >> hi
> >     >> is there a possibility for the queue to check its length before
> >     >> accepting new jobs, so that enqueue would return true or false,
> >     >> according to if the job really was enqueued or not?
> >     >> btw, can there be a problem if two or more rails instances are
> >     >> enqueueing simultaniously?
> >     >>
> >
> >
>
>

Re: [resque] check before enqueue + return success

From:
Date:
2011-01-28 @ 12:19
thanks for the information, these are good starting points.
however, in reality my situation is a little more complex. i would need
the queue length in terms of estimated processing time, which can only
be determined by the job properties (i know how to calculate the
estimated time). but where should i put such a method, are there some
kind of hooks, like 'before_enqueue'?

and what happens if 2 or more long running jobs get enqueued at the same
time (if the checks happen before any of these jobs IS in the queue,
would they all get enqueued)?


----------------
> What you're trying to do should probably involve only about 5 to 15
> lines of code in your application, if you organize it correctly.   The
> reason that nobody's written a plugin might have to do with the fact
> that a lightweight solution is well within reach without requiring a
> plugin.
>
> Resque exposes the redis instance to you, so your application code
> could have something like this (untested):
>
>   klass = ...
>   queue = Resque.queue_from_class(klass)
>   if Resque.redis.llen(queue) < 500
>     self.enqueue(...)
>   else
>     log("Resque queue is too long, so we are bailing")
>   end
>
> At work we say that resque was our gateway drug to getting hooked on
> redis.  Resque itself is wonderful, but once we read resque's code,
> which is fairly straightforward, we found that we were soon doing lots
> of powerful things with redis itself, using resque as our inspiration
> and guide.
>
> Here is a little peek beneath the covers of resque:
>
> https://github.com/defunkt/resque/blob/master/lib/resque.rb
>
> If you look at line 142 there, you'll see that Resque actually expose
> a size() method to you, so you actually don't even need to get down to
> the redis layer for your needs:
>
>   # Returns an integer representing the size of a queue.
>   # Queue name should be a string.
>   def size(queue)
>     redis.llen("queue:#{queue}").to_i
>
>   end
> To call any of the redis commands in the link I gave you earlier, you
> just follow the documentation and lowercase the command name for the
> Ruby library.
>
> Good luck with the deep dive.
>
> Cheers,
>
> Steve
>
> On Wed, Jan 26, 2011 at 6:31 PM, <commb07@googlemail.com
> <mailto:commb07@googlemail.com>> wrote:
>
>     hm, i guess ill have to dive deeper into this to be able to do what i
>     want. i was hoping there was a plugin or even functionality in
>     resque to
>     make it easier do this.
>
>
>     > You can call the redis command LLEN to find out the length of the
>     > queues, and check that before you enqueue tasks through resque.
>     >
>     > Resque is a pretty light encapsulation on top of redis (for the most
>     > part), and it's worth learning a bit of redis to understand what's
>     > going on under the hood.
>     >
>     > Here is some documentation on redis:
>     >
>     > http://redis.io/commands
>     >
>     > On Wed, Jan 26, 2011 at 2:56 PM, <commb07@googlemail.com
>     <mailto:commb07@googlemail.com>
>     > <mailto:commb07@googlemail.com <mailto:commb07@googlemail.com>>>
>     wrote:
>     >
>     >     i may not see it, but that does not seem to do what i want.
>     >     its not important to me how many jobs (even with the same
>     >     arguments) are
>     >     running, but the queue should not get too long. my jobs will
>     typically
>     >     take a while (some 10 seconds), so if there are already some
>     in the
>     >     queue, there shouldnt be added more, as it would take longer than
>     >     people
>     >     usually want to wait.
>     >
>     >     > You may want to checkout the resque-lock plugin, I believe
>     it does
>     >     > what you are looking for.
>     >     >
>     >     > https://github.com/defunkt/resque-lock
>     >     >
>     >     > On Wed, Jan 26, 2011 at 9:11 AM,  <commb07@googlemail.com
>     <mailto:commb07@googlemail.com>
>     >     <mailto:commb07@googlemail.com
>     <mailto:commb07@googlemail.com>>> wrote:
>     >     >> hi
>     >     >> is there a possibility for the queue to check its length
>     before
>     >     >> accepting new jobs, so that enqueue would return true or
>     false,
>     >     >> according to if the job really was enqueued or not?
>     >     >> btw, can there be a problem if two or more rails instances are
>     >     >> enqueueing simultaniously?
>     >     >>
>     >
>     >
>
>

Re: [resque] check before enqueue + return success

From:
David Kelso
Date:
2011-01-26 @ 23:53
I just took a look at the resque web interface and this is the command
it uses to get the queue length:
Resque.size queue_name

On Wed, Jan 26, 2011 at 3:04 PM, Steve Howell <steve@strings.com> wrote:
> You can call the redis command LLEN to find out the length of the queues,
> and check that before you enqueue tasks through resque.
> Resque is a pretty light encapsulation on top of redis (for the most part),
> and it's worth learning a bit of redis to understand what's going on under
> the hood.
> Here is some documentation on redis:
> http://redis.io/commands
> On Wed, Jan 26, 2011 at 2:56 PM, <commb07@googlemail.com> wrote:
>>
>> i may not see it, but that does not seem to do what i want.
>> its not important to me how many jobs (even with the same arguments) are
>> running, but the queue should not get too long. my jobs will typically
>> take a while (some 10 seconds), so if there are already some in the
>> queue, there shouldnt be added more, as it would take longer than people
>> usually want to wait.
>>
>> > You may want to checkout the resque-lock plugin, I believe it does
>> > what you are looking for.
>> >
>> > https://github.com/defunkt/resque-lock
>> >
>> > On Wed, Jan 26, 2011 at 9:11 AM,  <commb07@googlemail.com> wrote:
>> >> hi
>> >> is there a possibility for the queue to check its length before
>> >> accepting new jobs, so that enqueue would return true or false,
>> >> according to if the job really was enqueued or not?
>> >> btw, can there be a problem if two or more rails instances are
>> >> enqueueing simultaniously?
>> >>
>>
>
>