librelist archives

« back to archive

Different Concurrency Levels for Different Workers

Different Concurrency Levels for Different Workers

From:
bradford
Date:
2012-06-20 @ 20:04
Is there a way to say, I want SingleJobWorker processing only 1
message at a time, while I want ParallelJobWorker processing in
parallel?

Here's my situation.  I'm trialing sidekiq.  I have many jobs I want
to process asynchronously, that can be done in parallel.  But, I also
have this one kind of job that I need to processes only one at a time.
 The reason for this is because I'm clearing and writing to a file
over NFS, and I don't want collision (and have the file end up in a
bad state).

Thanks,
Bradford

Re: [sidekiq] Different Concurrency Levels for Different Workers

From:
Mike Perham
Date:
2012-06-22 @ 02:12
Sidekiq is not appropriate for the serial job and I don't want to make
it appropriate.  Different tools are useful for different reasons,
jack of all trades master of none, etc.

On Wed, Jun 20, 2012 at 1:04 PM, bradford <fingermark@gmail.com> wrote:
> Is there a way to say, I want SingleJobWorker processing only 1
> message at a time, while I want ParallelJobWorker processing in
> parallel?
>
> Here's my situation.  I'm trialing sidekiq.  I have many jobs I want
> to process asynchronously, that can be done in parallel.  But, I also
> have this one kind of job that I need to processes only one at a time.
>  The reason for this is because I'm clearing and writing to a file
> over NFS, and I don't want collision (and have the file end up in a
> bad state).

Re: [sidekiq] Different Concurrency Levels for Different Workers

From:
Jesse Cooke
Date:
2012-06-22 @ 01:29
For that I think you would need to set up 2 sidekiq servers (they can both
live on the same box) where one just your SingleJobWorker's jobs (which
would basically have a concurrency set to 1) and the other runs all of your
other jobs (concurrency set higher) that way you can assure yourself your
SingleJobWorker's jobs are being processed serially.

--------------------------------------------
Jesse Cooke :: N-tier Engineer
jc00ke.com / @jc00ke <http://twitter.com/jc00ke>


On Wed, Jun 20, 2012 at 1:04 PM, bradford <fingermark@gmail.com> wrote:

> Is there a way to say, I want SingleJobWorker processing only 1
> message at a time, while I want ParallelJobWorker processing in
> parallel?
>
> Here's my situation.  I'm trialing sidekiq.  I have many jobs I want
> to process asynchronously, that can be done in parallel.  But, I also
> have this one kind of job that I need to processes only one at a time.
>  The reason for this is because I'm clearing and writing to a file
> over NFS, and I don't want collision (and have the file end up in a
> bad state).
>
> Thanks,
> Bradford
>