librelist archives

« back to archive

does 1 sidekiq process with 50 threads use only 1 cpu core or multiple with ruby 2.0 ?

does 1 sidekiq process with 50 threads use only 1 cpu core or multiple with ruby 2.0 ?

From:
Michael Ni
Date:
2014-09-23 @ 03:12
does 1 sidekiq process with 50 threads use only 1 cpu core or multiple with
ruby 2.0 ?

ever since ruby 1.9, ruby uses native threads right?
although GIL still exists preventing true parallelism
I would think that a sidekiq process uses multiple cores.

But I read the following articles


http://stackoverflow.com/questions/22304965/can-sidekiq-take-advantage-of-multiple-cpu-cores

http://stackoverflow.com/questions/19018186/does-sidekiq-use-multiple-cpu-cores-and-can-it-be-run-on-multiple-machines

Re: [sidekiq] does 1 sidekiq process with 50 threads use only 1 cpu core or multiple with ruby 2.0 ?

From:
Jesse Cooke
Date:
2014-09-23 @ 03:55
The main process will use 1 core, but you may gain some parallelism
depending on the type of work the job is performing.
MRI releases the GIL for certain kinds of things, like some I/O.

If you really want to take advantage of multiple threads, look into
Rubinius or JRuby.

On Mon, Sep 22, 2014 at 8:12 PM, Michael Ni <michaelcni@gmail.com> wrote:

> does 1 sidekiq process with 50 threads use only 1 cpu core or multiple
> with ruby 2.0 ?
>
> ever since ruby 1.9, ruby uses native threads right?
> although GIL still exists preventing true parallelism
> I would think that a sidekiq process uses multiple cores.
>
> But I read the following articles
>
>
> 
http://stackoverflow.com/questions/22304965/can-sidekiq-take-advantage-of-multiple-cpu-cores
>
> 
http://stackoverflow.com/questions/19018186/does-sidekiq-use-multiple-cpu-cores-and-can-it-be-run-on-multiple-machines
>

Re: [sidekiq] does 1 sidekiq process with 50 threads use only 1 cpu core or multiple with ruby 2.0 ?

From:
Michael Ni
Date:
2014-09-23 @ 16:37
majority is mysql queries, then network calls,

maybe I will try splitting up into 3-4 workers

I tried JRuby over a year go, ran into too many issues getting gems to work.

Thanks for the confirmation.

On Mon, Sep 22, 2014 at 8:55 PM, Jesse Cooke <jesse@jc00ke.com> wrote:

> The main process will use 1 core, but you may gain some parallelism
> depending on the type of work the job is performing.
> MRI releases the GIL for certain kinds of things, like some I/O.
>
> If you really want to take advantage of multiple threads, look into
> Rubinius or JRuby.
>
> On Mon, Sep 22, 2014 at 8:12 PM, Michael Ni <michaelcni@gmail.com> wrote:
>
>> does 1 sidekiq process with 50 threads use only 1 cpu core or multiple
>> with ruby 2.0 ?
>>
>> ever since ruby 1.9, ruby uses native threads right?
>> although GIL still exists preventing true parallelism
>> I would think that a sidekiq process uses multiple cores.
>>
>> But I read th e following articles
>>
>>
>> 
http://stackoverflow.com/questions/22304965/can-sidekiq-take-advantage-of-multiple-cpu-cores
>>
>> 
http://stackoverflow.com/questions/19018186/does-sidekiq-use-multiple-cpu-cores-and-can-it-be-run-on-multiple-machines
>>
>
>

Re: [sidekiq] does 1 sidekiq process with 50 threads use only 1 cpu core or multiple with ruby 2.0 ?

From:
Michael Ni
Date:
2014-09-23 @ 20:03
One more question,

does the Global Interpreter Lock affect only 1 ruby process at a time? or
does it affect all ruby processes?

I'm thinking each ruby process would have a GIL but 2 separate ruby
processes should be running parallel right?

my problem is really I have super long running processes, I process large
gigabyte files and I don't separate it into small chunks because I need
some sort of control over order.

I'm thinking Resque may be a better solution.
It seems sidekiq is good for rapid small jobs, where loading the
environment etc would be extremely wasteful,

for me I have less than 200 jobs a day, but many of them are 10 min to 1 hr
long, mainly scripts that move data from csv files and insert them into a
db by key

On Tue, Sep 23, 2014 at 9:37 AM, Michael Ni <michaelcni@gmail.com> wrote:

> majority is mysql queries, then network calls,
>
> maybe I will try splitting up into 3-4 workers
>
> I tried JRuby over a year go, ran into too many issues getting gems to
> work.
>
> Thanks for the confirmation.
>
> On Mon, Sep 22, 2014 at 8:55 PM, Jesse Cooke <jesse@jc00ke.com> wrote:
>
>> The main process will use 1 core, but you may gain some parallelism
>> depending on the type of work the job is performing.
>> MRI releases the GIL for certain kinds of things, like some I/O.
>>
>> If you really want to take advantage of multiple threads, look into
>> Rubinius or JRuby.
>>
>> On Mon, Sep 22, 2014 at 8:12 PM, Michael Ni <michaelcni@gmail.com> wrote:
>>
>>> does 1 sidekiq process with 50 threads use only 1 cpu core or multiple
>>> with ruby 2.0 ?
>>>
>>> ever since ruby 1.9, ruby uses native threads right?
>>> although GIL still exists preventing true parallelism
>>> I would think that a sidekiq process uses multiple cores.
>>>
>>> But I read th e following articles
>>>
>>>
>>> 
http://stackoverflow.com/questions/22304965/can-sidekiq-take-advantage-of-multiple-cpu-cores
>>>
>>> 
http://stackoverflow.com/questions/19018186/does-sidekiq-use-multiple-cpu-cores-and-can-it-be-run-on-multiple-machines
>>>
>>
>>
>

Re: [sidekiq] does 1 sidekiq process with 50 threads use only 1 cpu core or multiple with ruby 2.0 ?

From:
Jesse Cooke
Date:
2014-09-23 @ 21:54
The GIL is a per-process lock, so each MRI process will have its own GIL.

If you use multiple Ruby processes, you're using the "process concurrency"
model. This will use more RAM, but you'll utilize more cores.
You can read a little about it here: http://ruby.io/strategies/processes/

It's hard to make recommendations w/o knowing exactly what kind of work
your jobs do. You'll have to experiment and let us know what ends up
working for your use case.


On Tue, Sep 23, 2014 at 1:03 PM, Michael Ni <michaelcni@gmail.com> wrote:

> One more question,
>
> does the Global Interpreter Lock affect only 1 ruby process at a time? or
> does it affect all ruby processes?
>
> I'm thinking each ruby process would have a GIL but 2 separate ruby
> processes should be running parallel right?
>
> my problem is really I have super long running processes, I process large
> gigabyte files and I don't separate it into small chunks because I need
> some sort of control over order.
>
> I'm thinking Resque may be a better solution.
> It seems sidekiq is good for rapid small jobs, where loading the
> environment etc would be extremely wasteful,
>
> for me I have less than 200 jobs a day, but many of them are 10 min to 1
> hr long, mainly scripts that move data from csv files and insert them into
> a db by key
>
> On Tue, Sep 23, 2014 at 9:37 AM, Michael N i <michaelcni@gmail.com> wrote:
>
>> majority is mysql queries, then network calls,
>>
>> maybe I will try splitting up into 3-4 workers
>>
>> I tried JRuby over a year go, ran into too many issues getting gems to
>> work.
>>
>> Thanks for the confirmation.
>>
>> On Mon, Sep 22, 2014 at 8:55 PM, Jesse Cooke <jesse@jc00ke.com> wrote:
>>
>>> The main process will use 1 core, but you may gain some parallelism
>>> depending on the type of work the job is perfo rming.
>>> MRI releases the GIL for certain kinds of things, like some I/O.
>>>
>>> If you really want to take advantage of multiple threads, look into
>>> Rubinius or JRuby.
>>>
>>> On Mon, Sep 22, 2014 at 8:12 PM, Michael Ni <michaelcni@gmail.com>
>>> wrote:
>>>
>>>> does 1 sidekiq process with 50 threads use only 1 cpu core or multiple
>>>> with ruby 2.0 ?
>>>>
>>>> ever since ruby 1.9, ruby uses native threads right?
>>>> although GIL still exists preventing true parallelism
>>>> I would think that a sidekiq process uses multiple cores.
>>>>
>>>> But I read th e following articles
>>>>
>>>>
>>>> 
http://stackoverflow.com/questions/22304965/can-sidekiq-take-advantage-of-multiple-cpu-cores
>>>>
>>>> 
http://stackoverflow.com/questions/19018186/does-sidekiq-use-multiple-cpu-cores-and-can-it-be-run-on-multiple-machines
>>>>
>>>
>>>
>>
>

Re: [sidekiq] does 1 sidekiq process with 50 threads use only 1 cpu core or multiple with ruby 2.0 ?

From:
Michael Ni
Date:
2014-09-24 @ 00:41
Thanks about the GIL per process information.

I will just fire up more sidekiq processes, I have 16 cores, so I may just
do 16 and divide up the queues.  Don't feel like reimplementing resque, I
think by firing up more sidekiq processes, I should technically have the
same effect, if i balance out my queues well.

I don't really mind having 16 copies of my app in memory.
I wonder if it is possible to first load 1 sidekiq process, then fork the
rest of the 15th with customizable queues?
Although I think with resque, you only have 1 process and forking going on


I would be interesting if sidekiq could support both process + fork and 1
process + threads but that may cause too much confusion.



On Tue, Sep 23, 2014 at 2:54 PM, Jesse Cooke <jesse@jc00ke.com> wrote:

> The GIL is a per-process lock, so each MRI process will have its own GIL.
>
> If you use multiple Ruby processes, you're using the "process concurrency"
> model. This will use more RAM, but you'll utilize more cores.
> You can read a little about it here: http://ruby.io/strategies/processes/
>
> It's hard to make recommendations w/o knowing exactly what kind of work
> your jobs do. You'll have to experiment and let us know what ends up
> working for your use case.
>
>
> On Tue, Sep 23, 2014 at 1:03 PM, Michael Ni <michaelcni@gmail.com> wrote:
>
>> One more question,
>>
>> does the Global Interpreter Lock affect only 1 ruby process at a time? or
>> does it affect all ruby processes?
>>
>> I'm thinking each ruby process would have a GIL but 2 separate ruby
>> processes should be running parallel right?
>>
>> my problem is really I have super long running processes, I process large
>> gigabyte files and I don't separate it into small chunks because I need
>> some sort of control over order.
>>
>> I'm thinking Resque may be a better solution.
>> It seems sidekiq is good for rapid small jobs, where loading the
>> environment etc would be extremely wasteful,
>>
>> for me I have less than 200 jobs a day, but many of them are 10 min to 1
>> hr long, mainly scripts that move data from csv files and insert them into
>> a db by key
>>
>> On Tue, Sep 23, 2014 at 9:37 AM, Michael N i <michaelcni@gmail.com>
>> wrote:
>>
>>> majority is mysql queries, then network calls,
>>>
>>> maybe I will try splitting up into 3-4 workers
>>>
>>> I tried JRuby over a year go, ran into too many issues getting gems to
>>> work.
>>>
>>> Thanks for the confirmation.
>>>
>>> On Mon, Sep 22, 2014 at 8:55 PM, Jesse Cooke <jesse@jc00ke.com> wrote:
>>>
>>>> The main process will use 1 core, but you may gain some parallelism
>>>> depending on the type of work the job is perfo rming.
>>>> MRI releases the GIL for certain kinds of things, like some I/O.
>>>>
>>>> If you really want to take advantage of multiple threads, look into
>>>> Rubinius or JRuby.
>>>>
>>>> On Mon, Sep 22, 2014 at 8:12 PM, Michael Ni <michaelcni@gmail.com>
>>>> wrote:
>>>>
>>>>> does 1 sidekiq process with 50 threads use only 1 cpu core or multiple
>>>>> with ruby 2.0 ?
>>>>>
>>>>> ever since ruby 1.9, ruby uses native threads right?
>>>>> although GIL still exists preventing true parallelism
>>>>> I would think that a sidekiq process uses multiple cores.
>>>>>
>>>>> But I read th e following articles
>>>>>
>>>>>
>>>>> 
http://stackoverflow.com/questions/22304965/can-sidekiq-take-advantage-of-multiple-cpu-cores
>>>>>
>>>>> 
http://stackoverflow.com/questions/19018186/does-sidekiq-use-multiple-cpu-cores-and-can-it-be-run-on-multiple-machines
>>>>>
>>>>
>>>>
>>>
>>
>

Re: [sidekiq] does 1 sidekiq process with 50 threads use only 1 cpu core or multiple with ruby 2.0 ?

From:
Jesse Cooke
Date:
2014-09-24 @ 00:46
If you don't care about the memory, then yeah, fire up 16 sidekiq processes.

My hunch is that forking would be a bad idea. Mike and Tony would know more.

One of the biggest benefits of Sidekiq is that it doesn't fork like Resque,
so you get the parallelism w/o the memory bloat.

On Tue, Sep 23, 2014 at 5:41 PM, Michael Ni <michaelcni@gmail.com> wrote:

> Thanks about the GIL per process information.
>
> I will just fire up more sidekiq processes, I have 16 cores, so I may just
> do 16 and divide up the queues.  Don't feel like reimplementing resque, I
> think by firing up more sidekiq processes, I should technically have the
> same effect, if i balance out my queues well.
>
> I don't really mind having 16 copies of my app in memory.
> I wonder if it is possible to first load 1 sidekiq process, then fork the
> rest of the 15th with customizable queues?
> Although I think with resque, you only have 1 process and forking going on
>
>
> I would be interesting if sidekiq could support both process + fork and 1
> process + threads but that may cause too much confusion.
>
>
>
> On Tue, Sep 23, 2014 at 2:54 PM, Jesse Cooke <jesse@jc00ke.com> wrote:
>
>> The GIL is a per-process lock, so each MRI process will have its own GIL.
>>
>> If you use multiple Ruby processes, you're using the "process
>> concurrency" model. This will use more RAM, but you'll utilize more cores.
>> You can read a little about it here: http://ruby.io/strategies/processes/
>>
>> It's hard to make recommendations w/o knowing exactly what kind of work
>> your jobs do. You'll have to experiment and let us know what ends up
>> working for your use case.
>>
>>
>> On Tue, Sep 23, 2014 at 1:03 PM, Michael Ni <michaelcni@gmail.com
>> <michaelcni@gmail+.com>> wrote:
>>
>>> One more question,
>>>
>>> does the Global Interpreter Lock affect only 1 ruby process at a time?
>>> or does it affect all ruby processes?
>>>
>>> I'm thinking each ruby process would have a GIL but 2 separate ruby
>>> processes should be running parallel right?
>>>
>>> my problem is really I have super long running processes, I process
>>> large gigabyte files and I don't separate it into small chunks because I
>>> need some sort of control over order.
>>>
>>> I'm thinking Resque may be a better solution.
>>> It seems sidekiq is good for rapid small jobs, where loading the
>>> environment etc would be extremely wasteful,
>>>
>>> for me I have less than 200 jobs a day, but many of them are 10 min to 1
>>> h r long, mainly scripts that move data from csv files and insert them into
>>> a db by key
>>>
>>> On Tue, Sep 23, 2014 at 9:37 AM, Michael N i <michaelcni@gmail.com>
>>> wrote:
>>>
>>>> majority is mysql queries, then network calls,
>>>>
>>>> maybe I will try splitting up into 3-4 workers
>>>>
>>>> I tried JRuby over a year go, ran into too many issues getting gems to
>>>> work.
>>>>
>>>> Thanks for the confirmation.
>>>>
>>>> On Mon, Sep 22, 2014 at 8:55 PM, Jesse Cooke <jesse@jc00ke.com> wrote:
>>>>
>>>>> The main process will use 1 core, but you may gain some parallelism
>>>>> depending on the type of work the job is perfo rming.
>>>>> MRI releases the GIL for certain kinds of things, like some I/O.
>>>>>
>>>>> If you really want to take advantage of multiple threads, look into
>>>>> Rubinius or JRuby.
>>>>>
>>>>> On Mon, Sep 22, 2014 at 8:12 PM, Michael Ni <michaelcni@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> does 1 sidekiq process with 50 threads use only 1 cpu core or
>>>>>> multiple with ruby 2.0 ?
>>>>>>
>>>>>> ever since ruby 1.9, ruby uses native threads right?
>>>>>> although GIL still exists preventing true parallelism
>>>>>> I would think that a sidekiq process uses multiple cores.
>>>>>>
>>>>>> But I read th e following articles
>>>>>>
>>>>>>
>>>>>> 
http://stackoverflow.com/questions/22304965/can-sidekiq-take-advantage-of-multiple-cpu-cores
>>>>>>
>>>>>> 
http://stackoverflow.com/questions/19018186/does-sidekiq-use-multiple-cpu-cores-and-can-it-be-run-on-multiple-machines
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

Re: [sidekiq] does 1 sidekiq process with 50 threads use only 1 cpu core or multiple with ruby 2.0 ?

From:
Michael Ni
Date:
2014-09-24 @ 00:58
Hmm,

I wonder if it is possible to fork more sidekiq processes from 1 process.
The only differences should be the queue settings.  Anyways thanks for the
info, hopefully Mike or Tony see's this.  For now I will just use 16
processes.

On Tue, Sep 23, 2014 at 5:46 PM, Jesse Cooke <jesse@jc00ke.com> wrote:

> If you don't care about the memory, then yeah, fire up 16 sidekiq
> processes.
>
> My hunch is that forking would be a bad idea. Mike and Tony would know
> more.
>
> One of the biggest benefits of Sidekiq is that it doesn't fork like
> Resque, so you get the parallelism w/o the memory bloat.
>
> On Tue, Sep 23, 2014 at 5:41 PM, Michael Ni <michaelcni@gmail.com> wrote:
>
>> Thanks about the GIL per process information.
>>
>> I will just fire up more sidekiq processes, I have 16 cores, so I may
>> just do 16 and divide up the queues.  Don't feel like reimplementing
>> resque, I think by firing up more sidekiq processes, I should technically
>> have the same effe ct, if i balance out my queues well.
>>
>> I don't really mind having 16 copies of my app in memory.
>> I wonder if it is possible to first load 1 sidekiq process, then fork the
>> rest of the 15th with customizable queues?
>> Although I think with resque, you only have 1 process and forking going on
>>
>>
>> I would be interesting if sidekiq could support both process + fork and 1
>> process + threads but that may cause too much confusion.
>>
>>
>>
>> On Tue, Sep 23, 2014 at 2:54 PM, Jesse Cooke <jesse@jc00ke.com> wrote:
>>
>>> The GIL is a per-process lock, so each MRI process will have its own GIL.
>>>
>>> If you use multiple Ruby processes, you& #39;re using the "process
>>> concurrency" model. This will use more RAM, but you'll utilize more cores.
>>> You can read a little about it here:
>>> http://ruby.io/strategies/processes/
>>>
>>> It's hard to make recommendations w/o knowing exactly what kind of work
>>> your jobs do. You'll have to experiment and let us know what ends up
>>> working for your use case.
>>>
>>>
>>> On Tue, Sep 23, 2014 at 1:03 PM, Michael Ni <michaelcni@gmail.com
>>> <michaelcni@gmail+.com>> wrote:
>>>
>>>> One more question,
>>>>
>>>> does the Global Interpreter Lock affect only 1 ruby process at a t ime?
>>>> or does it affect all ruby processes?
>>>>
>>>> I'm thinking each ruby process would have a GIL but 2 separate ruby
>>>> processes should be running parallel right?
>>>>
>>>> my problem is really I have super long running processes, I process
>>>> large gigabyte files and I don't separate it into small chunks because I
>>>> need some sort of control over order.
>>>>
>>>> I'm thinking Resque may be a better solution.
>>>> It seems sidekiq is good for rapid small jobs, where loading the
>>>> environment etc would be extremely wasteful,
>>>>
>>>> for me I have less than 200 jobs a day, but many of them are 10 min to
>>>> 1 h r long, mainly scripts that move data from csv files and insert them
>>>> into a db by key
>>>>
>>>> On Tue, Sep 23, 2014 at 9:37 AM, Michael N i <michaelcni@gmail.com>
>>>> wrote:
>>>>
>>>>> majority is mysql queries, then network calls,
>>>>>
>>>>> maybe I will try splitting up into 3-4 workers
>>>>>
>>>>> I tried JRuby over a year go, ran into too many issues getting gems to
>>>>> work.
>>>>>
>>>>> Thanks for the confirmation.
>>>>>
>>>>> On Mon, Sep 22, 2014 at 8:55 PM, Jesse Cooke <jesse@jc00ke.com> wrote:
>>>>>
>>>>>> The main process will use 1 core, but you may gain some parallelism
>>>>>> depending on the type of work the job is perfo rming.
>>>>>> MRI releases the GIL for certain kinds of things, like some I/O.
>>>>>>
>>>>>> If you really want to take advantage of multiple threads, look into
>>>>>> Rubinius or JRuby.
>>>>>>
>>>>>> On Mon, Sep 22, 2014 at 8:12 PM, Michael Ni <michaelcni@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> does 1 sidekiq process with 50 threads use only 1 cpu core or
>>>>>>> multiple with ruby 2.0 ?
>>>>>>>
>>>>>>> ever since ruby 1.9, ruby uses native threads right?
>>>>>>> although GIL still exists preventing true parallelism
>>>>>>> I would think that a sidekiq process uses multiple cores.
>>>>>>>
>>>>>>> But I read th e following articles
>>>>>>>
>>>>>>>
>>>>>>> 
http://stackoverflow.com/questions/22304965/can-sidekiq-take-advantage-of-multiple-cpu-cores
>>>>>>>
>>>>>>> 
http://stackoverflow.com/questions/19018186/does-sidekiq-use-multiple-cpu-cores-and-can-it-be-run-on-multiple-machines
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>