librelist archives

« back to archive

JRuby and Resque?

JRuby and Resque?

From:
Grant Birchmeier
Date:
2012-08-03 @ 22:59
Has anyone worked with JRuby and Rescue?

I'm currently working with it on a project, and I'm having jobs fail with "GC
overhead limit exceeded" / "java.lang.OutOfMemoryError: Java heap space".

So I tried launching my worker like so:
VVERBOSE=1 QUEUE=* jruby -J-Xmx2048m -J-XX:MaxPermSize=512m -S bundle exec
rake environment resque:work

I figure 2 Gigs should about cover it, but it's still occurring.

I've only begun working on this, but I thought I'd ask the list to see if
JRuby is considered a proven platform for Resque at this time.

-- 
Grant Birchmeier
*Connamara Systems, LLC*
*Made-To-Measure Trading Solutions.*
Exactly what you need. No more. No less.*
*
http://connamara.com

Re: [resque] JRuby and Resque?

From:
James Sanders
Date:
2012-08-03 @ 23:13
Hi Grant,

How much memory usage do you see just from "bundle exec rake environment" -
in my experience, it is loading the rails environment itself that tends to
be very memory-heavy, not resque.

-James

On Fri, Aug 3, 2012 at 4:59 PM, Grant Birchmeier
<gbirchmeier@connamara.com>wrote:

> Has anyone worked with JRuby and Rescue?
>
> I'm currently working with it on a project, and I'm having jobs fail with "GC
> overhead limit exceeded" / "java.lang.OutOfMemoryError: Java heap space".
>
> So I tried launching my worker like so:
> VVERBOSE=1 QUEUE=* jruby -J-Xmx2048m -J-XX:MaxPermSize=512m -S bundle exec
> rake environment resque:work
>
> I figure 2 Gigs should about cover it, but it's still occurring.
>
> I've only begun working on this, but I thought I'd ask the list to see if
> JRuby is considered a proven platform for Resque at this time.
>
> --
> Grant Birchmeier
> *Connamara Systems, LLC*
> *Made-To-Measure Trading Solutions.*
> Exactly what you need. No more. No less.*
> *
> http://connamara.com
>
>

Re: [resque] JRuby and Resque?

From:
Eric Tang
Date:
2012-08-04 @ 05:03
resque forks, so JRuby is probably not the best option (since every fork is
gonna be a new VM for Java).  Maybe try something like
Celluloid<http://celluloid.io/>
?

Eric


On Fri, Aug 3, 2012 at 7:13 PM, James Sanders <sanderjd@gmail.com> wrote:

> Hi Grant,
>
>
How much memory usage do you see just from "bundle exec rake environment" -
> in my experience, it is loading the rails environment itself that tends to
> be very memory-heavy, not resque.
>
> -James
>
>
> On Fri, Aug 3, 2012 at 4:59 PM, Grant Birchmeier <
> gbirchmeier@connamara.com> wrote:
>
>> Has anyone worked with JRuby and Rescue?
>>
>> I'm currently working with it on a project, and I'm having jobs fail with
>> "GC overhead limit exceeded" / "java.lang.OutOfMemoryError: Java heap
>> space".
>>
>> So I tried launching my worker like so:
>> VVERBOSE=1 QUEUE=* jruby -J-Xmx2048m -J-XX:MaxPermSize=512m -S bundle
>> exec rake environment resque:work
>>
>> I figure 2 Gigs should about cover it, but it's still occurring.
>>
>> I've only begun working on this, but I thought I'd ask the list to see if
>> JRuby is considered a proven platform for Resque at this time.
>>
>> --
>> Grant Birchmeier
>> *Connamara Systems, LLC*
>> *Made-To-Measure Trading Solutions.*
>> Exactly what you need. No more. No less.*
>> *
>> http://connamara.com
>>
>>
>

Re: [resque] JRuby and Resque?

From:
Grant Birchmeier
Date:
2012-08-04 @ 15:23
Ouch, so you're saying with Resque that, once my job gets going, I'm going
to have three VMs running at once?  (e.g. 1 for the main rails app, 1 for
the worker, and 1 for the worker's job)  That could get ugly.

Celluloid looks promising.  I'll look into it.

-Grant

On Sat, Aug 4, 2012 at 12:03 AM, Eric Tang <eric.x.tang@gmail.com> wrote:

> resque forks, so JRuby is probably not the best option (since every fork
> is gonna be a new VM for Java).  Maybe try something like 
Celluloid<http://celluloid.io/>
> ?
>
> Eric
>
>
> On Fri, Aug 3, 2012 at 7:13 PM, James Sanders <sanderjd@gmail.com> wrote:
>
>> Hi Grant,
>>
>>
> How much memory usage do you see just from "bundle exec rake environment"
>> - in my experience, it is loading the rails environment itself that tends
>> to be very memory-heavy, not resque.
>>
>> -James
>>
>>
>> On Fri, Aug 3, 2012 at 4:59 PM, Grant Birchmeier <
>> gbirchmeier@connamara.com> wrote:
>>
>>> Has anyone worked with JRuby and Rescue?
>>>
>>> I'm currently working with it on a project, and I'm having jobs fail
>>> with "GC overhead limit exceeded" / "java.lang.OutOfMemoryError: Java
>>> heap space".
>>>
>>> So I tried launching my worker like so:
>>> VVERBOSE=1 QUEUE=* jruby -J-Xmx2048m -J-XX:MaxPermSize=512m -S bundle
>>> exec rake environment resque:work
>>>
>>> I figure 2 Gigs should about cover it, but it's still occurring.
>>>
>>> I've only begun working on this, but I thought I'd ask the list to see
>>> if JRuby is considered a proven platform for Resque at this time.
>>>
>>> --
>>> Grant Birchmeier
>>> *Connamara Systems, LLC*
>>> *Made-To-Measure Trading Solutions.*
>>> Exactly what you need. No more. No less.*
>>> *
>>> http://connamara.com
>>>
>>>
>>
>


-- 
Grant Birchmeier
*Connamara Systems, LLC*
*Made-To-Measure Trading Solutions.*
Exactly what you need. No more. No less.*
*
http://connamara.com

Re: [resque] JRuby and Resque?

From:
Jose Cortinas
Date:
2012-08-04 @ 09:24
I would advise that you use Sideqik. It actually emulates the same API as 
resque so that developers can transition from on library to the other 
without issue. It also uses celluloid and therefore would support real 
threads in JRuby. I think it's definitely the best solution for you.

- Jose Cortinas

On Aug 4, 2012, at 12:03 AM, Eric Tang <eric.x.tang@gmail.com> wrote:

> resque forks, so JRuby is probably not the best option (since every fork
is gonna be a new VM for Java).  Maybe try something like Celluloid?
> 
> Eric
> 
> 
> On Fri, Aug 3, 2012 at 7:13 PM, James Sanders <sanderjd@gmail.com> wrote:
> Hi Grant,
>  
> How much memory usage do you see just from "bundle exec rake 
environment" - in my experience, it is loading the rails environment 
itself that tends to be very memory-heavy, not resque.
> 
> -James
> 
> 
> On Fri, Aug 3, 2012 at 4:59 PM, Grant Birchmeier 
<gbirchmeier@connamara.com> wrote:
> Has anyone worked with JRuby and Rescue?
> 
> I'm currently working with it on a project, and I'm having jobs fail 
with "GC overhead limit exceeded" / "java.lang.OutOfMemoryError: Java heap
space".
> 
> So I tried launching my worker like so:
> VVERBOSE=1 QUEUE=* jruby -J-Xmx2048m -J-XX:MaxPermSize=512m -S bundle 
exec rake environment resque:work
> 
> I figure 2 Gigs should about cover it, but it's still occurring.
> 
> I've only begun working on this, but I thought I'd ask the list to see 
if JRuby is considered a proven platform for Resque at this time.
> 
> -- 
> Grant Birchmeier
> Connamara Systems, LLC
> Made-To-Measure Trading Solutions.
> Exactly what you need. No more. No less.
> http://connamara.com
> 
> 
> 

Re: [resque] JRuby and Resque?

From:
Grant Birchmeier
Date:
2012-08-04 @ 15:44
Won't Sideqik be a separate-running VM as well?

Although, I guess it'd only be *one* additional VM, with no forking for
jobs, right?

On Sat, Aug 4, 2012 at 4:24 AM, Jose Cortinas <jacortinas@gmail.com> wrote:

> I would advise that you use Sideqik. It actually emulates the same API as
> resque so that developers can transition from on library to the other
> without issue. It also uses celluloid and therefore would support real
> threads in JRuby. I think it's definitely the best solution for you.
>
> - Jose Cortinas
>
>
> On Aug 4, 2012, at 12:03 AM, Eric Tang <eric.x.tang@gmail.com> wrote:
>
> resque forks, so JRuby is probably not the best option (since every fork
> is gonna be a new VM for Java).  Maybe try something like 
Celluloid<http://celluloid.io/>
> ?
>
> Eric
>
>
> On Fri, Aug 3, 2012 at 7:13 PM, James Sanders <sanderjd@gmail.com> wrote:
>
>> Hi Grant,
>>
>>
> How much memory usage do you see just from "bundle exec rake environment"
>> - in my experience, it is loading the rails environment itself that tends
>> to be very memory-heavy, not resque.
>>
>> -James
>>
>>
>> On Fri, Aug 3, 2012 at 4:59 PM, Grant Birchmeier <
>> gbirchmeier@connamara.com> wrote:
>>
>>> Has anyone worked with JRuby and Rescue?
>>>
>>> I'm currently working with it on a project, and I'm having jobs fail
>>> with "GC overhead limit exceeded" / "java.lang.OutOfMemoryError: Java
>>> heap space".
>>>
>>> So I tried launching my worker like so:
>>> VVERBOSE=1 QUEUE=* jruby -J-Xmx2048m -J-XX:MaxPermSize=512m -S bundle
>>> exec rake environment resque:work
>>>
>>> I figure 2 Gigs should about cover it, but it's still occurring.
>>>
>>> I've only begun working on this, but I thought I'd ask the list to see
>>> if JRuby is considered a proven platform for Resque at this time.
>>>
>>> --
>>> Grant Birchmeier
>>> *Connamara Systems, LLC*
>>> *Made-To-Measure Trading Solutions.*
>>> Exactly what you need. No more. No less.*
>>> *
>>> http://connamara.com
>>>
>>>
>>
>


-- 
Grant Birchmeier
*Connamara Systems, LLC*
*Made-To-Measure Trading Solutions.*
Exactly what you need. No more. No less.*
*
http://connamara.com

Re: [resque] JRuby and Resque?

From:
Jose Cortinas
Date:
2012-08-04 @ 15:53
It would be another VM for each worker you run, but the work itself would 
be able to do the job of a couple of resque workers. (at least)

You can also look into something like Torquebox, it's an amazing server 
environment for JRuby apps and has great support for background jobs, 
message queue and other features. Since it's made for this exact kind of 
situation it will only run 1 VM and do the proper Java stuff of having 
multiple threads and being concurrent and all of that nice JVM jazz.

The very last thing you could do is to quite literally just launch another
thread (or use something like celluloid to do it) from your main process 
and handle the backgrounding of workers yourself. Especially since you are
trying to stay within 1 JVM process. Most Ruby libraries that aren't JVM 
specific are going to provide more support for the multi-processs model 
than the multi-thread or evented concurrency models.

Hope this helps.

- Jose Cortinas

On Aug 4, 2012, at 10:44 AM, Grant Birchmeier <gbirchmeier@connamara.com> wrote:

> Won't Sideqik be a separate-running VM as well?
> 
> Although, I guess it'd only be *one* additional VM, with no forking for 
jobs, right?
> 
> On Sat, Aug 4, 2012 at 4:24 AM, Jose Cortinas <jacortinas@gmail.com> wrote:
> I would advise that you use Sideqik. It actually emulates the same API 
as resque so that developers can transition from on library to the other 
without issue. It also uses celluloid and therefore would support real 
threads in JRuby. I think it's definitely the best solution for you.
> 
> - Jose Cortinas
> 
> 
> On Aug 4, 2012, at 12:03 AM, Eric Tang <eric.x.tang@gmail.com> wrote:
> 
>> resque forks, so JRuby is probably not the best option (since every 
fork is gonna be a new VM for Java).  Maybe try something like Celluloid?
>> 
>> Eric
>> 
>> 
>> On Fri, Aug 3, 2012 at 7:13 PM, James Sanders <sanderjd@gmail.com> wrote:
>> Hi Grant,
>>  
>> How much memory usage do you see just from "bundle exec rake 
environment" - in my experience, it is loading the rails environment 
itself that tends to be very memory-heavy, not resque.
>> 
>> -James
>> 
>> 
>> On Fri, Aug 3, 2012 at 4:59 PM, Grant Birchmeier 
<gbirchmeier@connamara.com> wrote:
>> Has anyone worked with JRuby and Rescue?
>> 
>> I'm currently working with it on a project, and I'm having jobs fail 
with "GC overhead limit exceeded" / "java.lang.OutOfMemoryError: Java heap
space".
>> 
>> So I tried launching my worker like so:
>> VVERBOSE=1 QUEUE=* jruby -J-Xmx2048m -J-XX:MaxPermSize=512m -S bundle 
exec rake environment resque:work
>> 
>> I figure 2 Gigs should about cover it, but it's still occurring.
>> 
>> I've only begun working on this, but I thought I'd ask the list to see 
if JRuby is considered a proven platform for Resque at this time.
>> 
>> -- 
>> Grant Birchmeier
>> Connamara Systems, LLC
>> Made-To-Measure Trading Solutions.
>> Exactly what you need. No more. No less.
>> http://connamara.com
>> 
>> 
>> 
> 
> 
> 
> -- 
> Grant Birchmeier
> Connamara Systems, LLC
> Made-To-Measure Trading Solutions.
> Exactly what you need. No more. No less.
> http://connamara.com
>