librelist archives

« back to archive

HashRing?

HashRing?

From:
Mat Schaffer
Date:
2012-11-27 @ 20:04
Is it possible (or advisable) to run Resque on a hash ring of redis servers?

Does anyone have any general advice on ensuring queue availability across
multiple data centers (EC2 zones)?

Thanks!

-Mat

about.me/matschaffer

Re: HashRing?

From:
Mat Schaffer
Date:
2012-11-27 @ 20:34
Hrm... now that I look closer looks like it'd be a Redis::Distributed.


-Mat

about.me/matschaffer



On Tue, Nov 27, 2012 at 3:04 PM, Mat Schaffer <mat@schaffer.me> wrote:

> Is it possible (or advisable) to run Resque on a hash ring of redis
> servers?
>
> Does anyone have any general advice on ensuring queue availability across
> multiple data centers (EC2 zones)?
>
> Thanks!
>
> -Mat
>
> about.me/matschaffer
>
>

Re: [resque] Re: HashRing?

From:
David Balatero
Date:
2012-11-27 @ 20:39
I've configured redis_failover before with ZooKeeper and it can at least 
provide automated failover. There are cases in which the clients can die 
using failover... But you might be able to rescue + retry Redis N times 
when that happens, to at least avoid errors.

As far as multi data center... can't help there. I imagine you could get 
most of the way there with Redis Failover though.

David

On Nov 27, 2012, at 12:34 PM, Mat Schaffer <mat@schaffer.me> wrote:

> Hrm... now that I look closer looks like it'd be a Redis::Distributed.
> 
> 
> -Mat
> 
> about.me/matschaffer
> 
> 
> 
> On Tue, Nov 27, 2012 at 3:04 PM, Mat Schaffer <mat@schaffer.me> wrote:
>> Is it possible (or advisable) to run Resque on a hash ring of redis servers?
>> 
>> Does anyone have any general advice on ensuring queue availability 
across multiple data centers (EC2 zones)?
>> 
>> Thanks!
>> 
>> -Mat
>> 
>> about.me/matschaffer
> 

Re: [resque] Re: HashRing?

From:
Mat Schaffer
Date:
2012-11-28 @ 02:13
Thanks David, this setup makes sense to me. I'm curious how possible the
consistent hashing would be with resque given that some keys are probably
quite a bit hotter than others. Of course I haven't looked too hard at how
resque is implemented so maybe it takes this into account.


-Mat

about.me/matschaffer



On Tue, Nov 27, 2012 at 3:39 PM, David Balatero <dbalatero@gmail.com> wrote:

> I've configured redis_failover before with ZooKeeper and it can at least
> provide automated failover. There are cases in which the clients can die
> using failover... But you might be able to rescue + retry Redis N times
> when that happens, to at least avoid errors.
>
> As far as multi data center... can't help there. I imagine you could get
> most of the way there with Redis Failover though.
>
> David
>
> On Nov 27, 2012, at 12:34 PM, Mat Schaffer <mat@schaffer.me> wrote:
>
> Hrm... now that I look closer looks like it'd be a Redis::Distributed.
>
>
> -Mat
>
> about.me/matschaffer
>
>
>
> On Tue, Nov 27, 2012 at 3:04 PM, Mat Schaffer <mat@schaffer.me> wrote:
>
>> Is it possible (or advisable) to run Resque on a hash ring of redis
>> servers?
>>
>> Does anyone have any general advice on ensuring queue availability across
>> multiple data centers (EC2 zones)?
>>
>> Thanks!
>>
>> -Mat
>>
>> about.me/matschaffer
>>
>>
>

Re: [resque] Re: HashRing?

From:
Ashley Martens
Date:
2012-11-28 @ 14:19
The distributed client works fine for resque. The problem in your case is 
that it provides no HA. It just spreads the keys over the ring. And yes if
you only have a few keys then you will have hot nodes. 



On Nov 27, 2012, at 18:13, Mat Schaffer <mat@schaffer.me> wrote:

> Thanks David, this setup makes sense to me. I'm curious how possible the
consistent hashing would be with resque given that some keys are probably 
quite a bit hotter than others. Of course I haven't looked too hard at how
resque is implemented so maybe it takes this into account.
> 
> 
> -Mat
> 
> about.me/matschaffer
> 
> 
> 
> On Tue, Nov 27, 2012 at 3:39 PM, David Balatero <dbalatero@gmail.com> wrote:
>> I've configured redis_failover before with ZooKeeper and it can at 
least provide automated failover. There are cases in which the clients can
die using failover... But you might be able to rescue + retry Redis N 
times when that happens, to at least avoid errors.
>> 
>> As far as multi data center... can't help there. I imagine you could 
get most of the way there with Redis Failover though.
>> 
>> David
>> 
>> On Nov 27, 2012, at 12:34 PM, Mat Schaffer <mat@schaffer.me> wrote:
>> 
>>> Hrm... now that I look closer looks like it'd be a Redis::Distributed.
>>> 
>>> 
>>> -Mat
>>> 
>>> about.me/matschaffer
>>> 
>>> 
>>> 
>>> On Tue, Nov 27, 2012 at 3:04 PM, Mat Schaffer <mat@schaffer.me> wrote:
>>>> Is it possible (or advisable) to run Resque on a hash ring of redis servers?
>>>> 
>>>> Does anyone have any general advice on ensuring queue availability 
across multiple data centers (EC2 zones)?
>>>> 
>>>> Thanks!
>>>> 
>>>> -Mat
>>>> 
>>>> about.me/matschaffer
> 

Re: [resque] Re: HashRing?

From:
Mat Schaffer
Date:
2012-11-28 @ 20:30
Any guesses as to what Resque's key distribution might look like? I'm not
yet familiar with the internal key structure, but I can go code spelunking
if I need to.

Do you think it would make sense for HA to have each node on the ring be a
redis failover client with a slave or might that add too much complexity?

Thanks for the input, everyone. I suspect some prototyping will be in order
here too but any inside knowledge is that much less I need to discover on
my own.


-Mat

about.me/matschaffer



On Wed, Nov 28, 2012 at 9:19 AM, Ashley Martens <ashleym1972@gmail.com>wrote:

> The distributed client works fine for resque. The problem in your case is
> that it provides no HA. It just spreads the keys over the ring. And yes if
> you only have a few keys then you will have hot nodes.
>
>
>
> On Nov 27, 2012, at 18:13, Mat Schaffer <mat@schaffer.me> wrote:
>
> Thanks David, this setup makes sense to me. I'm curious how possible the
> consistent hashing would be with resque given that some keys are probably
> quite a bit hotter than others. Of course I haven't looked too hard at how
> resque is implemented so maybe it takes this into account.
>
>
> -Mat
>
> about.me/matschaffer
>
>
>
> On Tue, Nov 27, 2012 at 3:39 PM, David Balatero <dbalatero@gmail.com>wrote:
>
>> I've configured redis_failover before with ZooKeeper and it can at least
>> provide automated failover. There are cases in which the clients can die
>> using failover... But you might be able to rescue + retry Redis N times
>> when that happens, to at least avoid errors.
>>
>> As far as multi data center... can't help there. I imagine you could get
>> most of the way there with Redis Failover though.
>>
>> David
>>
>> On Nov 27, 2012, at 12:34 PM, Mat Schaffer <mat@schaffer.me> wrote:
>>
>> Hrm... now that I look closer looks like it'd be a Redis::Distributed.
>>
>>
>> -Mat
>>
>> about.me/matschaffer
>>
>>
>>
>> On Tue, Nov 27, 2012 at 3:04 PM, Mat Schaffer <mat@schaffer.me> wrote:
>>
>>> Is it possible (or advisable) to run Resque on a hash ring of redis
>>> servers?
>>>
>>> Does anyone have any general advice on ensuring queue availability
>>> across multiple data centers (EC2 zones)?
>>>
>>> Thanks!
>>>
>>> -Mat
>>>
>>> about.me/matschaffer
>>>
>>>
>>
>

Re: [resque] Re: HashRing?

From:
Ashley Martens
Date:
2012-11-29 @ 02:03
Resque does not have many keys unless you have a lot of queues. If you are
super worried about backup HA nodes then a slave for each node is about 
right. Use one of the existing tools for that. Maybe sentenial. 



On Nov 28, 2012, at 12:30, Mat Schaffer <mat@schaffer.me> wrote:

> Any guesses as to what Resque's key distribution might look like? I'm 
not yet familiar with the internal key structure, but I can go code 
spelunking if I need to.
> 
> Do you think it would make sense for HA to have each node on the ring be
a redis failover client with a slave or might that add too much 
complexity?
> 
> Thanks for the input, everyone. I suspect some prototyping will be in 
order here too but any inside knowledge is that much less I need to 
discover on my own.
> 
> 
> -Mat
> 
> about.me/matschaffer
> 
> 
> 
> On Wed, Nov 28, 2012 at 9:19 AM, Ashley Martens <ashleym1972@gmail.com> wrote:
>> The distributed client works fine for resque. The problem in your case 
is that it provides no HA. It just spreads the keys over the ring. And yes
if you only have a few keys then you will have hot nodes. 
>> 
>> 
>> 
>> On Nov 27, 2012, at 18:13, Mat Schaffer <mat@schaffer.me> wrote:
>> 
>>> Thanks David, this setup makes sense to me. I'm curious how possible 
the consistent hashing would be with resque given that some keys are 
probably quite a bit hotter than others. Of course I haven't looked too 
hard at how resque is implemented so maybe it takes this into account.
>>> 
>>> 
>>> -Mat
>>> 
>>> about.me/matschaffer
>>> 
>>> 
>>> 
>>> On Tue, Nov 27, 2012 at 3:39 PM, David Balatero <dbalatero@gmail.com> wrote:
>>>> I've configured redis_failover before with ZooKeeper and it can at 
least provide automated failover. There are cases in which the clients can
die using failover... But you might be able to rescue + retry Redis N 
times when that happens, to at least avoid errors.
>>>> 
>>>> As far as multi data center... can't help there. I imagine you could 
get most of the way there with Redis Failover though.
>>>> 
>>>> David
>>>> 
>>>> On Nov 27, 2012, at 12:34 PM, Mat Schaffer <mat@schaffer.me> wrote:
>>>> 
>>>>> Hrm... now that I look closer looks like it'd be a Redis::Distributed.
>>>>> 
>>>>> 
>>>>> -Mat
>>>>> 
>>>>> about.me/matschaffer
>>>>> 
>>>>> 
>>>>> 
>>>>> On Tue, Nov 27, 2012 at 3:04 PM, Mat Schaffer <mat@schaffer.me> wrote:
>>>>>> Is it possible (or advisable) to run Resque on a hash ring of redis
servers?
>>>>>> 
>>>>>> Does anyone have any general advice on ensuring queue availability 
across multiple data centers (EC2 zones)?
>>>>>> 
>>>>>> Thanks!
>>>>>> 
>>>>>> -Mat
>>>>>> 
>>>>>> about.me/matschaffer
> 

Re: [resque] Re: HashRing?

From:
Mat Schaffer
Date:
2012-11-29 @ 03:08
Gotcha. Thanks for the advice!


-Mat

about.me/matschaffer



On Wed, Nov 28, 2012 at 9:03 PM, Ashley Martens <ashleym1972@gmail.com>wrote:

> Resque does not have many keys unless you have a lot of queues. If you are
> super worried about backup HA nodes then a slave for each node is about
> right. Use one of the existing tools for that. Maybe sentenial.
>
>
>
> On Nov 28, 2012, at 12:30, Mat Schaffer <mat@schaffer.me> wrote:
>
> Any guesses as to what Resque's key distribution might look like? I'm not
> yet familiar with the internal key structure, but I can go code spelunking
> if I need to.
>
> Do you think it would make sense for HA to have each node on the ring be a
> redis failover client with a slave or might that add too much complexity?
>
> Thanks for the input, everyone. I suspect some prototyping will be in
> order here too but any inside knowledge is that much less I need to
> discover on my own.
>
>
> -Mat
>
> about.me/matschaffer
>
>
>
> On Wed, Nov 28, 2012 at 9:19 AM, Ashley Martens <ashleym1972@gmail.com>wrote:
>
>> The distributed client works fine for resque. The problem in your case is
>> that it provides no HA. It just spreads the keys over the ring. And yes if
>> you only have a few keys then you will have hot nodes.
>>
>>
>>
>> On Nov 27, 2012, at 18:13, Mat Schaffer <mat@schaffer.me> wrote:
>>
>> Thanks David, this setup makes sense to me. I'm curious how possible the
>> consistent hashing would be with resque given that some keys are probably
>> quite a bit hotter than others. Of course I haven't looked too hard at how
>> resque is implemented so maybe it takes this into account.
>>
>>
>> -Mat
>>
>> about.me/matschaffer
>>
>>
>>
>> On Tue, Nov 27, 2012 at 3:39 PM, David Balatero <dbalatero@gmail.com>wrote:
>>
>>> I've configured redis_failover before with ZooKeeper and it can at least
>>> provide automated failover. There are cases in which the clients can die
>>> using failover... But you might be able to rescue + retry Redis N times
>>> when that happens, to at least avoid errors.
>>>
>>> As far as multi data center... can't help there. I imagine you could get
>>> most of the way there with Redis Failover though.
>>>
>>> David
>>>
>>> On Nov 27, 2012, at 12:34 PM, Mat Schaffer <mat@schaffer.me> wrote:
>>>
>>> Hrm... now that I look closer looks like it'd be a Redis::Distributed.
>>>
>>>
>>> -Mat
>>>
>>> about.me/matschaffer
>>>
>>>
>>>
>>> On Tue, Nov 27, 2012 at 3:04 PM, Mat Schaffer <mat@schaffer.me> wrote:
>>>
>>>> Is it possible (or advisable) to run Resque on a hash ring of redis
>>>> servers?
>>>>
>>>> Does anyone have any general advice on ensuring queue availability
>>>> across multiple data centers (EC2 zones)?
>>>>
>>>> Thanks!
>>>>
>>>> -Mat
>>>>
>>>> about.me/matschaffer
>>>>
>>>>
>>>
>>
>