librelist archives

« back to archive

Userlevel Hotswap with multithreading

Userlevel Hotswap with multithreading

From:
Antonie Henning
Date:
2014-10-03 @ 23:59
Hi,

Is hotswap in userlevel supported when running multithread?

For example:
rs1::RandomSource( LENGTH 10, LIMIT 2, STOP false) -> Print(Cold-A) -> Discard;
rs2::RandomSource( LENGTH 10, LIMIT 2, STOP false) -> Print(Cold-B) -> Discard;
static1::StaticThreadSched(rs1 0, rs2 1);

click -R -j2 cold.click -p 13000 &


Produces:
Cold-A:   10 | 
Cold-A:   10 | 
Cold-B:   10 | 
Cold-B:   10 | 

Then telnet localhost 13000 and using the "write hotconfig" handler:
rs1::RandomSource( LENGTH 10, LIMIT 2, STOP true) -> Print(Hot-A) -> Discard;
rs2::RandomSource( LENGTH 10, LIMIT 2, STOP true) -> Print(Hot-B) -> Discard;
static2::StaticThreadSched(rs1 0, rs2 1);

Produces:
Click::ControlSocket/1.3
Hot-A: 10 | 
Hot-A: 10 | 
200 Write handler 'hotconfig' OK

Hot-B messages are missing. It seems the elements assigned to the second 
thread don't run after the hotswap. Running the configs without 
staticthread will produce the correct messages, but only one thread will 
run. Running click with -j1 will have the same result.

Am I using the hotswap incorrectly or is this not a supported configuration?

Any pointers would be most appreciated.

Thanks
A

Re: Userlevel Hotswap with multithreading

From:
Antonie Henning
Date:
2014-10-04 @ 00:30
A simpler method to produce the same result:

click -R -j2 -p 13000 -f /dev/null &
echo "write hotconfig s1::Script(print 'A'); s2::Script(print 'B'); 
StaticThreadSched(s1 0, s2 1);" | telnet localhost 13000

Thanks in advance for any input.


On Saturday, October 4, 2014 1:59 AM, Antonie Henning 
<antoniehenning@yahoo.com> wrote:
 


Hi,

Is hotswap in userlevel supported when running multithread?

For example:
rs1::RandomSource( LENGTH 10, LIMIT 2, STOP false) -> Print(Cold-A) -> Discard;
rs2::RandomSource( LENGTH 10, LIMIT 2, STOP false) -> Print(Cold-B) -> Discard;
static1::StaticThreadSched(rs1 0, rs2 1);

click -R -j2 cold.click -p 13000 &


Produces:
Cold-A:   10 | 
Cold-A:   10 | 
Cold-B:   10 | 
Cold-B:   10 | 

Then telnet localhost 13000 and using the "write hotconfig" handler:
rs1::RandomSource( LENGTH 10, LIMIT 2, STOP true) -> Print(Hot-A) -> Discard;
rs2::RandomSource( LENGTH 10, LIMIT 2, STOP true) -> Print(Hot-B) -> Discard;
static2::StaticThreadSched(rs1 0, rs2 1);

Produces:
Click::ControlSocket/1.3
Hot-A: 10 | 
Hot-A: 10 | 
200 Write handler 'hotconfig' OK

Hot-B messages are missing. It seems the elements assigned to the second 
thread don't run after the hotswap. Running the configs without 
staticthread will produce the correct messages, but only one thread will 
run. Running click with -j1 will have the same result.

Am I using the hotswap incorrectly or is this not a supported configuration?

Any pointers would be most appreciated.

Thanks
A

Re: Userlevel Hotswap with multithreading

From:
Antonie Henning
Date:
2014-10-04 @ 23:07
Some more notes after further experimentation with hotswap and multithread.

StaticThreadSched element appears to configure successfully during the 
hotswap, but only thread 0 elements run.
BalancedThreadSched appears to configure and run successfully, initially 
for a while, after the hotswap.
I had a look and saw StaticThreadSched does its magic during configure and
noticed BalancedThreadSched worked with a run timer and on the task level:

I tried to implement a timer in StaticThreadSched to run the scheduling 
similar to BalancedThreadSched and also in the init phase. This did not 
work, StaticThreadSched needs to run in the configure phase.
Since the BalancedThreadSched element works on the task level, and was 
providing some initial hope, I tried to run the config without any 
scheduling and then use the write handler home_thread to set the thread. 
Similar to BalancedThreadSched this works initially. It is possible to 
start a new hotswap config, then use the home_thread write handler to set 
the thread. This part seems to be relatively successful. Unfortunately, 
when loading a new hotswap config and setting the home_thread after the 
hotswap, traffic stops. It seems that something is needed to make thread 
scheduling safe in a hotswap scenario.

This can be reproduced by running the following commands one by one.
click -R -j2 -p 13000 -f /dev/null &

echo "write hotconfig rs1::RandomSource(LENGTH 10, LIMIT 2000000000, STOP 
false) -> Discard;" | telnet localhost 13000

echo "write rs1.home_thread 1" | telnet localhost 13000

echo "write hotconfig rs1::RandomSource(LENGTH 10, LIMIT 2000000000, STOP 
false) -> Discard;" | telnet localhost 13000

echo "write rs1.home_thread 1" | telnet localhost 13000


To see when traffic drops off, top can be used.
top -d 3 | grep CPU



If there is a specific way to implement multithreading in a hotswap 
scenario, I would appreciate any pointers very much. Hotswap is very 
useful when working with resource heavy elements.

Thanks
A



On Saturday, October 4, 2014 2:30 AM, Antonie Henning 
<antoniehenning@yahoo.com> wrote:
 


A simpler method to produce the same result:

click -R -j2 -p 13000 -f /dev/null &
echo "write hotconfig s1::Script(print 'A'); s2::Script(print 'B'); 
StaticThreadSched(s1 0, s2 1);" | telnet localhost 13000

Thanks in advance for any input.


On Saturday, October 4, 2014 1:59 AM, Antonie Henning 
<antoniehenning@yahoo.com> wrote:
 


Hi,

Is hotswap in userlevel supported when running multithread?

For example:
rs1::RandomSource( LENGTH 10, LIMIT 2, STOP false) -> Print(Cold-A) -> Discard;
rs2::RandomSource( LENGTH 10, LIMIT 2, STOP false) -> Print(Cold-B) -> Discard;
static1::StaticThreadSched(rs1 0, rs2 1);

click -R -j2 cold.click -p 13000 &


Produces:
Cold-A:   10 | 
Cold-A:   10 | 
Cold-B:   10 | 
Cold-B:   10 | 

Then telnet localhost 13000 and using the "write hotconfig" handler:
rs1::RandomSource( LENGTH 10, LIMIT 2, STOP true) -> Print(Hot-A) -> Discard;
rs2::RandomSource( LENGTH 10, LIMIT 2, STOP true) -> Print(Hot-B) -> Discard;
static2::StaticThreadSched(rs1 0, rs2 1);

Produces:
Click::ControlSocket/1.3
Hot-A: 10 | 
Hot-A: 10 | 
200 Write handler 'hotconfig' OK

Hot-B messages are missing. It seems the elements assigned to the second 
thread don't run after the hotswap. Running the configs without 
staticthread will produce the correct messages, but only one thread will 
run. Running click with -j1 will have the same result.

Am I using the hotswap incorrectly or is this not a supported configuration?

Any pointers would be most appreciated.

Thanks
A