librelist archives

« back to archive

Pipeline/cores interaction

Pipeline/cores interaction

From:
Omid Alipourfard
Date:
2014-09-10 @ 23:58
Hello,

Can you guys recommend any literature (beside reading the code) that
describes how Click distributes pipelines across core?  I think the
straw-man approach would be to use a multiplexer, and several click routers
each running a pipeline on a different core?

Thanks,
Omid

Re: [click] Pipeline/cores interaction

From:
Tom Barbette
Date:
2014-09-11 @ 12:50
Hi,

The original SMP Click paper is there :
http://pdos.csail.mit.edu/papers/click:usenix01/usenix01.pdf.

In practice, you can use StaticThreadSched to pin some FromDevice to
different threads. Another way to do it is to pin the Queue elements with
StaticThreadSched. There is also other ThreadSched Elements available but I
have the best performances with better-thinked static scheduling. In
userlevel Click, launching Click with the "-a" parameter will force each
thread to be pinned to different cores, so you'll have a full control of
the scheduling as the operating system won't do anything (only in the last
github version of click).

More recent papers such as the paper about Snap (
http://www.cs.utah.edu/~wbsun/snap.pdf), RouteBricks, or The power of
batching in click (
https://apsys2012.kaist.ac.kr/media/papers/apsys2012-final33.pdf) talk
about multiqueue (among lot of others things) capabilities to push
multi-threading further. You can use the actual original Click and not
their old/unstable version with multiqueue using the last Netmap, by
specifying "netmap:eth2-X" in a from device where X is the number of a
queue.

Does this answer your question?

*Tom*

2014-09-11 1:58 GMT+02:00 Omid Alipourfard <ecynics@gmail.com>:

> Hello,
>
> Can you guys recommend any literature (beside reading the code) that
> describes how Click distributes pipelines across core?  I think the
> straw-man approach would be to use a multiplexer, and several click routers
> each running a pipeline on a different core?
>
> Thanks,
> Omid
>

Userlevel click multithreading basics

From:
Antonie Henning
Date:
2014-09-11 @ 23:05
Thanks Tom, I found the info very useful to some questions I've had. 
Coincidentally Beyers gave similar advice yesterday.


Since the multithreading info has been slightly elusive, at least to me, 
I'll iterate my vague understanding for the archives.

The note about pinning Queue/Unqueue elements with StaticThreadSched is 
quite important. I'll give it a vote for going into StaticThreadSched.hh. 
Elements need to use Click's Task to be staticthreadsched eligible. 
Pinning any other element has no effect and causes all threads to run on 
the same CPU core regardless of '-jN' parameters. Statically pinning the 
packet path to a thread 'generally' provide the best control and 
performance.

BalancedThreadSched offers an easier method to spread the load over 
multiple cores, but offers less control/performance. Specifying -jN too 
high can cause one CPU thread to run high while others idle. Specifying 
-jN <= CPU cores causes click to randomly freeze without any error or 
output and requires a kill. The observation may be specific to my scenario
and the click pull version is a few months old and may have been resolved.
Not sure if this is good practice but it seems BalancedThreadSched offers 
the best/stable results with -j = CPU cores + 1 or 2.

Click -a is a new option to pin threads to CPU cores: -a, --affinity Pin 
threads to CPUs (default no)
Presumably, affinity needs to be used in conjunction with 
StaticThreadSchedule pinned Task elements. Affinity and 
balancedthreadsched seem to contradict each other and result is unknown. 
Result with affinity with no balance/static thread schedule elements is 
unknown. 

Click supports netmap when compiled with 
--with-netmap=pathto/netmap/netmap/sys/ and the netmap_lin.ko kernel 
module successfully load with 'insmod netmap_lin.ko'

Netmap can be used from click e.g. FromDevice(netmap:eth2, METHOD NETMAP) 
and ToDevice(netmap:eth2, METHOD NETMAP)
Netmap suffixes
^               bind the host (sw) ring pair
*               bind host and NIC ring pairs (transparent)
-NN             bind individual NIC ring pair
{NN             bind master side of pipe NN
}NN             bind slave side of pipe NN

Statically pinning a ring to a cpu thread offers the best/ultimate throughput.

netmap_user.h comments offers some documentation.

Userlevel click multithreading is enabled when compiling click with 
--enable-user-multithread



On Thursday, September 11, 2014 2:51 PM, Tom Barbette 
<tom.barbette@ulg.ac.be> wrote:
 


Hi,

The original SMP Click paper is there : 
http://pdos.csail.mit.edu/papers/click:usenix01/usenix01.pdf.

In practice, you can use StaticThreadSched to pin some FromDevice to 
different threads. Another way to do it is to pin the Queue elements with 
StaticThreadSched. There is also other ThreadSched Elements available but 
I have the best performances with better-thinked static scheduling. In 
userlevel Click, launching Click with the "-a" parameter will force each 
thread to be pinned to different cores, so you'll have a full control of 
the scheduling as the operating system won't do anything (only in the last
github version of click).

More recent papers such as the paper about Snap 
(http://www.cs.utah.edu/~wbsun/snap.pdf), RouteBricks, or The power of 
batchin g in click 
(https://apsys2012.kaist.ac.kr/media/papers/apsys2012-final33.pdf) talk 
about multiqueue (among lot of others things) capabilities to push 
multi-threading further. You can use the actual original Click and not 
their old/unstable version with multiqueue using the last Netmap, by 
specifying "netmap:eth2-X" in a from device where X is the number of a 
queue.

Does this answer your question?


Tom

2014-09-11 1:58 GMT+02:00 Omid Alipourfard <ecynics@gmail.com>:

Hello,
>
>Can you guys recommend any literature (beside reading the code) that 
describes how Click distributes pipelines across core?  I think the 
straw-man approach would be to use a multiplexer, and several click 
routers each running a pipeline on a different core?
>
>
>Thanks,
>Omid