From 14b9294b89e85a609d04a3b838682b31ee70701e Mon Sep 17 00:00:00 2001 From: Thomas Schoebel-Theuer Date: Mon, 29 Jun 2015 07:57:09 +0200 Subject: [PATCH] doc: describe socket bundling --- docu/images/socket-bundling-long-summary.png | Bin 0 -> 4378 bytes docu/images/socket-bundling-short-summary.png | Bin 0 -> 3975 bytes docu/mars-manual.lyx | 250 +++++++++++++++++- 3 files changed, 246 insertions(+), 4 deletions(-) create mode 100644 docu/images/socket-bundling-long-summary.png create mode 100644 docu/images/socket-bundling-short-summary.png diff --git a/docu/images/socket-bundling-long-summary.png b/docu/images/socket-bundling-long-summary.png new file mode 100644 index 0000000000000000000000000000000000000000..d48f094c05fe95051e873c3109bb4854e0396cd4 GIT binary patch literal 4378 zcmcgv2~-nlwoX_i4cZVzw6TFGjSC`q2nre?L0M&yeGvs3S!7>i5d>0_Zc!s7BZwlP zNP`eo6#+?DwGjkKR04=0pdz9P1VTEpQ?C;3Gx*;0c{AtCyqro>sr>hT_kQ2~zdCi% z-p*p3tg}{OQKoEqk6ciMQzFJ{00BnVU zFdnZJ2kY#FnFPHBTTw6o*9w6D;W^4O_hU0XGe2M)j>G?*Xs ziQh)trcHDc|0e$MLnKrR4AzrI=fXrKf-w3EP%#t4+fM@kXa-vVqCaIPPJg)ER3Gy2 zyqr)tQ~;pBVi+%#mj&x(7qH3pIF}GKpa=sL6#*P{0Dw;`-~$4p`f*?ZbohV;kzi{p z2=xQD_SD~;2B!<^dfLFYp4PibAg%yJH8(dugOjXbF!@?5QxoSid6R5ncCJCa+0S|_ zf)z&6LB4d-|&9rvqdc@_D?;uY8$IJ52JZbXu$XrH{D!20Wj~?8B zkZvM=qN%!N9q%Zfuf)&Y*?XrZ3r%JC4jNfgRziisYYQ$l*WiO2dDz|BXP!27FD)l~ zhPQZ4wfK$)t}M3Ll<)5w)vF)*RLv^R=H|1U>24vrh_UQruBvABpPS)LFKZ%HZ zzhXN0DYQFW_!tq5kF6oPRG+RLZmPAxR99)qfqw&DwI3qXOE`Y^-HZOi2^aSoeUK9q zxu?9tF(SdEz9Ni!q9R?*o&WKe|BIe%h9SPUsw4HY$-<0%qERaId`H&Q7VZI+b6_2$szU$gDNdNckhe6 zdkj}fIU1re1wVfX2dt4uMV>lZQZ*%v4W(=cL-@Y2BPi6|rKazlKYMA>_6(cjB6soh z?=POkQ+sSV*0HnH`w!#2G~9=hX;cfpSk`jzcks62olbpC<17pvZ5%Q2(5rA|U^Hqa zBJr)F!tqO*sNQpc;hHN)?6;HwOJN>L+&Kd9#<4>3eq(!NU!j&KdLs*XMyCmf-65#Y z=HwOz_OymV|_!ffzAG`JR6BP+hBKS!N)yU7 zIpe0aBczI!9}&&DHTbFLr&b0VuLEyvyFOoe)T7I=mM!tmZQ}bHSMn7r7`M^dn4#}= za)rkFG0Cp&dgylvpSgW*cdO~~*c+~ajU_=_$ebB+e%q%N)8u=d)auE~c(0_Wmv@V;LJpdm>c(dO)-etr)YTejUfU_i36?<{qqvsY(Kw(nY*&+-|v& zvPmyoHf_7mKaI4}MXlsU+4mkwCuRm$1mO!k+Wp&5JBEvqgqj0$Cnj7bkK@jq&0`Jh z@%Ad1JLS?u#bh)duVdb3u`!GMNf4F){1#gVVn9IOLS8mrkpg+k#Z;c#UH;yF$0O1x~$H;i<+{MQ*G z8-S%L&&05uj{hP|+oB3BR23^ox&RdHh=`wX3@2V49}T>D6shpXdv?aMrTKFGQA0I# z)W@fRGp|-|Dhy7 zT53z)&WXY&E=aO=@bQiCPyh7k)qoD|4oowEusm{1g{Gx#u{3mAQzG@|1Ig$l1X2NR zbrFbh7%ekcBTA`5>=WTrPqxfP#ln%~EIpFR@JFGUaPd*5Vc=HOM=LIht#bibN=%uvp(0IjDf$^E)Y2#ysUo7XTYG|tfUEuO zmv$u3$8NyCb5>VWYiOb>-`;Fe`l3|_u}}GWd!e}`{I=-5vdJ`DND2Py6OKIQbMivG z7vxwn4)JR{y9KCO+5uM&64tz*vznbkRA|M@ z3KQV-_nhjh59_4blS+98sd^iJLNO1QrCo0?o4wyGXteh6r#O{bAUaR_==4|{`U?(Vb>mY=+Ozn8ms-CD*TKx{9 zY9J08b<;FZ%*UlYzGk9!sh+r%q;_XoyAGp4j7{a5BvM+Q*8K|D%NKszKJBIz_J%zC zwyIf}(rz&mP1PU5^b9^8xY{aWX;wwtqc4X}b2nGugiyZnl}|EJu2vJ~9T6)72}Ruz z_*MHA)3+;fy#l>=NJh>Vc!A8FxGh5xj^156MZ*B;7`7<{N!HJBYu+XvhH>flC3ayD zcLj-gCl#P-?2R62WfNjJKld3~kO!)-e<)UUX3wLRQ4)uYE_~(J=y&aYecKjwaRy`g z^)4P9MGi9Ho_GK_^l_8RDCU48#&R`lf2wd+`gb;)f~M>zZWM=wt6QGj&>XfbeQwo+ zVAgyYBIK75kq{#U*SP=n6n$Ot&G1pVJ7hhpRcG_8#8jP4@hO`-zl&4;CzD@dI8_~} z=HJK*!GGrGDUqXd+Yj~&r9~PuPLf;Fwx%sK|8E5sAqB60qhLORVBla0?T!)MPj-rq z-ZqntnX5;hrdmW(BCdiuKD3b0Ad?mtpX#@B6GX zp>8OuKPz#YuR~h%nI{;8#-yO82cjn98qR*%FH^n;GNG+r`&EU*KN3P&s^Rk*Vx?4l zl|l&O<@+MU%W2Z@=l0mXzIpmPk+DE~<3})@TsQ6~u^TdgFsUzsp8Vnfx$T!$ZK!|3 zB>s`S|1ivZD@w9uF=qMPLg?%v$N--*CDxDK5n0bpeX%Pwh@Gd)R%}Y)97V{lqMbyZ zl$9&Ct0Bme{+%I4=D2WgF9Y-*k*;Y7%LZ)r*oLjSXLz)u18Sv=IJDm_=^{h5?sqw4 zTf#TIWu^E;(=>Ifmg*=kODsQXW&SRgFtANvMDre|toic`gB z_30wN5r%}S_gCu{-J2jv_Zpoi1>%6LBt>%21_hX$l+>9vBV{vVz=*dPFasAqR{++6 z$GbWsKD4~%MA)UQV1#ZPgUVVARhAx;bdIPr^m#9DW}J*r{}8gpJL28B^azS)AiBe) zUNicUs|>}n8-IfAfWKT4g>i_cmGa}lk|?sCVOA9FBts=$_QL!^Ei=}P?t)I)c&oxo z&yd#tH_zpE5?`-N04~Pa5oa5NJ@16&%J-GuV&!mR-R+#NJ*@QFO?Qq+IeYwF#~j=B zmU(SDhFdMi4-Y zcW&-?F}6OTmg#k>e3W_4=zVvkae9*i+B=^&I%2q1b*L4ptp6la!+iPGbVyjNp9v4# z98u0FeAZ!(*{V&t79r2DSCi$pIAYN+*4n$l8=kvbl&Ph!VQ~z24&n3&Z2i?63)?sX zgZ0nl58K!-GnjvWL<8trRlv^P2TiJv=DnAEk5G(DG*GQqqFXl8hE%&I%VRsaUuQh?YBMcnAO`k)8E+T8Xe2!c=~BO^m}= zZwiHyl9EzVQo`kOo12?QM@OL+vxg2b1a;qn~MQWaq0%MZu^j6x9g2s8n|@)g7dY_#1&4(B%$h{0efp zAT$riZ7ENm1!ptLTAqW?TewfcK%)#0sHv%G-oiKogGv5uYK*hJ&idRRpHy6ihPxMs z2*6aU)rU-24Puw!E_UwY;3;lW%4?bBozYt$MZSGPm>W+yGijV2u+Y3XOjx?oKZwgT zaeD1$zRp>tE8ZDk-ryZDb=$$>apd4m;ldvj_J-Wi13y_u}hU@c1XXj!VDkb}zM{51q9aeCX#{7^^L^1(k zaU#D$*&w%JCdGe&HCl4*U^t$gv!pBcen!D}>axyKck)L&w!S(==K776`H%bJTPrVo zKaX3i7}tJsR&}N|J>5x?iJd;=GYmPd8|rIvSVJa9kbN_HZjL4y}Idj zmm4Aci(^Tj7N>=s-Qq@2pZ5@=Z6E&RK58(0Wm>p{b3rLQl`LPTlIGb+`aB;Vw*LHr zwA^G5-hCg|D$NTY$;}~EMXNE7LUCAfpWCJw(Zq3GOA&ISsl zU#GG107{PeiKN+RT3c*sEkMLXuj=PIM|;9ek;l)Cb>4OBHjJ;ni!81Xgt-=#Vos+X z%2n%7L-;l56m%E(x;gUlD}#QjkZStZ2zGw&p z2C)qlRXwIm;@tE0dNefru!E4AI1#R-mcGKp*hziqwP7yF1^hEralKFJZrEk_xjt^~ ztrwrOd6U1XDa$9A>}w=;b;)*k#2*CI|M^shRmzMvvwbY~JTkG)nJ>hlNd#d`d3 zNbt6>V9u&GG5BTZPf76CmOnT-9Wj_`u#!n(Jbn7=p0}nX+Is+W^DK%B&EkD7lE-T& zlnN{4@j8g;af1cgjt(;~1tAG%ziWP37M`Nsr(C#WVC_kj_vxO`T2-{Q1P<_`w|kV* z`1I8rEpOi)hO3MxM>#aYp%z#$%b6pSdDFbkD%!{iPjbjdrbU%Kb3WUY-23rUL~;2! zi5{vsk13yvH8ig!-LylEynL(`c!eH6Fl5)+_Zd&uDn7P#-^CWZcAPpw=S=piuDXl>ag-<8qY z_Q4~lG$SJ%knv6XZDL5VE&o=SWzU=qoNvSU{%P9Z6lyszDK%F+Y3=ZT3vgQT9r1GL z{&4{Q(X2{5*o@KBe!c3}xyw>&2#4PDOozksg-J({$CVSRU*c67DD^~Cl;oYrmEmXK z`AU&*(2NELUH8jPuHERDn`}+?n)7#s-~aKdBuW)=?_YcMHx#4brfk{MxQVm2R9jX; zg|Z25!q}PF>hKTb@nqCCu`(Rst|_zagRJL3U#E1G#&3PVG1W{UNJYRMm#~qlEVYSX zIQ$Id>TI9&IEjkV=%K?SG61mPm|^XFCM2`F?o7~5UcZPo1cwxhm^dBMdwKa9{Qj9Q z1RSLNlkkk$(+RY;n-`V(br9HqVS~voU`?lQ(-1vLsKApqEGp-W%7}FVQconrwuBb_ zKf@v-;UvMvy)k>KN)*Mgp=uateSE}Iu`LG(A}*{00kyYd>FF4R%h7y**LVsy!5vH4 zJ4eEF0YXi~G~HjAwArnaGh_E$z4Aveh(oQI_(dRx4el5bbFmjXU=>%S1H7^w@9I^7qF@Q$Ldk z#6Vg{RXa+(<5aHRmxsKv)D~@{`==>^D`BpqjFCkt+YM>`+I2rdel@Nxe>3NrsLkM@ z56a*C5fDUJP%kgTEqOs&0+iEO~9_r3iz|VsrEga*k?DPY0eNbYUCT_W`ars#N(Gw=lmI?w(>-2lgm5^ zxzIXoxGLbUBrnY_s8SVkI%Jx+l6M@rOH(Jlw_@6=5xaF{ zffcKiKZGgx>Hqe7l&>_Y_~isnM<(smpR@w6euYNv?D{=O9NFr02|a3iTO`3IPo4VO zD@B(0OXxrQG>T7%*oSTF`zD(!!3B!cX~!r{=Bn$&A1#yp)X5_H!*9x_o!V0iIpn$2 z`4qpOrr1;|4V9)QZGz|7I3G-v@$-u`ftyBZH>C4&Y)lI2*}y79bdSEQ znn8eH7Nz2y)-S_)4fU6T7s!?Gv>GP`kSmW`#Pn{F8wnj4Uy;I{&;6M3f3@U1n1!?+ zz4QV4%5G^+{8*Rfrb;1h5v^D`&y;hQv+U$~(hrD<+9DGgbv94eyl)!(omYPI+ijTi z%)XIp`GD-n@`wC^#fN;zq@XnPx(gef4`d#r+cWV!nN>0WEw>sIoPwgOUJJxMU{^Su zmNo9DI`cg<0I?w3VEx(&-VQ}ubr*ylT~s*WVXK1sFRc#X;sV11bE7) z29|PX{PklDkA?2&ojW*jnDL8aWK4~RP`wL?q{Y+_h2cnol8aE{!HhL?+|{*;baFs* zMF#F@#U&p+#nZ4YQllKAn!HUl&*n^k0_>4-VcL940Y; z_p5d-WP+(s9eIdjuXq8MaV8phXg^X6h1&Mvy1mk*cpj-nr{f3FUs`l|FcCzrMF&*P`?d}%PwBzeV{*veqL9?tTx5h(GS+;~KO511~0FW|OL z>D$FjwFKtynec?=_R%KCjkXsR8=>mMUs`K+8*G$Y20i+59Y0%^R5_+Oa<$tpX+@v@ zIX$d7;dWN{A<5yR0__=>2W~sHGfQY%rEj$#l!OY3-gd`)4PZ2>;)Bld9UeLo_?aMH z>tC-1J}Q=!(=VRr@2-39Ms_q?CfyyjFyv)7Z7si*?nB6Z)xs`0$M(7JJ)T*|Fe{9W z3vk4IusnL~f&#g@1W}myD#$8$JJwgVE(18AtJyI0IScyDZJO6*)RbY1Loa2CUa4!R Yz>)xplzBqjkD{igCoPQ&j=R(T2N=Dd4*&oF literal 0 HcmV?d00001 diff --git a/docu/mars-manual.lyx b/docu/mars-manual.lyx index 04ee9a75..65b83843 100644 --- a/docu/mars-manual.lyx +++ b/docu/mars-manual.lyx @@ -28540,7 +28540,7 @@ name "sec:The-/proc/sys/mars/-Expert" \end_layout \begin_layout Standard -In general, you shouldn't need to deal with any tweaks in +In many case, you will not need to deal with tweaks in \family typewriter /proc/sys/mars/ \family default @@ -28549,8 +28549,26 @@ In general, you shouldn't need to deal with any tweaks in \family typewriter mars.ko \family default - kernel module at runtime. - Thus it is + kernel module at +\emph on +runtime +\emph default +. + This means, the values will be reset to default at +\family typewriter +rmmod mars +\family default + or at reboot. + If you need some persistence, implement it by yourself, e.g. + at startup scripts. +\end_layout + +\begin_layout Standard + +\family typewriter +/proc/sys/mars/ +\family default + is \emph on not \emph default @@ -28564,6 +28582,223 @@ This section describes only those tweaks intended for sysadmins, not those for developers / very deep internals. \end_layout +\begin_layout Subsection +Tuning Network Performance +\end_layout + +\begin_layout Standard +Starting with MARS Light series 0.2, a new feature called +\begin_inset Quotes eld +\end_inset + +socket bundling +\begin_inset Quotes erd +\end_inset + + is available. +\end_layout + +\begin_layout Standard +It is mostly intendend for lines showing high packet loss. + By using multiple TCP sockets in parallel for emulating a single logical + connection, throughput can be significantly increased. +\end_layout + +\begin_layout Standard +Example for setting the socket parallelism to 4: +\end_layout + +\begin_layout Itemize + +\family typewriter +echo 4 > /proc/sys/mars/parallel_connections +\end_layout + +\begin_layout Standard +The following graphics shows the throughput of a non-fast +\begin_inset Foot +status open + +\begin_layout Plain Layout +The fast fullsync algorithm would not saturate the +\family typewriter +eth0 +\family default + link with traffic from a single resource. +\end_layout + +\end_inset + + fullsync of a +\emph on +single +\emph default + 100GiB resource over a loaded long-distance line between Europe/Germany + and USA/Midwest. + In order to compensate highly varying load at the line, all the experiments + were repeated more than 10 times and averaged. + Each bar shows the throughput for a particular socket parallelism. +\end_layout + +\begin_layout Standard +\noindent +\align center +\begin_inset Graphics + filename images/socket-bundling-long-summary.png + width 70col% + +\end_inset + + +\end_layout + +\begin_layout Standard +\noindent +Notice that the uplinks of the two servers are only 1 GBit/s respectively. + When the uplink is saturated, about 100 MByte/s is the maximum possible + peak throughput in theory. + You can easily recognize that the peak throughput is almost reached with + a parallelism degree of 2, but using even more sockets appears to be slightly + counter-productive. + One of the reasons is that more sockets will increase contention on the + line, and thus increasing packet loss. + Another potential reason is that higher parallelism at sockets will lead + to higher parallelism in disk reads, in turn leading to more permutations + of disk read positions (more +\emph on +random +\emph default + reads instead of purely sequential reads), which is counter-productive + for disk readahead strategies. +\end_layout + +\begin_layout Standard +The next graphics shows the same, but over a medium distance of about 50km. + This line is even more heavily loaded with respect to the number of TCP + connections running in parallel (probly some 10,000 or even 100,000 if + not more), and there is some kind of +\begin_inset Quotes eld +\end_inset + +traffic shaping +\begin_inset Quotes erd +\end_inset + + at some intermediate network gear which will +\begin_inset Quotes eld +\end_inset + +punish +\begin_inset Quotes erd +\end_inset + + those traffic sources disproportionally increasing overall packet loss. + This can explain the even higher counter-productive effect of using too + much sockets and thus injecting additional packet loss: +\end_layout + +\begin_layout Standard +\noindent +\align center +\begin_inset Graphics + filename images/socket-bundling-short-summary.png + width 70col% + +\end_inset + + +\end_layout + +\begin_layout Standard +\noindent +\begin_inset Graphics + filename images/lightbulb_brightlit_benj_.png + lyxscale 12 + scale 7 + +\end_inset + +In general, the optimum value for +\family typewriter +/proc/sys/mars/parallel_connections +\family default + may depend on many runtime factors such as other load running over some + (parts of) physical equipment. + You will need to determine optimum values yourself. +\end_layout + +\begin_layout Standard +\noindent +\begin_inset Graphics + filename images/MatieresCorrosives.png + lyxscale 50 + scale 17 + +\end_inset + + Notice that socket bundling is conceptually the +\begin_inset Quotes eld +\end_inset + +opposite +\begin_inset Quotes erd +\end_inset + + of traffic shaping. + You are trying to get +\emph on +more +\emph default + bandwidth, at the cost of +\emph on +other +\emph default + traffic competing for the same network resources. +\end_layout + +\begin_layout Standard +\noindent +\begin_inset Graphics + filename images/MatieresCorrosives.png + lyxscale 50 + scale 17 + +\end_inset + + If you are operating masses of servers, don't set the MARS socket parallelism + +\series bold +too high +\series default +everywhere. + You might +\begin_inset Quotes eld +\end_inset + +steal +\begin_inset Quotes erd +\end_inset + + too much bandwidth from other applications when starting masses of syncs + in parallel, e.g. + after an incident. + Best practice is to start with a default value of 1, and to increase it + only +\emph on +on demand +\emph default +, and/or preferably +\emph on +only +\emph default + at those servers where high load really occurs or where some urgent actions + need a +\emph on +temporary +\emph default + boost. +\end_layout + \begin_layout Subsection Syslogging \end_layout @@ -28608,7 +28843,14 @@ Logging to Files \end_layout \begin_layout Standard -These classes are used to produce status files +This feature will likely disappear when MARS goes to kernel upstream. + It was mostly intended for debugging during early beta phases and is no + longer needed for stable operation. + Developers may use it for spotting potential problems. +\end_layout + +\begin_layout Standard +The classes may be used to produce status files \family typewriter $class.*.status \family default