From ca30de0bde294bf60f22ce51b366e0e5a291c112 Mon Sep 17 00:00:00 2001 From: gmarek Date: Tue, 11 Aug 2015 16:58:24 +0200 Subject: [PATCH] Initial kubemark proposal --- docs/proposals/Kubemark_architecture.png | Bin 0 -> 30417 bytes docs/proposals/kubemark.md | 190 +++++++++++++++++++++++ 2 files changed, 190 insertions(+) create mode 100644 docs/proposals/Kubemark_architecture.png create mode 100644 docs/proposals/kubemark.md diff --git a/docs/proposals/Kubemark_architecture.png b/docs/proposals/Kubemark_architecture.png new file mode 100644 index 0000000000000000000000000000000000000000..479ad8b11f490d49fe977e28a253e6f1a4c4b263 GIT binary patch literal 30417 zcmeFZRX~(o_clIsm%z}i0>V&AH;AB;V!+TnG)VUV5;9UEEh&N`%@9hf2$G}ZfP^3n z-JRb(@I3GHd*Ao>Klq=02j7VY<9+XY@3q%n>sr^kHqUjn)hS4sNkJeG#a)ft4?rM% z2nd9ePD}{=Qo7#V4+43o-My{+(06R5i7dg`=3|TO%V%Gox!te71{YPSC%8dP9QIr` z(Oe3ji@WpD%hUMG106cM^p}#FzPt_nX z%<*CL+M@>;gQ60d!S`PChKoK2wl-cm{hIxnKKL#|!nGTs?ykrKTj`Dxbl-w>i@uAOzhMnIJCXed% zz~u*}HN&dYA3bad>kU?%?_Rw09v4}EzP4_SK0l2ahAwl7Osl~D+JBUaV`&8V(i2D^ z$YCfM zAxPcJv0-^N!_fbKA02XqZSEAxZtVhZ?)M`4(pXS4lce7@R@20BkkPMfv#)G50IP1>|0R8s%{2ejgW~&`yIVEv z{_9oULgzWt`F^<(;-bjmu?$9F)QorG>-uV?LHSpa)PI}cp^?`{&9HYLc15L4!Z+T& z0<7bV(Zm{T>^L7Vi|hf^QPxg{tR)lbxEiDfSPd|&tlfXT(j%dTwZtSt@`HQWxBu&b z|7A>IL;rrh)}s(Rhs!Up1-ib3j!V-*iT^bXV4ueN(y@D5v#$p102!5aR+kS6B@3zgPDee0pzI&5x_6Sc1pI!yw z9RaRX&>=ZAaV*`PmR;oVH+yuCKd&7{)Wf45*XgP`)e`JCG)I~4kAkZrh#wO=9G6l; zIwK9A{=ig8n)zw!av@;>u-^%?r8Rra#egHje$b2PC~ncBUsx3;3vF-J5^b?Ofs+A4 zDv>#3?qT<0?6~;Bz4sO*kj|=wcVf4#=eEeI@p#uW73YLi2$qc8*33*B`KBHhcnaM3?tW{fWW@cUS&uwNJnN;-U*59eM z1=zio;5A#DUk@%dGKo1L_l?~*SbMEj^k?MRX+e#!R7d@pd*4am+Vh63W!es-J?W~` zp%0I}g9usm8$jH$Pq0UHc=8G0WHq8l&dya}UHIn8rO&yMxr78v%#C~BY2*5Y6xMg- zjKW*?r(9K@fOIMX50L!mG|`56oWny*phSpblM9?Qj;FR=42Oug!Y?bRGTtDL#etQF zI(+0OGg}8ofc4O}Ez62szx+2$hs1FK)1gTE#JAYq;nRbo6QAMuZHq%br7oS6_#ijU zlRvJtskoGpo)ubs^5!_>wmEc?vHj*V&*Zm9>V-~v#q9!&w;jYGGp})2oszbn&zk2f zid<1JtNs3z6>r(CL~ul(=CSOPTKj)RpDR%ELli#l&I~mTF0?Pc=tK=hgJx>?h-aCY za4hdVKigdTu-%dq=;>USpGhvsl~5)wch!^!kE{1-v=u+$2L6n-8wl(`52n~iRLOwj z;_q1teOCgA2FM`npYM9R`qlReHR= z3|Qqrz+yWoi~zTDapSirg%f}7c7T4RJuZ@*NK_@^R40EQy=#zNbkcZd7{~l~C52*F zHHicL3_ff_a$F!Yj%=YdA@LSmI4lv03gcq8L2dRlhe>_6DVLko zC8_V_1BqLLkvQPdAQs_2U2Uut*j$Sx7R=I2Aco8x~&0F!iabtcu#Y?5`c+`5>WL`&Jr1$+tZ(mh~Pa#pMs-H`)6H~og@>GjOaIxfmH&^pdwM+}sc;+!=)NAOxUKbb4_K51k9XNW z4(y-`YpZ};qPj{LUsD;!;JnMW3K6uW{A4&X6h8ay<$JUxalUec&kD%lU>&E zS|b?KCng_6m`Kv*pZNX3NyD&oY8FSwSIm`;HgWB0>lI`E3ox|);U2#Yk(nbliHO2c zxaJ*~$(1p3d@(>^f4JGVw(j8tP5|JQ3gTs0Jr>6O;vg2nW~Dp;KoLXk4P*E%$RmCN zV6MKfEM?#3U(b^{S_0?``@=;FXGItO<;jnMbcoIIi6{WU0U@}BMeC2<6r}Gl6j2&0~ zg;wI8fwA%T;uY*FvUsjx{bM!AkpJz(Ye4Ot7zy zOJDg5!T_JaF3&seGFDv9{{M4109Xw$0}OJ3z0_66u!9De{l{{s3Z_Er9BX$m`_>#M zyVuI2E^P+Pz19PZ@W^Wg&Q)dodeTg)nL0GPgBy>QT$GPrh4ap__gJxZ1~s2X@osIK z006_=e;vrjkg4?(zv?ZU+H(i1US|_Obn4bz_N1r!nMujM?Dt)hfNn{`9p-O-TP+V9 zKVp$k5RpUHIsgGyp_COjsXKb?KJNuU%OpqsWA17QWyI-zW&%<4dto3ipLk80eLVTh z6#=f+$w&;S$Rofxd6E*$6>!kZvEs?r)rvQ=-1_}^zD2S2xB8*ijGa!0TVKBZ5@{b1 z3pDU}>8(7w$4w=~r15|54TKOv{IGj6V87IEODXVACk~uSs@t3_%3?aJ->svk23nD66F8m+TQh2hRSmH$<2CGxJPRF_|*792;{dP%V6^i5{s3 zFZxvlgED}`*s7|Y%*73qOPL{On<1$x2r0#$N;eY&+oQys&dGq|k1n~t53J%>K2f?u zMqL6W+)c?J-D`1dpDp$-#o@B;nMulIVX|R& zar=p-NahWqm}zOG@E8r^K%!yRVLHPCwptr9sPH*BYkPkz?_^2pa0QUU4;sG*NAu3$ z7vpE)e@>*(HIB@|5yJw-3{==D%}!x?rSqO%ASnkXarogoum-KK`gQ#2!m*76e<11J zcGH>}!_xzi;Y1zbLh@5I+@p!7@Dw=8%xIu)+BhG6IsGF>12W14UoFAVsn3>`XPBw{l>oqzkCuh_pP0DkUZ|E zTWRw(8tuZi*0)92P@JclGmG-dPDdZ<2Tt$y$N%QDLE$QWPCBky$3au#udeLB7?@wd z*J6C|l5pJy07=(uI_TZP^7z_S@>um{Rtlh;S+-V##rAr_NFW7O-xwYvhEwO}3ZwF! zqKPYTVQu0B~pNA}jD&{8mtnIV!!5XQ`;b%&SnN9a> z;||6xUAg{AB}5e$cVM>rnavl~pN@T)9`SfhMEy6M5N@)9B9+%_ocfKmO&^CBy<+W+ zzSo*(4cZsRtv>rZtzG5@Ehz9YrFR67{KDHZ+r(%Z(Lf%^lEzU8sb;6K2AON&aS43_ zIHx#?>qyQy`3sBa{9Y0^;;-sueW7AIgb1?!!pN1vH~lEdv9{p$OAO`X)8Y0 ztO+BdTdblrtgqkRQ)WVf4qgRiFN&8D$S$SJ7wDn5hd<0!F<<}pRECg%gW*weU7B=y zoiB51of<^DiCW>UX)B1%Pa6A7SJ>`a zbbIFDw<*O|jZwGK#DwDBCocO=%|{6_*HD4LXlghe2tGKaw#ubIjwi)&p|6Hkqitqo z)b}*d$bwinIMJjH^HsWtiBApIX4ktb3@*nvEf|rCB)}nyIVKWwUt28U!vprFmTW=#RLU7PyJ>)HT8}$JXAYxQm|h=7h>OYxsyW zNV3s(%u3r|6${bUQ^1i>s*!s3c0r2QLG~5FVp9Ydf2lDQOc9F~pJB$=eQvL)4{P%) za*HHSPP2H^g5u8Fj2IcfeGeQngRfrNMRWur^wow3PHxaI&phS=X9C1fC~n(l{K2X9 zDegV;6c#_QNXWrJh0G;I?waK|w656ir36a99esfSZgZ~M?bJ4#tYgC<;%8=Na1jKJ}z0fkYwVoK#>9SzIKyn=KQT+H_r6dYh z_j5px(iNT4?_$5jbBQ~csd-49zIjn@(4qxTdS!4_V%!RaQQVQybU0cx(61F#a1zYd z7h-`ftSp!m$~Ro$IIV3Gp}J}P5&e99&LX)4H5xHfZSSZ)Z;s!&JjQA`k3|T5deVN# zgCl=_qSxe2)^-w=AHP6*Kjf}!AnZk#MfcxVrOW!__SberKH9jI*;-KoFwE3Wc(laV zznk(6C-IH*>TFB9WJg!Cw*_Y;Qv=`RTc@S#aHEaP0kKH2$ED8N8j7(Q)JhH6r=tm% z>G#tK=NdV_kP*iXv>DMpx{K5~bsyFenTn*OfIrNj};L>3TW;<1| zn=(VJhRk$hLVFXY7V-^UQkCLN$ml!4pB0X6sGNco#?M;L3cb1mdOt-~ zSI&N;7)82Y08qrd_!kw8?Ra|~zv4Bo9~VqyF+m=IAvj<~RKfMnde9&KaFdD0lgQm6 zAvYR*{ILti8Ji-vi*^#I7DQeUo(>i;nt^iBt6w>v)H!}5nA)tG3A!A9@76BkfpN?( zKk}0P1Ynk>g8veTaeB#nw@qImJEVUg%=vFDd^y-t7!AeAqp6SW(~s*)L!Sorvm6c% z{f+D|NYrqANb|n156iEmTJh?bTQcvE7nt>LVio}Sc@fSTmvrAn@xb`YAur-@?0-S) z0$CeibpQ28Rk)%jz@n;WcS$%0)mV(r8uZKUbK|)~kQM*48bCB-Ex7HBCj`6?%fqJr z3zpc1sKq{)V4M#4E#iXXxL66>iE>18~;+RY+tzU`^G*cn_&&gyE zUT5{|aX<7ASd-^sX*p&;T<7h}PHj4v55aVUz&9l0ekD=L)jI}pc0?!gcx=n8SI5!? zmNXx?Ijz*Y9|jz=iAbM+obK_!F2>@rr2Pxh3{g~o_!nj|n$$L0dnK)@^heHyWHq04L?oNlQ6r@7}0G0W331VHvPPiBD3%S#u91W1_-5N#1f&VhYFE}2l{mab+6#`q_-{O-9LuFiO ziQ&H6$WtW>Q@r4>j^!sK$A?bbJqoxRzUJH8CQM9?PkOchy8S=g`IZ0f1l2q|w1>D* z%EWi?@X(FY)W~c6Q`ubb)`&Gb{Gv9*Cm&7S>&pa zT|>es6`L00P7G>>V|RhNg;$Om+g4%bgDG2Rh8s@762XvW`$&CxBnXz*O!8Z3>PGn>Q$6hri+Wl&oR;fz@g6MmvoS^;QmMMTU z@4OfQyJleFKx1a;9*VoSHu?owWQ^`?*vxBaT}Js)9|Wx^)WR(boQ+S-BiLH^bajGZ z<$^{4m9M^274YS&U*a|MM|~_h4ApU4UX4u3jCW5=2Jl zaqB32l5pLraj^+GKbkxA@#8fAJ2XOoMchL4ZSyA&Yt2&OGvHAH9hAR=GwSp4Q^bT4 zLzd`B(PJo89;Pqd-kcMO0!SBZti^7xN`RRJA8L%SbMLOB^Y~?>C;~j0NA#$)qTewE?#hkJLMpO7qHc;SzddFxZ_Mfd5uejdK@bk zPz?RYTo(k8(R9AqH?|vwKZ$0GJEnZO-zY3;r@PUYsbhonNQEM5`w8-3ptpnu7z0NZ zP5>u*$Gt*XrYT`5EN3n9^=@Z&x<;T3M>7@mic;LGi{;fGy=m0sV6z8wJ*sf76gxn4 zA6=3+z;afR<>3I%vv5Vr)&eBZbPJ%K1Lu6&JvY8 z2Iz`W5Xqw&@Vgrq_Qq~35~^Eq`h=wc-d?054H-Wg%c)vDUv`DoHI9@7NU_Me7@tj? zb^W}2(d9<2|1i@we(}XI!{$@eXBL=FcF@lkZ0!hbGwa+B<9+dGGjE-~b|xPByv`$# zP*QnRno06bsCmKT(q(`(W8aoUL2+=##~s(3su|8U>|~QZLF8TAm=5<}Ly0e6YRjx` z9wQK1GYu&HXLq@f8-!5XZ-dO#3*P(b7}LQG!jV?*3HRCN^;)J_*MQe*{{BScZ|DwAlFT`f2xsH#-oN z2sPdn?{a3&ybkQ=F!o9M_c`1^0+Eu1bDXzyye_E98S9hew=QT(a#CO{M#fPE`^&_9 zG4`di0tqk&aH!RP8%u<5L5%B5-z4FheWj0@Gc1uD*h3>F-q%5l-g-dn@Uj&@+yh zQ}folHFJ-Ry;Mh@+IC(T1MxT%>^NMq=+J-!QbcDWPo#J~;Ng5o&fMv{n5KBSWX0qU zyRTmI`U^&3k(G~s&M7tP#xY)IPJilbO?GWZ6*&PvyS`^9`p!|XoU*k^iq&Bim#C51 zv3@gq_uDU;2EkZNGvuQ<>dcn?`P-p8drh|K<8dG3@T${08+0QLPC;k3EXEwc@w5E0 z9U?huf*sPSO&mE4Bh1yOwv?o}4_U=_bIb134sQ_>_KqtgK5zBcyk>YZxd~MC)2E?5 z^^APPGJ_~w;%6TqKBIaa)>a*{*p@XGIY9^@mO=l7mtlaX=SISmVVk{~F01}OnXcpI z5npvV`H_VN4#Vj4!OgF?!V*S1q#1;JWxE!i%IRr^yM)n31{ZX@vAR#52(BUEa=31B zmGx_0ijB@iG938#2alrX*8CRi4g>0@&J83i|8$mFS!SxUdl=U_ z9#jsX(YpQ_j#RTB&H*c_LGI3@t~J}*{aMb1ITRiGa_VJ4?wKd1b&aD|w)ezAhQFx1Vv2D%_83-OYx*G|K|;vjHY1YGNm4D)krH2d|1n{Z z6sPaz<%?(lwhL7qVddZI4oVrBiNt-c_b> zQB-{%FY8IlEF;OZVTxv?A6ef5>5e{3Ik#Wm_(4B9#M+U%u}GP&gNzeoTcIU0P^tCp zw|d9Nde^w)g<0*+cv;*;5RD+q@f`GyTib4~Rqv6ptb%0RO)*(qXoXhF9P?8MUW(g` z_ErLo&!W{P=C?QftA#r7eYs6uTQR-EEo!15Ee?$_@kZP5!t@+ZCi8F3e|x(Bm^m+R z$7#HX%2A;pwt$jkg_=MBJNoT=s`ljx-64|qxLrjhgN4*XG142}cJFMjp~TBj@WWW!p# z%m@GRE`D4%1w>me*uNo;;UxENYkNYSPN11#(c@ty^nia@VDlIJP5h5YDkP7S!0FUn zgO+*Exx)Dy_4Q8P4efI~T;eOa>fcnxym#>-2OL)uFa`(b zZ%p2BUSFlpEf}5LRMg(+JJ%>hXImonu25*(eeB#sOGg9`Rv{5 z3aL7S6vxipIcE2u!xuoqw7GQ>KJFwNc~)WCIyHKI)tA&2!4%&58{j#TY9;ovz_PJ> z&z3bad+mmEYPBy=)n6H}F_e-Tnfv9NzfIwrRKj{s(5$^&*0;gMfSRBFB;;MWYG~RP z;)BIeLXo`L?*rm88#er3Z2%gMV_|_IURGvPFEzYRSm#Rj&!|}vn?|r2E{U?I zalC(+aLQfT?V1PK14@FCEw|x`why1pe!tnlkYn$8qL26Z=ei!gdCih1`J+OD2zcj9 zMp4V#@gnfK0}lReKzd~LQ7iepJ8x_Fx&v`kV-TtDXy&^52^C%S4VbL23YB&A`IiG1 zbzRm%=ey(KReM)+)!Y|I&jW_W#KvMei9Y}YV}qWYg87}+mpcb=s`ev!d1VCrqy0P~ zHMgmN3RzVLEF4k&ykcrbBBBmc_+Sq%gHQaN`#*kxCp1Wk+bmI3~eSTeVB7o4RkGa~1V*csO-5>i1_8G^W_qB#o; z2EzJeb;t&knglzsT$2a>xp92edm=x7qG^899A9qM>j@U<1(A;C9P%iOq+% zz zd@Rf~N#E`BY2{>c>9>QamHKx16tlf~Mhwd_Z17dtB@Dd4_^{s;NRnIvow59ZC#95kUOb7TGvUNzdvq4}Z~|BkImpd3 z5}_)vkXEGt=QP)N9Uu-5RNEsX_Qw+#O}nbq ziM;ZYvG;%!q{^_^Ccw=Z*SsRI#xYTW!o?d#6MUXkie7pofE6H(7e`1%fFm+!o7mVN zEFWhK=kbLT28Dv#x1YCuDoL{{Ny2S|qVltU-(K9@44E*3(X8H=x&3U)$_Sua_abUz zdy?O&(m1=(Kws6^xd-XPa6Ab=C9u(~+MT=zs%~-EFW!Ns3~{}e_$3W+2cSA@a`NeYlgEsUBei@8xO4(O_zK2l$=JTF()cVlHxI@h8>rZ-%e>fwWnZmRA z0|8Sz7pEHNC8aq3veUL_Jb5D0oVWZ3T008yHA!9OZ1d_ZF_tcwZ>%2p#K38Lki$dY zxbhs35xk21PPpcF!moL+|5}8w@vQHI37||o!3v_V+A{QJ2RlC!_IrAEBNMD!;KJRE?jRyCARGRq}kqOVbMG%K`*s31I}H8dI6rauxxVDT_@*f^*j z#91cb9AcKIhTIg?F(Sz=)QT&K-r2s<{o*|#1oq^ENgWK)4T?d1R!F@l@W4=CGZ%MQ z=%U6pT?yl}4%(B>7;~m7!5CGx8GCE|WqYoVbQ&m{`B4z4gn@0T(_Di~nR&Eh@27hg zO$FV}Ys7t_0#dlKRej$0toCEl3E>`sagG=R>Gk?D_g`v!w1!vp8q3I`UxN`=e`f2u za|t?<6c%Hh@nM!GahXLK{JQtTn>Acr)nJJ-OQv%5YuAC+VvP_r*Sg#dQRLZ#;=vtIgJbdj? zBa8}@kX)l}-+qx7?QUvcYf#xVH%T)h)=X>Tb)&xMHjW;)jDr1mjk0Wyq5rGc$WuGd zBgmTb9lO(!z-u9zB1e%n`<_QpT1aNKkwcsxm-DMazJp>LfzC!gZqe zZ#S!r@kv*5;zT-#JEJwD0p=N};6w9o7x~>fE&ffLJXLOS66W#mbr4@Q##xrU;f!!y= zc5(@o-_FdLr`EQqNcMuyRv6`S+vo}>cXnTD+xO1knAIIHwBQ_|u+-=(|D4NFd?r~s z41s`gO9>z|QZcHtAi_mK#?uHe4i7GM(Kx>f!Xd)XUKQvw;z}{tna_-=8F6$Z*=D^R zk?0-N$aAL&E7IlfQ@@Csw-}J4%|QVY_CK%wRRp3eF~~6}!LNmtYR$??d^WpbglMm4 z=OT5X&e~yCkyRp1tC%(Yh722K)YbD7$*obs?tza*;MLxl7Qe;o<}6}i;NwE1@^fQdz{RT7CnRQF&WNx_U>An7EoU`iCe93En`Wi z;@6m#N$r(s)wk+PoV7Xj+&Fx_m{6@vClU(^52Afm5r_`{sYx{)zxvb!&a^1Ts#D54 z0YtoZfU-DgA<}$fGTqSQ18)^~V5cKHAmscLLYPZBN-p&s2*CPr0BNWccaZaTB7gM* zA>=^<6M^6CdG^T9^YZLmHFzA11<&9q>q$T13uk;}v|&b!>H?swn#NzfK61sw3FpoL zI(~_b{YsATNUX&3{udHLF_c9We7SfdN!!sDu2M}YPP_^?%a}UM2#2u+V4j+XG9rSE z!MjH7Uv8-X@v#=!#PK3K&!~es86?u|n64Uw@ASfnJNGt?R3Tgh_Y9P)RzZ>=fLF`P z)8;+s72Q*imIr7M_6-WhN@xZ9H@}^cyh3z?_o=A)2oJ4qRT$%MxKA!t5*U>2k9`Y{ zQJwHDP0PDtelsr1k;Pm(S9BR91IH6&F=VvakZfwM3<<`~vwX~7CVGxxrDYOmcLv$` z$;UCgGm_8}Zc5IyEYGla zw=M-o99~q#%#r9m;`c=2YBq*pnV_2r4-Y2zFui09^=c zRckzKkY!%H*{9IoSL$MV$zKr9>qknPO=nZ2cidcS?~Qm?;it-8C$Fh=wgIZ%CxHwhnyEn%TM`Pk=+bd-DSEmn}?e3kJ(E`GOi)}TBqxBSpaAx3Aakwa-)7Rprml@6CsZ+9y3|WG zk%5(Xf62pf2*B>f>H?a#7Y1`Tzi_m_NLxk;fWicPw%9>l#I(s_4&F-o&wpF=0Gue$ zTLS2xTIVXj*I_qdm6q+m6Tq@Bm`oN%tPL*vWAkC)@jBQ3>slFCv4rsNAat&hBOV^` z@THdRhcU|X_vZ+c8e3qY*vg584(5zyUH?*tW#B@c&{fyWTnFEoMEvhh5J8Q7CcF;) zITWq2EwR5CQVMdc`Ba_d!Yy2^5wT383QX@vzQ>mJW~n+{ z^jDr*Nx!~L@Ca1$TeWR3^VW;%-+^YPk3S|l0HiiX{C*M8Ubx`nGH9R$WrQNnz{6ir zpG+Jr|EB~bpCGqgh~rS>5&SQv86e*C)Qpr6K9hkru;MX;Cu(&iIe=ho51{X{w$ES9 zZ4Yo&Ol`t)wAh|NR?xhizxd$RAKG{tAX))Y1>pNp`CfShY`<@6XIfFvPk^(fEc%^x zi<8krAwl1_hBnA2RJUodAgM7Z**iX8__W)-+yU=@+i(zq7D1N~_%^Wa#a;mt1c4!R!7Z zaL}s43pH0s$B2*4_SVR-z7^iAVCdKyhRLoM2+6Tv7DzQqGYOnrL<1EUR;c!B^<&3@ zsWNejBG9P_Jhdw1fYOcy0RlO6gRA_&dItE3sh>^rB$B5P2%v!HV%3u53dmf2w$_<} zqSqO;y&wAH`1o9WU%wJ~i+0 z^yso-v#|1UUQ}?}5ca@dT>OEF!cbv?WVLmeN!ltbjg4nAk;23f*g+z*%1R*&YePj( zo$n>GQg58~PMt!%=WNWnaB#l}blzh;{vq0gOJn_3v?P4D9}wlCOVZWn%9GgIL>f}- zK=`+k=I;RJrO##B)4 zwJSJMe1t{VzdOc*Yyf9CiX7@R)hwTR%l_qGVF74gxuCe24?i?}AliV8r2jf;UE2Q8 zJ1(O-h#>dnTcujuRhAKKgca?#<@|s{O;h9JkH=learitiND1E%yb`sF$KCn4-hb{I zM0{>hR|U~bcnZjC$QXoHwl`jRM}d=gb^20-OrX1`>g;}3&E`KXmo4wsqJ@^fO*w_P z)H#n!@L{}HL#_J&(M^K;?gZtzVj(ohL8&<#6A7+$q%yF|u)p$>DhAF+0L-NFWRzC1a+Ehl%AseqGuX%x?46fKSH2w>mp#=MR{lzqtQlkWSWIB86nX!?KsrOUKsqB2(HlThvBL%p z+;5ob4UerduVKn-*VRcM(?;QnxUGG}*z(+Yv2Zjs?^F9FpeYm_uXyI`{gp=Dvz+&) zyXA-*WvwI-1sf9_{fz_9yW-^ca5OYR@mup(wgtk$fkkiJoYJfengjhLQLXC?#%{`W z%dHQM#*=7!Y;{_7v zhR4cirQb%gZEwcOdMt~i?Ld)Ee`>>WS;MEjj;H)0#P1ToM_)@22qoO;HtNI~kwk8! zIY$(`Ka7&Q?BjgK3feBGw`JTe+t0R{6~narO?vwj!(3r(oVg>1o-z$sP0r!P0GD{6 zJ!zsmQ&cV3c>9*{shz&CXsq49t4~sjbRqx_1!DCb^mOYb{$c99mU15 zeNh-a3)AM9sM(`(jv8x9Gb6v@AmAKEeqnvkiIWwt{B5_JQ~NBm(bv6hR;NG7c_V}eK;Rddx{vY7f^QI+Qeidr?(;YHf zg*YCU9xHuGj}v{MQuO5ESriV9;_^;+a!GVas%_TP!&-MUo6PPT$7x1rOxj}yvk0&% z6+~exG2ZjEFUbM77by{n%J<772$jSm8T>nOK1NevX$8J~6p9lI?U2 z3!kvkYcKDOH$QBlN`;`4Q&$cqU6^~Pg}N=!BF>O+wCNZ~yq{t+DFCA()c(WvyvvO` zFM?eVws}D>jc>eJS$IP^K|=Z3d+Gbzb4dU?P(cwB9&=ANeEp3XOBxC~ABuJ-|M*T{fLA8vd%UzM(fZ9rl{%AgBUk-QPOg(we5 zNFr4cISLJ*@bl66ix-7s~+B%k{$8o3Wq zpo57tC-p|1{&cPY{m(7Os9Kqr{m-1?p$>r~x zr_WbjdyDJ5^vf4S!h&G#KvKYj=r8Bh3zl!Pe^FQp1#dR-jk{V4eAT5@!f8`fy#Z$c za(9mzkpW1kbIIZzftQ4T2Gdox>ko|Q|0L;@kwO@c=+N}mMuTJ!DWNJz@eoCu{bK-6 zTqFpDsS09)CMrnIXn%HjyC@Jq0vO}QTV_53!%rG0==t#ofir(tB;1vX^LU34`c*+O zq;2Mem5WfANU z1@HH7@xR0It>6fLD%VOSrU&QXub+m3Bz~UKArhBDL&1t6PR$NOJN0)`20f3~T3 zo6#@jto_K179Pc!WR8@g#T7a;OR!hA{RSv-a!L&<^UY8edv;)ANCpV{%k2w=>{?>o zRP#Vp<8UH*IyT6&anI&CE%h^gqtTBQ`@^%&Qvyx?LS{>K7e|?nTxw*B%5}L`a2c1;fY&tO=&j;>w<*$7?MyPE;Ei=O`ACE#!PUEY1l{19@;j|62$Fzx z3zV^f;9E(h0?CRwJMmU6-vH}-f!i_HyzC4E0Tt3jq#Ww(5eqn!zZfd<%JRFKEUv{~ zR`c1gW9Aa`CmPj=vA5KAG!F~f77XpOhKh0(9r!e5EkB|1fkwm^MmS*)DpA|DYrXMX z08Pn0K4hjE0&c>Wt6-~Q|2gtLBy>PC{+$mF-E_mqfM;R51@~rCtWCZtp{mx;Ii?xp z`PngM`KzCPQg~3|uJA!tPkhz(3*0;lC2|bY7t_B?eQ#oXE%d%?9s!!prcGXz-}m%~ zS>xabv?`oDvGNBCj3&zfL_l3Iwvp%n=bSwLKHKjd4(5Li;1WaTP3A93zOnx7_9m3( zcia`N!!}H<^=}tOC|S7WwU@oQBXJahIbYm4&N?fPW2+8mH27YBFeU3v9#G%%Gx+3w zM;!m{^7T=@qM7?_O)Qq~8tSKCB==m^_?)=<0F~sBl3H5P&o_5sm-JB_%70vDJIJse zSbzqT`*D83*$z<86R`o(N>4bW#8(ljD!$GK(e}FOcW%k2q(!_jXuN5=JS}S=jMSJC z7NSkR0*JCnJ0LNeg+ zl{@F_{#&}NArs;X$5u)Aeg-N2TwSlU>iEONlxkcd(m%RDhK!!h7R1B-ETYQho0Z3- z22-w8{v>F7fDh*uWAi72fC$O^qdjL5&Z`%6aM((g`Cdg=J$FQDXO30=y$6(O z5Tk)G(ZK>8#r>})hfXA?@9mZEs47KQf#gXI3elu&@`FgePm&Q*XsUFM!9-#yILSkO zm>$01dmf{FW@Ib*akOR(pUOS0SURZN+R$Do56f+s=uU*xld}5U>YGMpCJ4_PopISw z&=d8JI4&w{W8LK$bR3&%x9P$`g+zM*1wP06lUk;m@lNlR;saVSARU*rNsgrT*?nOrviAZcP1yt+R)U~M8Kk%kX5$8J%q@jsPLK!sW1 zu!;o1YS0>Rg2AeHwWaUc^mGad3Oo6ZXPN2jX!9o3lwxPx#a>*EU(o2e z5x?Zw*Cx_yMiJm%3J5pn;X|8)DRZiR7(UY6zc((J-1W!b9EOI@wPP|!+FA7*v36ML z9Pyf=ki&iQhlK=8RDvDkSK&l4ex{qNk7RqnSKPvY+)8Bdd?t-Q5XfB#@>kfVJns#4 z9)SicT&wg(i7*_;^Wrxl8MILILR?$I=Yafn<(;)+u<@Q^OQ5z+77y-bL~;(*BQbi1 zQ22TE6;$<#j?Af25eF71=eC$iZDe*yoP8V+kO_KRWcLts$Lv-mF177L_-Di9Jr}jN zy@wVZefl*Lf&N2jGfC;U-~I47lg{ZaT581p#fTAB*sDJLE#U7~9LWbnq4EzM@9p@U z(}i3AtD|s1@HMC+I?bce68@xX4B0_va22vhZx9EFz<%S4d!eeNrF0dWCDZWl8vm#$ zAT-t{Y9~QoPu?oiOPj+1FK%6EM}K2^os}eB)-0lLrwgoXrX(k>CDyDb&3D|+mFI#u zzk1Q@RusJCch`T(r6A`pk(>w_6URn^u}S8kdS;^I@@Byr9zcepUd8W=bi63At? zA2;-b3^GOs6=5M<4mwG|{=o1- z5yKbi@C!0Q3lBnA>4AS$s=2B(qzn}8BjdoZeWG;@5hh5uWTL5A+d-YkMBiDNkE_96 zT(YgLei|WdRt8TNPd*LId;0oVQa>}rRz|mjx0MKmw=Hvaw%fgJ^ZqU_Txu36E&XaV zq=jvOJ$}mB=}RjSC&7m3X{XMgV=3pe?}rNSCF!Pl;soNhf%~{~9lauN0fFI%C*^&Q z%Z$_I0VWE9134ly`I`s@*kATIRDK4WioC|)F%D77!aMm#O99C%&t-2^%p|eRi3@ce zqyeM|v3k#gr>-`~u3m#rUT?1_3nrR*21?a9Xug+u`^M*H%&nmnF6;Xss>#5)O4jST zgHj+>5YaRqH7+C0>xFkc!^rXaZ=qDR9kq6>Hz7nEO5nY_wIm&G+deHR90TFYE=#x^ zI8Oc3f5O;m(#y~q7)2rKt% zM)^ftoX1a6cp7F{=wVgwUk3f3=Ds_g%Kz{GAR|sjA+jkE$EL{4ipXp@%61$($;u{D zvdi9OBqHZz&+M#YWF{jcTQ1!9BSdm!))kV@ z(wPY4KVPC2M!)SeJ*l@bXw{vXLdvLNsSc|JNe#P(;B&*xVN960G+)`#5_`kK4CMu` z@Ee)5CMTJ6GRRyB-7}|hlyrT}=E zJ~CR^WrYIdTP%Go0<&eVZXIkNsbx-7^#B z_(F-fN6gsn}t#bCzV#=TpL;;P-?SqYNPsFG*% zy-ST3w3eI>|93HMW6XX=b*yK&+}wvU@wIbjni`6n#j?y>G`I(!^8*T)t zqm+zy?$qN#ui=jd)tg&ZJ^gscP=p)|5uEWB*MuGgW-#gCGahFsbHNV?ybfl^SK8}a zE_VOY30Zee!#`rRfU$PT;+O0I0yGQ|z+y&G)O4iT@!^gm+Np0%ft2M0G&)fcWGJt8 z=xLUejuXVB%-kx?O4ygku2}B8aA1m)Of>R8Kab7-YV**pP(H%;8nuPG+>YFr=&^bU z%<*!63e`!6_GR07peymAZ>r_+d z#unjGiswrC9xnR12x5E4Ji==azgjD2_{HpB3(Yt1q(^WTDpIg#q;S;T_U})k4w-tgbu^+G#WSh72c>P~wU0AWIjCuyEf#diQM%GG zj7phLt}v3_3T>vF(x{$G^>p=ref^jFdsA@)$`=K}Ri~+sU@mMr5gP$);@caFF-o6F zD&MZzw3=K#cgx%UDeq-J-9Yd)NdH>?dmgk4~P!O2@d~dOgj$h zo)xv_moj&d{Y}yrJ`hvz1;6XG*BzrJ&0ToGX=lWC<68o4j|DePhO#%#tgw3f8Zq6+ zZmGq5+2vy>BPFcZM$`8X7{`~@|KtUI{4#NwNiG?np?9_Ab0SNlA%TZven_T2j3Pk$ zrYccpe5^Q1B!k*YtFKr@^BN|K|5HA`!T zW0R*gA=9=1V3UVE?Sc|aGguT`HW|$H{0!Y>XVP^m=s?hSlH!Dk*a418Oo4=%^@VgH0BI9>j1 zU_2nViBrT^`Md`x7ztT}E7rk9bjHak(fkz5NdzcylVrlZ7fgNw1?bdZsx$dT&i#%o zL-Ro_V>y6fTs%#oAw!DoigS!mL7)X5KiNKd~?9YmN$yZ;E-KvbFh0Ys?bPXQUU z43;u=@V?k$nVR>jwV1EVz!*YyF>LRTDjt4L!fpsh91={9gz>3a{h@7>whDm3y? zlfAMCHtLakLT`Bu7ZD8KC8H+)Le^DuoRPm(Ku_K=peOVUkQ(xjKbGfgQy@Xi9tBkFfJidCc=xAOtsRgxB3$!F zzf(DSe4%)>X_*`HKZ?;niPRVUD!_1Gftyefka1BQG=SeCA z6D3e!P^|Bc&++`ZK})Vj92$`}m~bqV4YbypynWtKCzXkmON|4$oje6p~) ziY)C^?;XL;95HCQDy9&udLp0Rv|Ku#TDvn9Dkve24cSV#I}B8=pFTPPAgPCmC#?G3 z&{tNezH+;0lG`V+HkKWmOE)d5e@4E3pY3?)gBOpQYFOg={tT1bUG>`UQ2z8X zjBAA0$%k^sd)-#;af01l(irPu0+HnwEXp2aCrb^5m~7S;;A_h!({^< z?z>BQp8x36{uh}eGi_0CF(5O(O645f#MfOXX#y6St3-o9Y4W5D>L&SVC&gYDyutU+ZQy zE0C(@Xy=09fxvN+J*A2tQ;X5&LM2d^Q|+pnAT|`ik0~kRjLAlaCqJdRQvhSG;`@sXZ64@q4Qzg8k ztK8js`@3(oAT=OW|J%tuH)0kva^mf6$MR4rWU0>}+av6^{rLRX&ye?v1r`!w-|wE?w0s#WrMc&c%kf8wphxE(ruu5?mh@^C+K`wbMF{=*aA?=Wa z>(8>E|Lb*veHmFXqgx+rXFKfL9@Hd`S79B*jfuI&ug9&H(sUU3HmG({0 zBcyif>80DNP-SzwbtEg!4>+0*wZ~uVj^;zxmy>UJ8kJ1tF;y&R3O)P1ago34GYCG$ z1~Ozx@Kq}>OI^5rD9XliDkE6#8iaSgw$8`ua*J1`uYwe&3bSyJjZz7+_kN^ zy)tD3NT^otD+U5{!X}QZ7S2yt?E=#v(OFiD^HH?ry>TFW2DmU1-P5?N_vp9x_E@PQ zslwv@?L-%>3c;iyt4AMpytt|gBAQZB(qrZ0o~z$wRG{MB5!J?#dvNl z#h|8JDh7=_?*%cMw}f|QYo(=X-Fx0CK4<}HRdbkF)X@sD#83b;#KH2b_aZG%8hekTjC;t%-q%>@LgN z`8ju>o0`MlCeVU0mxw>ae1cGO#5sf?ora~$#PlJvbN*--P}1g3S`PuTfqfbjruulR zHEjkLjEeu}Y~F_^GF%m{Q$Rhpy*!F(Ve^$f_MY#}8iqE7R~N3{uWnfdWQXA+{A$|q zUHh}_b&kIHQ3Zecx(fqI&;Lw%Q|4U~uF3-3?DWJpfg&-QkpRW;I6{%BIvM^ok`Dj_ z*5J|s3nSgf-!cQlWNV)ABpCcj2OQH{_#K3-rz3%9X9J$7=F#&S ze?Ur78kFXh_w=4+G~ih85EjIuQ?}tTb|bbQ76Lq%YFG!)76P}2lmYX+ADbge18T#?5M_lY{+61Z^HdFf zR?FvL4!3O4>gE@!LPeDO7U1oCa6@6@ZS zC{a{LD@HJ{9$qXm+nE$+<)jqcAea1nKk`Q-nix%^VE%P7aqnv(yc^^Wl?D`tUCoAD zGt*hm4AaSzux3XSd|QM?50T(CF_#xUTI=u&AW0All5t_Wl7t4Lu^IosSxjzEGqwuCF&h0nF|(r^MK(gjhO^f_*xNxvnb))5@NAK4Jz5l7Gnv*_TV0|KIghW;-mo@oX}IA8Ly zl3A7h<`ahH$-cya&2>J6s7RvGLZWS0>%R3Zh%4~+)*Biwm1&G8KLEhTNf*EP01$_R z*W+=}-`^O(4EgM^+mnwA$p2v$X~e8TL(C0i4#Lj24Y&zSI7RrT#6@U^9G|D^v`kHZ zFJhih24}Da?YtDxxx|5<5K;>mEsFV-DVwlGwjIgavb(ZkK-)niPHeb1Ne$6cOHPE& zFX#`+_UC+DO^;%bj8)A=Q6Kekwr?O7O<%Ia9(#8Ca=z~ld^d|fmmCm{Mb6- zD?AEr9U(M&H*yS!WfUmE@YXi|yf1Uiv5g)`YC#33++)W^q zOVzTi0KukL8L5!j`gGep(`InQOQcf#cwf(awj;qq0eAleyqjyVG(I5lAhnpa>I) z4<6^Q*5QA?{HUsM7*Zw|*V$jcKTjQ*n^nJPSGzP6nx39$j<=jkq{rpEb}_qsJ-p`` zLHOP;vDn%e;%DdLZb2mD5eDR1e{N2=nO%hCr+b2Ic73Lqee}*fJ6o?J|LtIJihV0z zl!9VnpM>=UG>5A&;P*J(vNkL1?~aru|GKi82%oF-e&KtC2~S&&+@W=ba#B(Uy=>0z zXZdAS0;69@ybdo3}fL%+uN{9=sBL|rsR!h@~;UFwQ(-%22gFBDRu{S$h{_KN_& zL3qqsT>dYhMo4(M-;vtNcDO4_`1Q#am$Fr5eU+kzmM#Cg>j?3GL>`M zSx5q8W4}d{nL!kB6su~7TDO?(&)WuZc~koGT7y=2*txvdY~DZ5?mZP&W`c2&2N3cB z^XnS|eu8zJT`$x0RdYJSY=8~UD70ROy1t+X>z@N_{N{j$&d6Xo3d+9dos!i}sY#vr2j~olARb7J*sW?$zOTN@hfUYy zN_w*tlKY00kv`pc375QsSiMucFN^EtsT_vZP&y8;`l*err~$Ep$L+?Q4lVZ<4H$mvScXxQG}NZrrxwd-2p_Y#0;)Oe*xjDAkWa{JcsKSD*qn-@;yGpfUKh; zMEvB(?j8`|Y~C0HWS}e@AzD6BRExqIlj5aks5$4~9IJJYvC17Og zkki6Qq)p$jXOJOa6dxY8tVIb8)6eT^h+N&pr@r(Vu$LqxNxfCb~ahmw`qxJykVO0dw*&W6j1< zuL+lw_lJfipV!>D+raM3l86Wx&nSnHPxnn*xd>QCoCH~{B0-cWb2)wBH- z$q^4(a}@$v8HQ7kQJORv5e{zndvh^o z*`OL1;GNUt{~p2hDZo7BP$^0rS~LD*y8<1oh{zC-FItW4e0S3f=M^awi<!hk`BSl#VXU8%*G1ZC6%i$`lnr+o`~ zB73T!uGUtbQTjrnZNcfa&Te@1$v!-!X8L&fXh?> z74}rMwCTh4bqP0KXrKCf=1+*!rKSo04DS4-&lmgBr{X!3(55CPMEx}YfEeW-iT)=_06AU%`Wnqg)IzkR39zP}H8wydizrk4RA| zPgDGMGx1`R1Y;=cD4a+L9ssonV5gtDEoJK9%uZ-&K(PYkyAo*OAr!+|gA4R&eIz6e z6i98+Z>vQP5A;IYPJEcqkJ0u0nJ2sWNecQ`+)J?dhp-G^&>(JgilzJ72hR32xT!RO z$`Np%A=~xEn42s1JBM|rw=1B(IQ?$+svdJ?8W+Qv0kCfvaLL#0t;h0=KuEB?zgS2C z{CAK<$VNL35$HGyri9PRC=f*bxdiVX`NBUFAY~nuGEC59T z2$U^0W?j-M3_(%>Vm3c^?i3#e(L5yV5pyG#NRhHJmEI4HTbj1fD|)Ge-#of$h!KA0 z*9GSr3lE2XgS0xQO8{=?=%;h{a-ZH9@P*o8ykz&^(u+0y=YWyDC!rF-8p8s7)I7$o zE>u~MsmuDd6)%Ale&q!5aoP!f0JlOrZMeC4MN)LjcHafIL#kHt=pN*9AK*&Q4#p=0 zLPk11aW01OggrpEg5}Ec`Z=`J`G6I3B0}5j15kenhn6dz{MCIy)Z&OG$0!sSR*?|9Lk7u$mp|S(a{lA`0qa$JRck~LKT#++4Ey>J zzX?A3#A0Kv^Qm2q17EkSwTdD~-pN$%8PUt$rb7Z>f0HzR0larbifA^k5FeJIS)m@% zQ-v{0W>TDPTae?BglXF0`sy}-98XMw^dNI-i%$;9b1`EPf26su`6kC<6utpj`r* zxl^yG(DkkP9R`s2Pb1bwn`-0UIngL8V;wH~g3m3HV1CE|9e{v=6405xpzkC^xSN4`G0FV(jo&^Syg@Xj# zE#E4-tM}9P#@!r--X3>KkaE?99p0=`nH` zd9-jHF0-~Sb2weuOXdHns2PAm-aRl%D1=hpAi>TMy#~x5XB7F2b{o<_I;5m#JLkY9 zIJ|1zc$JC)i#`Jkvnje#%zdYV#(z9|X)411z@9<%D=18^O9~{8hMeC{Eagfx)j@>D zp#AS!)EwdX9IZfI4f3GH!2VR3DMrx`7*yTx{M&iFLWgw+6jl@@fUv#EkXy7+;_?_j z;=Fox_{*`cr`sUsoWV;1>C5T5I$HhNTS1whgI>#^=lx0Fg;D0mD-ZNuktF+Uwh$0- zOHJv#6UEdWVr%_NoV5G`&CFyZn{`})R(~m`8zUxOGL*A!7_WG*FQG(4lf3bV6PZgL zqJ?UxxtizWI~%V?h%vu3sCCcr zvs;UW{CG`j&w~dQW|?w*ndG5ku{q#%)8#olZT1`E2g$!hD6)*q$zMN84dB8EukU(< zd1x+^q#w52HH~W5b)WX`vTapKDIdF7d+@Gcn91dlLlXXT%Y9H_IJr_|UEN$vNH=$y zwR$Mm{wm7Fe~w7aCTRJV@R2_CAldx-=*o;diW3`Ay7~1% zDS7Dh*VW@E4q3|TDH&IId3i<#hm2oGi*Naf&2s1_6H;f*c+BF52+kPT>~A zh^;##D$#LwKVVybXS?5?k2beLAc9jpU&pu(tE<0CF)~L5O%Ek1^vb##Df)E5-)8jT zN@haAP;kN(bpKYor!*UTWqk1+J*PSa)SeB-kEUgkxI+bN%L=X7h8JZJHn_6B^P$Kt zAvNo$^XUk}U=)A9M(|IoScCo=I$l;0Lyn+3GDfK9AJ}teXk6HM`EAl(b$oBN7meXc z^6k<^E^YpF_NdMgXR;{m$BXJyYKq{MtuwGU_?w!7LzX4=ZVVsESU z#?lYpewLHK--go&eZG?LxVUebpGpS_ONSl}zGW$k#(zG%qI+#KqEk2gQ5>gfbT4O; z+V=IKe2bz~a|~_fc5iNrCU2xA-?D+yLXGW4*O+=JE%Vo5tI7kj(q7I8$+q>#j$i0i zE~Xfibvw?s1OKky8(aD*OF&UsiIr>6>cZWupk6{4)Xf0}-B{}#nordcZ|Omk8X+yz zR7e&LVtd#^23gOE6HS2*lkB)lzj^1Uz zuPXc}cay^7)HGXq)vWC}hLI3V?JM*$fv%`hF~{78!Y|%wqPAC*)pTc+d}|}Fp^K-X zhK0#nb7W#~{F*I}h=;m)=uT*f-EIe_+a+)v{u9&B8JFTcbI_P0-3brmugt}FZzb$S z9&+ol_6H}vTYY?_=4OWyz~*e_&&BlF|)3Fl1;bNW0{kNSG<<}Q|&bB z6yWkLCF{u@5kK(LOHvoy167V9ix_snOCilQ$D1ZW#-*9uI2@`tG(K)w6V$mn4_g~6 z%|>5gQv(;9{_=ptzv*k{tp0hUcl59hOxX~1ly|2jn=^h&i^G z)8OI5KMtNg%Qs2No=&e1+$^oA9naB9V;!$_)3PRj!PF@dwyTXEKVE+QQneE;mRNxu z`uUSNP0GdQQ<>23TjA)T?atBcnuD*cyQZ1#&k$dv=rZ1ggwhv~D{a3(>_O$Yv zK(YSx65m{9r=;_Jny7}@cE>8oqu$=(zZbNsYVkL{K~DA~$^bMiG*&g8qISFz`D+fQ zG{CGV%gy5CRK0Rh;8A=9ys6W6Ar2ao{-1oiz>C>mj zP8^&YZD=vw(~6 + + + +WARNING +WARNING +WARNING +WARNING +WARNING + +

PLEASE NOTE: This document applies to the HEAD of the source tree

+ +If you are using a released version of Kubernetes, you should +refer to the docs that go with that version. + + +The latest 1.0.x release of this document can be found +[here](http://releases.k8s.io/release-1.0/docs/proposals/kubemark.md). + +Documentation for other releases can be found at +[releases.k8s.io](http://releases.k8s.io). + +-- + + + + + +# Kubemark proposal + +## Goal of this document + +This document describes a design of Kubemark - a system that allows performance testing of a Kubernetes cluster. It describes the +assumption, high level design and discusses possible solutions for lower-level problems. It is supposed to be a starting point for more +detailed discussion. + +## Current state and objective + +Currently performance testing happens on ‘live’ clusters of up to 100 Nodes. It takes quite a while to start such cluster or to push +updates to all Nodes, and it uses quite a lot of resources. At this scale the amount of wasted time and used resources is still acceptable. +In the next quarter or two we’re targeting 1000 Node cluster, which will push it way beyond ‘acceptable’ level. Additionally we want to +enable people without many resources to run scalability tests on bigger clusters than they can afford at given time. Having an ability to +cheaply run scalability tests will enable us to run some set of them on "normal" test clusters, which in turn would mean ability to run +them on every PR. + +This means that we need a system that will allow for realistic performance testing on (much) smaller number of “real” machines. First +assumption we make is that Nodes are independent, i.e. number of existing Nodes do not impact performance of a single Node. This is not +entirely true, as number of Nodes can increase latency of various components on Master machine, which in turn may increase latency of Node +operations, but we’re not interested in measuring this effect here. Instead we want to measure how number of Nodes and the load imposed by +Node daemons affects the performance of Master components. + +## Kubemark architecture overview + +The high-level idea behind Kubemark is to write library that allows running artificial "Hollow" Nodes that will be able to simulate a +behavior of real Kubelet and KubeProxy in a single, lightweight binary. Hollow components will need to correctly respond to Controllers +(via API server), and preferably, in the fullness of time, be able to ‘replay’ previously recorded real traffic (this is out of scope for +initial version). To teach Hollow components replaying recorded traffic they will need to store data specifying when given Pod/Container +should die (e.g. observed lifetime). Such data can be extracted e.g. from etcd Raft logs, or it can be reconstructed from Events. In the +initial version we only want them to be able to fool Master components and put some configurable (in what way TBD) load on them. + +When we have Hollow Node ready, we’ll be able to test performance of Master Components by creating a real Master Node, with API server, +Controllers, etcd and whatnot, and create number of Hollow Nodes that will register to the running Master. + +To make Kubemark easier to maintain when system evolves Hollow components will reuse real "production" code for Kubelet and KubeProxy, but +will mock all the backends with no-op or very simple mocks. We believe that this approach is better in the long run than writing special +"performance-test-aimed" separate version of them. This may take more time to create an initial version, but we think maintenance cost will +be noticeably smaller. + +### Option 1 + +For the initial version we will teach Master components to use port number to identify Kubelet/KubeProxy. This will allow running those +components on non-default ports, and in the same time will allow to run multiple Hollow Nodes on a single machine. During setup we will +generate credentials for cluster communication and pass them to HollowKubelet/HollowProxy to use. Master will treat all HollowNodes as +normal ones. + +![Kubmark architecture diagram for option 1](Kubemark_architecture.png?raw=true "Kubemark architecture overview") +*Kubmark architecture diagram for option 1* + +### Option 2 + +As a second (equivalent) option we will run Kubemark on top of 'real' Kubernetes cluster, where both Master and Hollow Nodes will be Pods. +In this option we'll be able to use Kubernetes mechanisms to streamline setup, e.g. by using Kubernetes networking to ensure unique IPs for +Hollow Nodes, or using Secrets to distribute Kubelet credentials. The downside of this configuration is that it's likely that some noise +will appear in Kubemark results from either CPU/Memory pressure from other things running on Nodes (e.g. FluentD, or Kubelet) or running +cluster over an overlay network. We believe that it'll be possible to turn off cluster monitoring for Kubemark runs, so that the impact +of real Node daemons will be minimized, but we don't know what will be the impact of using higher level networking stack. Running a +comparison will be an interesting test in itself. + +### Discussion + +Before taking a closer look at steps necessary to set up a minimal Hollow cluster it's hard to tell which approach will be simpler. It's +quite possible that the initial version will end up as hybrid between running the Hollow cluster directly on top of VMs and running the +Hollow cluster on top of a Kubernetes cluster that is running on top of VMs. E.g. running Nodes as Pods in Kubernetes cluster and Master +directly on top of VM. + +## Things to simulate + +In real Kubernetes on a single Node we run two daemons that communicate with Master in some way: Kubelet and KubeProxy. + +### KubeProxy + +As a replacement for KubeProxy we'll use HollowProxy, which will be a real KubeProxy with injected no-op mocks everywhere it makes sense. + +### Kubelet + +As a replacement for Kubelet we'll use HollowKubelet, which will be a real Kubelet with injected no-op or simple mocks everywhere it makes +sense. + +Kubelet also exposes cadvisor endpoint which is scraped by Heapster, healthz to be read by supervisord, and we have FluentD running as a +Pod on each Node that exports logs to Elasticsearch (or Google Cloud Logging). Both Heapster and Elasticsearch are running in Pods in the +cluster so do not add any load on a Master components by themselves. There can be other systems that scrape Heapster through proxy running +on Master, which adds additional load, but they're not the part of default setup, so in the first version we won't simulate this behavior. + +In the first version we’ll assume that all started Pods will run indefinitely if not explicitly deleted. In the future we can add a model +of short-running batch jobs, but in the initial version we’ll assume only serving-like Pods. + +### Heapster + +In addition to system components we run Heapster as a part of cluster monitoring setup. Heapster currently watches Events, Pods and Nodes +through the API server. In the test setup we can use real heapster for watching API server, with mocked out piece that scrapes cAdvisor +data from Kubelets. + +### Elasticsearch and Fluentd + +Similarly to Heapster Elasticsearch runs outside the Master machine but generates some traffic on it. Fluentd “daemon” running on Master +periodically sends Docker logs it gathered to the Elasticsearch running on one of the Nodes. In the initial version we omit Elasticsearch, +as it produces only a constant small load on Master Node that does not change with the size of the cluster. + +## Necessary work + +There are three more or less independent things that needs to be worked on: +- HollowNode implementation, creating a library/binary that will be able to listen to Watches and respond in a correct fashion with Status +updates. This also involves creation of a CloudProvider that can produce such Hollow Nodes, or making sure that HollowNodes can correctly +self-register in no-provider Master. +- Kubemark setup, including figuring networking model, number of Hollow Nodes that will be allowed to run on a single “machine”, writing +setup/run/teardown scripts (in [option 1](#option-1)), or figuring out how to run Master and Hollow Nodes on top of Kubernetes +(in [option 2](#option-2)) +- Creating a Player component that will send requests to the API server putting a load on a cluster. This involves creating a way to +specify desired workload. This task is +very well isolated from the rest, as it is about sending requests to the real API server. Because of that we can discuss requirements +separately. + +## Concerns + +Network performance most likely won't be a problem for the initial version if running on directly on VMs rather than on top of a Kubernetes +cluster, as Kubemark will be running on standard networking stack (no cloud-provider software routes, or overlay network is needed, as we +don't need custom routing between Pods). Similarly we don't think that running Kubemark on Kubernetes virtualized cluster networking will +cause noticeable performance impact, but it requires testing. + +On the other hand when adding additional features it may turn out that we need to simulate Kubernetes Pod network. In such, when running +'pure' Kubemark we may try one of the following: + - running overlay network like Flannel or OVS instead of using cloud providers routes, + - write simple network multiplexer to multiplex communications from the Hollow Kubelets/KubeProxies on the machine. + +In case of Kubemark on Kubernetes it may turn that we run into a problem with adding yet another layer of network virtualization, but we +don't need to solve this problem now. + +## Work plan + +- Teach/make sure that Master can talk to multiple Kubelets on the same Machine [option 1](#option-1): + - make sure that Master can talk to a Kubelet on non-default port, + - make sure that Master can talk to all Kubelets on different ports, +- Write HollowNode library: + - new HollowProxy, + - new HollowKubelet, + - new HollowNode combining the two, + - make sure that Master can talk to two HollowKubelets running on the same machine +- Make sure that we can run Hollow cluster on top of Kubernetes [option 2](#option-2) +- Write a player that will automatically put some predefined load on Master, <- this is the moment when it’s possible to play with it and is useful by itself for +scalability tests. Alternatively we can just use current density/load tests, +- Benchmark our machines - see how many Watch clients we can have before everything explodes, +- See how many HollowNodes we can run on a single machine by attaching them to the real master <- this is the moment it starts to useful +- Update kube-up/kube-down scripts to enable creating “HollowClusters”/write a new scripts/something, integrate HollowCluster with a Elasticsearch/Heapster equivalents, +- Allow passing custom configuration to the Player + +## Future work + +In the future we want to add following capabilities to the Kubemark system: +- replaying real traffic reconstructed from the recorded Events stream, +- simulating scraping things running on Nodes through Master proxy. + + + +[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/proposals/kubemark.md?pixel)]() +