Kafka性能測試報告

Kafka性能測試報告本測試主要是測試kafka的性能，同時也了解一下kafka在虛擬機環(huán)境下的性能。本次測試Kafka使用的是最新版本,支持的scala版本為2.10Kafka的部署環(huán)境如下：Work1.work2,work3,work4部署了4個broker，hmaster和work6做為客戶端來測試kafka的寫入速度。Work的配置為8核16g，hmaster的配置為4核16g。我們首先建立兩個topic:test2,test3,兩者都是12個分區(qū)(patrtition)，前者復制數(shù)為2，后置復制為1。先以傳統(tǒng)的同步api方式來進行性能測試，kafka提供了性能測試工具，kafka-producer-perf-test.sh，該工具可配置的參數(shù)較多。我們配置的參數(shù)為線程數(shù)20，bacthsize為2000，消息體大小分別為100個字節(jié)和200個字節(jié)，每次測試時發(fā)送200萬個消息。首先在hamster下測試LroDt^tuustaj：bLn.]fd/ka.￡kA-produacr-porf-tabsJi■--brckar-listudtIcIi1e)woE-kle30-^--ba-tch-sizs衛(wèi)。。山一xejz-agej2QOQOOO--thr*j-±320―reportlog-intervis120DQ—shcnr-detailed-stats--topiEStest3--csv-zepoitisr-en^zled--reqoe-st-Dm-acfeg0—iressaf-B-slz-fr100SLF4J; toclajs 5114j,mifd. r",ELF4J：!DataultLQjtonD-opar=.ticti1KOPJLoggeriim:>1awntjtiDnSLF4JiSeehttpz/,>'xwvhs11:4lj.-org/code5uhtolfSt^ticIxigg^rBindErfarfurtherdctai13ustait.tifie,etidutiiri&i 職冬w旬*心』latch.,size,total.data.si-nt-Ln.BGrhb.a&e,t*tal-data.si-iit.Ln,ftMsg,“Msg.ge兀虹呻-；￡3lhLn53：；§L&NL4-呻-跆匕1叩「L知」麗e源艮網(wǎng)如的995LTL11消息體大小100，topic為test3[toDtflluuscetblnjf?/kaCti-prodijoer-perC-reac.sh--broket-LIacworkl19DS2rwoik2s?0S2rwoik:31?092rworkai?092—batcti-slze20(J&—昵與跪邪古3QOOOOO—thi-eadsm--x?portlJig-lfitt￡val2000--sbcw-detalled-atata—topicstflst3—^sv-Kport-er-eni&led--xequest-riiiJiL-acks0--iTfi5=-iaga-5iiaZDOEirdJ;Failedtoloadclaf?”口匚9.日1E4j 日■fcitLcggssrBiind蘭r"-slf4j:ceriiltLbgtoiio-operjtion(ndpJlogij&fifiplsmentitionSLF4J:Seekttpt//wwra1￡4]r<?rg/c?4e5BlitnrdfstaticL^aeiBladexCorfucthex'letalls,■st^rt.tins?end.tinifi..rcnipr?52j.ani!raassagQ-siiQg.batch.ai^Q,tota!.data.s?nt.in.MB sec,,tata1.d^ta.jant.in.nMsq,nM^g.sc-r況140卜W#1*L如SEgL之L&lQgSH%L河上MM,第1ML 的如如上N砧郭.,鴕皿消息體大小200，topic為test3IroQtShribast^Etiln]I,/kafka-prwhicer-pert-teat,sh—Stoker-listworkl:5052rwork^iworJc3?^32rworWs?D?2一batcK-slz?2000一niassaga^2QOl]dOO--thrnad.520■-raparting-intcL20dd--shew-dstailad-st^ts--topicsta^t2--csv-repartaE-BDAbled—rfi-quest-n'jn.~a.ckjs0--tresjage-size100snjfi-dJ;FAiledtoloadclauJiiflj.inpl.St-BticLoggerBijider^^SLE4J1Eietaulti>ono-opeiacloniNOP)loggermpl-einetitatSLF4JiSeehttps//www,al14j,org/^xlesrhtnilIfitaticLc^erDlRd-effortuitherdetaila,start.tme.end,timerccco(p￡e53i.MrDiessage,siz<3rbatctt,31z€rtotal.>lataBsentBIn.M3rffi.sec!BOata,sent.inrnM5grnMag.sec2014-03-2313：13:51:038^2014-DB-231312Q：OD￡B9^r1QO#ZDCQ,15*0.73,19-.34Z3r200DCDDPZD2B13.LE67消息體大小100，topic為test2LfCDt^tuustarbin]]. ka-przduci&r-pGrf-Left.：.sh--btoksr-iLstw^rkl:^09*2HDrk2z9D&2,?UDrk3:^￡!^2fMDrkt:92■-r-^tizh-3j.zs2400--□,jsjMqej20QOODO一一th匚亡旦dh20--teportlog-inteival2000--show-detiiled-ztats—topicstwt2--csv-reporter-enable'd一一icquest-nimi-arCfcE!0—iies3ige-3i2e2Q0sinJiFailed8loalclaa?Foi-g,sinj■?mpl-.st-atlcLc-ggerBlEidei".SLF4J:Deraijltbigtono-or-e￡ati<?ei.gp*L^iaeriraplemeritationSLT4-J]Smtit^i!Amt?Jlf4J-arg/codes-htmlVStaticLoggarBinilBrforfurthardata!12.stArt.tiuE^God：tins,.cocj:>r435ioei.j .jsLzcbdtch..fiiq,total^aata.scutin^HE^ ?.aata.*?nt.i:nsnM^r fee】L4-l]g-：2313:W0;也2;:0曰E,aO14-(!B：2313=2口； J乏g乏。口■,331-4L aCiDUClOO-：2衛(wèi)BD5CLLIID .消息體大小200，topic為test2在Work6的測試結果如下圖所示[tooL0Hc-rk€bln]f./tarki-prodoc^t-perr-teat.sh--biotei-listwcirk:li,9Ciy2?witK2￡9(P^2ruork3:??2,wcirkflt<i0y2--baich-aize20Qa■-n網(wǎng)mgg2OD0QOQ--thre&daM--rflp&￡ting-lnterTaL2QQQ--ahoM-det-alled-st-ats—topicsteats--csv-reportsi-eniblM—request-eiuj-acJcs0??nu&53迥"slrm1叩ELF4J:FailedtoleadElas3■'□rg.51f4].imr：'!?StaticLnggarBindar"1-SLF4jeDeEaultinqtono-operati.onfNGFJloggerimrE'len&ntationELT4J:E?http://wwusifj-org/c-zdcs?.htmlFStatidx:gqerBinder￡diturthrsi .9tart?tlmerend.tiireic<Mnp￡essl<Mirmess-age.size.,l>atcbr￡lEertotal.ilat3ratJil：..inrMf!.r totalrdata,sent.in,nHa^r.s-bc如14FE-2313*3：35=珥孔EUFH-23"23:如沔丁心LLM。嘩。七15D-T乳我，。3拓.2DMDQD』消息體大小100，topic為test3l：r<K>t^wciilcCbiji]?./katla-pcoduc^t-peir-test.ah--Moiei-listw^rkl;9052^w&r￡21woriss5052,wcK4s503-2—l^atch-oize2000--ue2DDOOOD■-thrcad2￡0一一rcpzTtirig-iatGnraL2DOO--show-cLstlcd-2tats-■tcpLcataxtl-■csv-rapDrtar-S'n.abl&l--r&Twast-num-ickji0--rrstJM'gie-sii*20DSLF4J；Failedtoloadclass"org.alC4].iitpltstarlcuig^erBirider"-SjjE4JsCierauLtinQtoJio-or?etatton(WOP)Iom-gkmplenieritationSLF4J:Ssahttp://uw.iltflj..口:rgFucd口milrtruLkStmtl匚LxDggPTEizitlprCarfuxthardstail^.ft-ari：dtinKgierpi.tini^conpreiafLDnroe：3ageb=ize(hatch^sizettotaludatahneni:in..KB(MBurecjtotzl.datz.sentuin□ciHsg^口M三勺.占已匚201a-OS-2313￡2asl?￡fi33r2014-00-2313：24:2459?0,0r200,2000-.361.a7r13,0430.2000000.3081^6.757?消息體大小200,topic為test3[eoo匕目warJcE■Died*u/leafka-prsduccr-psrf-t?st-sh■-brskar-listWDikli^092work2:!-9092(.wark3s9042^ipzrije2--Mtcli-sizo2000?-rr史3^dg*320000flO--threads20--reporting-iatervcil2000--show-detBiiled-stats--topicsteshS—Bv^rEpotter-enabledl—rwpjest-nun.-aci[&D--~3H3sa^e-31ze10d5LF-4J:FmLlmdtol?d "'erg.sif4ji.impl-StaticLoggarBirxlflr*.SLr4■iSefE.ultiQ^taDD--zperatian-IWOF[Lc-jgerinpleoRntationSLF4J：see r<irg/cod&B.hcooi#$tatieLo^tain<^cforcurerwtdetails-st^rt,tin.e^■end-timerranpresslan,, total,data,sentr1n■,MB# t-ot-a1,dst-a.^ent,in-nMsgrnMsg^ec卜滔1打踣WL4-皓-找L打笠dllME-iMh如Mh19LT乳粕-罪WWMlOA隊郭村透。了：2消息體大小100,topic為test2[rcctSwarkSbioI#d/kafka-proditeer-ptrf-testah■■broker-listworkl3*9092^Mork2j$092^varkS;K92riwozkle9*Q$2--batch-size2.Q00--me2000(100--threids20--ceportlEig-lnterval2000—stKW-detalled-stats--topleatesta—cav-repfitter-fruabled—re^piest-niwn-acJts5—3H3sage-3ize200SLF4iJ:FmLLMtcClaris*<?tg,si14j,bo(pl,Staticlo^iBlnril-ei：F?SLF4J:Peraultingtono-operatian4*睡*lojaerimplwwRtstioriEir4-3iSee/codas.htmlVStaticLagga rforfurthardaLails.start,tine^ezid^timeFccopressiaD^ jage..jsiz*^hatch.siz-±^total.daca.sentbi:n晚日匚『tot^a1udata.^ent.indrl^sgf曲日g」5岐匚網(wǎng)14-04:2:11*找;:21涕。丸知訝-皓-找13:%;網(wǎng)WEfl『JMS就NOMlOCh3Cl%5W.a3tl消息體大小200,topic為test2機器topic消息體大小每秒消息數(shù)每秒MBHamster(4core16g)Test3（無復制）10020995220.0220020669739.42Test2（復制1份）10020182919.3420022805043.49Work6(8core16g)Test3（無復制）10046178744.0420038819974.04Test2（復制1份）10041867339.9220030335257.86從上述測試結果可以看到，work6的處理能力和吞吐量大致為hmaster的兩倍，可以判斷出producer的性能瓶頸不是磁盤IO，而可能是cpu。我們使用nmon來對cpu進行監(jiān)控，發(fā)現(xiàn)在性能測試階段，兩臺機器的所有cpu都達到了100%，證實了我們的這個判斷。另外從上述的測試結果還可以看到，對topic的消息進行復制對性能的影響是比較小的，同時消息體的大小對于吞吐量的影響較大。我們同時在這兩臺機器上執(zhí)行性能測試[raot^hmastarbinJfa/l,Aflu-prEKiuc,Qr-ps-rC-tD3t-5b--btalrar-listvozlrL works=90-9Zrunrlc32*9092pwurk4!90S2--batch-siza2DOO-HUK^foges200H0Q0--threads-20--repartiiijg-inte-rvaL2Q00--ahjarw-dfi-tail-sd-stats---topicst?t3--cav^DqpcKEter-enaibLed--reqw=a：t-nUAfi-acks0--tneasage-size200SLF4JEFailedtoloadclaasPrg,310],impLStatmLogg玲rdlndleL-SLF4JEDefaultingt&ao-ciperatlon(NOP)1凹供!rImplwmeHtatiQOSLF4J?Beehttp?,5114j,arg/todes.JitmltSt-aticLoggerBindsrforfurther(fetalis,.starti.tirrjQ,end^ccazf-rejssiDii,.ncssagG.siza^ba.tcti.￡12a^totalsd^ta.：5c：Dttotald^ta?santinnNsg>sac20HT4；23Lh沮=3口滂抵，2014如》L3i：Hgij35』Lg』遍9』前LM.4LM?&網(wǎng)OMlQL213利9.559 hmaster

[￡ODt8wi：?rk6bio]f,/kafka-picdoc*c-perl-test.ah workl4-5052rworkss?Q?2rworkl45052rwcrk4s3052--batch-size2000--me20DCDOO--threadsZO■-roportiDj-iatcrvaL2DOO—show-djntailed-stata—tapirstcst3-■rav-rc[:artGE-onablad—roquast-nuin-ack:0--mejsriage-sLze200slf4jjrallrf3load^lassnorg..sir4J.inplSLHJ:DerAultln.<rtono-operation(NOP)lo^￡jjopleiientatLOJi5LT4J;Seehttp?//wuwrs1C4j,^rg/€^xJ&5rlLtnlOt^tlcDvggprBinCferCorfurtheEdetailsrst^rt.txinCgiQnd：.tirafijD3npr?S￡LDnrn￡SsagQ.EiEi&lbatch>.512^^tat^l>>lats..s^ntizi.MErHE.sac,tct^l.sent.in?nNs^^sac2心14-皓技注1;網(wǎng)瀉駝.網(wǎng)1U-如■打L3;27滔土第W』L網(wǎng)LW3L了』的上2?如網(wǎng)MlOUL加.找找浦Work6可以看到測試結果和只在某臺機器上執(zhí)行的結果差不多，而整個磁盤的吞吐量可以達到100M左右在kafka的最新版本中，提供了一個異步的ProducerAPI，該功能對性能的提升很大。性能測試的執(zhí)行命令為./kafka-run-class.shorg.apache.kafka.clients.tools.ProducerPerformance該測試工具只能對topic為test的進行測試，可以提供的參數(shù)是消息數(shù)目和消息大小。我們在work6中對寫入性能進行測試，分別在replication-factor為1和2的情況下進行測試，消息體大小為100字節(jié)和200字節(jié)，消息數(shù)目為1000萬。測試結果如下[rootguuirkEbin]|ka-run-ci^ss一rhorg-auahs,Jem土ka.clisntsatools?I'r=-±JCE：Tl^arfc>inuncQWDTkleM-&2agr心s=idS2『workQ泗aaloo&oooo1000nanlatency=15?nus^avglat-enc^=iKQEQlfi104000GmsxlateDcy-14:n^davglatency-latanEy=C?UOO9-3latency=LClMMlateMy-Q,OQL11latanry=C.00132l-stemry=U.001411-lteiWY-0.00053latenEy=0^0010320000D0tnaxlateocylatanEy=C?UOO9-3latency=LClMMlateMy-Q,OQL11latanry=C.00132l-stemry=U.001411-lteiWY-0.00053latenEy=0^001033000000maxlatancy=Bru^avqaOOOOOCllatency口4心』svg5OQQQ0O酢非lateocy-弓『耳avqEX10DDD0 max latQDcy = 4ru^airg70000-00 max Ifltenry ? S avgBqqqqqq 暗】；lateocy - 2 avgBOODODO max lat^DEy = 2 a.vgIQODOODD工巴euidasentLn134:31ns皓』FQ《4d(h!5Lmcciidspersekxic!，日5』99nbfaee#?[ra。匚i8wQFk《性Ln]乎|replication-factor=1message-size=100[tcot^work^bLn]*■./kafIca-Eun-^lass?ibJi瞬g[tcot^work^bLn]*■./kafIca-Eun-^lass?ibJi瞬g?apadns?taflca-.clients,toola.ProducerPerf-onnaticew^rkl=劉92「works:如姓,w*ck3?9。呢ruorkd:IQOOOOOD2DO0mxlatoncy=l€-4avglatency=D.00D1E-lOOClOOC-2D000003加頓4DaaoDc>5OQOOO06&000007000000BDaaaoD^oaoooci1M00000ma肅it如ITSmaxtna及丫如rr林Latezijcy-2^JUjavglatency-16ns#wjlatent■15gavQM塊jl做=lia?H5PjvglatazicyLatency.Lateacy-latency=Latezicy■C-.D0235OrMl-CICr00143=D-004Da6nL5|avglatitncy-OlOiJISIlatene^■6n￡)av^flawn-cy■0-0015-3Latency-12jus.,avalatency=Or001^8ls.t?zicy=EnL5,ivglat?n.cy=D.CD13.5Latezijcy-5ruiavglatency■Ol001414EECDr<3，凸吐燈匚In193后3ms5L54.48.9Q1reeordA□吐r3配口燈日41。乜_42!Irib/WEC：)?[lool&wtRWbLii]platency-1+ns#avqlatency-0,00161latency=2Q叫武wlatency-1+ns#avqlatency-0,00161latency=2Q叫武wlatency二a,00H4lAtancy=8ns.,avglatancy=a.00L3Dlatcriry=9砧 avgidtsney=O.OD427latency,380ns#avqlat-sncy"0?.00267liatEncy■-■14njw心四latEncy*O4OOLB6lareBcy-7as』avglatency-OuOOLSllatency?14ns*wvglatency■OjJOZOGlaterwy-11好,avglatency-LDDL41inajfmaxmaxe邱tfiaxmaxnaif _琵COIZdaMJLt BMHJ?44)日01-爭8 pe?r H.T.H3Cfc/wm。)n[.Eoat^uDrlcE-bin]i?/kAfka-nm-cLass..2herg.jpacbB.kaCkd.cliant^-tools.DraduogrPcrfDrouncQuorkl2?092?vork-Zs9DB2rwurkZ：:9Q-S2rwnrkiiM92l￡>00000020Q0ruxlat?ijcy-1BEru』latency=O.DOOl^1QQ00002DOO0OO3叫。。。04naaooD50000006000000vnaooooeogaoQO丸MDQD1DaaooDDmaxinwmaxmaxmaxma^maxinajfmaxlatency-30lat-eney-12lat^Acy=latency=13latency=12latenzy=13laten-ey-13lat-e