文本大數(shù)據(jù)-學(xué)習(xí)public phpapp

ApacheKafka0.8basictrainingMichaelG.Noll,Verisign/@migunoJuly20142Update2015-08-01:

Shamelessplug!SincepublishingthisKafkatrainingdeckaboutayearago

IjoinedConfluentInc.astheirDeveloperEvangelist.ConfluentistheUSstartupfoundedin2014bythecreatorsofApacheKafkawhodevelopedKafkawhileatLinkedIn(seeForbesaboutConfluent).Nexttobuildingtheworld’sbeststreamdataplatformwearealsoprovidingprofessionalKafkatrainings,whichgoevendeeperaswellasbeyondmyextensivetrainingdeckbelow.

IcansaywithconfidencethatthesearethebestandmosteffectiveApacheKafkatrainingsavailableonthemarket.Butyoudon’thavetotakemywordforit–feelfreetotakealookyourselfandreachouttousifyou’reinterested.

—MichaelKafka?Part1:IntroducingKafka“WhyshouldIstayawakeforthefulldurationofthisworkshop?”Part2:KafkacoreconceptsTopics,partitions,replicas,producers,consumers,brokersPart3:OperatingKafkaArchitecture,hardwarespecs,deploying,monitoring,P&StuningPart4:DevelopingKafkaappsWritingtoKafka,readingfromKafka,testing,serialization,compression,exampleappsPart5:PlayingwithKafkausingWirbelsturmWrappingup3Part1:IntroducingKafka4OverviewofPart1:IntroducingKafkaKafka?KafkaadoptionandusecasesinthewildAtLinkedInAtothercompaniesHowfastisKafka,andwhy?Kafka+XforprocessingStorm,Samza,SparkStreaming,customapps5Kafka?

OriginatedatLinkedIn,opensourcedinearly2011ImplementedinScala,someJava9corecommitters,plus~20contributors6

Kafka?LinkedIn’smotivationforKafkawas:“Aunifiedplatformforhandlingallthereal-timedatafeedsalargecompanymighthave.”MusthavesHighthroughputtosupporthighvolumeeventfeeds.Supportreal-timeprocessingofthesefeedstocreatenew,derivedfeeds.Supportlargedatabacklogstohandleperiodicingestionfromofflinesystems.Supportlow-latencydeliverytohandlemoretraditionalmessagingusecases.Guaranteefault-toleranceinthepresenceofmachinefailures.7

Kafka@LinkedIn,20148

(Numbershaveincreasedsince.)Dataarchitecture@LinkedIn,Feb20139

(Numbersareaggregated

acrossalltheirclusters.)Kafka@LinkedIn,2014Multipledatacenters,multipleclustersMirroringbetweenclusters/datacentersWhattypeofdataisbeingtransportedthroughKafka?Metrics:operationaltelemetrydataTracking:everythingauserdoesQueuing:betweenLinkedInapps,e.g.forsendingemails

TotransportdatafromLinkedIn’sappstoHadoop,andbackIntotal~200billionevents/dayviaKafkaTensofthousandsofdataproducers,thousandsofconsumers7millionevents/sec(write),35millionevents/sec(read)<<<mayincludereplicatedeventsBut:LinkedInisnoteventhelargestKafkauseranymoreasof201410

Kafka@LinkedIn,201411

“Forreference,herearethestatsononeof

LinkedIn'sbusiestclusters(atpeak):15brokers15,500partitions(replicationfactor2)400,000msg/sinbound70MB/sinbound400MB/soutbound”Staffing:Kafkateam@LinkedInTeamof8+engineersSitereliabilityengineers(Ops):atleast3Developers:atleast5SRE’saswellasDEV’sareoncall24x712

KafkaadoptionandusecasesLinkedIn:activitystreams,operationalmetrics,databus400nodes,18ktopics,220Bmsg/day(peak3.2Mmsg/s),May2014Netflix:real-timemonitoringandeventprocessingTwitter:aspartoftheirStormreal-timedatapipelinesSpotify:logdelivery(from4hdownto10s),HadoopLoggly:logcollectionandprocessingMozilla:telemetrydataAirbnb,Cisco,Gnip,InfoChimps,Ooyala,Square,Uber,…13

Kafka@Spotify14(Feb2014)HowfastisKafka?“Upto2millionwrites/secon3cheapmachines”Using3producerson3differentmachines,3xasyncreplicationOnly1producer/machinebecauseNICalreadysaturatedSustainedthroughputasstoreddatagrowsSlightlydifferenttestconfigthan2Mwrites/secabove.TestsetupKafkatrunkasofApril2013,but0.8.1+shouldbesimilar.3machines:6-coreIntelXeon2.5GHz,32GBRAM,6x7200rpmSATA,1GigE15

WhyisKafkasofast?Fastwrites:WhileKafkapersistsalldatatodisk,essentiallyallwritesgotothe

pagecacheofOS,i.e.RAM.Cf.hardwarespecsandOStuning(wecoverthislater)Fastreads:VeryefficienttotransferdatafrompagecachetoanetworksocketLinux:sendfile()systemcallCombinationofthetwo=fastKafka!Example(Operations):OnaKafkaclusterwheretheconsumersaremostlycaughtupyouwillseenoreadactivityonthedisksastheywillbeservingdataentirelyfromcache.16

WhyisKafkasofast?Example:,whorunKafka&Co.onAmazonAWS“99.99999%ofthetimeourdataiscomingfromdiskcacheandRAM;onlyveryrarelydowehitthedisk.”“Oneofourconsumergroups(8threads)whichmapsalogtoacustomercanprocessabout200,000eventsperseconddrainingfrom192partitionsspreadacross3brokers.”Brokersrunonm2.xlargeAmazonEC2instancesbackedbyprovisionedIOPS17

Kafka+Xforprocessingthedata?Kafka+Stormoftenusedincombination,e.g.Twitter

Kafka+custom“Normal”Javamulti-threadedsetupsAkkaactorswithScalaorJava,e.g.Ooyala

Recentadditions:Samza(sinceAug’13)–alsobyLinkedInSparkStreaming,partofSpark(sinceFeb’13)Kafka+CamusforKafka->Hadoopingestion18

Part2:Kafkacoreconcepts19OverviewofPart2:KafkacoreconceptsAfirstlookTopics,partitions,replicas,offsetsProducers,brokers,consumersPuttingitalltogether20AfirstlookThewhoiswhoProducerswritedatatobrokers.Consumersreaddatafrombrokers.Allthisisdistributed.ThedataDataisstoredintopics.Topicsaresplitintopartitions,

whicharereplicated.21Afirstlook22

Broker(s)Topics23newProducerA1ProducerA2ProducerAn…Producersalwaysappendto“tail”(think:appendtoafile)…Kafkaprunes“head”basedonageormaxsizeor“key”O(jiān)ldermsgsNewermsgsKafkatopicTopic:feednametowhichmessagesarepublishedExample:“zerg.hydra”Broker(s)Topics24newProducerA1ProducerA2ProducerAn…Producersalwaysappendto“tail”(think:appendtoafile)…OldermsgsNewermsgsConsumergroupC1Consumersusean“offsetpointer”totrack/controltheirreadprogress(anddecidethepaceofconsumption)ConsumergroupC2TopicsCreatingatopicCLIAPI

Auto-createviaauto.create.topics.enable=trueModifyingatopic

Deletingatopic:DON’Tin0.8.1.x!25$kafka-topics.sh--zookeeperzookeeper1:2181--create--topiczerg.hydra\

--partitions3--replication-factor2\

--configx=yPartitions26Atopicconsistsofpartitions.Partition:ordered+

immutablesequenceofmessages

thatiscontinuallyappendedtoPartitions27#partitionsofatopicisconfigurable#partitionsdeterminesmaxconsumer(group)parallelismCf.parallelismofStorm’sKafkaSpoutviabuilder.setSpout(,,N)ConsumergroupA,with2consumers,readsfroma4-partitiontopicConsumergroupB,with4consumers,readsfromthesametopicPartitionoffsets28Offset:messagesinthepartitionsareeachassignedaunique(perpartition)andsequentialidcalledtheoffsetConsumerstracktheirpointersvia(offset,partition,topic)tuplesConsumergroupC1Replicasofapartition29Replicas:“backups”ofapartitionTheyexistsolelytopreventdataloss.Replicasareneverreadfrom,neverwrittento.TheydoNOThelptoincreaseproducerorconsumerparallelism!Kafkatolerates(numReplicas-1)deadbrokersbeforelosingdataLinkedIn:numReplicas==21brokercandieTopicsvs.Partitionsvs.Replicas30

Inspectingthecurrentstateofatopic--describethetopicLeader:brokerIDofthecurrentlyelectedleaderbrokerReplicaID’s=brokerID’sISR=“in-syncreplica”,replicasthatareinsyncwiththeleader

Inthisexample:Broker0isleaderforpartition1.Broker1isleaderforpartitions0and2.Allreplicasarein-syncwiththeirrespectiveleaderpartitions.31$kafka-topics.sh--zookeeperzookeeper1:2181--describe--topiczerg.hydra

Topic:zerg2.hydraPartitionCount:3ReplicationFactor:2Configs:Topic:zerg2.hydraPartition:0Leader:1Replicas:1,0Isr:1,0Topic:zerg2.hydraPartition:1Leader:0Replicas:0,1Isr:0,1Topic:zerg2.hydraPartition:2Leader:1Replicas:1,0Isr:1,0Let’srecapThewhoiswhoProducerswritedatatobrokers.Consumersreaddatafrombrokers.Allthisisdistributed.ThedataDataisstoredintopics.Topicsaresplitintopartitionswhicharereplicated.32Puttingitalltogether33

Sidenote(opinion)DrawingaconceptuallinefromKafkatoClojure'score.asyncCf.talk"Clojurecore.asyncChannels",byRichHickey,at~31m54

34Part3:OperatingKafka35OverviewofPart3:OperatingKafkaKafkaarchitectureKafkahardwarespecsDeployingKafkaMonitoringKafkaKafkaappsKafkaitselfZooKeeper"Auditing"Kafka(not:securityaudit)P&StuningOps-relatedKafkareferences36KafkaarchitectureKafkabrokersYoucanrunclusterswith1+brokers.Eachbrokerinaclustermusthave

auniquebroker.id.37KafkaarchitectureKafkarequiresZooKeeperLinkedInruns(old)ZK3.3.4,

butlatest3.4.5works,too.ZooKeeperv0.8:usedbybrokersandconsumers,butnotbyproducers.Brokers:generalstateinformation,leaderelection,etc.Consumers:primarilyfortrackingmessageoffsets(cf.later)v0.9:usedbybrokersonlyConsumerswillusespecialKafkatopicsinsteadofZooKeeperWillsubstantiallyreducetheloadonZooKeeperforlargedeployments38Kafkabrokerhardwarespecs@LinkedInSolelydedicatedtorunningKafka,runnothingelse.1Kafkabrokerinstancepermachine2x4-coreIntelXeon(infooutdated?)64GBRAM(upfrom24GB)Only4GBusedforKafkabroker,remaining60GBforpagecachePagecacheiswhatmakesKafkafastRAID10with14spindlesMorespindles=higherdiskthroughputCacheonRAID,withbatterybackupBeforeH/Wupgrade:8xSATAdrives(7200rpm),notsureaboutRAID1GigE(?)NICsEC2example:m2.2xlarge@$0.34/hour,withprovisionedIOPS39ZooKeeperhardwarespecs@LinkedInZooKeeperserversSolelydedicatedtorunningZooKeeper,runnothingelse.1ZooKeeperinstancepermachineSSD’sdramaticallyimproveperformanceInv0.8.x,brokersandconsumersmusttalktoZK.Inlarge-scaleenvironments(manyconsumers,manytopicsandpartitions)thismeansZKcaneabottleneckbecauseitprocessesrequestsserially.AndthisprocessingdependsprimarilyonI/Operformance.1GigE(?)NICsZooKeeperinLinkedIn’sarchitecture5-nodeZKensembles=tolerates2deadnodes1ZKensembleforallKafkaclusterswithinadatacenterLinkedInrunsmultipledatacenters,withmultipleKafkaclusters40DeployingKafkaPuppetmodule

patible,rspectests,TravisCIsetup(e.g.totestagainstmultipleversionsofPuppetandRuby,Puppetstylechecker/lint,etc.)RPMpackagingscriptforRHEL6

DigitallysignedbyRPMisbuiltonaWirbelsturm-managedbuildserverPublic(Wirbelsturm)S3-backedyumrepo

41DeployingKafkaHieraexample42OperatingKafkaTypicaloperationstasksinclude:AddingorremovingbrokersExample:ensureanewlyaddedbrokeractuallyreceivesdata,whichrequiresmovingpartitionsfromexistingbrokerstothenewbrokerKafkaprovideshelperscripts(cf.below)butstillmanualworkinvolvedBalancingdata/partitionstoensurebestperformanceAddnewtopics,re-configuretopicsExample:Increasing#partitionsofatopictoincreasemaxparallelismAppsmanagement:newproducers,newconsumersSeeOps-relatedreferencesattheendofthispart43LessonslearnedfromoperatingKafkaatLinkedInBiggestchallengehasbeentomanagehypergrowthGrowthofKafkaadoption:moreproducers,moreconsumers,…Growthofdata:moreusers,moreuseractivity,…TypicaltasksatLinkedInEducatingandcoachingKafkausers.ExpandingKafkaclusters,shrinkingclusters.Monitoringconsumerapps–“Hey,mystuffstopped.Kafka’sfault!”44

KafkasecurityOriginaldesignwasnotcreatedwithsecurityinmind.DiscussionstartedinJune2014toaddsecurityfeatures.Coverstransportlayersecurity,dataencryptionatrest,non-repudiation,A&A,…See[DISCUSS]KafkaSecuritySpecificFeaturesAtthemomentthere'sbasicallynosecuritybuilt-in.45MonitoringKafka46MonitoringKafkaNothingfancybuiltintoKafka(e.g.noUI)butsee:

47KafkaOffsetMonitorKafkaWebConsoleMonitoringKafkaUseofstandardmonitoringtoolsmendedGraphitePuppetmodule:JavaAPI,alsousedbyKafka:JMX

CollectloggingfilesintoacentralplaceLogstash/KibanaandfriendsHelpswithtroubleshooting,debugging,etc.–notablyifyoucancorrelateloggingdatawithnumericmetrics48MonitoringKafkaappsAlmostallproblemsaredueto:ConsumerlagRebalancing<<<wecoverthislaterinpart449MonitoringKafkaapps:consumerlagLagisaconsumerproblemTooslow,toomuchGC,losingconnectiontoZKorKafka,…BugordesignflawinconsumerOperationalmistakes:e.g.youbroughtup6serversinparallel,eachoneinturntriggeringrebalancing,thenhitKafka'srebalancelimit;

cf.rebalance.max.retries(default:4)&friends50Broker(s)newProducerA1ProducerA2ProducerAn……OldermsgsNewermsgsConsumergroupC1Lag=howfaryourconsumerisbehindtheproducersMonitoringKafkaitself(1of3)Under-replicatedpartitionsForexample,becauseabrokerisdown.Meansclusterrunsindegradedstate.FYI:LinkedInrunswithreplicationfactorof2=>1brokercandie.OfflinepartitionsEvenworsethanunder-replicatedpartitions!Seriousproblem(dataloss)ifanythingbut0offlinepartitions.51MonitoringKafkaitself(1of3)DatasizeondiskShouldbebalancedacrossdisks/brokersDatabalanceevenmoreimportantthanpartitionbalanceFYI:Newscriptinv0.8.1tobalancedata/partitionsacrossbrokersBrokerpartitionbalanceCountofpartitionsshouldbebalancedevenlyacrossbrokersSeenewscriptabove.52MonitoringKafkaitself(1of3)LeaderpartitioncountShouldbebalancedacrossbrokerssothateachbrokergetsthesameamountofloadOnly1brokerisevertheleaderofagivenpartition,andonlythisbrokerisgoingtotalktoproducers+consumersforthatpartitionNon-leaderreplicasareusedsolelyassafeguardsagainstdatalossFeatureinv0.8.1toauto-rebalancetheleadersandpartitionsincaseabrokerdies,butitdoesnotworkthatwellyet(SRE'sstillhavetodothismanuallyatthispoint).NetworkutilizationMaxednetworkonereasonforunder-replicatedpartitionsLinkedIndon'trunanythingbutKafkaonthebrokers,sonetworkmaxisduetoKafka.Hence,whentheymaxthenetwork,theyneedtoaddmorecapacityacrosstheboard.53MonitoringZooKeeperEnsemble(=cluster)availabilityLinkedInrun5-nodeensembles=tolerates2deadTwitterrun13-nodeensembles=tolerates6deadLatencyofrequestsMetrictargetis0mswhenusingSSD’sinZooKeepermachines.Why?BecauseSSD’saresofasttheytypicallybringdownlatencybelowZK’smetricgranularity(whichisper-ms).OutstandingrequestsMetrictargetis0.Why?BecauseZKprocessesallingrequestsserially.Non-zerovaluesmeanthatrequestsarebackingup.54"Auditing"KafkaLinkedIn'swaytodetectdatalossetc.55“Auditing”KafkaLinkedIn'swaytodetectdatalossetc.inKafkaNotpartofopensourcestackyet.Maycomeinthefuture.Inshort:customproducer+consumerappthatishookedintomonitoring.ValuepropositionMonitorwhetheryou'relosingmessages/data.Monitorwhetheryourpipelinescanhandletheingdataload.56

LinkedIn'sAuditUI:afirstlookExample1:CountdiscrepancyCausedbymessagesfailingtoreachadownstreamKafkaclusterExample2:Loadlag57“Auditing”KafkaEveryproducerisalsowritingmessagesintoaspecialtopicabouthowmanymessagesitproduced,every10mins.Example:"Overthelast10mins,IsentNmessagestotopicX.”ThismetadatagetsmirroredlikeanyotherKafkadata.Auditconsumer1auditconsumerperKafkaclusterReadseverysinglemessageoutof“its”Kafkacluster.Itthencalculatescountsforeachtopic,andwritesthosecountsbackintothesamespecialtopic,every10mins.Example:"IsawMmessagesinthelast10minsfortopicXinTHIScluster”Andthenextauditconsumerinthenext,downstreamclusterdoesthesamething.58“Auditing”KafkaMonitoringauditconsumersCompletenesscheck"#msgsaccordingtoproducer==#msgsseenbyauditconsumer?"Lag"Cantheauditconsumerskeepupwiththeingdatarate?"Ifauditconsumersfallbehind,thenallyourtrackingdatafallsbehindaswell,andyoudon'tknowhowmanymessagesgotproduced.59“Auditing”KafkaAuditUIOnlyreadsdatafromthatspecial"metrics/monitoring"topic,butthisdataisreadsfromeveryKafkaclusteratLinkedIn.Whattheyproducerssaidtheywrotein.Whattheauditconsumerssaidtheysaw.Showscorrelationgraphs(producersvs.auditconsumers)Foreachtier,itshowshowmanymessagestherewereineachtopicoveranygivenperiodoftime.Percentageofhowmuchdatagotthrough(fromclustertocluster).Ifthepercentagedropsbelow100%,thenemailsaresenttoKafkaSRE+DEVaswellastheirHadoopETLteambecausethatstopstheHadooppipelinesfromfunctioningproperly.60LinkedIn'sAuditUI:aclosinglookExample1:CountdiscrepancyCausedbymessagesfailingtoreachadownstreamKafkaclusterExample2:Loadlag61Kafkaperformancetuning62OStuningKerneltuningDon’tswap!vm.swappiness=0(RHEL6.5onwards:1)Allowmoredirtypagesbutlessdirtycache.LinkedInhavelotsofRAMinservers,mostofitisforpagecache(60of64GB).Theyletdirtypagesbuiltup,butcacheshouldbeavailableasKafkadoeslotsofdiskandnetworkI/O.Seevm.dirty_*_ratio&friendsDiskthroughputLongercommitintervalonmountpoints.(ext3orext4?)Normalintervalforext3mountpointis30s(?)betweenflushes;LinkedIn:120s.Theycantoleratelosing2minsworthofdata(becauseofpartitionreplicas)sotheyratherpreferhigherthroughputhere.Morespindles(RAID10w/14disks)63Java/JVMtuningBiggestissue:garbagecollectionAnd,mostofthetime,theonlyissueGoalistominimizeGCpausetimesAka“stop-the-world”events–appsarehalteduntilGCfinishes64JavagarbagecollectioninKafka@Spotify65

BeforetuningAftertuningJava/JVMtuningGoodnews:useJDK7u51orlaterandhaveaquietlife!LinkedIn:OracleJDK,notOpenJDKSilverbulletisnewG1“garbage-first”garbagecollectorAvailablesinceJDK7u4.SubstantialimprovementoverallpreviousGC’s,atleastforKafka.66$java-Xms4g-Xmx4g-XX:PermSize=48m-XX:MaxPermSize=48m-XX:+UseG1GC-XX:MaxGCPauseMillis=20-XX:InitiatingHeapOccupancyPercent=35KafkaconfigurationtuningOftennotmuchtodobeyondusingthedefaults,yay.Keycandidatesfortuning:67num.io.threadsshouldbe>=#disks(starttestingwith==#disks)work.threadsadjustitbasedon(concurrent)#producers,#consumers,andreplicationfactorKafkausagetuning–lessonslearnedfromothersDon'tbreakthingsupintoseparatetopicsunlessthedatainthemistrulyindependent.Consumerbehaviorcan(andwill)beextremelyvariable,don’tassumeyouwillalwaysbeconsumingasfastasyouare

producing.Keeptimerelatedmessagesinthesamepartition.Consumerbehaviorcanextremelyvariable,don'tassumethelagonallyourpartitionswillbesimilar.Designapartitioningscheme,sothattheownerofonepartitioncanstopconsumingforalongperiodoftimeandyourapplicationwillbeminimallyimpacted(forexample,partitionbytransactionid)68

Ops-relatedreferencesKafkaFAQ

Kafkaoperations

Kafkasystemtools

Consumeroffsetchecker,getoffsetsforatopic,printmetricsviaJMXtoconsole,readfromtopicAandwritetotopicB,verifyconsumerrebalanceKafkareplicationtools

Caveat:Somesectionsofthisdocumentareslightlyoutdated.Controlledshutdown,preferredleaderelectiontool,reassignpartitionstoolKafkatutorial

69Part4:DevelopingKafkaapps70OverviewofPart4:DevelopingKafkaappsWritingdatatoKafkawithproducersExampleproducerProducertypes(async,sync)MessageackingandbatchingofmessagesWriteoperationsbehindthescenes–caveatsahead!ReadingdatafromKafkawithconsumersHigh-levelconsumerAPIandsimpleconsumerAPIConsumergroupsRebalancingTestingKafkaSerializationinKafkaDatacompressioninKafkaExampleKafkaapplicationsDev-relatedKafkareferences71WritingdatatoKafka72WritingdatatoKafkaYouuseKafka“producers”towritedatatoKafkabrokers.AvailableforJVM(Java,Scala),C/C++,Python,Ruby,etc.TheKafkaprojectonlyprovidestheJVMimplementation.HasriskthatanewKafkareleasewillbreaknon-JVMclients.

Asimpleexampleproducer:Fulldetailsat:

73ProducersTheJavaproducerAPIisverysimple.We’lltalkabouttheslightlyconfusingdetailsnext.74ProducersTwotypesofproducers:“async”and“sync”SameAPIandconfiguration,butslightlydifferentsemantics.Whatappliestoasyncproduceralmostalwaysappliestoasync,too.Asyncproducerispreferredwhenyouwanthigherthroughput.Importantconfigurationsettingsforeitherproducertype:75client.ididentifiesproducerapp,e.g.insystemlogsproducer.typeasyncorsyncrequest.required.acksackingsemantics,cf.nextslidesserializer.classconfigureencoder,cf.slidesonAvrousagemetadata.broker.listcf.slidesonbootstrappinglistofbrokersSyncproducersStraight-forwardsoIwon’tcoversyncproducersherePleasegotoMostimportantthingtoremember:producer.send()willblock!76AsyncproducerSendsmessagesinbackground=noblockinginclient.Providesmorepowerfulbatchingofmessages(seelater).Wrapsasyncproducer,orratherapoolofthem.Communicationfromasync->syncproducerhappensviaaqueue.Whichexplainswhyyoumayseeducer.async.QueueFullExceptionEachsyncproducergetsacopyoftheoriginalasyncproducerconfig,includingtherequest.required.ackssetting(seelater).Implementationdetails:Producer,async.AsyncProducer,async.ProducerSendThread,ProducerPool,async.DefaultEventHandler#send()77AsyncproducerCaveatsAsyncproducermaydropmessagesifitsqueueisfull.Solution1:Don’tpushdatatoproducerfasterthanitisabletosendtobrokers.Solution2:Queuefull==needmorebrokers,addthemnow!Usethissolutioninfavorofsolution3particularlyifyourproducercannotblock(asyncproducers).Solution3:Setqueue.enqueue.timeout.msto-1(default).Nowtheproducerwillblockindefinitelyandwillneverwillinglydropamessage.Solution4:Increasequeue.buffering.max.messages(default:10,000).In0.8anasyncproducerdoesnothaveacallbackforsend()toregistererrorhandlers.Callbackswillbeavailablein0.9.78ProducersTwoaspectsworthmentioningbecausetheysignificantlyinfluenceKafkaperformance:

MessageackingBatchingofmessages791)MessageackingBackground:InKafka,amessageisconsideredcommittedwhen“anyrequired”ISR(in-syncreplicas)forthatpartitionhaveappliedittotheirdatalog.Messageackingisaboutconvey