3-6-OS 造成的長(zhǎng)時(shí)間非典型 JVM GC 停頓：深度分析和解決-莊振運(yùn)

OS-causedLongJVMPauses-DeepDiveandSolutionsZhenyunZhuang/in/zhenyunLinkedInCorp/in/zhenyunZhenyun@2016-4-212016-4-213Lessonslearned4§Javaispopularinproductiondeployments§LinuxfeaturesinteractwithJVMoperations§Uniquechallengescausedbyconcurrentapplications口LongJVMpausescausedbyLinuxOS§Productionissues,inthreescenarios§Rootcauses§Solutions§EnsuringHigh-performanceofMission-criticalJavaApplicationsinMulti-tenantCloudPlatforms,IEEECloud2014§EliminatingLargeJVMGCPausesCausedbyBackgroundIOTrafic,LinkedInEngineeringBlog,2016(Toomanytweetsbringingdownatwitterserver!:)5 §OracleHotSpotJVM§Generations§Garbagecollectors§Paging(Regularpage,Hugepage)§Swapping(Anonymousmemory)§Pagecachewriteback(Batched,Periodic)6Scenarios§Startupstate§Steadystatewithmemorypressure§SteadystatewithheavyIO§Javaapplicationkeepsallocating/de-allocatingobjects§BackgroundapplicationstakingmemoriesorissuingdiskIO§Applicationthroughput(Kallocations/sec)§JavaGCpauses7Cpausesspikeduringthesameperiod8JavaheapisgraduallyallocatedWithoutenoughmemory,JavaheapisgraduallyallocatedWithoutenoughmemory,directpagescanningcanhappenHeapisswappedoutandinItcauseslargeGC9SolutionsJVM“-XX:AlwaysPreTouch”apspacesfrombeingswappedout?=0forkernelversionbefore2.6.32-303?=1forkernelversionfrom2.6.32-303tionsPreallocatingHeapingHeapmreal=54.83secs]口However,swappingisnotenough[real=54.83secs]口However,swappingisnotenough[Times:user=0.12sys=54.67,口DuringGCpauses,swappingactivitiespersist口SwappinginJVMpagescausesGCpauses§ExcessiveGCpauses(i.e.,55seconds)§Highsys-cpuusage(swappingisnotsys-cpuintensive)4KB4KB4KB4KB4KB4KB4KB4KB4KB4KB4KB4KBPages(THP)22MB§ImprovedTLBcache-hitss§THPsareallocatedfirst,butsplitduringmemorypressure§RegularpagesarecollapsedtomakeTHPs§CPUheavy,andthrashing!SolutionsrsationsDynamicTHP§DynamicTHPdeliverssimilarperformanceasTHPisonMechanismTHPOffTHPOnDynamicTHPThroughput(Kallocations/sec)§DynamicTHPhassomeperformanceoverhead§PerformanceislessthanTHP-off§ButbetterthanTHP-onMechanismTHPOffTHPOnDynamicTHPThroughput(Kallocations/sec)口Productionissue§Onlineproducts§Applicationshavelightworkload§BothCMSandG1garbagecollectors口Preliminaryinvestigations§Examinedmanylayers/metrics§Theonlysuspect:diskIOoccasionallyisheavy§ButallapplicationIOareasynchronouslem口WorkloaddNosinglelonger-than-200mspauseCaseII:WithbackgroundIO TimelinesAttime35.04(line2),ayoungGCstartsandtakes0.12secondstocomplete.tics5)tTheyoungGCfinishesattime35.16andJVMtriestooutputtheyoungGCstatiogclogfilebyissuingawrite()systemcalltics5)tThewrite()callfinishesattime36.64afterbeingblockedfor1.47seconds(lineWhenwrite()callreturnstoJVM,JVMrecordsattime36.64thisSTWpauseofseconds(i.e.,0.12+1.47)(line3).InteractionbetweenJVMandOSNon-blockingIOcanbeblocked§Forfile-backedwriting,OSwritestopagecachefirst§OShaswrite-backmechanismtopersistdirtypages§Ifapageisunderwrite-back,thepageislocked§Journalsaregeneratedforjournalingfilesystem§WhenappendingGClogfilesneedsnewblocks,journalsneedtobecommitted§CommitmentmightneedtowaitBackgroundIOactivities口OSactivitysuchasswapping§Datawritingtounderlyingdisks口Administrationandhousekeepingsoftware§System-levelsoftwaresuchasCFEnginealsoperformdiskIO口Otherco-locatedapplications§Co-locatedapplicationsthatsharethediskdrives,thenotherapplicationscontendonIO口IOofthesameJVMinstance§TheparticularJVMinstancemayusediskIOinwaysotherthanGCloggingSolutionsThegood,thebad,andtheugly口Thegood:lowrealtime§Lowusertimeandlowsystime§[user=0.18sys=0.01,real=0.04secs]口Thebad:non-low(butnothigh)realtime§Highusertimeandlowsystime§[user=8.00sys=0.02,real=0.50secs]口Theugly:highrealtime§Highsystime[u