2015pg大會(huì)內(nèi)容-源代碼主題05postgresql checkpoint源碼分析

PostgreSQLcheckpoint源碼分析-2015-05-0710:53:59|作者：|分類：PgSQL2015年度PG大象會(huì)報(bào)名地址：PostgreSQL中國(guó)社區(qū)： PostgreSQL用戶 *Flush*Flushalldirtyblocksinbufferpooltodiskatcheckpoint*Note:temporaryotparticipateincheckpoints,so **needtobeCheckPointBuffers(int{ //bufferCheckpointStats.ckpt_write_t= CheckpointStats.ckpt_sync_t=// syncCheckpointStats.ckpt_sync_end_t= buffercheckpoint}BufferSyncbuffercheckpointflush的checkpointFLUSH但是需要注意，第一次設(shè)置為needcheckpoint的塊，有一個(gè)計(jì)數(shù)，第二次在刷*BufferSync--Write*BufferSync--Writeoutalldirtybuffersinthe*ThisiscalledatcheckpointtimetowriteoutalldirtysharedThecheckpointrequestflagsshouldbepassed Ifisset,wedisabledelaysbetweenwrites;ifCHECKPOINT_END_OF_RECOVERYorCHECKPOINT_FLUSH_ALLisset,wewriteunloggedbuffers,whichareotherwise TheremainingcurrentlyhavenoeffectstaticBufferSync(int{ mask /*MakesurewecanhandlethepininsideSyncOneBufferUnlessthisisashutdowncheckpointorwehavebeenwewriteonlypermanent,dirty Butatshutdownorendrecovery,wewritealldirtyif(!((flags&(CHECKPOINT_IS_SHUTDOWN|CHECKPOINT_END_OF_RECOVERY)mask|= Loopoverallbuffers,andmarktheonesthatneedtobe Countthemaswego(num_to_write),socanestimatehowmuchworkneedstobeThisallowsustowriteonlythosepagesthatweredirtywhencheckpointbegan,andnotthosethatgetdirtiedwhileitWheneverapagewithBM_CHECKPOINT_NEEDEDiswrittenout,eitherlaterinthisfunction,orbynormalbackendsorthescan,theflagis AnybufferdirtiedafterthishavetheflagNotethatifwefailtowritesomebuffer,wemayleaveBM_CHECKPOINT_NEEDEDstill ThisisOKsinceanysuchcertainlyneedtobewrittenforthenextcheckpointattempt,num_to_write //BM_CHECKPOINT_NEEDEDforbuf_id0;buf_idNBuffers; {volatileBufferDesc*bufHdr=HeaderspinlockisenoughtoexamineBM_DIRTY,comment ifbufHdr->flags&mask) {bufHdr->flags|=}}if(num_to_write== nothingtodoTRACE_POSTGRESQL_BUFFER_SYNC_START(NBuffers, Loopoverallbuffersagain,andwritetheones(still)marked Inthisloop,westartattheclocksincewemightaswelldumpsoon-to-be-recycledbuffersNotethatwe'treadthebufferalloccountherethatleftuntouchedtillthenextBgBufferSync()buf_id=StrategySyncStart(NULL,num_to_scan=num_written=while(num_to_scan syncbuffer{volatileBufferDesc*bufHdr=only

* 'tneedtoacquirethelockhere,becauseatasinglebit.It'spossiblethatsomeoneelsetheandclearstheflagrightafterwecheck,butdoesn'tsinceSyncOneBufferwillthen thereisfurtherracecondition:it'sconceivablethatthetimeexaminethebithereandthetimeacquiressomeoneelsenotonlywrotethebufferbutreplacedwithpageanddirtied InthatimprobableSyncOneBufferwritethebufferthoughwedidn'tneed Itseemguardingagainstthis,if(bufHdr->flags& {if(SyncOneBuffer(buf_id,false) SyncOneBuffer{ Weknowthereareatnum_to_writebuffersBM_CHECKPOINT_NEEDEDset;socanstopscanningnum_writtenNotethatnum_writtenincludebufferswrittenotherbackends,orbybgwritercleaningscan.meansthattheestimateofmuchprogresswe'vemadeconservative,andalsothistestwilloftenfail Butitseemsmakingif(num_written SleeptothrottleourI/O(double)num_written （num_to_write），100（num_written） 0.1);,假 里,progress pletionTarget;=0.1*0.5= elapsed_xlogs=(recptr-ckpt_start_recptr))/XLogSegSize)/ progress progress0.5，num_writtennum_to_write1,10.5 }}if(++buf_id>=buf_id=}Updatecheckpointstatistics.Asnotedabove,thisdoesn'tbufferswrittenbyotherbackendsorbgwriterCheckpointStats.ckpt_bufs_written+= E(NBuffers, BM_CHECKPOINT_NEEDEDflush}*SyncOneBuffer--processasinglebufferduring**Ifskip_recently_usedistrue,'twritecurrently-pinnedbuffers,buffersmarkedrecentlyused,asthesearenotreplacement*Returnsabitmaskcontainingthefollowingflag*BUF_WRITTEN:wewrotethe*BUF_REUSABLE:bufferisavailableforreplacement,ie,it*pincount0andusagecount*(BUF_WRITTENcouldbesetinerrorifFlushBuffersfindsthebufferafterlockingit,but 'tcareallthatNote:callermust estaticintSyncOneBuffer(intbuf_id,bool{volatileBufferDesc*bufHdr=result=Checkwhetherbufferneeds*Wecanmakethischeckwithouttakingthebuffercontentlock*aswemarkpagesdirtyinaccessmethods*before*loggingXLogInsert():ifsomeonemarksthebufferdirtyjustaftercheck'tworrybecauseourcheckpoint.redopointsbeforelogingchangesandsowearenotrequiredtowritesuchif(bufHdr->refcount==0&&bufHdr->usage_count==result|=elseif{/*Callertoldusnottowriterecently-usedbuffersreturn}if(!(bufHdr->flags&BM_VALID)||!(bufHdr->flags&{/*It'sclean,sonothingtodoreturn}

*Pinit,share-lockit,write (FlushBuffer othing*bufferiscleanbythetimewe'velockedLWLockAcquire(bufHdr->content_lock,FlushBuffer(bufHdrFlushBuffer(bufHdr UnpinBuffer(bufHdr,true);returnresult|}FlushBufferBUFFERcheckpointWAL前，必須寫到磁盤。**Physicallywriteoutashared*NOTE:thisactuallyjustpassesthebuffercontentstothekernel;realwritetodiskwon'thappenuntilthekernelfeelslike isokayfromourpointofviewsincewecanredothechangesfromHowever,wewillneedtoforcethechangestodiskviafsyncwecancheckpoint checkpointWAL，bufferThecallermustholdapinonthebufferandhaveshare-lockedbuffer (Note:ashare-lockdoesnotpreventupdateshintbitsinthebuffer,sothepagecouldchangewhiletheisinprogress,butweassumethatthatwillnotinvalidatetheIfthecallerhasansmgrreferenceforthebuffer'srelation,passasthesecond Ifnot,passstaticFlushBuffer(volatileBufferDesc*buf,SMgrRelation{ ErrorContextCallback Acquirethebuffer'sio_in_progress Iffalse,thensomeoneelseflushedthebufferbeforewecould,sonotdoif(!StartBufferIO(buf,/*Setuperrortracebacksupportforereport()errcallback.callback=errcallback.arg=(void*)errcallback.previous=error_context_stack=/*Findsmgrrelationforbufferif(reln==reln=smgropen(buf->tag.rnode, RunPageGetLSNwhileholdingheaderlock,sincewe'thavebufferlockedexclusivelyinallrecptr BUFFER/*Tocheckifblockcontentchangeswhileflushing.-vadimbuf->flags&=ForceXLOGflushuptobuffer's Thisimplementsthe XLOGbufferlsnrulethatlogupdatesmusthitdiskbeforeanyofthedata- XLOGtheydescribeHowever,thisruledoesnotapplytounloggedrelations,whichlostafteracrash MostunloggedrelationpagesLSNssinceweneveremitWALrecordsforthem,andupthroughthebufferLSNwouldbeuseless, GiSTindexesuseLSNsinternallytotracksplits,unloggedGiSTpagesbear"fake"LSNsgenerated ItisunlikelybutpossiblethatLSNcountercouldadvancepasttheWALinsertionpoint;andifhappen,attemptingtoflushWALthroughthatlocationwoulddisastroussystem-wide Tomakesurethatskiptheflushifthebufferisn'tif(buf->flags&Nowit'ssafetowritebuffertodisk.Notethatnooneelsehavebeenabletowriteitwhilewewerebusywithlogwehavetheio_in_progressbufBlock=Updatepagechecksumif Sincewehaveonlysharedonbuffer,otherprocessesmightbeupdatinghintbitsinit,socopythepagetoprivatestorageifwedobufToWrite=PageSetChecksumCopy((Page)bufBlock,buf-ifbufToWriteiseitherthesharedbufferoracopy,as if{INSTR_TIME_SUBTRACT(io_time,INSTR_TIME_ADD(pgBufferUsage.blk_write_time,}Markthebufferasclean(unlessBM_JUST_DIRTIED eendtheio_in_progressTerminateBufferIO(buf,true, //單bufferflush/*Poptheerrorcontextstackerror_context_stack=}*smgrwrite()--*smgrwrite()--Writedbuffer**Thisistobe lyforupdatingalready-blocksof*relation(ie,thosebeforethecurrent Toextend*use**Thisisnotasynchronouswrite--theblockis*ondiskatreturn,onlydumpedoutto *provisionswillbemadetofsyncthewritebeforethe**skipFsyncindicatesthatthecallerwillmakeprovisions**fsynctherelation,soweneedn't relations*otrequiresmgrwrite(SMgrRelationreln,ForkNumberforknum,BlockNumberchar*buffer,bool{(*(smgrsw[reln->smgr_which].smgr_write))(reln,forknum,buffer,}*smgrsync()--Sync*smgrsync()--Syncfilestodiskduring{for(i=0;i<NSmgr;{if(*(smgrsw[i].smgr_sync))}}smgr_sync mdsync()--Syncpreviouswritestostable{staticboolmdsync_in_progress=HASH_SEQ_STATUSPendingOperationEntry /*Statisticsonsynctimes processed= longest= total_elapsed=Thisisonlycalledduringcheckpoints,andcheckpoints occurinprocessesthathavecreatedaifelog(ERROR,"cannotsyncwithoutaIfweareinthecheckpointer,thesynchadbetterincluderequeststhatwerequeuedbybackendsuptothis raceconditionthatcouldoccuristhatabufferthatmustandfsync'dforthecheckpointcouldhavebeendumpedbyabeforeitwasvisitedby Weknowthebackendqueuedanfsyncrequestbeforeclearingthebuffer'sdirtybit,aresafeaslongaswedoanAbsorbaftercompletingToavoidexcessfsync'ing(intheworstcase,maybeacheckpoint),wewanttoignorefsyncrequeststhatareentered

hashtableafterthispointtheyshouldbeprocessednext Weusemdsync_cycle_ctr loldentriesapartones:newoneswillhavecycle_ctrequaltotheincrementedInnormalcircumstances,allentriespresentinthetableatwillhavecycle_ctrexactlyequaltothecurrent(abouttobevalueof However,ifwefailpartwayfsync'ingloop,thenoldervaluesofcycle_ctrmightremainwhencomebackheretotry Repeatedcheckpointfailureseventuallywrapthecounteraroundtothepointwhereanoldmightappearnew,causingustoskipit,possiblyallowingtosucceedthatshouldnot Toforestallwraparound,timepreviousmdsync()failedtocomplete,runthroughthetableforciblysetcycle_ctr=Thinknottomergethisloopwiththemainloop,astheproblemexactlythatthatloopmayfailbeforehavingvisitedallFromaperformancepointofviewitdoesn'tmatteranyway,aswillneverbetakeninasystemthat'sfunctioningif{/*priortryfailed,soupdateanystalecycle_ctrhash_seq_init(&hstat,while((entry=(PendingOperationEntryhash_seq_search(&hstat))!={entry->cycle_ctr=}}/*Advancecountersothatnewhashtableentriesaredistinguishable/*Setflagtodetectfailureifwe'treachoftheloop*/mdsync_in_progress=true;/*Nowscanthehashtableforfsyncrequeststoprocessabsorb_counter=hash_seq_init(&hstat,while((entry=(PendingOperationEntry*)hash_seq_search(&hstat)){

Iftheentryisnew 'tprocessitthistime;containmultiplefsync-requestbits,buttheyare "continue"bypassesthehash-removecallattheoftheif(entry->cycle_ctr==/*Elseassertwehaven'tmisseditAssert((CycleCtr)(entry->cycle_ctr+1)ScanovertheforksandsegmentsrepresentedbyThebitmapmanipulationsareslightlytricky,wecanAbsorbFsyncRequests()insidetheloopandthatresultbms_add_member()modifyingandevenre-palloc'ingThisisokaybecauseweunlinkeachbitmapsetfromentrybeforescanning Thatmeansthatingrequestswillbeprocessednowiftheyreachthebeforebegintoscantheirfor(forknum=0;forknum<=MAX_FORKNUM;{ *requests entry->requests[forknum]=entry->canceled[forknum]=while((segno=bms_first_member(requests)){ havetobotheropening

Iffsyncisoffthen fileat (WecheckinguntilthispointsochangingfsynconthebehavesifIfincheckpointer,wewantabsorbpendingeverysooftentooverflowofthefsync Itiswhethernewly-addedentriesbutwe'tcaresince

bevisitedby'tneedtoprocessif(--absorb_counter<={absorb_counter}Thefsynctablecouldrequeststofsyncthathavebeen(unlinked)bythetimewegetthem.RatherthanjustanENOENT(orEACCESWindows)errorcanbewhatwedoonerrorabsorbpendingrequeststhen Sincequeuesa"cancel"beforeactuallyunlinking,fsyncrequestisguaranteedbemarkedcanceledafterabsorbifitreallywascase.DROPDATABASEhastolustoforgetrequestsbeforeitfor(failures=0;;failures++)loopexitsat"break"{SMgrRelation FindorcreatesmgrhashentryforthisThismayseemaunclean--mdcalling it'sreallybest Itensuresthatopenfileisn'tpermanentlyleakedifweanerrorhere.maysay"butanSMgrRelationstillaleak!"Notreally,becauseonlycaseinacheckpoint ebyathatisn'taboutshutdownisintheanditperiodicallydosmgrcloseall().Thisjustifiesourclosingtherelninthesuccesseither,whichisgoodthingsinceinnon-caseswesafelydorelnsmgropen(entry->rnode,/*Attempttoopenfsyncthetargetsegmentseg_mdfd_getseg(reln,(BlockNumber)segno*(BlockNumber)false,if(seg!=NULLg->mdfd_vfd)>={/*updatestatisticsaboutsynctimingsync_diffSUBTRACT(sync_diff,elapsed(elapsed>longest=ed+=;elog(DEBUG1,"checkpointsync:number=%dfile=%stime=%.3f(double)elapsed/ outofretryloop}/*Computefileforuseinmessagesave_er