LecNote-8-并行計(jì)算中的通信：同步與數(shù)據(jù)傳輸

第八講并行計(jì)算中的通信：同步與數(shù)據(jù)傳輸并行計(jì)算中的通信：communication消息交換：messageexchanging數(shù)據(jù)傳輸：datatransmittingCELLBE的消息傳輸機(jī)制：郵箱MailboxCELLBE的數(shù)據(jù)傳輸機(jī)制：DMA并行計(jì)算中的通信并行計(jì)算：多個處理器/執(zhí)行內(nèi)核協(xié)作，共同完成一個問題的求解，每個處理器/執(zhí)行內(nèi)核執(zhí)行其中的一個子任務(wù)處理器/執(zhí)行內(nèi)核在完成一個子任務(wù)時，需要解決算法設(shè)計(jì)方面的問題子任務(wù)是什么？它涉及哪些數(shù)據(jù)、有多大的規(guī)模這個子任務(wù)與整個問題、其它子任務(wù)的關(guān)系是什么樣的？算法實(shí)現(xiàn)方面的問題所涉及數(shù)據(jù)的存儲空間能夠被直接訪問嗎？如果不能夠，如何將數(shù)據(jù)傳輸?shù)教幚砥?執(zhí)行內(nèi)核能夠直接訪問的存儲空間？以求素?cái)?shù)問題為例，子任務(wù)I：計(jì)算[baseScope+chunkSize*IbaseScope+chunkSize*(I+1))間的素?cái)?shù)已經(jīng)求得[2baseScope]范圍內(nèi)的素?cái)?shù)，存儲在primes[]中數(shù)組primes[]中，第I個子任務(wù)的結(jié)果要存儲在第I+1個子任務(wù)的結(jié)果之前于是，需要解決同步問題：每個子任務(wù)分別交給一個處理器/執(zhí)行內(nèi)核子任務(wù)的描述問題：primes[]的地址、baseScope、chunkSize、第I個子任務(wù)的結(jié)果在primes[]中起始下標(biāo)數(shù)據(jù)訪問問題：primes[]的元素訪問并行計(jì)算對通信的需求同步：任務(wù)分配，實(shí)現(xiàn)負(fù)載平衡；滿足進(jìn)度要求(數(shù)據(jù)相關(guān))，保證結(jié)果唯一、正確求素?cái)?shù)問題：子任務(wù)I+1需要子任務(wù)I的結(jié)果(求得的素?cái)?shù)總數(shù))，確定自己結(jié)果在primes[]中的下標(biāo)其始位置被交換數(shù)據(jù)(規(guī)模、存儲地址)的描述被交換的數(shù)據(jù)值問題數(shù)據(jù)子集1數(shù)據(jù)子集2數(shù)據(jù)子集1處理器/執(zhí)行內(nèi)核1的直接存儲空間數(shù)據(jù)子集2處理器/執(zhí)行內(nèi)核2的直接存儲空間同步：誰做子任務(wù)1、子任務(wù)2；誰可以讀寫問題空間的數(shù)據(jù)；何時可以開始做了源地址、目的地址、規(guī)模源地址、目的地址、規(guī)模并行計(jì)算中通信的分類實(shí)現(xiàn)并行計(jì)算的算法邏輯子任務(wù)的同步信息子任務(wù)的描述信息實(shí)現(xiàn)并行計(jì)算的問題域數(shù)據(jù)存儲與訪問通信在共享存儲并行計(jì)算機(jī)中的實(shí)現(xiàn)共享存儲并行計(jì)算機(jī)：基于Cache的multi-core處理器、SMP、基于軟件的NUMA并行計(jì)算機(jī)(COW)等編程模型：POSIX線程庫、OpenMP等通信的實(shí)現(xiàn)實(shí)現(xiàn)并行計(jì)算的算法邏輯子任務(wù)的同步信息，提供互斥鎖機(jī)制，實(shí)現(xiàn)臨界區(qū)、保證操作的原子性信號量機(jī)制，實(shí)現(xiàn)數(shù)據(jù)依賴關(guān)系、保證結(jié)果的確定性子任務(wù)的描述信息：提供全局存儲空間，用專門的變量存儲，所有處理器/執(zhí)行內(nèi)核均可直接訪問實(shí)現(xiàn)并行計(jì)算的問題域數(shù)據(jù)存儲與訪問：提供全局存儲空間，所有處理器/執(zhí)行內(nèi)核均可直接訪問通信在CELLBE中的實(shí)現(xiàn)：MFC實(shí)現(xiàn)并行計(jì)算的算法邏輯：郵箱Mailbox子任務(wù)的同步信息子任務(wù)的描述信息實(shí)現(xiàn)并行計(jì)算的問題域數(shù)據(jù)存儲與訪問：DMA數(shù)據(jù)存取CELLBE中的存儲結(jié)構(gòu)Threetypesofstoragedomainsmain-storagedomain,8SPElocalstoredomains8SPEchanneldomainsThemain-storagedomain,whichistheentireeffective-addressspace,canbeconfiguredbythePPEoperatingsystemtobesharedbyallprocessorsandmemory-mappeddevicesinthesystem(allI/Oismemory-mapped)Local-storageandchannelproblem-state(user-state)domainsareprivatetotheSPU,LS,andMFCofeachSPEMFCCommandsMainmechanismforSPUstoaccessmainstorage(DMAcommands)maintainsynchronization

withotherprocessorsanddevicesinthesystem(Synchronizationcommands)CanbeissuedeitherSPUviaitsMFCbyPPEorotherdevice,asfollows:CoderunningontheSPUissuesanMFCcommandbyexecutingaseriesofwritesand/orreadsusingchannelinstructions-readchannel(rdch),writechannel(wrch),andreadchannelcount(rchcnt).CoderunningonthePPEorotherdevicesissuesanMFCcommandbyperformingaseriesofstoresand/orloadsto

memory-mappedI/O

(MMIO)registersintheMFCMFCcommandsarequeuedinoneoftwoindependentMFCcommandqueues:MFCSPUCommandQueue—Forchannel-initiatedcommandsbytheassociatedSPUMFCProxyCommandQueue—ForMMIO-initiatedcommandsbythePPEorotherdeviceCommunicationBetweenthePPEandSPEsThreeprimarycommunicationmechanismsbetweenthePPEandSPEsMailboxesQueuesforexchanging32-bitmessagesTwomailboxes(theSPUWriteOutboundMailboxandtheSPUWriteOutboundInterruptMailbox)areprovidedforsendingmessagesfromtheSPEtothePPEOnemailbox(theSPUReadInboundMailbox)isprovidedforsendingmessagestotheSPESignalnotificationregistersEachSPEhastwo32-bitsignal-notificationregisters,eachhasacorrespondingmemory-mappedI/O(MMIO)registerintowhichthesignal-notificationdataiswrittenbythesendingprocessorSignal-notificationchannels,orsignals,areinbound(toanSPE)registersTheycanbeusedbyotherSPEs,thePPE,orotherdevicestosendinformation,suchasabuffer-completionsynchronizationflag,toanSPEDMAsTotransferdatabetweenmainstorageandtheLSPPEcommunicateswithSPEsthroughMMIOregisterssupportedbytheMFCofeachSPESoftwareontheSPE’sSPUinteractswiththeMFCthroughchannels,whichenqueueDMAcommandsandprovideotherfacilities,suchasmailboxes,signalnotification,andaccessauxiliaryresourcesMFCDMACommandsMFCcommandsthattransferdataarereferredtoasDMAcommandsAnSPEorPPEperformsdatatransfersbetweentheSPE’sLSandmainstorageprimarilyusingDMAtransferscontrolledbytheMFCDMAcontrollerforthatSPEEachMFCcanmaintainandprocessmultiplein-progressDMAcommandrequestsandDMAtransfersTheMFCcanautonomouslymanageasequenceofDMAtransfersinresponsetoaDMA-listcommandMemory-mappedmailboxesoratomicMFCsynchronizationcommandscanbeusedforsynchronizationandmutualexclusion.DMAtransferrequestscontainbothanlocalstoreaddressandaneffectiveaddressThus,theycanaddressbothanSPE’sLSandmainstorageandtherebyinitiateDMAtransfersbetweenthedomainsData-transferdirectionforDMAcommandsisalwaysreferencedfromtheperspectiveofanSPEget:transferdataintoanSPE(frommainstoragetolocalstore)put:transferdataoutofanSPE(fromlocalstoretomainstorage)PPE和SPE要使用不同的API接口來實(shí)現(xiàn)DMA傳輸和DMA列表傳輸。PPE使用libspe

提供的API接口SPE有兩種API接口可以使用MFC應(yīng)用API接口DMA復(fù)合內(nèi)控指令A(yù)PI接口SPEProxy命令A(yù)PI接口使用者：PPE這些函數(shù)是由libspe

提供的，它提供了PPE發(fā)起的DMA傳輸功能，它利用SPEMFCProxy命令發(fā)送機(jī)制。PPE線程可以使用這些函數(shù)訪問SPE的本地存儲spe_mfcio_get()spe_mfcio_getb()spe_mfcio_getf()spe_mfcio_put()spe_mfcio_putb()spe_mfcio_putf()要使用這些API，必須包含頭文件libspe2.h注意這些API名字是基于SPE為中心的視角而命名的put意味著從SPE到系統(tǒng)內(nèi)存的數(shù)據(jù)傳輸get則是相反方向的傳輸MFC應(yīng)用API接口這些函數(shù)是為了編程方便而提供的，不是必需的。這些函數(shù)通過宏定義或者編譯器內(nèi)置函數(shù)方式實(shí)現(xiàn)。供SPE發(fā)起DMA數(shù)據(jù)傳輸時使用DMA傳輸APImfc_put()mfc_putb()mfc_putf()mfc_get()mfc_getb()mfc_getf()DMA列表傳輸APImfc_putl()mfc_putlb()mfc_putlf()mfc_getl()mfc_getlb()mfc_getlf()要使用這些API，必須包含頭文件spu_mfcio.h注意這些API名字是基于SPE為中心的視角而命名的put意味著從SPE到系統(tǒng)內(nèi)存的數(shù)據(jù)傳輸get則是相反方向的傳輸DMA復(fù)合內(nèi)控指令A(yù)PI接口使用者：SPE這些DMA復(fù)合內(nèi)控指令是由一系列底層內(nèi)控指令構(gòu)成，主要使用了通道控制的內(nèi)控指令。不需要頭文件spu_mfcdma32()spu_mfcdma64()DMAExample:ReadintoLocalStoreinlinevoiddma_mem_to_ls(unsigned

int

mem_addr,volatilevoid*ls_addr,unsignedintsize){unsignedinttag=0;unsignedintmask=1;mfc_get(ls_addr,mem_addr,size,tag,0,0);mfc_write_tag_mask(mask);mfc_read_tag_status_all();}Readcontentsofmem_addrintols_addrSettagmaskWaitforalltagDMAcompletedDMAExample:WritetoMainMemoryinlinevoiddma_ls_to_mem(unsigned

int

mem_addr,volatilevoid*ls_addr,unsignedintsize){unsignedinttag=0;unsignedintmask=1;mfc_put(ls_addr,mem_addr,size,tag,0,0);mfc_write_tag_mask(mask);mfc_read_tag_status_all();}Writecontentsofls_addrintomem_addrSettagmaskWaitforalltagDMAcompletedMFCDMA語義：以SPU端的mfc_get、mfc_put為例DMAgetfrommainmemoryintolocalstore(void)mfc_get(volatilevoid*ls,uint64_tea,uint32_tsize,uint32_ttag,uint32_ttid,uint32_trid)DMAputintomainmemoryfromlocalstore(void)mfc_put(volatilevoid*ls,uint64_tea,uint32_tsize,uint32_ttag,uint32_ttid,uint32_trid)ls=targetaddressinSPUlocalstoreforfetcheddata(SPUlocaladdress)ea=effectiveaddressfromwhichdataisfetched(globaladdress)size=transfersizeinbytestag=tag-groupidentifiertid=transfer-classidrid=replacement-classidDMAreadandwritecommandsarenon-blockingReplacementClassIDandTransferClassIDtransfer-classid：allowsapplicationsoftwaretoinfluencetheallocationofbusbandwidthforanMFCcommandreplacement-classid：allowsprivilegedsoftwaretoinfluenceL2-cacheandtranslationlookasidebuffer(TLB)replacementforcachemissescausedbytheMFCcommand.ThedefaultclassID(‘0’)isusedforallundefinedorinvalidclassIDs.AninvalidclassIDdoesnotgenerateanexception.DMA-CommandTagGroupsEachDMAcommandhasa5-bittagcommandswithsametagvalueforma“taggroup”Tagmaskisusedtoidentifytaggroupsforstatuschecks:a32-bitword,eachbitinthetagmaskcorrespondstoaspecifictagidtag_mask=(1<<tag_id)Tags,taggroups,andtagmasksusedfor:checkingstatusofDMAcommandswaitingforcompletionofDMAcommandsTaggingisoptionalbutcanbeusefulwhenusingbarrierstocontroltheorderingofMFCcommandswithinasinglecommandqueue.SynchronizationofDMAcommandswithinataggroup:fenceandbarrierExecutionofafencedcommandoptionisdelayeduntilallpreviouslyissuedcommandswithinthesametaggrouphavebeenperformed.Executionofabarriercommandoptionandallsubsequentcommandsisdelayeduntilallpreviouslyissuedcommandsinthesametaggrouphavebeenperformed.barrierandfence在同一個命令組中，barrier命令會等待所有barrier命令之前發(fā)起的命令完成之后，再執(zhí)行命令組中后繼的命令。在同一個命令組中，一個fence命令保證在此命令之前發(fā)起的命令執(zhí)行完成，但fence命令之后的命令也可能在fence命令之前完成barrier之后的命令等待barrier之前的所有命令完成之后再開始執(zhí)行fence命令前面的所有命令都已完成，而且后面的兩個命令在也在fence命令之前完成了ToensureorderofDMArequestexecution:mfc_putf:fenced(allcommandsexecutedbeforewithinthesametaggroupmustfinishfirst,lateronescouldbebefore)mfc_putb:barrier(thebarriercommandandallcommandsissuedthereafterarenotexecuteduntilallpreviouslyissuedcommandsinthesametaggrouphavebeenperformed)DMACommandStatus(SPE)Settagmaskunsignedint

tag_mask;mfc_write_tag_mask(tag_mask);tagmaskremainssetuntilchangedFetchtagstatusunsignedintresult;result=mfc_read_tag_status();/*ormfc_stat_tag_status();*/tagstatusislogicallyANDedwithcurrenttagmasktagstatusbitof‘1’indicatesthatnoDMArequeststaggedwiththespecifictagid(correspondingtothestatusbitlocation)arestilleitherinprogressorintheDMAqueueWaitingforDMACompletion(SPE)WaitforanytaggedDMA:mfc_read_tag_status_any():waituntilanyofthespecifiedtaggedDMAcommandsiscompletedWaitforalltaggedDMA:mfc_read_tag_status_all():waituntilallofthespecifiedtaggedDMAcommandsarecompletedSpecifiedtaggedDMAcommands=commandspecifiedbycurrenttagmasksettingDMACharacteristicsDMAtransfers：TheMFCsupportsnaturallyalignedtransfersizesof1,2,4,or8bytes,andmultiplesof16-bytes,withamaximumtransfersizeof16KBtransfersizescanbe1,2,4,8,andn*16bytes(ninteger)maximumis16KBperDMAtransfer128Balignmentispreferable：PeakperformancecanbeachievedwhenboththeEAandLSAare128-bytealignedandthesizeofthetransferisanevenmultipleof128bytesDMAcommandqueuesperSPU16-elementqueueforSPU-initiatedrequests8-elementqueueforPPE-initiatedrequestsSPU-initiatedDMAisalwayspreferableDMAtags：EachDMAcommandistaggedwitha5-bitTagGroupID.ThisidentifierisusedtocheckorwaitonthecompletionofallqueuedcommandsinoneormoretaggroupseachDMAcommandistaggedwitha5-bitidentifiersameidentifiercanbeusedformultiplecommandstagsusedforpollingstatusorwaitingoncompletionofDMAcommandsDMAlistsasingleDMAcommandcancauseexecutionofalistoftransferrequests(inLS)listsimplementscatter-gatherfunctionsalistcancontainupto2KtransferrequestsMailboxesTocommunicatemessagesupto32bitsinlength,suchasbuffercompletionflagsorprogramstatuse.g.,WhentheSPEplacescomputationalresultsinmainstorageviaDMA.AfterrequestingtheDMAtransfer,theSPEwaitsfortheDMAtransfertocompleteandthenwritestoanoutboundmailboxtonotifythePPEthatitscomputationiscompleteCanbeusedforanyshort-datatransferpurpose,suchassendingofstorageaddresses,functionparameters,commandparameters,andstate-machineparametersCanalsobeusedforcommunicationbetweenanSPEandotherSPEs,processors,ordevicesPrivilegedsoftwareneedstoallowoneSPEtoaccessthemailboxregisterinanotherSPEbymappingthetargetSPE’sproblem-stateareaintotheEAspaceofthesourceSPE.Ifsoftwaredoesnotallowthis,thenonlyatomicoperationsandsignalnotificationsareavailableforSPE-to-SPEcommunication.EachMFCprovidesthreemailboxqueuesPPE(“SPUwriteoutbound”)mailboxqueueSPEwrites,PPEreads1deepSPEstallswritingtofullmailboxPPE(“SPUwriteoutbound”)interruptmailboxqueuelikePPEmailboxqueue,butaninterruptispostedtothePPEwhenthemailboxiswrittenSPU(“SPUreadinbound”)mailboxqueuePPEwrites,SPEreads4deepcanbeoverwrittenEachmailboxentryisafullword:32bitPPEAccesstoMailboxesPPEcanderive“addresses”ofmailboxesfromspethreadidPPEmailboxcallsusespe_idtoidentifydesiredSPE’smailboxChannels:SPE端的訪問接口SPE(outgoing)writethe32-bitmessagevaluetoeitheritstwooutboundmailboxchannelsSPE(incoming)readsamessageintheinboundmailboxMMIORegisters:PPE端的訪問接口PPEandotherdevices(incoming)readmessageinoutboundmailboxbyreadingtheMMIOregisterintheSPE’sMFCPPEandotherdevices(outgoing)sendbywritingtheassociatedMMIOregisterForinterruptsassociatedwiththeSPUWriteOutboundInterruptMailbox,noorderingoftheinterruptandpreviouslyissuedMFCcommandsMailbox的訪問MailboxesAPI–libspe2SPUWriteOutboundMailboxSPEwrite將數(shù)據(jù)信息data發(fā)送到SPU_WrOutMbox，如果SPU_WrOutMbox已滿，指令被阻塞voidspu_write_out_mbox(unsignedintdata)返回SPU_WrOutMbox可接收信息數(shù)據(jù)的數(shù)目，如果返回0，表示SPU_WrOutMbox已滿unsignedint

spu_stat_out_mbox(void)PPEread從spe_id

指定的SPE的SPU_WrOutMbox中讀出可用信息數(shù)目。如該SPU_WrOutMbox為空，返回?cái)?shù)值0。返回值如果為非零正整數(shù)，表明郵箱中未讀出的信息個數(shù)int

spe_out_mbox_status(spe_context_ptr_t

spe_id)從spe_id

指定的SPE的SPU_WrOutMbox中讀出至多count條可用信息，結(jié)果放在mbox_data

指向的緩沖區(qū)中，返回值是實(shí)際讀出的信息個數(shù)。該函數(shù)不會阻塞。如果可用的信息數(shù)目少于count，只將可用的信息讀出。

int

spe_out_mbox_read(spe_context_ptr_t

spe_id,unsignedint*mbox_data,intcount)Thevaluewritten

totheSPUWriteOutboundMailboxchannelSPU_WrOutMboxisenteredintotheoutboundmailboxintheMFCifthemailboxhascapacitytoacceptthevalue.Ifthemailboxcanaccept

thevalue,thechannelcountforSPU_WrOutMboxisdecremented

by‘1’.Iftheoutboundmailboxisfull,thechannelcountwillreadas‘0’.IfSPEsoftwarewritesavaluetoSPU_WrOutMboxwhenthechannelcountis‘0’,theSPUwillstall

onthewrite.TheSPUremainsstalled

untilthePPEorotherdevicereadsamessagefromtheoutboundmailboxbyreadingtheMMIOaddressofthemailbox.Whenthemailboxisread

throughtheMMIOaddress,thechannelcountisincrementedby‘1’.SPUWriteOutboundInterruptMailboxSPEwrite將數(shù)據(jù)信息data發(fā)送到SPU_WrOutIntrMbox，如果SPU_WrOutIntrMbox已滿，指令被阻塞voidspu_write_out_intr_mbox(unsignedintdata)返回SPU_WrOutIntrMbox可接收信息數(shù)據(jù)的數(shù)目，如果返回0，表示SPU_WrOutIntrMbox已滿unsignedint

spu_stat_out_intr_mbox(void)PPEread從spe_id指定的SPE的SPU_WrOutIntrMbox中讀出至多count條可用信息，結(jié)果放在mbox_data指向的緩沖區(qū)中，返回值是實(shí)際讀出的信息個數(shù)。由behavior指定該函數(shù)是否是阻塞的int

spe_out_intr_mbox_read(spe_context_ptr_t

spe_id,unsignedint

*mbox_data,intcount,

unsignedintbehavior)Possiblevaluesforbehaviorare:SPE_MBOX_ALL_BLOCKING：Thecallblocksuntilallcountmailboxmessageshavebeenread.SPE_MBOX_ANY_BLOCKING：Thecallblocksuntilatleastonemailboxmessagehasbeenread.SPE_MBOX_ANY_NONBLOCKING：Thecallreadsasmanymailboxmessagesaspossibleuptoamaximumofcountwithoutblocking.從spe_id指定的SPE的SPU_WrOutIntrMbox中讀出可用信息數(shù)目。如果該SPU_WrOutIntrMbox為空，返回?cái)?shù)值0。返回值如果為非零正整數(shù)，表明郵箱中未讀出的信息個數(shù)int

spe_out_intr_mbox_status(spe_context_ptr_t

spe_id)Thevaluewritten

totheSPUWriteOutboundInterruptMailboxchannel(SPU_WrOutIntrMbox)isenteredintotheoutboundinterruptmailboxifthemailboxhascapacitytoacceptthevalue.Ifthemailboxcanaccept

themessage,thechannelcountforSPU_WrOutIntrMboxisdecrementedby‘1’,andaninterruptisraised

inthePPEorotherdevice,dependingoninterruptenablingandrouting.ThereisnoorderingoftheinterruptandpreviouslyissuedMFCcommands.Iftheoutboundinterruptmailboxisfull,thechannelcountwillreadas‘0’.IfSPEsoftwarewritesavaluetoSPU_WrOutIntrMboxwhenthechannelcountis‘0’,theSPUwillstall

onthewrite.TheSPUremainsstalled

untilthePPEorotherdevicereadsamailboxmessagefromtheoutboundinterruptmailboxbyreadingtheMMIOaddressofthemailbox.Whenthisisdone,thechannelcountisincremented

by‘1’.SPUReadInboundMailboxPPEwrite向spe_id指定的SPE的SPU_RdInMbox寫入最多count個信息，mbox_data

指向數(shù)據(jù)源，由behavior指定該函數(shù)是否是阻塞的int

spe_in_mbox_write(spe_context_ptr_t

spe_id,unsignedint*mbox_data,intcount,unsignedintbehavior)

PossiblevaluesforbehaviorareSPE_MBOX_ALL_BLOCKING:ThecallblocksuntilallcountmailboxmessageshavebeenwrittenSPE_MBOX_ANY_BLOCKING:ThecallblocksuntilatleastonemailboxmessagehasbeenwrittenSPE_MBOX_ANY_NONBLOCKING:Thecallwritesasmanymailboxmessagesaspossibleuptoamaximumofcountwithoutblocking從spe_id指定的SPE的SPU_RdInMbox中讀出可寫入的信息數(shù)目。如果SPU_RdInMbox已滿，返回值為0。返回值如果為非零正整數(shù)，表明郵箱中可以寫入的信息個數(shù)int

spe_in_mbox_status(spe_context_ptr_t

spe_id)SPEread從SPU_RdInMbox中讀出下一個數(shù)據(jù)信息，如果郵箱隊(duì)列為空，該指令被阻塞unsignedint

spu_read_in_mbox(void)返回SPU_RdInMbox中有效信息的數(shù)目，如果返回值為非零正整數(shù)，則SPU_RdInMbox中包含未被讀出的數(shù)據(jù)信息unsignedint

spu_stat_in_mbox(void)MailboxisFIFOqueueIftheSPUReadInboundMailboxchannel(SPU_RdInMbox)hasamessage,thevaluereadfromthemailboxistheoldestmessagewrittentothemailbox.MailboxStatus(empty:channelcount=0)If