字節(jié)跳動云原生機器學習離線訓練實踐-單既喜

InfInfoQInfInfoQArchSummit全球架構(gòu)師峰會字節(jié)跳動云原生J機器學習離線訓練實踐既喜InfoQInfoQInfoQInfoQ個人簡介2014-2016SchlumbergerMaxwell2016-2020—點資訊基礎(chǔ)架構(gòu)部（調(diào)度系統(tǒng)、Spark、Hive、Presto）2020-2022字節(jié)跳動基礎(chǔ)架構(gòu)-計算（Primus、Spark、FeatureStore）ArchSummit仝球嬰枚i］而峰會簡介云原生離線訓練案例與最佳實踐總結(jié)ArchSummit全球架構(gòu)師峰會ArchSummit全球架構(gòu)師峰會簡介InfoQInfoQ業(yè)務(wù)背景與架構(gòu)ReckonReckon訓練平臺?范圍：推薦、廣告、搜索?機器學習訓練引擎（計算+數(shù)據(jù)）Lagrange(TF)Lagrange-Lite(TF)蒲公英（GPU）美洲豹LagrangeX（訓練流程編排）Norbert（訓練流程編排）Ream（資源Quota管理與準入）YARNHDFSYARNHDFS注［:Lagrange、Lagrange-Lite為字節(jié)跳動自研的TF深度學習訓練框架；注2：蒲公英為GPU架構(gòu)（SparseDense分離的）深度學習框架，高速訓練注3：美洲豹為GPU架構(gòu)（AIIReduce架構(gòu)）的高速訓練框架

Primus（云原生離線訓練框架）KubernetesArchSummit全球架構(gòu)師峰會?集中式昂線訓練?DockeronYARN離線訓練架構(gòu)1.0?服務(wù)化ParameterServer(PS)?0->150wvCoresReckon+Lagrangex丁MesosdockeronYARNHDFSArchSummit全球架構(gòu)師峰會Q

fo云原生離線訓練架構(gòu)2.0PrimusAppPrimusApp-分布式離線訓練-伴生(PS)Norbert?聲明式API-Top3離線計算框架Reckon+Lagrangex /APIServer—Worker PSPrimusAppPrimusAppHDFSdockeronYARNHDFSdockeronYARNQ：Top3其他2個計算框架是什么?Spark、Flink、PrimusArchSummit厶*時構(gòu)

云原生離線訓練架構(gòu)3.0?集中式APIServer+CRDs?自研Godel(Kubernetes)?30%PrimusAppNorbert■HWorker PSReckon+LagrangexNorbertNorbertAPIServer

k8sPrimusAppWorkerPSPrimusAppWorkerPSKubernetes云原生離線訓練架構(gòu)UIPrimusDoctorPrimusUIPrimusJobsPrimusFlowPrimusDoctorPrimusUIDetailedviewQAuto-Refresh(30s)ResourcesConf(clicktodownload)(23items}Job(clicktodownload)(4items}Data(clicktodownload)(4items}SummaryJobProgressStartTimeFinishTimeStatusExitCodeDiagnosticDashboardAMMonitorJobMonitorgip_cvr_mftc_3tower_MMCN_fusion_lr_warmup_r1868665_wangzhifeng.zac,webshell96.69387%2022/07/0718:55:23IN_PROGRESS-GrafanaDTOPDTOPTasks(excludingsuccessfultasks)搜索IdState ▼#AttemptProgressLastAssignTimeFinishTimeURIExecutor131456RUNNING195.8%2022/07/1014:46:32/20210326/part-09431.pb.snappy[start=O,length=971320923]executor_sailor_165_2018132126RUNNING144.3%2022/07/1014:46:47/20210326/part-10101.pb.snappy[start=O,length=970553002]executor_sailor_13_1860132290RUNNING198.8%2022/07/1014:46:50|/20210326/part-10265.pb.snappy[start=0,length=970371409]executor_sailor_140_1993134343RUNNING182.2%2022/07/1014:47:32/20210326/part-12318.pb.snappy[start=0,length=966761143]executor_sailor_140_1993134578RUNNING199.3%2022/07/1014:47:37/20210326/part-12553.pb.snappy[start=0,length=966683380]executor_sailor_140_1993136589RUNNING157.0%2022/07/1014:48:18/20210326/part-14564.pb.snappy[start=0,length=961961699]executor_sailor_140_1993136641RUNNING199.8%2022/07/1014:48:19/20210326/part-14616.pb.snappy[start=0,length=961752319]executor_sailor_140_1993136694RUNNING192.7%2022/07/1014:48:22/20210326/part-14669.pb.snappy[start=0,length=961779966]executor_sailor_140_1993136726RUNNING197.1%2022/07/1014:48:23/20210326/part-14701.pb.snappy[start=0,length=961680145]executor_sailor_65_1915136768RUNNING17.2%2022/07/1014:48:24|/20210326/part-14743.pb.snappy[start=0,length=961170671]executor_sailor_192_2045顯示第1到第10條記錄，總共12825條記:錄每頁顯示10-條記錄< 1 2 3 4 5... 1283 >scm-version:1.0.0.517ArchSummit全球架構(gòu)師峰會省時查報告小程序全網(wǎng)最全、最新、最專業(yè)的行研報告庫，每天實時更新，超百萬份行研報告限時免費下載。10:30行業(yè)i名￡變言，2019-2025年中國數(shù)據(jù)中心10:30行業(yè)i名￡變言，2019-2025年中國數(shù)據(jù)中心(IDC)行O業(yè)發(fā)展態(tài)勢與前景展望研究報告2019年中國人工晉能產(chǎn)業(yè)知識產(chǎn)權(quán)發(fā)展fe白皮書-胡潤研究院2019音樂人報告-傳媒大學-清科-2019音樂人報告-傳媒大學-清科-201911-pdf￡化妝品行業(yè)專題：高成長、長賽道、拼耐□ 0涵蓋教育、電商、短視頻、房地產(chǎn)、新媒體、區(qū)塊鏈、

人工智能、5G、互聯(lián)網(wǎng)、物聯(lián)網(wǎng)、創(chuàng)業(yè)、醫(yī)療、金

融、零售、餐飲、旅游、汽車等數(shù)十個行業(yè)。掃碼關(guān)注公眾號，回復(fù)：入群

加入“省時查報告〃微信群RrimusJot>sIdNcxte,oo■MOC5OT_Miftor_■OO_2■6499■x?c5or_mbUot_99_21?7,02■x?c5orMdkor_1Q2_21TC1O1?x?c5orMUftor-IOI104■x—5or_Md?or_*104_2,173103wx^cotor_Mii?or_103_2171106?xM：L>tor_Mdaor_lO0_217^105■X9c5or_MMftor_'105_2.TZg■x^c5or_Mfe0ftor_32_21605?x^CL/tor_MdBor_91_J2159.w?*>sr?oM.w?t>sr?oM?示■■mmio j&mU4i 示io-*te*Id Ncxlwttx^CMtor <785?x?cutor_fM_S_786executor 787ox^cutor 77&6O ?x?cutor_p?_0_7B*1?x?cutor_p?—1_782oxocutorox^cutor_p?_3_784■2 *12_7^13 ?X0Otutor 13_7fl.wot3?r>?iiId NoctoO ?x^cwtor_^ilot_*?rvic?_0_204C8 eoc?cwtor_|9ilot_s?rvftc?_0_2OS77 oxocwtor_^ilot_?0rvic9_7_2OSC6 ?M^cwtor_s?rvic^_6_2O&￡5 ?M?cvrtor_^>Mot_aerv*c^_S_206ifi4 otxocwtor_.|3iIot_aorvic?_4_2O&23 oxocwtor_^ilot_S0rvftc^_3_2O&22 ?x?cwtor_pilot_*?rvic?_2_2O611 euc^cwtor_s?rvic?_1_2OSC,2 eJcoovftor_|3Hot_a?rvlC9_12_2O€ArchSuwhit全球架構(gòu)師峰會云原生離線訓練架構(gòu)UI2022707/102O22ZD7/102022Z07/102O22ZD7/102022707/1020S2Z07/102022Z07/102022Z07/102022X)7/102022/07V.OOA?Qrwo*4*■5:55:34RUZZ，f\?OISISISISISSS04S5:MS724FUJZZ，ff>?GRMMNIfXSse&8:24S7S,lSI20^2/07700O9:S8:S82O22Z07/0aO9:SB:S82022/07/0eO9:SB:&82O22/O，/OeO9:S8:S82022/07700O9:SB:S82022/07700O9:SB:S82022Z07/OeO9:S8:&82O22A)7/0aO9:S8:SS2022/07700O9:S8:&82O22Z07/0eO9:Sa：&83303RUIMIMIMGOTOF?DTO<>DTOF?OTOF>OTOF?OTO<>OTO<>OTOF?DTO<>DTO<>2 3 4S— *12S??*■Iirv??S3t?RUNNIMOEjcltOod.Oi*or?o*4*O??Kt?o*r<lGrafarxaDTOF>RLJNVMIMOGratanADTORFtLJNNINGGrsfar?ierror*F^LJNNIMGGrsfar^aDTOf>RMNNIIMODTOf>RCJhJNIMOGrmtarwiDTORRUIMNIMGGrwfarwaDTOf?RMNINIIMGGrafATidiDTOf>RUIMNIMGGraf*r>?OTORRUNIMIMGGrsfATMDTOF?<1 2 34S一 iOOL^%jr>c4tTarv>02O22A17/1O14:S7l29TtrrwSVst? EjcHOod?RVMMNIMGOTOR2022^)7/101^:S7L2eGrwtenaOTORw^t>sr?eM2022/077*1。RCMMIMIMGGrafanaOTOR2022^07/1014:S，:29RVMMIMIMOGra-fAnAOTOR2022^)7/1015:01^29ROtMNIfstGGraten—OTOR2022^7/10ROMNIt^GGrafaoMOTOR2022/07/10'l?4:S7^eRVNXNIMGGrafariAOTOR2022/077*，。RCMMNIMGGraf^n^iOTOR2O22AJ7/1O1?4:S7^eROMIMIMGGrwtenAOTOR2O22/O7Y1O1<?:S7l29RMfMNIMGGrafanaOTORInfoQInfoQ作業(yè)規(guī)模10000作業(yè)/天4000單最大作業(yè)數(shù)400wvCore總量30wvCoreK8sArchSii?wnit全球架構(gòu)師峰會云原生離線訓練■計算調(diào)度-數(shù)據(jù)編排ArchSummit全球架構(gòu)師峰會InfoQArchSummit全球架構(gòu)師峰會容器化多角色+異構(gòu)計算調(diào)度PrimusOperator彈性API-Server微服務(wù)InfInfoQInfInfoQPrimusOperator整體架構(gòu)JobCRDMetaSpecStatusArchSummit全球架構(gòu)師11$KubebuilderInfoQInfoQInfoQInfoQPrimusOperator+Framework PrimusJobCRDPrimusDataCRD PrimusOperatorPrimusRole NorbertDriverPrimusAMPodTFPSTFworkers TFGPUWorkersArchSummit全球架構(gòu)師峰厶zx訓練多角色調(diào)度-PrimusRoleCRDPrimusRolecrdWorkerPrimusRolecrdPSPrimusRolecrdGPURoleStatusRoleStatusShardStatusName ExecutorStatArchSummit全球架構(gòu)師峰RestartCntEndpoint多角色異構(gòu)調(diào)度-調(diào)度朿略DynamicPolicyGangPolicyOrderPolicyLagrangeLagrangeLiteLagrangeLite訓練彈性調(diào)度Norbert

DriverPrimusRoleworkerPodGroupArchSummit全球架構(gòu)師峰會PrimusOperator彈性調(diào)度-價值與收益啟動時強穩(wěn)定性、中期高訓練速度更大的資源池InfoQ啟動時強穩(wěn)定性、中期高訓練速度更大的資源池InfoQ?混部SmartResource:CPU利用率20%-->70%?SlowStart:?潮汐/反潮汐訓練ArchSummit全球架構(gòu)師峰會數(shù)據(jù)編排數(shù)據(jù)編排InfoQInfoQ云原生離線訓練■計算調(diào)度-數(shù)據(jù)編排ArchSummit全球架構(gòu)師峰會InfoQ復(fù)雜數(shù)據(jù)源(day/hour/customize)HDFSFeatureStore(Iceberg)Kafka編排策略GroupByFiltenjBIIIIH^HIShuffleWithinPartitionGlobalShuffle1^1ArchSummit仝球嬰枚臨UfS會元數(shù)據(jù)編排頭條000102000000010100010202020102頭條000102000000010100010202020102Shuffle+GroupBy+追新iSumIIInfoQInfoQInfoQInfoQ訓練Task傳輸DataStreamBDataStreamADataStreamBPrimusDataCRD2020-2022KafkaPrimusDataCRD2020-2022Kafka2020-2022Norbert

DriverArchSummit全球架構(gòu)師峰厶zx數(shù)據(jù)傳輸實踐（IPC）?管道：競爭、序列化反序列化Socket:內(nèi)核態(tài)用戶態(tài)ShareMemory:多進程管理JNI:統(tǒng)一進程ArchSummit全球架構(gòu)師峰會案例與最佳實踐■服務(wù)化PSvs.云原生PSPrimusFlow■批流一體■云原生混部訓練NativeLibArchSummit全球架構(gòu)師峰會PSService服務(wù)化PSvs.云原生PS訓練PSServicePSonYARN/K8SCheckpointOOPSonYARN/K8SSavePointOOOPSonYARN/K8SSavePointO同機房撮合資源利用率低運維與部署難度大偏離性差（網(wǎng)絡(luò)、內(nèi)存帶寬、CPU）a全鏈路IncrementalCheckpointaPS單點Kill/恢復(fù)aSmartResourceaPSGang性調(diào)度a服務(wù)發(fā)現(xiàn)（consul-＞自研）、健康檢查A單作業(yè)+容器化部署A性能：NumaBindInfoQaSavePoint機制InfoQArchSummit

PrimusFlow「8PrimusRow

Job1Sparkl-0 ArchSummitModel

Dump特征調(diào)研, a行級ShuffleA邊計算邊訓練?豐富數(shù)據(jù)源支持aHudi、TOS、HBase?多數(shù)據(jù)流編排IA普通流與預(yù)處理流混合訓練構(gòu)師峰會InfoQ批流一體架構(gòu)Data,ioMasterPODFlink->PrimusStreaming（抖音晚高峰+GPU）Data,ioMasterPODDataRunnerData.ioMasterDataLoader

PyRunnerStream

ManagerCheckpoin

Manager多角色異構(gòu)流式訓練?N個角色：GPU+CPU+伴生PS?Forward+Rebalance批流一體?Forward+Rebalance批流一體+批流協(xié)同Sailor '、、Data,ioExecutorLib:KafkajRecieverCheckpoint:Manageri/二二//y//SailorPod?Batch?StreamingKafkaBrokerHDFSSavePointsPSSavepoint丄……,"Partition0PartitionlInfoQInfoQArchSummit全球架構(gòu)師峰

混部訓練場景（資源問題）計算管理：-1818等退出碼識別（混部資源隨時退出的特點）數(shù)據(jù)管理：樣本丟失或重復(fù)問題（緩存問題，心跳丟失問題）TF

WORKERTF

WORKERFile:

Pos:File:

Pos:6%->0.5%ExecutorExecutorExecutorInfoQInfoQInfoQInfoQPrimusNativeDataMaster?編程范式aConfigurationaDeclarativeprogrammingaPy4JUDF,AutoTypeConvertDataExecutorSerialize/DeserializePod->多進程、多ContainerTF

WORKERExecutorArchSummit全球架構(gòu)師峰TF

WORKERExecutor?總吞吐量從3.3GB/S提升至13.5GB/S!提升4x?單節(jié)點吞吐率從411