2024微軟人工智能系統(tǒng)深度學(xué)習(xí)中的分布式訓(xùn)練算法

單一設(shè)備Top-5accuracyvs.computationalcomplexity(singleforwardpass)BenchmarkAnalysisofRepresentativeDeepNeuralNetworkArchitectures/pdf/1810.00736.pdf模型相關(guān)，相對固定 可變因素可變因素工作重點Moore定律，可變因素工作重點相對有限本章內(nèi)容本章內(nèi)容下章內(nèi)容 TensorFlow

Data-FlowGraph(DFG)x=tf.placeholder(tf.float32)W=x=tf.placeholder(tf.float32)W=b=tf.Variable(tf.float32)m=W*xs=m+by=tf.reduce_sum(s)grad_W,grad_b=tf.gradients(y,[W,b])update=optimizer.apply_gradients({[W,grad_W],[b,grad_b]})xWb??W??b*m*??++????ms??sΣΣ??y x=tf.placeholder(tf.float32)W=b=tf.Variable(tf.float32)

多個樣本

Data-FlowGraph(DFG)m=W*xs=m+by=tf.reduce_sum(s)grad_W,grad_b=tf.gradients(y,[W,b])update=optimizer.apply_gradients({[W,grad_W],[b,grad_b]})

x W *m

??W ??b*????m+ +??s

多個操作Σ y

張量計算算子間并行數(shù)據(jù)并行：多個樣本并行執(zhí)行模型并行：多個算子并行執(zhí)行組合并行：多種并行方案組合疊加https://blog.skymind.ai/distributed-deep-learning-part-1-an-introduction-to-distributed-training-of-neural-networks/ im2colconvolution im2colconvolution JosephE.GonzalezAI-SystemsDistributedTraining算子內(nèi)并行算子并行：并行單個張量計算子內(nèi)的計算（GPU多處理單元并行）https://blog.skymind.ai/distributed-deep-learning-part-1-an-introduction-to-distributed-training-of-neural-networks/AllReducex W b *

AllReduce??W ??b*??

AllReducex W b*

AllReduce??W ??b*??m +

??m

m+

??ms ??s

s ??sΣ y

GPU0

Σ y

GPU1xWb??W??b*m*??+xWb??W??b*m*??++????mGPU0s??sΣΣ??yGPU1非并行數(shù)據(jù)并行模型并行樣本數(shù)據(jù)量11/N1傳輸數(shù)據(jù)量0模型大小激活大小存儲占用1N1負(fù)載平衡度-強(qiáng)弱并行限制-單步樣本量算子數(shù)量具有同步障不含同步障具有限定的寬松同步障ComputeCommunicateComputeIterationComputeCommunicateComputeIterationw1wIterationw2wIterationw3BarrierwBarrierWasteIterationWasteIterationWasteIterationwwMachine2wwMachine3wwJosephE.GonzalezAI-SystemsDistributedTrainingComputeCommunicateComputeIterationComputeCommunicateComputeIterationIterationIterationWasteIterationWasteIterationWasteIterationMachine2Machine3Barrier BarrierJosephE.GonzalezAI-SystemsDistributedTrainingMachine1Machine2

Compute Communicate

ComputeIterationIterationIterationIterationIterationIterationIterationIterationJosephE.GonzalezAI-SystemsDistributedTraining具有同步障AAA不含同步障