高性能計(jì)算發(fā)展概述

高性能計(jì)算及應(yīng)用任課教師王云嵐EMAIL：wangyl@

趙天海EMAIL：zhaoth@nwpu.高性能計(jì)算研究與發(fā)展中心辦公室:勇字樓3樓 電話(huà)：88493434（O）2課程目標(biāo)掌握高性能計(jì)算編程工具，解決相關(guān)問(wèn)題課程主要內(nèi)容：介紹高性能計(jì)算系統(tǒng)體系結(jié)構(gòu)、高性能并行程序程序設(shè)計(jì)方法及高性能計(jì)算技術(shù)最新方向。主要包括：高性能處理機(jī)、多處理機(jī)系統(tǒng)；集群計(jì)算系統(tǒng)、Linux集群系統(tǒng)配置方法，集群資源管理與作業(yè)調(diào)度，多線(xiàn)程編程及性能優(yōu)化等；并行編程程序工具：OpenMP、MPI、CUDA、MapReduce等。交流平臺(tái)2013年高性能計(jì)算課程qq群：158463721作業(yè)高性能計(jì)算相關(guān)研究熱點(diǎn)的技術(shù)報(bào)告云計(jì)算CPU/GPU技術(shù)虛擬化…實(shí)驗(yàn)報(bào)告集群環(huán)境構(gòu)建并行應(yīng)用編程：MPI，openMP，Cuda…高性能計(jì)算及應(yīng)用課程1：高性能計(jì)算發(fā)展概述課程內(nèi)容提綱應(yīng)用需求計(jì)算機(jī)體系結(jié)構(gòu)的發(fā)展高性能計(jì)算的核心技術(shù)：并行計(jì)算并行編程的重要性應(yīng)用需求

Highperformancecomputing高性能計(jì)算與科研，產(chǎn)業(yè)…——需求與意義基礎(chǔ)科研領(lǐng)域的計(jì)算需求物理化學(xué)生物材料工業(yè)領(lǐng)域的需求銀行輔助設(shè)計(jì)醫(yī)藥石油氣象在線(xiàn)服務(wù)信息安全傳統(tǒng)的科學(xué)研究difficult,例如建造大型風(fēng)洞expensive,例如建造樣機(jī)slow,例如等待氣候的變化，天體的演化dangerous,例如武器開(kāi)發(fā)，藥品，大氣試驗(yàn)，電力系統(tǒng)分析基于計(jì)算科學(xué)的科學(xué)研究物理原理和數(shù)值方法理論分析設(shè)計(jì)試驗(yàn)富有挑戰(zhàn)性的計(jì)算問(wèn)題遍及科學(xué)與工程的各個(gè)領(lǐng)域ScienceGlobalclimatemodelingAstrophysicalmodelingBiology:genomics;proteinfolding;drugdesignComputationalChemistryComputationalMaterialSciencesandNanosciencesEngineeringCrashsimulationSemiconductordesignEarthquakeandstructuralmodelingComputationfluiddynamics(airplanedesign)Combustion(enginedesign)OilfieldapplicationsBusinessFinancialandeconomicmodelingTransactionprocessing,webservicesandsearchenginesDefenseNuclearweapons--testbysimulationsCryptographyUnitsofHighPerformanceComputing計(jì)算能力存儲(chǔ)能力全球氣候模擬計(jì)算問(wèn)題:f(經(jīng)度,緯度,海拔,時(shí)間)

溫度,氣壓,適度,風(fēng)速做法:域的離散化分解,10公里解析度(Discretizethedomain,e.g.,ameasurementpointevery10km)給定時(shí)間t設(shè)計(jì)算法預(yù)測(cè)t+dt的天氣(Deviseanalgorithmtopredictweatherattimet+dtgivent)應(yīng)用:主要事件預(yù)測(cè)(Predictmajorevents,e.g.,ElNino)用于確定大氣散射標(biāo)準(zhǔn)(Useinsettingairemissionsstandards)大氣環(huán)流模擬需求解Navier-Stokes方程1分鐘時(shí)間間隔100個(gè)浮點(diǎn)運(yùn)算/網(wǎng)格點(diǎn)對(duì)計(jì)算的需求為確保時(shí)效需1分鐘執(zhí)行5x1011flops= 8Gflop/s以天為單位的7天天氣預(yù)報(bào)需要

56Gflop/s以月為單位的50年氣候預(yù)測(cè)需要 4.8Tflop/s以12小時(shí)為單位的50年預(yù)測(cè) 288Tflop/s如果提高網(wǎng)格解析度則計(jì)算復(fù)雜性將呈8x,16x增加更高的精確預(yù)測(cè)模型則需要綜合考慮大氣,海洋,冰川,陸地,加上地球化學(xué)等因素千年氣候模型分析目前無(wú)法對(duì)此進(jìn)行有效計(jì)算全球氣候模擬高性能計(jì)算已經(jīng)成為復(fù)雜系統(tǒng)工程的必備手段航空高性能計(jì)算領(lǐng)域高端需求主要集中在CAE領(lǐng)域氣動(dòng)力計(jì)算結(jié)構(gòu)計(jì)算氣動(dòng)彈性分析多學(xué)科設(shè)計(jì)優(yōu)化飛行載荷計(jì)算隱身設(shè)計(jì)計(jì)算穩(wěn)定性和操縱計(jì)算需求飛行仿真其他高性能計(jì)算需求數(shù)字化裝配數(shù)字樣機(jī)主要特點(diǎn)計(jì)算能力vs計(jì)算規(guī)模先導(dǎo)性研究vs工程應(yīng)用超音速巡航大攻角機(jī)動(dòng)武器系統(tǒng)內(nèi)埋式發(fā)射CFD終極目標(biāo)：虛擬飛行試驗(yàn)虛擬風(fēng)洞(CFD)設(shè)計(jì)經(jīng)驗(yàn)風(fēng)洞試驗(yàn)虛擬飛行試驗(yàn)計(jì)算設(shè)備/用戶(hù)/內(nèi)容Today2015Source：IDF2012大數(shù)據(jù)現(xiàn)象“Dataarebecomingthenewrawmaterialofbusiness:aneconomicinputalmostonaparwithcapitalandlabor”

—TheEconomist,2010“Informationwillbethe‘oilofthe21stcentury”

—Gartner，2010Source：IDF20122015CloudVisionCoexistenceofOpportunitiesandChallengesSource：IDF2012TrendstoExascalePerformanceRoughly10xperformanceevery4years,predictsthatwe’llhitExascaleperformancein2018-19Source：IDF2012計(jì)算機(jī)體系結(jié)構(gòu)的發(fā)展計(jì)算機(jī)體系結(jié)構(gòu)的發(fā)展趨勢(shì)體系結(jié)構(gòu)的改進(jìn)將技術(shù)創(chuàng)新轉(zhuǎn)變?yōu)橛?jì)算機(jī)的處理性能計(jì)算機(jī)體系結(jié)構(gòu)歷史：電子管、晶體管、集成電路、大規(guī)模集成電路超大規(guī)模集成電路(VeryLargeScaleIntegration)的發(fā)展階段可以看做為并行處理的探索過(guò)程并行處理是提高計(jì)算機(jī)處理性能的核心技術(shù)體系結(jié)構(gòu)的發(fā)展:并行方法的探索GreatesttrendinVLSIgenerationisincreaseinparallelism1970-1985:位級(jí)并行（bitlevelparallelism）

4-bit->8bit->16-bitslowsafter32bitadoptionof64-bitnowunderway,128-bitfar(notperformanceissue)80年代中期to90年代中期:指令級(jí)別并行（instructionlevelparallelism）pipeliningandsimpleinstructionsets,+compileradvances(RISC)on-chipcachesandfunctionalunits=>superscalarexecutiongreatersophistication:outoforderexecution,speculation,predictiontodealwithcontroltransferandlatencyproblemsNow:線(xiàn)程級(jí)并行（threadlevelparallelism）VLSI三個(gè)階段Threephases:Bit-levelInstruction-levelThread-levelVLSITechnologyTrendsIntelannouncedthattheyhavereach1.7billionwithItaniumprocessorGigascaleIntegration(GSI)=1billiontransistorsperchip/~jeff/ece4420/technology.pdf單處理器的性能增長(zhǎng)變化VAX:25%/year1978to1986RISC+x86:52%/year1986to2002RISC+x86:??%/year2002topresent處理器功耗發(fā)展的趨勢(shì)不在提供時(shí)鐘頻率，而轉(zhuǎn)變?yōu)槊總€(gè)芯片的CPU數(shù)量風(fēng)冷芯片最大功耗的瓶頸RecentIntelProcessors“Wearededicatingallofourfutureproductdevelopmenttomulticoredesigns.Webelievethisisakeyinflectionpointfortheindustry.”IntelPresidentPaulOtellini,IDF2005ProcessorsYearFabrication(nm)Clock(GHz)Power(W)Pentium420001801.80-4.0035-115PentiumM200390/1301.00-2.265-27Core2Duo2006652.60-2.9010-65Core2Quad2006652.60-2.9045-105Corei7(Quad)2008452.93-3.6095-130Corei5(Quad)2009453.20-3.6073-95PentiumDual-Core2010452.80-3.3365-130Corei3(Duo)2010322.93-3.3318-732ndGeni3(Duo)2011322.50-3.4035-652ndGeni5(Quad)2011323.10-3.8045-952ndGeni7(Quad/Hexa)2011323.80-3.9065-1303rdGeni3(Duo)201222/322.80-3.4035-553rdGeni5(Quad)201222/323.20-3.8035-773rdGeni7(Quad/Hexa)201222/323.70-3.9045-77XeonE5(8-cores)2013221.80-2.9060-130XeonPhi(60-cores)2013221.10300Intel'sManyCoreandMulti-coreIntel80-coreTeraScaleProcessor(Vangaletal.2008)億級(jí)處理器developedasolver(singleprecision)forthischipthatranat1TFLOPwithonly97WattsSource：TimMattson,IntelLabsTrendsareputtingallontoonechipThefuturebelongstoheterogeneous,manycoreSOCasthestandardbuildingblockofcomputingSOC=systemonachipSource：TimMattson,IntelLabs集群系統(tǒng)的發(fā)展趨勢(shì)Large-ScaleComputingSystems大規(guī)模集群計(jì)算系統(tǒng)Franklin(NERSC-5):CrayXT49,532computenodes;38,128coresEachnodehasanAMDquadcoreprocessor and8GBofmemory~25Tflop/sonapplications;352Tflop/speakHPSSArchivalStorage40PBcapacity4TapelibrariesNERSCGlobalFilesystem(NGF)UsesIBM’sGPFS1.5PB;5.5GB/sClusters105TflopstotalCarverIBMiDataplexclusterPDSF(HEP/NP)Linuxcluster(~1Kcores)MagellanCloudtestbedIBMiDataplexclusterAnalyticsEuclid(512GBsharedmemory)DiracGPUtestbed(48nodes)Hopper(NERSC-6):CrayXE6Phase1:CrayXT5,668nodes,5344coresPhase2:>1Pflop/speak(2sockets/node,12cores/socket)Tianhe-I(A)6,144computenodes;24576cores2560AMDRadeonHD4870*2GPU98TBmemoryintotalRpeak:4.700pflops;Rmax:2.566pflopsJaguar:(CrayXT5)224,256x86-basedAMDOpteronprocessorcoresRpeak:2.331pflops;Rmax:1.759pflops西工大高性能計(jì)算中心高性能集群設(shè)備浪潮天梭TS10000NX5440刀片計(jì)算節(jié)點(diǎn)浪潮TS10KClusters計(jì)算能力：73Tflopstotal153計(jì)算刀片3MIC加速節(jié)點(diǎn)4GPU加速節(jié)點(diǎn)并行存儲(chǔ)179TB光纖存儲(chǔ)系統(tǒng)40TBLinux操作系統(tǒng)集群的基本組成光纖存儲(chǔ)系統(tǒng)管理、登錄、IO節(jié)點(diǎn)計(jì)算節(jié)點(diǎn)并行存儲(chǔ)Top10listinJune2012RankSiteComputer1DOE/NNSA/LLNL

UnitedStatesSequoia-BlueGene/Q,PowerBQC16C1.60GHz,Custom

IBM2RIKENAdvancedInstituteforComputationalScience(AICS)

JapanKcomputer,SPARC64VIIIfx2.0GHz,Tofuinterconnect

Fujitsu3DOE/SC/ArgonneNationalLaboratory

UnitedStatesMira-BlueGene/Q,PowerBQC16C1.60GHz,Custom

IBM4LeibnizRechenzentrum

GermanySuperMUC-iDataPlexDX360M4,XeonE5-26808C2.70GHz,InfinibandFDR

IBM5NationalSupercomputingCenterinTianjin

ChinaTianhe-1A-NUDTYHMPP,XeonX56706C2.93GHz,NVIDIA2050

NUDT6DOE/SC/OakRidgeNationalLaboratory

UnitedStatesJaguar-CrayXK6,Opteron627416C2.200GHz,CrayGeminiinterconnect,NVIDIA2090

CrayInc.7CINECA

ItalyFermi-BlueGene/Q,PowerBQC16C1.60GHz,Custom

IBM8ForschungszentrumJuelich(FZJ)

GermanyJuQUEEN-BlueGene/Q,PowerBQC16C1.60GHz,Custom

IBM9CEA/TGCC-GENCI

FranceCuriethinnodes-BullxB510,XeonE5-26808C2.700GHz,InfinibandQDR

Bull10NationalSupercomputingCentreinShenzhen(NSCS)

ChinaNebulae-DawningTC3600BladeSystem,XeonX56506C2.66GHz,InfinibandQDR,NVIDIA2050

Dawning2011年6月，我國(guó)進(jìn)入Top500的高性能計(jì)算機(jī)2NationalSupercomputingCenterinTianjinNUDTProprietaryProprietary4NationalSupercomputingCentreinShenzhen(NSCS)DawningInfinibandInfinibandQDR33InstituteofProcessEngineering,ChineseAcademyofSciencesIPE,Nvidia,TyanInfinibandInfinibandQDR40ShanghaiSupercomputerCenterDawningInfinibandInfinibandDDR82ComputerNetworkInformationCenter,ChineseAcademyofScienceLenovoInfinibandInfiniband97TsinghuaUniversityInspurInfinibandInfinibandQDR143NetworkCompanyIBMGigabitEthernetGigabitEthernet164InternetServiceIBMGigabitEthernetGigabitEthernet199WebCompany(C)Hewlett-PackardGigabitEthernetGigabitEthernet201InternetServiceIBMGigabitEthernetGigabitEthernet202InternetServiceIBMGigabitEthernetGigabitEthernetIPE：中國(guó)科學(xué)院過(guò)程工程研究所（原化工冶金研究所）RankSiteSystemCoresRmax(TFlop/s)Rpeak(TFlop/s)Power(kW)10NationalSupercomputingCentreinShenzhen(NSCS)

ChinaNebulae-DawningTC3600BladeSystem,XeonX56506C2.66GHz,InfinibandQDR,NVIDIA2050

Dawning1206401271.02984.3258026NationalSupercomputingCenterinJinan

ChinaSunwayBlueLight-SunwayBlueLightMPP,ShenWeiprocessorSW1600975.00MHz,InfinibandQDR

NationalResearchCenterofParallelComputerEngineering&Technology137200795.91070.2107437InstituteofProcessEngineering,ChineseAcademyofSciences

ChinaMole-8.5-Mole-8.5Cluster,XeonX55204C2.27GHz,InfinibandQDR,NVIDIA2050

IPE,Nvidia,Tyan29440496.51012.654094ShanghaiSupercomputerCenter

ChinaMagicCube-Dawning5000A,QCOpteron1.9Ghz,Infiniband,WindowsHPC2008

Dawning30720180.6233.5122Government

ChinaSunway4000HCluster,XeonX56xx(Westmere-EP)2.93GHz,InfinibandQDR

NationalResearchCenterofParallelComputerEngineering&Technology14280145.6167.4127ResearchCenter

ChinaClusterPlatformSL250sGen8,XeonE5-26608C2.200GHz,InfinibandFDR,NVIDIA2090

Hewlett-Packard8064135.4270.7132InternetService

ChinaxSeriesx3650Cluster,XeonE56496C2.530GHz,GigabitEthernet

IBM23316131.4236.0707.32012年6月，我國(guó)進(jìn)入TOP500的部分超級(jí)計(jì)算機(jī)/sublistTOP500(2011年6月)中的集群

星群系統(tǒng)(Constellations)包含了一個(gè)超大容量交換系統(tǒng)，可以同時(shí)管理數(shù)千個(gè)計(jì)算引擎之間的高速數(shù)據(jù)傳輸;大規(guī)模并行機(jī)（MPP)：由許多松耦合的處理單元組成,每個(gè)單元內(nèi)的CPU都有自己私有的資源，如總線(xiàn)，內(nèi)存，硬盤(pán)等，每個(gè)處理單位只有一個(gè)微內(nèi)核；集群（Cluster）：每個(gè)節(jié)點(diǎn)有完整的操作系統(tǒng)。2012年6月數(shù)據(jù)，TOP500中有407套系統(tǒng)為ClusterArchitectureCountShare%RmaxSum(GF)RpeakSum(GF)ProcessorSumConstellations20.40%9497011294717648MPP8717.40%19293725255504292984630Cluster41182.20%39541331595165734777646Totals500100%58930025.5985179949.007779924Top500國(guó)家分布TOP500過(guò)去19年體系結(jié)構(gòu)演化TOP500過(guò)去19年體系結(jié)構(gòu)演化2013年6月，cluster

417，MPP

83

從TOP500看集群系統(tǒng)在高性能計(jì)算領(lǐng)域，集群系統(tǒng)已經(jīng)成為主流的系統(tǒng)結(jié)構(gòu)，并將進(jìn)一步擴(kuò)大其所占份額在Top500中，集群結(jié)構(gòu)占了絕對(duì)大多數(shù)，說(shuō)明在構(gòu)建超大規(guī)模計(jì)算系統(tǒng)的時(shí)候，集群是主要的系統(tǒng)構(gòu)成方式集群系統(tǒng)的發(fā)展趨勢(shì)64位系統(tǒng)逐漸成為主流多種商業(yè)化的高速互連網(wǎng)絡(luò)SAN系統(tǒng)作為集群的存儲(chǔ)設(shè)備64位：突破2GB的系統(tǒng)內(nèi)存瓶頸科學(xué)計(jì)算大規(guī)模模擬應(yīng)用三維網(wǎng)格模擬應(yīng)用所需的內(nèi)存可以輕易突破2GB生物信息學(xué)基因拼接等應(yīng)用需要大量的內(nèi)存，實(shí)際應(yīng)用中內(nèi)存不足是主要問(wèn)題之一素?cái)?shù)運(yùn)算需要用到大量64位整數(shù)運(yùn)算和大內(nèi)存商業(yè)應(yīng)用海量數(shù)據(jù)處理DBinmemory媒體播放服務(wù)器大內(nèi)存高內(nèi)存帶寬減少訪問(wèn)磁盤(pán)次數(shù)，可將性能提高近一個(gè)數(shù)量級(jí)64位：突破2GB的系統(tǒng)內(nèi)存瓶頸64位：新的設(shè)計(jì)理念引發(fā)新的設(shè)計(jì)理念現(xiàn)有的很多算法是基于內(nèi)存不足設(shè)計(jì)的，因此很多精力花費(fèi)在用時(shí)間換取空間上64位系統(tǒng)提供了訪問(wèn)更大內(nèi)存的機(jī)會(huì)，因此很多應(yīng)用可能要基于新的理念進(jìn)行設(shè)計(jì)，以獲得64位所帶來(lái)的好處64位：不是萬(wàn)能靈藥并非所有用戶(hù)都需要現(xiàn)在就轉(zhuǎn)向64位代碼膨脹，性能反而可能會(huì)下降需要根據(jù)自己的應(yīng)用特性來(lái)分析是否需要2GB以上的內(nèi)存是否有大量64位整數(shù)運(yùn)算如果上述問(wèn)題的答案都是否，那么不一定能夠從64位系統(tǒng)中得到預(yù)期的好處某些應(yīng)用可以從特定的64位處理器獲得很大的性能提高，但這不是64位本身的特性，而是依賴(lài)于特定處理器，需要具體分析實(shí)際情況集群系統(tǒng)的互連網(wǎng)絡(luò)評(píng)價(jià)互連網(wǎng)絡(luò)的指標(biāo)延遲帶寬功能支持價(jià)格集群系統(tǒng)的互連網(wǎng)絡(luò)InterconnectInterfaceMPILatency(us)Uni-directionalBandwidth(MB/s)說(shuō)明GBEtherPCI~30-50100最便宜MyrinetPCI-X6248SCIPCI1.4326延遲最小QuadricsIIIPCI5340InfiniBand4xPCI-X7.5805帶寬最高集群系統(tǒng)的互連網(wǎng)絡(luò)功能支持都支持MPI，除GBEthernet外都實(shí)現(xiàn)了高效率的通信協(xié)議SCI和Quadrics還提供了共享內(nèi)存的支持，但是其遠(yuǎn)程通信延遲仍然在us數(shù)量級(jí)，對(duì)于細(xì)粒度的共享內(nèi)存程序，仍然無(wú)法很好地支持（對(duì)比SGIAltrix系列的遠(yuǎn)程訪問(wèn)延遲在200ns以下）集群系統(tǒng)所面臨的挑戰(zhàn)能耗問(wèn)題不僅僅是集群系統(tǒng)的問(wèn)題從芯片，單機(jī)和集群系統(tǒng)等多個(gè)層次來(lái)共同解決這個(gè)問(wèn)題管理性監(jiān)控自我修復(fù)管理信息的過(guò)濾與提取分區(qū)ExecutionisnotjustabouthardwareModernprogrammerdoesnotseeassemblylanguageManydonotevensee“l(fā)ow-level”languageslike“C”什么是并行編程?WhyparallelprogrammingWhatisParallelComputing?Traditionally,softwarehasbeenwrittenforserialcomputationToberunonasinglecomputerhavingasingleCentralProcessingUnit(CPU)AproblemisbrokenintoadiscreteseriesofinstructionsInstructionsareexecutedoneafteranotherOnlyoneinstructionmayexecuteatanymomentintimeForexample：發(fā)工資程序ParallelComputing同時(shí)使用多個(gè)計(jì)算資源來(lái)處理一個(gè)計(jì)算任務(wù)ToberunusingmultipleCPUsAproblemisbrokenintodiscretepartsthatcanbesolvedconcurrentlyEachpartisfurtherbrokendowntoaseriesofinstructionsInstructionsfromeachpartexecutesimultaneouslyondifferentCPUsExampleExampleThecomputeresourcesmightbeAsinglecomputerwithmultipleprocessorsAnarbitrarynumberofcomputersconnectedbyanetworkAcombinationofbothThecomputationalproblemshouldbeabletoBebrokenapartintodiscretepiecesofworkthatcanbesolvedsimultaneouslyExecutemultipleprograminstructionsatanymomentintimeBesolvedinlesstimewithmultiplecomputeresourcesthanwithasinglecomputeresource加速比Goalofapplicationsinusingparallelmachines:SpeedupForafixedproblemsize(inputdataset),performance=1/time并行編程的重要性WhyparallelprogrammingNowwecanget:single-sourceapproachtomulti-andmany-coreSource：IDF2012However,the

ParallelizingCompilersAfter30yearsofintensiveresearchonlylimitedsuccessinparallelismdetectionandprogramtransformationsinstruction-levelparallelismatthebasic-blocklevelcanbedetectedparallelisminnestedfor-loopscontainingarrayswithsimpleindexexpressionscanbeanalyzedanalysistechniques,suchasdatadependenceanalysis,pointeranalysis,flowsensitiveanalysis,abstractinterpretation,...whenappliedacrossprocedureboundariesoftentakefartoolongandtendtobefragile,i.e.,canbreakdownaftersmallchangesintheprograminsteadoftrainingcompilerstorecognizeparallelism,peoplehavebeentrainedtowri