Topsummit大數(shù)據(jù)的虛擬化之路-VMware張君遲

大數(shù)據(jù)的虛擬化之路演講者：張君遲來自VMware虛擬化？現(xiàn)實。Source:Gartner“MagicQuadrantforx86ServerVirtualizationInfrastructure”byThomasJ.Bittman,GeorgeJ.Weiss,MarkA.Margevicius,PhilipDawson,June11,2012虛機部署的百分比2005200620072008200920102011201380%70%60%50%40%30%20%10%020152010年超越Empowerpeopleandorganizationsbyradically

simplifyingITthroughvirtualizationsoftware通過虛擬化軟件創(chuàng)新，徹底地簡化IT虛擬化絕對的領(lǐng)導(dǎo)者超過50萬家客戶超過5.5萬家合作伙伴約1.3萬名員工#3什么是虛擬化？–名詞解釋：當(dāng)初x86體系計算機硬件設(shè)計思想是單臺運行一個操作系統(tǒng)和一個應(yīng)用，造成大多數(shù)此類計算機的利用率偏低。虛擬化使得多個虛擬機能夠運行在同一個物理計算機上，每個虛擬機共享物理機的資源。虛擬機可以支持大多類型的操作系統(tǒng)和各式各樣的應(yīng)用，最終它們都是運行在同一臺物理計算機上。傳統(tǒng)架構(gòu)虛擬化架構(gòu)圖解……OSExchangeOperatingSystem虛擬化OSSAPERPOperatingSystem虛擬化OSFile/PrintOperatingSystem虛擬化OSOracleCRMOperatingSystem虛擬化虛擬化基礎(chǔ)架構(gòu)網(wǎng)絡(luò)交換池CPU池內(nèi)存池存儲池傳統(tǒng)視角虛擬化架構(gòu)動畫解……OracleCRMOperatingSystemSAPERPOperatingSystemFile/PrintOperatingSystemExchangeOperatingSystem虛擬化基礎(chǔ)架構(gòu)網(wǎng)絡(luò)交換池CPU池內(nèi)存池存儲池動畫解……交付的改變存儲計算網(wǎng)絡(luò)安全管理過去現(xiàn)在按

周、天計按分鐘、秒計為什么要大數(shù)據(jù)的虛擬化？設(shè)備越來越多!應(yīng)用越來越多！社交越來越多！數(shù)據(jù)能創(chuàng)造巨大價值，但保留和處理數(shù)據(jù)是有成本的……大數(shù)據(jù)時代Source:Gartner2020年，非結(jié)構(gòu)化數(shù)據(jù)10倍于結(jié)構(gòu)化數(shù)據(jù)的增長結(jié)構(gòu)化數(shù)據(jù)非結(jié)構(gòu)化數(shù)據(jù)花10倍的投入買這些硬件，無以為繼。換一種思路解決……大數(shù)據(jù)的虛擬化將大數(shù)據(jù)的工作負(fù)載運行或遷移到虛擬化的基礎(chǔ)環(huán)境中，繼承虛擬化的優(yōu)點。MPP

DBHadoopHBase虛擬化平臺

Hadoop虛擬化平臺

HBase

MPP監(jiān)控易于管理集群安裝和配置監(jiān)控硬件規(guī)劃和部署集群安裝和配置硬件規(guī)劃和部署虛擬化平臺集群整合共享資源，降低CAPEXΣ(Max)Max(Σ)效率對比物理集群虛擬化集群集群構(gòu)建采購服務(wù)器搭建數(shù)據(jù)中心復(fù)雜手工步驟無需精確了解業(yè)務(wù)對資源消耗中心化IT管理完全端到端自動化操作集群運維故障發(fā)生需要立即反饋高容錯自動故障轉(zhuǎn)移容量計劃需要為未來做好規(guī)劃，預(yù)留未使用資源只需為現(xiàn)在準(zhǔn)備，所用即所需，無需預(yù)留資源增加計算/存儲能力需要重新采購和搭建服務(wù)器一鍵觸發(fā)，自動向資源池申請資源擴展容量減少運維成本（OPEX）減少資產(chǎn)投入（CAPEX）高回報（ROI）17動態(tài)伸縮Hadoop－合理利用資源不同租戶部署各自的計算集群，共享分布式文件系統(tǒng)（HDFS）根據(jù)優(yōu)先級和可用資源動態(tài)Adhocdatamining動態(tài)資源控制數(shù)據(jù)層HDFSHostHostHostHostHostHostProductionrecommendationengine虛擬平臺計算層ComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVM測試集群生產(chǎn)集群ComputeVMJobTrackerJobTracker為什么要大數(shù)據(jù)的虛擬化？簡化操作共享基礎(chǔ)架構(gòu)利用現(xiàn)有投入vSphereBigDataExtensionsVMware的BigData解決方案-BDEVMwarevSphereBigDataExtensions（簡稱BDE）于2013年9月22日作為vSphere5.5的新功能正式上市。全新的BigDataExtensions件作為vSphere的插件發(fā)布。管理員可以直接從vCenter上部署、監(jiān)控和管理Hadoop集群。提高了Hadoop運行效率。幾分鐘內(nèi)部署大數(shù)據(jù)集群服務(wù)器準(zhǔn)備操作系統(tǒng)安裝網(wǎng)絡(luò)配置大數(shù)據(jù)集群的安裝和配置手工部署流程自動化的界面部署流程一鍵即可橫向擴展集群輕松自定義配置集群Resourceconfiguration

ClusterSpecificationFile

"groups":[{"name":"master","roles":["hadoop_namenode","hadoop_jobtracker”],"storage":{"type":"SHARED”,sizeGB":20},"instance_type":MEDIUM,"instance_num":1,"ha":true},{"name":"worker","roles":["hadoop_datanode","hadoop_tasktracker"],"instance_type":SMALL,"instance_num":5,"ha":false

…Storageconfiguration

ChoiceofsharedstorageorLocaldiskHighavailabilityoption

#ofHadoopnodes

PredefinedSpecforStandardizationandEaseofConsumptionShipwithanumberofcommonclusterspecificationfilesPredefinespecssuitableforvaryingneedsoftheirusersEaseofconsumption–Itjustworks!StandardizationDeveloper3HadoopnodesCloudera,Pivotal

MapRSmallVMLocalstorageNoHA…DataScientist5HadoopnodesCloudera,PivotalHive,PigMediumVMHA…Highpriority50HadoopnodesClouderaHive,PigLargeVMHA…………YourChoiceofHadoopDistributionsandToolsCommunityProjectsDistributionsFlexibilitytochooseandtryoutmajordistributionsSupportformultipleprojectsOpenarchitecturetowelcomeindustryparticipationContributingHadoopVirtualizationExtensions(HVE)toopensourcecommunityAutomationofHadoopClusterLifecycleManagementDeployCustomizeLoaddataExecutejobsTuneconfigurationScaling…vSphereBigDataExtensionsChallengesofRunningHadoopinEnterprisesProductionTestExperimentationDeptA:recommendationengineDeptB:adtargetingProductionTestExperimentationLogfilesSocialdataTransactiondataHistoricalcustbehaviorPainPoints:ClustersprawlingRedundantcommondatainseparateclustersInefficientuseofresourcs,someclusterscouldberunningatcapacitywhileotherclustersaresittingidleNoSQLRealtimeSQL…Onthehorizon…Whatifyoucan…Experimentation

ProductionrecommendationengineProductionAdTargetingTest/DevProductionTestProductionTestExperimentationRecommendationengineAdtargetingExperimentationOnephysicalplatformtosupportmultiplevirtualbigdataclustersToday’sChallengesonHadoopInfrastructureFixedcomputeandstorageleadstolowutilizationandinflexibilityComputeandstoragelinkedtogetherwithfixedratiobasedonhardwarespecNotalljobsarecreatedequal(puteintensive)InflexibleinfrastructureleadstowasteToolittlecomputepowerslowprocessingToomuchcomputepowersittingidleProblemcompoundswithlargerclustersSowhathappens?Yahoo-averageCPUutilizationofHadoopclustersis<15%Twitter–usedifferenthardwareforclusters,expensivewaytoachievedefficiencyServerCompute

NodeData

NodeServerCompute

NodeData

NodeServerCompute

NodeData

NodeServerCompute

NodeStorage

NodeServerCompute

NodeGettingmoreoutofyourinfrastructureDecouplethelinkagebetweencomputeandstorageStatelesscomputecanelasticgrowandshrinkDatalocalityispreserved,placethecomputewheredataresidesExtracomputecapacitycanbeusedforotherworkloadsVMStorage

NodeVMComputelayerComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMStorageVMStorageVMStorageVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMStorageVMStorageVMStorageVMComputeVMStorageVMStoragelayerRunotherworkloadsRunHadoopStorageElastic,Multi-tenantHadoopwithVirtualizationComputeCombinedStorage/ComputeStorageT1T2VMVMVMVMVMVMUnmodifiedHadoop

nodeinaVMVMlifecycle

determined

byDatanodeLimitedelasticitySeparateComputefrom

StorageSeparatecompute

fromdataStatelesscomputeElasticcomputeSeparateVirtualComputeClusters

pertenantSeparatevirtualcomputeComputeclusterpertenantStrongerVM-gradesecurity

andresourceisolationHadoopNodeUsecase1:ElasticHadoopwithTierredSLAProductionworkloadshashighpriorityExperimentationworkloadshaslowerpriorityExperimentationDynamicresourcepoolDatalayerProductionrecommendationengineComputelayerComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMExperimentationProductionComputeVMExperimentation

MapreduceProduction

MapreduceVMwarevSphere+SerengetiUsecase2:ElasticHadoopforMultipledepartmentsCentralizeITisofferingHadooptomultipledepartmentsExperimentationDynamicresourcepoolDatalayerProductionrecommendationengineComputelayerComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMDepartment1Department2ComputeVMMapreduceMapreduceVMwarevSphere+SerengetiUsecase3:ElasticBigDataHadoopecosystemevolvingquicklytoincludemoreandmorecomputingengines(Hbase,streaming,interactivesqletc.)ExperimentationDynamicresourcepoolDatalayerProductionrecommendationengineComputelayerComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMHbaseRPHadoopResourcePoolComputeVMHbaseMapreduceVMwarevSphere+SerengetiHDFS

(HadoopDistributedFileSystem)HBase(Key-Valuestore)MapReduce(JobScheduling/ExecutionSystem)Pig(DataFlow)Hive(SQL)BIReportingETLToolsManagementServerZookeepr(Coordination)HCatalogRDBMSNamenodeJobtrackerHiveMetaDBHcatalogMDBServervSphereHAisbattle-testedhighavailabilitytechnologySinglemechanismtoachieveHAfortheentireHadoopstackOneclicktoenableHAand/orFTAchieveHAfortheEntireHadoopStackHybridstoragemodeltogetthebestofbothworldsMasternodes:Namenode,jobtrackeretc.onsharedstorageLeveragevSpherevMotion,HAandFTSlavenodesTasktracker/datanodeonlocalstorageLowercost,scalablebandwidthLocalStorageSharedStorageLeveragingIsilonasExternalHDFSTimetoresults:AnalysisofdatainplaceLowerriskusingvSpherewithIsilonScalestorageandcomputeindependentlyDataLayer–HadooponIsilonElasticVirtualComputeLayerProactivemonitoringwithvCOPsProactivelymonitoringthroughVCOPsGaincomprehensivevisibilityElimin