Topsummit大數(shù)據(jù)的虛擬化之路-VMware張君遲

大數(shù)據(jù)的虛擬化之路演講者：張君遲來自VMware虛擬化？現(xiàn)實(shí)。Source:Gartner“MagicQuadrantforx86ServerVirtualizationInfrastructure”byThomasJ.Bittman,GeorgeJ.Weiss,MarkA.Margevicius,PhilipDawson,June11,2012虛機(jī)部署的百分比2005200620072008200920102011201380%70%60%50%40%30%20%10%020152010年超越Empowerpeopleandorganizationsbyradically

simplifyingITthroughvirtualizationsoftware通過虛擬化軟件創(chuàng)新，徹底地簡(jiǎn)化IT虛擬化絕對(duì)的領(lǐng)導(dǎo)者超過50萬家客戶超過5.5萬家合作伙伴約1.3萬名員工#3什么是虛擬化？–名詞解釋：當(dāng)初x86體系計(jì)算機(jī)硬件設(shè)計(jì)思想是單臺(tái)運(yùn)行一個(gè)操作系統(tǒng)和一個(gè)應(yīng)用，造成大多數(shù)此類計(jì)算機(jī)的利用率偏低。虛擬化使得多個(gè)虛擬機(jī)能夠運(yùn)行在同一個(gè)物理計(jì)算機(jī)上，每個(gè)虛擬機(jī)共享物理機(jī)的資源。虛擬機(jī)可以支持大多類型的操作系統(tǒng)和各式各樣的應(yīng)用，最終它們都是運(yùn)行在同一臺(tái)物理計(jì)算機(jī)上。傳統(tǒng)架構(gòu)虛擬化架構(gòu)圖解……OSExchangeOperatingSystem虛擬化OSSAPERPOperatingSystem虛擬化OSFile/PrintOperatingSystem虛擬化OSOracleCRMOperatingSystem虛擬化虛擬化基礎(chǔ)架構(gòu)網(wǎng)絡(luò)交換池CPU池內(nèi)存池存儲(chǔ)池傳統(tǒng)視角虛擬化架構(gòu)動(dòng)畫解……OracleCRMOperatingSystemSAPERPOperatingSystemFile/PrintOperatingSystemExchangeOperatingSystem虛擬化基礎(chǔ)架構(gòu)網(wǎng)絡(luò)交換池CPU池內(nèi)存池存儲(chǔ)池動(dòng)畫解……交付的改變存儲(chǔ)計(jì)算網(wǎng)絡(luò)安全管理過去現(xiàn)在按

周、天計(jì)按分鐘、秒計(jì)為什么要大數(shù)據(jù)的虛擬化？設(shè)備越來越多!應(yīng)用越來越多！社交越來越多！數(shù)據(jù)能創(chuàng)造巨大價(jià)值，但保留和處理數(shù)據(jù)是有成本的……大數(shù)據(jù)時(shí)代Source:Gartner2020年，非結(jié)構(gòu)化數(shù)據(jù)10倍于結(jié)構(gòu)化數(shù)據(jù)的增長(zhǎng)結(jié)構(gòu)化數(shù)據(jù)非結(jié)構(gòu)化數(shù)據(jù)花10倍的投入買這些硬件，無以為繼。換一種思路解決……大數(shù)據(jù)的虛擬化將大數(shù)據(jù)的工作負(fù)載運(yùn)行或遷移到虛擬化的基礎(chǔ)環(huán)境中，繼承虛擬化的優(yōu)點(diǎn)。MPP

DBHadoopHBase虛擬化平臺(tái)

Hadoop虛擬化平臺(tái)

HBase

MPP監(jiān)控易于管理集群安裝和配置監(jiān)控硬件規(guī)劃和部署集群安裝和配置硬件規(guī)劃和部署虛擬化平臺(tái)集群整合共享資源，降低CAPEXΣ(Max)Max(Σ)效率對(duì)比物理集群虛擬化集群集群構(gòu)建采購(gòu)服務(wù)器搭建數(shù)據(jù)中心復(fù)雜手工步驟無需精確了解業(yè)務(wù)對(duì)資源消耗中心化IT管理完全端到端自動(dòng)化操作集群運(yùn)維故障發(fā)生需要立即反饋高容錯(cuò)自動(dòng)故障轉(zhuǎn)移容量計(jì)劃需要為未來做好規(guī)劃，預(yù)留未使用資源只需為現(xiàn)在準(zhǔn)備，所用即所需，無需預(yù)留資源增加計(jì)算/存儲(chǔ)能力需要重新采購(gòu)和搭建服務(wù)器一鍵觸發(fā)，自動(dòng)向資源池申請(qǐng)資源擴(kuò)展容量減少運(yùn)維成本（OPEX）減少資產(chǎn)投入（CAPEX）高回報(bào)（ROI）17動(dòng)態(tài)伸縮Hadoop－合理利用資源不同租戶部署各自的計(jì)算集群，共享分布式文件系統(tǒng)（HDFS）根據(jù)優(yōu)先級(jí)和可用資源動(dòng)態(tài)Adhocdatamining動(dòng)態(tài)資源控制數(shù)據(jù)層HDFSHostHostHostHostHostHostProductionrecommendationengine虛擬平臺(tái)計(jì)算層ComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVM測(cè)試集群生產(chǎn)集群ComputeVMJobTrackerJobTracker為什么要大數(shù)據(jù)的虛擬化？簡(jiǎn)化操作共享基礎(chǔ)架構(gòu)利用現(xiàn)有投入vSphereBigDataExtensionsVMware的BigData解決方案-BDEVMwarevSphereBigDataExtensions（簡(jiǎn)稱BDE）于2013年9月22日作為vSphere5.5的新功能正式上市。全新的BigDataExtensions件作為vSphere的插件發(fā)布。管理員可以直接從vCenter上部署、監(jiān)控和管理Hadoop集群。提高了Hadoop運(yùn)行效率。幾分鐘內(nèi)部署大數(shù)據(jù)集群服務(wù)器準(zhǔn)備操作系統(tǒng)安裝網(wǎng)絡(luò)配置大數(shù)據(jù)集群的安裝和配置手工部署流程自動(dòng)化的界面部署流程一鍵即可橫向擴(kuò)展集群輕松自定義配置集群Resourceconfiguration

ClusterSpecificationFile

"groups":[{"name":"master","roles":["hadoop_namenode","hadoop_jobtracker”],"storage":{"type":"SHARED”,sizeGB":20},"instance_type":MEDIUM,"instance_num":1,"ha":true},{"name":"worker","roles":["hadoop_datanode","hadoop_tasktracker"],"instance_type":SMALL,"instance_num":5,"ha":false

…Storageconfiguration

ChoiceofsharedstorageorLocaldiskHighavailabilityoption

#ofHadoopnodes

PredefinedSpecforStandardizationandEaseofConsumptionShipwithanumberofcommonclusterspecificationfilesPredefinespecssuitableforvaryingneedsoftheirusersEaseofconsumption–Itjustworks!StandardizationDeveloper3HadoopnodesCloudera,Pivotal

MapRSmallVMLocalstorageNoHA…DataScientist5HadoopnodesCloudera,PivotalHive,PigMediumVMHA…Highpriority50HadoopnodesClouderaHive,PigLargeVMHA…………YourChoiceofHadoopDistributionsandToolsCommunityProjectsDistributionsFlexibilitytochooseandtryoutmajordistributionsSupportformultipleprojectsOpenarchitecturetowelcomeindustryparticipationContributingHadoopVirtualizationExtensions(HVE)toopensourcecommunityAutomationofHadoopClusterLifecycleManagementDeployCustomizeLoaddataExecutejobsTuneconfigurationScaling…vSphereBigDataExtensionsChallengesofRunningHadoopinEnterprisesProductionTestExperimentationDeptA:recommendationengineDeptB:adtargetingProductionTestExperimentationLogfilesSocialdataTransactiondataHistoricalcustbehaviorPainPoints:ClustersprawlingRedundantcommondatainseparateclustersInefficientuseofresourcs,someclusterscouldberunningatcapacitywhileotherclustersaresittingidleNoSQLRealtimeSQL…Onthehorizon…Whatifyoucan…Experimentation

ProductionrecommendationengineProductionAdTargetingTest/DevProductionTestProductionTestExperimentationRecommendationengineAdtargetingExperimentationOnephysicalplatformtosupportmultiplevirtualbigdataclustersToday’sChallengesonHadoopInfrastructureFixedcomputeandstorageleadstolowutilizationandinflexibilityComputeandstoragelinkedtogetherwithfixedratiobasedonhardwarespecNotalljobsarecreatedequal(puteintensive)InflexibleinfrastructureleadstowasteToolittlecomputepowerslowprocessingToomuchcomputepowersittingidleProblemcompoundswithlargerclustersSowhathappens?Yahoo-averageCPUutilizationofHadoopclustersis<15%Twitter–usedifferenthardwareforclusters,expensivewaytoachievedefficiencyServerCompute

NodeData

NodeServerCompute

NodeData

NodeServerCompute

NodeData

NodeServerCompute

NodeStorage

NodeServerCompute

NodeGettingmoreoutofyourinfrastructureDecouplethelinkagebetweencomputeandstorageStatelesscomputecanelasticgrowandshrinkDatalocalityispreserved,placethecomputewheredataresidesExtracomputecapacitycanbeusedforotherworkloadsVMStorage

NodeVMComputelayerComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMStorageVMStorageVMStorageVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMStorageVMStorageVMStorageVMComputeVMStorageVMStoragelayerRunotherworkloadsRunHadoopStorageElastic,Multi-tenantHadoopwithVirtualizationComputeCombinedStorage/ComputeStorageT1T2VMVMVMVMVMVMUnmodifiedHadoop

nodeinaVMVMlifecycle

determined

byDatanodeLimitedelasticitySeparateComputefrom

StorageSeparatecompute

fromdataStatelesscomputeElasticcomputeSeparateVirtualComputeClusters

pertenantSeparatevirtualcomputeComputeclusterpertenantStrongerVM-gradesecurity

andresourceisolationHadoopNodeUsecase1:ElasticHadoopwithTierredSLAProductionworkloadshashighpriorityExperimentationworkloadshaslowerpriorityExperimentationDynamicresourcepoolDatalayerProductionrecommendationengineComputelayerComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMExperimentationProductionComputeVMExperimentation

MapreduceProduction

MapreduceVMwarevSphere+SerengetiUsecase2:ElasticHadoopforMultipledepartmentsCentralizeITisofferingHadooptomultipledepartmentsExperimentationDynamicresourcepoolDatalayerProductionrecommendationengineComputelayerComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMDepartment1Department2ComputeVMMapreduceMapreduceVMwarevSphere+SerengetiUsecase3:ElasticBigDataHadoopecosystemevolvingquicklytoincludemoreandmorecomputingengines(Hbase,streaming,interactivesqletc.)ExperimentationDynamicresourcepoolDatalayerProductionrecommendationengineComputelayerComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMHbaseRPHadoopResourcePoolComputeVMHbaseMapreduceVMwarevSphere+SerengetiHDFS

(HadoopDistributedFileSystem)HBase(Key-Valuestore)MapReduce(JobScheduling/ExecutionSystem)Pig(DataFlow)Hive(SQL)BIReportingETLToolsManagementServerZookeepr(Coordination)HCatalogRDBMSNamenodeJobtrackerHiveMetaDBHcatalogMDBServervSphereHAisbattle-testedhighavailabilitytechnologySinglemechanismtoachieveHAfortheentireHadoopstackOneclicktoenableHAand/orFTAchieveHAfortheEntireHadoopStackHybridstoragemodeltogetthebestofbothworldsMasternodes:Namenode,jobtrackeretc.onsharedstorageLeveragevSpherevMotion,HAandFTSlavenodesTasktracker/datanodeonlocalstorageLowercost,scalablebandwidthLocalStorageSharedStorageLeveragingIsilonasExternalHDFSTimetoresults:AnalysisofdatainplaceLowerriskusingvSpherewithIsilonScalestorageandcomputeindependentlyDataLayer–HadooponIsilonElasticVirtualComputeLayerProactivemonitoringwithvCOPsProactivelymonitoringthroughVCOPsGaincomprehensivevisibilityElimin