大數(shù)據(jù)技術(shù)年會(huì)

開(kāi)放平臺(tái)

ApsaraCloudPlatformAboutAliyunChina’slargestcloudserviceprovider100softhousandsofcustomersBillionsofaccesseseverydayAlibaba-operatedIDCPartner-operatedIDCApsaraCloudPlatformACE3rd-partyPlatformServicesMap,Mail,Search3rd-partyApplicationServicesCustomersISVandSIDevelopersProvidingFoundationServicesoftheCloudEco-systemPaybyusageElasticitySafety(like“tapwater”)TheNatureofCloudComputingScale大規(guī)模Economy低成本PublicUtility服務(wù)運(yùn)營(yíng)Internet-scalecomputing2.5EBgeneratedperday,doublingevery40monthsBillionsoftxnsonTaobaoeveryday,mustbeprocessedin6hoursEconomymeansmorethanlowpricesLeadingtobehaviorchanges(like“telephone”)Keyisscheduling(like“powergrid”)TwoDesignPrinciplesLarge-scalegeneralcomputingplatformasthebaseOnesystemsupportingbothofflineandonlineservicesMulti-tenancy,resourcesharing,loadshiftingWeb-basedAPIasthedeliverymechanismOnlineactivation,pay-by-usageLocation-transparencyLinuxClusterIDCResourceManagement(伏羲)Security(鐘馗)RPC(夸父)Naming/Coordination(女?huà)z)ClusterDeployment(大禹)ClusterMonitor(神農(nóng))DistributedFileSystem(盤(pán)古)JobScheduling(伏羲)ACEOSSOTSODPSECS/SLBRDSMap,Mail,Search,etcCloudMartOtherCloudServicesOSPSCloudComputingServicesElasticComputing彈性計(jì)算ECS:virtualizedinstancesofserversthatcanbecreatedandtailoredtomeetapplicationrequirementsSLB:softwareloadbalancingtechnologythatcanelasticallyexpandservicecapacityondemandACE:ConvenientandefficientexecutionenvironmentforWebservices,supportingJava,PHP,Node.jsStorageandDatabases海量存儲(chǔ)和數(shù)據(jù)庫(kù)Large-scaleComputing大規(guī)模計(jì)算CloudComputingServicesElasticComputing彈性計(jì)算StorageandDatabases海量存儲(chǔ)和數(shù)據(jù)庫(kù)OSS:large-scaleobjectstorageserviceforunstructureddatasuchasphotos,music,orvideoOTS:largescalestorageserviceforstructuredorsemi-structureddatastorageandreal-timequeryRDS:managedinstancesforrelationaldatabaseswithautomaticbackupandfailoverLarge-scaleDataComputing大規(guī)模計(jì)算AComparisonofStorageandDatabaseServicesOSSOTSRDSDataModelUnstructuredSemi-structuredFully-structuredTargetDataVolume~10PB~100TB/table~TB/dbTxnSupportNoneLimitedsupportFullSupportProgrammingInterfaceRESTfulAPIRESTfulAPISQLCloudComputingServicesElasticComputing彈性計(jì)算StorageandDatabase海量存儲(chǔ)和數(shù)據(jù)庫(kù)Large-scaleComputing大規(guī)模計(jì)算ODPS:large-scaledatabatchprocessingandcomputation,supportingSQLandMapReducestyleprogramminglanguagesOSPS:streamdataprocessingservice,supportingSQL-likequerylanguageandautomaticfailurerecoveryApsaraTechnicalHighlightsAcommonplatformsupportingbothofflineandonlineservicesSearch:24Bpagesprocessed,13BonlineindexMail:100Mmailsreceived,10Mmailssent,10mslatencyCapability-basedsecuritymanagementframework,enforcingthePrincipleofLeastPrivilegeDistributeddeployment,monitoringanddiagnosticsZeroSPOF(single-point-of-failure):availability>99.9%Alldatahas3replicas:datareliability>99.99999999%5K2013/08/15:First-ever5000-nodeApsaracluster(ODPS)wentintoproduction100KCPUcores,100PBrawstorageProcessingpetabytesperday2013/09/24:OpenedaccesstoODPSfor4universities&researchinstitutionsSorting100TBin30minutesCurrentknownrecord:72minutes(Yahoo!,2013/07/03)Pangu:Large-scaleDistributedFileSystemMaster-SlaveArchitectureMasterformetadatamgmt,Slave(ChunkServer)forIOmgmtPaxos-basedmulti-masterarchitecture,failurerecoverytime<1minuteEnd-to-endinlinechecksumScalesto1billionfilesCSCSCSCSCSMMMPaxosSeparatedIOPipelineandStorageMgmtAdaptiveIOPipelineReplicationmaster:chunkservervsclientReplicationpolicy:chainingvsstar-replicationChunkingpolicy:fixed,variable,orRAIDDurabilityguarantee:txnloggingvssequentialwriteCommonStorageManagementPhysicalIOmanagementPriorityandQoSBackgroundre-replicationChunkplacementStagedEvent-drivenPhysicalIOMgmtChunkServerwouldrearrangeIOrequeststosupportpriority,QoS,andreduceIOseekoverheadDistributedRe-replication1TB1TB1TBTypical:Mirroring(10hours)1TB1TB1TBPangu:Distributedre-replication(20min,50-nodes)1TBIntelligentschedulingBalancedstorageBandwidththrottlingMinimizingdatalossRAIDBuiltintothecoresysteminsteadofanadd-onlayer(asinHDFSRAID)Bettermanagementofdataintegrity,recovery,andchunkplacementSynchronousredundancyblockgenerationLow-latencyfailurerecoverySmallfilesupportFuxiMasterFuxiMaster...AppMasterAPPWorkerAppMasterAPPWorker...Client.........TuboTuboJobcontrolResourcerequestsNodecontrolJobsubmissionAPPWorkerAPPWorkerAPPWorkerTuboTuboFuxiResourceSchedulingMulti-dimensionresourcesElasticquotaCGroup-basedisolationFuxiMasterHAAppMasterfailoverIncrementalschedulingFuxiJobProgrammingModelJob:ADAGVertex:TaskEachtaskmayhavemultipleinstancesbasedoninputdatachunksEdge:dataflow,eachtaskmayhavemultipleinput/outputflowsAdataflowconnectingtwotasksrepresentsdatashufflingDAinput1input2output2BCDAABBCDoutput1MapReduceoutputinputMapMapReduceMapReduceisadegeneratedcaseExample:FindBest-SellersSELECTprod_id,Sum(count)ASquantityFROMordersGROUPBYprod_idORDERBYquantityDESC;order_idprod_idunit_pricecount0001042151003000203343101000003012345010000404215805…………prod_idquantity0251831790075632845109641229430421520043……ordersFuxiMapReduceComparisonofJob-executionPlanInputMMR1R2R2prod_idcountprod_idquantityprod_idquantityInputMMR1R2R2prod_idcountprod_idquantityprod_idquantityM2prod_idquantityFuxiMapReduceComparisonofJobExecutionR1MR1MM2R2R2ABriefHistory02/04/2009FirstlineofApsaracode08/27/2010Apsarabecamethecommonplatfor