1-1開(kāi)源大數(shù)據(jù)技術(shù)之演進(jìn)-Fangjin Yang

EvolutionofOpenSourceDataInfrastructurePast,Present,andFutureFangjinYangCofounder@Imply2016-4-232016-4-23OverviewSimplertimeswithsmalldataTheriseofopensourceCurrentopensourcelandscapeWhereareweheaded?DataInsightsBroadly,wecareabouttwousecases:-OLTP-OLAPOLTP-businessprocessing-dealingwithtransactionsOLAP-reporting-businessintelligenceOLAPdata-dimensions&measuresSmallData++SmallDataAnalyticsTableau-Solutionsareverysimple-Fastandeasytoextractinsights-EasytocreatedifferentcustomvisualizationsDataGrowthabases-Oracle,Teradata,IBM,Microsoft,etc.Proprietarydatabasesareexpensive!TheRiseofHadoopHadoopGoogleGFSpaperpublishedin2003GoogleMapReducepaperpublishedin2004Nutchprojectstartedin2005atYahooNutchbecameHadoopandwasopensourcedin2006CommunityquicklygrewEarlyOpenSourceStacksDataDatabaseApplications/usersDataHadoopApplications/usersStorageStorageHadoopDataPProcessing(MapReduce)Applications/usersHadoopWhenonetechnologybecomesveryadopted,itslimitationsalsobecomemorewellknownHadoopisaveryflexiblesolutionMostcommonlyusedfordataprocessingNotoptimizedformanythings-manyinefficiencies!HadoopData QueriesApplications/usersRiseofOpenSourceDataInfrastructureThingsHadoopisn’tgoodat:-Fastqueries-Deliver(streamsof)events-Streamprocessing-In-memorycomputationTheselimitationsledtonewtechnologiestobecreatedDataInfrastructureSpaceTodayModernOpenSourceStacksData DeliveryProcessingStorageQueryingApplications/usersModernOpenSourceStacksData DeliveryProcessingStorageQueryingApplications/usersDataDeliveryDataproducers DeliveryDataconsumersDataDeliveryFocusisstoringdataforalimitedtimeanddeliveringitelsewhereThreedifferentapproaches-ApacheKafka-publish/subscribe,transactionqueues-RabbitMQ-publish/subscribe,distributedqueues-ApacheFlume-push-basedeventdeliveryDataDeliveryDataDataStorageDDeliveryStorageStorageDistributedfilesystemsStoredataindefinitelyStandard:HDFSCanoverlapwithdeliverysystems(e.g.Kafka)ProcessingProcessingsystemsaredesignedtotransformdataHasoverlapwithqueryingsystems-Querysystems:outputsetsmallerthaninputset-Processingsystems:outputsetsamesizeasinputset-HavingseparationismorestandardnowadaysDataDataDataDataStreamProcessingSStreamProcessingDeliveryStorageSStreamProcessingQueryingDeliveryStreamProcessingSystemsdealwithunboundedmessages/eventsDifferentapproaches-SparkStreaming-Storm-Samza-KafkaStreams-etc.StreamProcessingDataDataBatchProcessingBBatchProcessingDeliveryStorageBatchProcessingManipulate(large)staticsetsofdataDifferentapproaches-Spark-HadoopBatchProcessingQueryingDataDeliveryApplications/users ProcessingDataDeliveryApplications/usersDataDeliveryApplications/users ProcessingStorageQueryingDataDeliveryApplications/usersQueryingLargestandmostcomplex(broadrangeofusecases)Let’sfocusonthemostcommonusecase:-Businessintelligence/datawarehousing/OLAPSignificantoverlapwithstorage-SeparationisbecomingmorecommonSQL-on-HadoopEnablead-hocqueriesondifferentinputformatsExamples:Impala,Hive,SparkSQL,Drill,PrestoSQL-on-HadoopSQL-on-HadoopAdvantages:-Flexible/wfullSQLsupportDisadvantages:-Slow-serialization/deserializationcanhaveoverheadManynewstorageformats-ApacheParquet,ApacheKudu,ApacheArrow,etc.Key/ValueStoresVeryfastwritesVeryfastlookupsTimeseriesdatabasesoftenhaveK/VstorageenginesKey/ValueStoresPre-computation-Pre-computeeverypossiblequery-Pre-computeasetofqueries-ExponentialscalingcostsKey/ValueStores-Primarykey:dimensions/attributes-Value:measures/metrics(thingstoaggregate)-Stilltooslow!ColumnstoresLoad/scanexactlywhatyouneedforaqueryDifferentcompressionalgorithmsfordifferentcolumns-Encodingforstringcolumns-CompressionformeasurecolumnsDifferentindexesfordifferentcolumnsDruidTargetedforextremelylowlatencyqueries-poweringuser-facinganalyticapplicationsCustomcolumnformatoptimizedforeventdataandBIqueriesSupportslotsofconcurrentreadsStreamingdataingestionSomanychoices!Doestheprojectsolveyourusecase?Isthereanactiveandgrowingcommunity?10xfasteror10xcheaper--upgrade!TheNextFewYearsGeneralTrendsNumberofprojectsreachingsaturation