文案分析與火花

Agenda

WhatClouderadoesforSparkEcosystem

AdvancedAnalyticswithSpark

SparkEngineeringinCloudera

ClouderaembracedSparkinearly2014

EngineeringwithInteltobroadenSparkecosystem

Hive-on-Spark

Pig-on-Spark

Spark-over-YARN

SparkStreamingReliability

GeneralSparkOptimization

HiveonSpark

Technology

Hive:“standard”SQLtoolinHadoop

Spark:next-gendistributedprocessingframework

Hive+Spark

Performance

Minimumfeaturegap

Industry

AlotofcustomersheavilyinvestinHive

WanttoleveragetheSparkengine

DesignPrinciples

NoorlimitedimpactonHive’sexistingcodepath

Maximizecodereuse

Minimumfeaturecustomization

Lowfuturemaintenancecost

ClassHierarchy

TaskCompiler

generates

Task

describedby

Work

MapRedCompiler

TezCompiler

MapRedTask

TezTask

MapRedWork

TezWork

SparkCompiler

SparkTask

SparkWork

Work–MetadataforTask

MapReduceWorkcontainsoneMapWorkandapossibleReduceWork

SparkWorkcontainsagraphofMapWorksandReduceWorks

MapWork1

ReduceWork1

MRJob1

MapWork2

ReduceWork2

MRJob2

Query:selectname,

MapWork1

ReduceWork1

ReduceWork2

sum(value)asvfromdec

groupbynameorderbyv;

SparkJob

SparkClientandSparkContext

SparkClient

TalkingtoSparkcluster

Supportlocal,local-cluster,standalone,yarn-cluster,yarn-client

Jobsubmission,monitoring,errorreporting,statistics,metrics,counters

SparkContext

CoreofSparkclient

Heavy-weighted,thread-unsafe

Designedforasingle-userapplication

Doesn’tworkinmulti-sessionenvironment

RemoteSparkContext

BeingcreatedandlivingoutsideHiveServer2

Inyarn-clustermode,Sparkcontextlivesinapplicationmaster(AM)

Otherwise,Sparkcontextlivesinaseparateprocess(otherthanHS2)

AM(RSC)

Node1

AM(RSC)

Node2

Node3

HiveServer2

YARNCluster

Session2

User2

Session1

User1

DataProcessingviaSpark

TreatTableasHadoopRDD(inputRDD)

ApplythefunctionthatwrapsMR’smap-sideprocessing

ShufflemapoutputusingSpark’stransformations(groupByKey,sortByKey,etc)

ApplythefunctionthatwrapsMR’sreduce-sideprocessing

SparkPlan

MapInput–encapsulateatable

MapTran–map-sideprocessing

ShuffleTran–shuffling

ReduceTran–reduce-sideprocessing

Query:Selectname,sum(value)asvfromdecgroupbynameorderbyv;

CurrentStatus

AllfunctionalityinHiveisimplemented

Firstroundofoptimizationiscompleted

Mapjoin,SMB

Splitgenerationandgrouping

CBO,vectorization

Moreoptimizationandbenchmarkingcoming

BetainCDH

/cloudera-labs/hive-on-spark/

/content/cloudera/en/documentation/hive-

spark/latest/PDF/hive-spark-get-started.pdf

AdvancedAnalyticswithSpark

WrittenbyClouderadatascienceteam

FirsteverbookbridgingMLwithHadoopecosystem

Focusingonusecasesandexamplesratherthanamanual

Targetfordatascientistsolvingrealwordanalysisproblems

GenerallyavailableinMay2015

AnalyzingBigData

Buildingamodeltodetectcreditcardfraudusingthousandsoffeaturesandbillionsoftransactions

Intelligentlyrecommendmillionsofproductstomillionsofusers

Estimatefinancialriskthroughsimulationsofportfoliosincludingmillionsofinstruments

Easilymanipulatedatafromthousandsofhumangenomestodetectgeneticassociationswithdisease

ChallengesofDataScience

Datapreprocessing

Variousfastdatafrommultiplesourcerequirespowerfuldatapipeline

Iteration

Fundamentalpartofdatascience

Acceleratingdiskdataloadingismuchhelpful

Fromlabtoproduction

Makedatausefultonon-datascientists

Modelsbecomepartoftheproductionserviceandmayneedtoberebuiltperiodicallyoreveninrealtime.

ValueatRisk

VaR（風(fēng)險價值或者風(fēng)險收益）

指在一定的持有期和給定的置信水平下，利率、匯率等市場風(fēng)險要素發(fā)生變化時可能對某項資金頭寸、資產(chǎn)組合或機構(gòu)造成的潛在最大損失。

例如，在持有期為1天、置信水平為99%的情況下，若所計算的風(fēng)險價值為10萬人民幣，則表明該銀行的資產(chǎn)組合在1天中的損失有99%的可能不會超過10萬人民幣。

IntroducedbyHarryMarkowitzin1952,NobelPrizeinEconomicsin1990

IllustrationforVaR

MethodsforCalculatingVaR

Variance-Covariance

HistoricalSimulation

MonteCarloSimulation

EstimatingthroughMonteCarloSimulation

instrument

Normalize

marketfactors Input

Modeling MonteCarlo

Simulation

Oil Bonds SNP SNP

Sampling

MonteCarloSimulationwithSpark

Normalizedata

Fillthemissingvalue

Transformthehistoricaldatatotwo-weeks’return

Modeling

Definefactorfeatures

Useregressionmodeltocomputefactorweights

Sampling

Takethecorrelationinformationbetweenthefactorsintoaccount

IfS&Pisdown,theDowislikelytobedownaswell

Simulation

Broadcastinstrumentstoeachnode

Parallelizetrialcomputationacrossworkers

Compu