煉數(shù)hadoop數(shù)據(jù)分析平臺(tái)十一、hbase hive pig zookeeper性能調(diào)校

1、-Hive Hive執(zhí)行的總體性能進(jìn)行了調(diào)研，下面以分條的形式列舉出來(lái)。hive.map.aggr=trueHiveQL 聚合的執(zhí)行性能。這個(gè)設(shè)置可以將頂層的聚合操作Map Reduce hiveSEThiveSELECT-Hive Hive執(zhí)行的總體性能進(jìn)行了調(diào)研，下面以分條的形式列舉出來(lái)。hive.map.aggr=trueHiveQL 聚合的執(zhí)行性能。這個(gè)設(shè)置可以將頂層的聚合操作Map Reduce hiveSEThiveSELECTcount(*),avg(salary)FROMemployeesgroupby*having MapReduce程序。而對(duì)于其它類型的查詢，比如指定字段或

2、者包含表關(guān)聯(lián)的查詢等，Hive MapReduce 查詢的MapReduce 執(zhí)行，可以設(shè)置hive.exec.m （陷阱）hiveSELECTname,salary,deductionsFederalFROMemployeesWHEREdeductionsFederalTaxes 查出的數(shù)據(jù)實(shí)際上是=0.2 。原因：IEEE 0.200000000001,0.2 cast函數(shù)。WHEREdeductionsFederalTaxescast(0.2ASHive Join Hive streaming反之，該表多次，特別是包含Join SELECT /*+ STREAMTABLE(s) */ s

3、.ymd, s.symbol, s.price_close, d.dividend FROMstockssJOINdividendsdONs.ymd=d.ymdANDs.symbol=d.symbol WHERE s.symbol = AAPL;STREAMTABLEHivewhere SELECTS*FROMstocksJOINdividendsON條件，Hive會(huì)對(duì)兩SELECT*FROMstocksJOINWHEREwhere SELECTS*FROMstocksJOINdividendsON條件，Hive會(huì)對(duì)兩SELECT*FROMstocksJOINWHEREstock.symbol

4、dividends.symbolandstock.symbol=AAPL;/ Hivewhere 條件之前執(zhí)行，所以這條語(yǔ)句與上一條語(yǔ)句執(zhí)行時(shí)間相當(dāng)！SELECT*FROMstocksJOINdividendsONstock.symboldividends.symbol;/InnerJoin ap Map-sideSELECT/*+MAPJOIN(d)*/s.ymd,s.symbol,s.price_close,FROMstockssJOINdividendsdONs.ymd=d.ymdANDs.symbol=d.symbol WHERE s.symbol = AAPL;Hivev0.7之前，

5、MAPJOIN()會(huì)將指定的表，一般是較小的表，加載到內(nèi)存中，這樣整 Reduce 操作，進(jìn)而提高整體性能。在Hive v0.7 之后，需要設(shè)置 hive.auto.convert.jorue，開(kāi)啟MapJoin功能。注：另外也可以進(jìn)行bucketMapJoin 的優(yōu)化，具體理解，待調(diào)研。Orderbyvs.Sortbyvs.DistributeByvs.ClusterOrderByreducer上執(zhí)行，得到的結(jié)果是整體有序的。但是由于 MapReduceHash SortBy reducer 失。DistributeBy reducerreducerreducer的SELECTs.ymd,s

6、.symbol,s.price_close FROM stocks sDISTRIBUTEBYSORTs.symbolASC,s.ymdClusterBy=DistributeBy+SortOrderBy 之外產(chǎn)生的所有排序結(jié)果默認(rèn)情況下（的值）Hive 支持抽樣查詢例一hiveSELECT*fromnumbers24UCKET3OUTOF10ONrand()hiveHive 支持抽樣查詢例一hiveSELECT*fromnumbers24UCKET3OUTOF10ONrand()hiveSELECT*fromnumbers2UCKET3OUTOF10ONnumber) 如果不用 Bucket

7、語(yǔ)句中，分母(eg10)。SELECT*FROMnumbersflatLE(0.1PERCENT)HDFSUNION ALL SELECTlog.ymd,log.level,FROMSELECT l1.ymd, l1.level, l1.message,Log1ASsourceFROMlog1l1 UNION ALLSELECT l2.ymd, l2.level, l2.message,Log2ASsourceFROMlog1)SORTBYlog.ymd注：要求兩個(gè)表查詢結(jié)果的字段的個(gè)數(shù)和類型必須一致（技巧）EXPLAIN/EXPLAINEXTENDEDLimit Whentryingasma

8、llersubsetofdataforsimplehowmuchsizeweneedtoguaranteeeachrowtoWhentryingasmallersubsetofdataforsimpleumnumberoffilesweWhentryingasmallersubsetofdataforsimpleumnumberoffileswecan本地模式localset mapred.job.tracker=local;setSELECT*frompeople set并行執(zhí)行 Hive query MapperReducer Hivereducer數(shù)量，但這個(gè)hive.exec.redu

9、cers.bytes.per.reducer1GB，也可以改變這edwardetl02$hadoopdfs-count/user/media6/fracture/ins/*|tail-4 1 8 2614608737 hdfs:/./user/media6/fracture/ins/hit_date=20120118172742992546 1172656878252 12362657644 Hadoop hadoop-mapper reducerreducer 0JVM JVM mapreduce jvm Howmanytaskstorunperjvm.Ifsetto-1,thereisno

10、limit. JVM Reuse | 139 Hadoop會(huì)啟動(dòng)同一個(gè)任務(wù)的多個(gè)副本在集群上執(zhí)行，但它會(huì)丟入If true, then multiple instanofsomemaptasks may be executed in parallel.If true, then multiple instanofsomereduce Hadoop會(huì)啟動(dòng)同一個(gè)任務(wù)的多個(gè)副本在集群上執(zhí)行，但它會(huì)丟入If true, then multiple instanofsomemaptasks may be executed in parallel.If true, then multiple instanofsomereducetasks may be executed in parallel.Hive 提供了三個(gè)虛擬列：INPUT FILE NAME, BLOCK OFFSET INSIDE FILE ROW OFFSET INSIDE BLOCKROW OFFSET INSIDE BLOCK hiveSELECTINPUT FILE NAME,BLOCK OFFSET INSIDE FI