hadoop+hbase集群搭建 詳細(xì)手冊(cè)V0.9

Hadoop，ZooKeeper，HBase分布式搭建前提準(zhǔn)備注：hbase的安裝需要hadoop和zookeeper和hbase，生產(chǎn)環(huán)境下需要將zookeeper獨(dú)立安裝，hbase軟件選擇cloudera的cdh3u0，這樣不會(huì)出現(xiàn)版本不兼容等問(wèn)題。apache的版本需要重新編譯hadoop0.20.2-appender版以保證沒(méi)有數(shù)據(jù)丟失。1、機(jī)器集群結(jié)構(gòu)分布使用8臺(tái)曙光服務(wù)器搭建集群，ip為*.*.83.1-8，hostname為hadoop-node1到hadoop-node8，其中機(jī)器名IP所安裝的程序hadoop-node1 hadoopnamenode/hbaseHMasterhadoop-node2*.*.83.2 hadoopdatanode/hbaseHRegionServerhadoop-node3 *.*.83.3 hadoopdatanode/hbaseHRegionServerhadoop-node4 *.*.83.4hadoopdatanode/hbaseHRegionServerhadoop-node5 *.*.83.5hadoopdatanode/hbaseHRegionServerhadoop-node6 *.*.83.6hadoopdatanode HRegionServer zookeeperhadoop-node7 *.*.83.7hadoopdatanode HRegionServer zookeeperhadoop-node8 *.*.83.8hadoopsecondNamenode/hbaseHMasterzookeeper2.硬件環(huán)境：8臺(tái)曙光服務(wù)器，cpu8核，每臺(tái)機(jī)器都有兩個(gè)500GB硬盤。操作系統(tǒng)：SUSEenterprise11操作系統(tǒng)分區(qū)：硬盤1：sda/boot400MB/Swap10GB/余下的sda空間硬盤2：/sdb2都掛載在/sdb2上3.提前準(zhǔn)備相關(guān)軟件Jdk： jdk-6u30-linux-x64.binHadoop： hadoop-0.20.2-cdh-3u**Zookeeper： zookeeper-3.3.5-cdh3u**Hbase： hbase-0.906-c*******詳細(xì)操作：當(dāng)前用戶hadoop/home/hadoop/app/app/hadoop/hadoop1.0.0//app/hbase/hbase0.92.0//app/zookeeper/zoookeeper3.3.4/Step1安裝SUSE及搭建基本環(huán)境操作系統(tǒng)具體安裝可以隨意，盡量保證每臺(tái)機(jī)器一致，如每臺(tái)的機(jī)器名，和ip配置都要符合上面所規(guī)劃，這樣方便之后的操作在所有機(jī)器上，創(chuàng)建hadoop用戶保證每臺(tái)suse服務(wù)器上的ssh服務(wù)都已經(jīng)啟動(dòng)在每臺(tái)機(jī)器上/home/hadoop/下新建如下一些目錄/program/java;/app/hadoop;/app/hbase;/app/zookeeper;/app/zookeeperdata將相應(yīng)的軟件上傳到實(shí)體機(jī)，在hadoop-node1上解壓hadoop和hbase#tarzxvfhadoop-0.20.2-cdh3u0.tar.gz#tarzxvfhbase-0.90.1-cdh3u0.tar.gzhadoop-node6在上解壓zookeeper#tarzxvfzookeeper-3.3.3-cdh3u0.tar.gz在節(jié)點(diǎn)1-5,8上修改/etc/profileexportJAVA_HOME=/home/hadoop/program/java/jdk1.6.0_30exportPATH=$JAVA_HOME/bin:$PATHexportCLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$CLASSPATHexportHADOOP_HOME=/home/hadoop/app/hadoop/hadoop-0.20.2-cdh3u4exportHBASE_HOME=/home/hadoop/app/hbase/hbase-0.90.6-cdh3u4在節(jié)點(diǎn)6,7上修改/etc/profileexportJAVA_HOME=/home/hadoop/program/java/jdk1.6.0_30exportOOKEEPER_HOME=/home/hadoop/app/zookeeper/zookeeper-3.3.5-cdh3u4exportPATH=$JAVA_HOME/bin:$ZOOKEEPER_HOME/conf:$PATHexportCLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$CLASSPATHssh免密碼登錄用hadoop用戶登錄所有機(jī)器，在/home/hadoop/下建立.ssh目錄；運(yùn)行#ssh-keygen-tdsa-P''-f~/.ssh/id_dsa這樣會(huì)在~/.ssh/生成兩個(gè)文件：id_dsa和id_dsa.pub。#cat~/.ssh/id_dsa.pub>>~/.ssh/authorized_keys拷貝authorized_keys到2到8#scp/home/hadoop/.ssh/authorized_keyshadoop@*.*.83.2:/home/hadoop/.ssh/……#scp/home/hadoop/.ssh/authorized_keyshadoop@*.*.83.8:/home/hadoop/.ssh/修改hosts文件,每一臺(tái)實(shí)體機(jī)配置都如下#sudovim/etc/hosts從1登錄1,2…8用hostname（hadoop-nodeX），第一次需要確認(rèn)，以后就可以直接登錄。#sshhadoop-node1#sshhadoop-node2#sshhadoop-node3#sshhadoop-node4#sshhadoop-node5#sshhadoop-node6#sshhadoop-node7#sshhadoop-node8部分操作截圖：1．上傳jdk最好是使用64系統(tǒng)2．上傳hadoop，hbase，zookeeperStep2安裝JDK#cdjava#chmod+xjdk-6u30-linux-x64.bin./jdk-6u24-linux-x64.binHadoop安裝1．在節(jié)點(diǎn)1和節(jié)點(diǎn)8上分別都創(chuàng)建/datamkdir/home/hadoop/data2.在節(jié)點(diǎn)2和5上分別都創(chuàng)建/disk1,/disk2,/disk3即mkdir/home/hadoop/disk1disk2disk33.vim/home/hadoop/app/hadoop/hadoop-0.20.2-cdh3u0/conf/hadoop-env.sh4.修改/home/hadoop/app/hadoop/hadoop-0.20.2-cdh3u0/conf/core-site.xml添加,截圖如下：5.修改/home/hadoop/app/hadoop/hadoop-0.20.2-cdh3u0/conf/hdfs-site.xml截圖：6.vihome/hadoop/app/hadoop/hadoop-0.20.2-cdh3u0/conf/mapred-site.xml截圖：7.修改/home/hadoop/app/hadoop/hadoop-0.20.2-cdh3u0/conf/masters截圖：8.修改/home/hadoop/app/hadoop/hadoop-0.20.2-cdh3u0/conf/slaves截圖：9.很關(guān)鍵的一步：拷貝節(jié)點(diǎn)1的hadoop到2-5,8#scp–r./hadoop-0.20.2-c**/hadoop@*.*.*.2:/home/hadoop/app/hadoop/…#scp–r./hadoop-0.20.2-c**/hadoop@*.*.*.5:/home/hadoop/app/hadoop/…10.格式化hadoop文件系統(tǒng)#hadoopnamenode–format11.啟動(dòng)hadoop，在1上運(yùn)行#start-all.sh注：運(yùn)行jps查看各個(gè)節(jié)點(diǎn)運(yùn)行情況截圖：節(jié)點(diǎn)1(主節(jié)點(diǎn))：節(jié)點(diǎn)2或3、4、5(數(shù)據(jù)節(jié)點(diǎn))：節(jié)點(diǎn)6或7(Zookeeper節(jié)點(diǎn))：節(jié)點(diǎn)8(secondName和HbaseMaster節(jié)點(diǎn))：查看集群狀態(tài):http://*.*.83.1:50070/dfshealth.jsp查看集群狀態(tài):http://*.*.83.1:50030/dfshealth.jspZookeeper安裝1.在節(jié)點(diǎn)6,7上創(chuàng)建/home/hadoop/zookeeperdata目錄修改6的/home/hadoop/app/zookeeper/zookeeper-3.3.3-cdh3u0/conf/zoo.cfg2.拷貝6的zookeeper到7#scp–r./zookeeper-3.3.3-*/hadoop@*.*.*.7:/home/hadoop/app/zookeeper/3.在6,7的/home/hadoop/zookeeperdata目錄下建myid文件，內(nèi)容分別為1,2,3啟動(dòng)zookeeper，4.在6,7上分別執(zhí)行#zkServer.shstart5.啟動(dòng)后可以使用#zkServer.shstatus查看狀態(tài)Hbae安裝在節(jié)點(diǎn)1上修改/home/hadoop/app/hbae/hbase-0.90.1-cdh3u0/conf/hbase-env.shexportJAVA_HOME=/home/hadoop/program/java/jdk1.6.0_30exportHBASE_MANAGES_ZK=false2.在節(jié)點(diǎn)1上修改/home/hadoop/app/hbase/hbase-0.90.1-cdh3u0/conf/hbase-site.xml<configuration><property><name>hbase.rootdir</name><value>hdfs://hadoop-node1:9000/hbase</value></property><property><name>hbase.cluster.distributed</name><value>true</value></property><property><name>hbase.master.port</name><value>60000</value></property><property><name>hbase.zookeeper.quorum</name><value>hadoop-node6,hadoop-node7</value></property></configuration>3.在1上修改/home/hadoop/app/hbase/hbase-0.90.1-cdh3u0/conf/regionservers在regionservers中添加如下內(nèi)容：hadoop-node2hadoop-node3hadoop-node4hadoop-node54.拷1的hbase到2,3,4,58#scp-r./hbase-0.90.1-cdh**/hadoop@*.*.*.2:/home/hadoop/app/hbae/…..5.啟動(dòng)hbase在節(jié)點(diǎn)1上執(zhí)行#start-hbase.sh6.啟動(dòng)hbase的第二個(gè)HM