【Hadoop實(shí)驗(yàn)一】三節(jié)點(diǎn)完全模式安裝

1、Hadoop 1.0.4 Installation GuideHadoop 1.0.4 安裝指南Create: LinouLast Update: Version: VersionPersonDateComments1.0Linou2013/1/2Contents1環(huán)境準(zhǔn)備31.1 環(huán)境介紹31.2 建立Hadoop運(yùn)行帳號(hào)31.3 配置SSH免密碼登錄31.4 下載并解壓Hadoop安裝包到指定目錄42配置Hadoop52.1 配置namenode, 修改.xml文件52.2 配置hadoop-env.sh62.3 配置master和slaves文件62.4 向各個(gè)節(jié)點(diǎn)復(fù)制Hadoop63格

2、式化namenode73.1 格式化namenode74啟動(dòng)Hadoop84.1 啟動(dòng)hadoop84.2 檢查后臺(tái)進(jìn)程85測(cè)試95.1 上傳文件到HDFS95.2 創(chuàng)建目錄95.3 顯示文件內(nèi)容95.4 刪除目錄，文件91 環(huán)境準(zhǔn)備1.1 環(huán)境介紹 本次實(shí)驗(yàn)使用Oracle VM VirtualBox搭建虛擬機(jī)。本機(jī)環(huán)境：虛擬機(jī)環(huán)境：機(jī)器IPname內(nèi)存硬盤Master01master2048M30Gslave102slave11024M30Gslave203slave21024M30G1.2 建立Hadoop運(yùn)行帳號(hào)分

3、別在各個(gè)節(jié)點(diǎn)執(zhí)行下面腳本#!/bin/bashgroupadd hadoopuseradd hadoop -g hadoop;echo hadoop|passwd -stdin hadoop 1.3 配置SSH免密碼登錄首先所有節(jié)點(diǎn)上執(zhí)行：su - hadoop ssh-keygen -t dsa然后分別在各個(gè)節(jié)點(diǎn)上執(zhí)行：master： cd .ssh/ cat id_rsa.pubauthorized_keys scpauthorized_keys slave1:/home/hadoop/.ssh/slave1: cd .ssh/ cat id_rsa.pubauthorized_keys

4、scpauthorized_keys slave2:/home/hadoop/.ssh/slave2: cd .ssh/ cat id_rsa.pubauthorized_keys scpauthorized_keys master:/home/hadoop/.ssh/master: scpauthorized_keys slave1:/home/hadoop/.ssh/ scpauthorized_keys slave2:/home/hadoop/.ssh/遇到一個(gè)問(wèn)題，怎么設(shè)置都還是問(wèn)我要密碼！最后查到是因?yàn)閍uthorized_keys的權(quán)限問(wèn)題，把權(quán)限設(shè)置為600chmod 600 a

5、uthorized_keys一切OK！1.4 下載并解壓Hadoop安裝包到指定目錄首先在master節(jié)點(diǎn)上完成配置之后拷貝到各個(gè)節(jié)點(diǎn)上去oracle192 hadoop$ cp -r hadoop-1.0.4 /u01/app/2 配置Hadoop2.1 配置namenode, 修改.xml文件cd /u01/app/hadoop/conf/vi hdfs:/01:9000vi hdfs-site.xmldfs.data.dir/u01/app/data/hdfsdfs.replication2vi map

6、red-site.xmlmapred.job.trackerhdfs:/01:90012.2 配置hadoop-env.sh2.3 配置master和slaves文件vi master01vi slaves02032.4 向各個(gè)節(jié)點(diǎn)復(fù)制Hadoopscp -r ./hadoop/ 02:/u01/appscp -r ./hadoop/ 03:/u01/app3 格式化namenode3.1 格式化namenodeoracle192 hadoop$ bi

7、n/hadoop namenode -format13/01/02 11:48:49 INFO namenode.NameNode: STARTUP_MSG: /*STARTUP_MSG: Starting NameNodeSTARTUP_MSG: host = 01/01STARTUP_MSG: args = -formatSTARTUP_MSG: version = 1.0.4STARTUP_MSG: build = /repos/asf/hadoop/common/branches/branch-1

8、.0 -r 1393290; compiled by hortonfo on Wed Oct 3 05:13:58 UTC 2012*/13/01/02 11:48:49 INFO util.GSet: VM type = 64-bit13/01/02 11:48:49 INFO util.GSet: 2% max memory = 17.77875 MB13/01/02 11:48:49 INFO util.GSet: capacity = 221 = 2097152 entries13/01/02 11:48:49 INFO util.GSet: recommended=2097152,

9、actual=209715213/01/02 11:48:49 INFO namenode.FSNamesystem: fsOwner=oracle13/01/02 11:48:49 INFO namenode.FSNamesystem: supergroup=supergroup13/01/02 11:48:49 INFO namenode.FSNamesystem: isPermissionEnabled=true13/01/02 11:48:49 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=10013/01/02 11:4

10、8:49 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)13/01/02 11:48:50 INFO namenode.NameNode: Caching file names occuring more than 10 times 13/01/02 11:48:50 INFO common.Storage: Image file of size 112 saved in 0 seconds.13/01/02

11、 11:48:50 INFO common.Storage: Storage directory /tmp/hadoop-oracle/dfs/name has been successfully formatted.13/01/02 11:48:50 INFO namenode.NameNode: SHUTDOWN_MSG: /*SHUTDOWN_MSG: Shutting down NameNode at 01/01*/4 啟動(dòng)Hadoop4.1 啟動(dòng)hadooporacle192 hadoop$ bin/start-all.sh start

12、ing namenode, logging to /u01/app/hadoop/libexec/./logs/hadoop-oracle-namenode-01.out02: starting datanode, logging to /u01/app/hadoop/libexec/./logs/hadoop-oracle-datanode-02.out03: starting datanode, logging to /u01/app/hadoop/libexec/./logs/hadoop-o

13、racle-datanode-03.out01: starting secondarynamenode, logging to /u01/app/hadoop/libexec/./logs/hadoop-oracle-secondarynamenode-01.outstarting jobtracker, logging to /u01/app/hadoop/libexec/./logs/hadoop-oracle-jobtracker-01.out02: starting

14、tasktracker, logging to /u01/app/hadoop/libexec/./logs/hadoop-oracle-tasktracker-02.out03: starting tasktracker, logging to /u01/app/hadoop/libexec/./logs/hadoop-oracle-tasktracker-03.out4.2 檢查后臺(tái)進(jìn)程oracle192 hadoop$ jps3404 kvstore-2.0.23.jar3472 ManagedService3575

15、 ManagedService9544 NameNode9820 JobTracker9734 SecondaryNameNode9930 Jps5 測(cè)試5.1 上傳文件到HDFSoracle192 test$ hadoop dfs -put test.txt /testoracle192 test$ hadoop dfs -ls /testFound 1 items-rw-r-r- 2 oracle supergroup 17 2013-01-02 12:15 /test/test.txt5.2 創(chuàng)建目錄oracle192 test$ hadoop dfs -ls /useroracle19

16、2 test$ hadoop dfs -mkdir /user/testoracle192 test$ hadoop dfs -ls /userFound 1 itemsdrwxr-xr-x - oracle supergroup 0 2013-01-02 13:03 /user/test5.3 顯示文件內(nèi)容oracle192 test$ hadoop dfs -cat /test/test.txthadoop test file5.4 刪除目錄，文件oracle192 test$ hadoop dfs -rmr /test/test.txtDeleted hdfs:/

17、01:9000/test/test.txtoracle192 test$ hadoop dfs -ls /testoracle192 test$ hadoop dfs -rmr /testDeleted hdfs:/01:9000/testoracle192 test$ hadoop dfs -ls /testls: Cannot access /test: No such file or directory.5.5 測(cè)試WorldCount程序oracle192 hadoop$ hadoop dfs -put /home/oracle/test/test.txt /t

18、estoracle192 hadoop$ hadoop jar hadoop-examples-1.0.4.jar wordcount /test/test.txt /test/out13/01/03 10:47:46 INFO input.FileInputFormat: Total input paths to process : 113/01/03 10:47:46 INFO util.NativeCodeLoader: Loaded the native-hadoop library13/01/03 10:47:46 WARN snappy.LoadSnappy: Snappy nat

19、ive library not loaded13/01/03 10:47:46 INFO mapred.JobClient: Running job: job_201301021214_000213/01/03 10:47:47 INFO mapred.JobClient: map 0% reduce 0%13/01/03 10:48:00 INFO mapred.JobClient: map 100% reduce 0%13/01/03 10:48:12 INFO mapred.JobClient: map 100% reduce 100%13/01/03 10:48:17 INFO map

20、red.JobClient: Job complete: job_201301021214_000213/01/03 10:48:17 INFO mapred.JobClient: Counters: 2913/01/03 10:48:17 INFO mapred.JobClient: Job Counters 13/01/03 10:48:17 INFO mapred.JobClient: Launched reduce tasks=113/01/03 10:48:17 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=1616413/01/03 10:48:

21、17 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=013/01/03 10:48:17 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=013/01/03 10:48:17 INFO mapred.JobClient: Launched map tasks=113/01/03 10:48:17 INFO mapred.JobClient

22、: Data-local map tasks=113/01/03 10:48:17 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=1141113/01/03 10:48:17 INFO mapred.JobClient: File Output Format Counters 13/01/03 10:48:17 INFO mapred.JobClient: Bytes Written=2313/01/03 10:48:17 INFO mapred.JobClient: FileSystemCounters13/01/03 10:48:17 INFO m

23、apred.JobClient: FILE_BYTES_READ=4113/01/03 10:48:17 INFO mapred.JobClient: HDFS_BYTES_READ=13713/01/03 10:48:17 INFO mapred.JobClient: FILE_BYTES_WRITTEN=4330313/01/03 10:48:17 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=2313/01/03 10:48:17 INFO mapred.JobClient: File Input Format Counters 13/01/03 1

24、0:48:17 INFO mapred.JobClient: Bytes Read=3213/01/03 10:48:17 INFO mapred.JobClient: Map-Reduce Framework13/01/03 10:48:17 INFO mapred.JobClient: Map output materialized bytes=4113/01/03 10:48:17 INFO mapred.JobClient: Map input records=113/01/03 10:48:17 INFO mapred.JobClient: Reduce shuffle bytes=013/01/03 10:48:17 INFO mapred.JobClient: Spilled Records=613/01/03 10:48:17 INFO mapred.JobClient: Map output bytes=5613/01/03 10:48:17 INFO mapred.JobClient: CPU time spent (ms)=362013/01