hadoop實(shí)驗(yàn)報(bào)告

1、1 基于 hadoop 的大規(guī)模文本處理技術(shù)實(shí)驗(yàn)專 業(yè) 班 級(jí) ：軟件 1102 學(xué) 生 姓 名 ：張國(guó)宇學(xué)號(hào) ： 201116040221 2 setup hadoop on ubuntu 11.04 64-bit 提示：前面的 putty 軟件安裝省略；直接進(jìn)入jdk 的安裝。1. install sun jdk 由于 sun jdk 在 ubuntu 的軟件中心中無(wú)法找到， 我們必須使用外部的ppa 。打開(kāi)終端并且運(yùn)行以下命令：sudo add-apt-repository ppa:ferramroberto/java sudo apt-get update sudo apt-get i

2、nstall sun-java6-bin sudo apt-get install sun-java6-jdk add java_home variable: 先輸入粘貼下面文字：sudo vi /etc/environment 再將下面的文字輸入進(jìn)去：按i 鍵添加， esc鍵退出，x保存退出；如下圖：export java_home=/usr/lib/jvm/java-6-sun-6 3 test the success of installation in terminal: sudo . /etc/environment java version 2. check ssh

3、setting ssh localhost 如果出現(xiàn) “connection refused ”，你最好重新安裝ssh（如下命令可以安裝） ：sudo apt-get install openssh-server openssh-client 如果你沒(méi)有通行證ssh到主機(jī)，執(zhí)行下面的命令：ssh-keygen -t dsa -p -f /.ssh/id_dsa cat /.ssh/id_dsa.pub /.ssh/authorized_keys 4 3. setup hadoop 安裝 apache2 sudo apt-get install apache2 下載 hadoop：sudo wg

4、et /hadoop/common/hadoop-1.0.4 /hadoop-1.0.4.tar.gz解壓 hadoop 所下載的文件包：tar xvfz hadoop-1.0.4.tar.gz 下載最近的一個(gè)穩(wěn)定版本，解壓。編輯 / hadoop-env.sh定義java_home“use/library/java-6-sun-6”作為 hadoop 的根目錄：sudo vi conf/hadoop-env.sh 將以下內(nèi)容加到文件最后：# the java implementation to use. required.expo

5、rt java_home=/usr/lib/jvm/java-6-sun-6 pseudo-distributed operation (single-node cluster): 將下面中間部分插入： （得到如圖：）5 cat conf/core-site.xml: hdfs:/localhost:9000 cat conf/hdfs-site.xml: 6 dfs.replication 1 cat conf/mapred-site.xml: mapred.job.tracker localhost:9001 switch to hadoop

6、root directory and format a new distributed file system: bin/hadoop namenode format 注意：這一步一定不能掉，否則后面會(huì)出錯(cuò)！youll get info like “storage directory /tmp/hadoop-ubuntu/dfs/name has been successfully formatted.” remember this path is the hdfs home 7 directory of namenode.（你會(huì)得到提示：你已經(jīng)成功格式化。 ）start and stop h

7、adoop daemons （啟動(dòng)和終止進(jìn)程。）: bin/start-all.sh (start the hadoop) bin/stop-all.sh (stop the hadoop) web interfaces for the namenode and the jobtracker: namenode http:/localhost:50070/= http:/ ip address|domain name:50070 提示：如果前面沒(méi)有格式化成功，這個(gè)網(wǎng)址會(huì)出現(xiàn)就不去的現(xiàn)象！ ！如果進(jìn)不去就先結(jié)束進(jìn)程，重新格式化然后再開(kāi)始進(jìn)程。jobtracker http:/localhost:

8、50030/= http:/ ip address|domain name:50030 4. deploy an example map-reduce job 創(chuàng)建路徑 /home/ubuntu/mapreduce/wordcount/ ，并在該目錄下放置文本文件 worldcount.txt。然后將這些文件復(fù)制到hdfs 目錄：bin/hadoop dfs -copyfromlocal /home/ubuntu/mapreduce/wordcount /tmp/hadoop-ubuntu/dfs/name/wordcount 8 bin/hadoop dfs -ls /tmp/hadoop-

9、ubuntu/dfs/name/wordcount run the job: bin/hadoop jar hadoop*examples*.jar wordcount /tmp/hadoop-ubuntu/dfs/name/wordcount /tmp/hadoop-ubuntu/dfs/name/wordcount-output if the output info looks no problem, copy the output file from hdfs to local directory: bin/hadoop dfs -getmerge /tmp/hadoop-ubuntu/dfs/name/wordcount-output /home/ubuntu/mapreduce/wordcount/ now you can open the output file in your local directory to view the results. 實(shí)