hadoop單機(jī)部署、集群部署及win7本地Eclipse遠(yuǎn)程配置管理.doc

準(zhǔn)備工作：Window版hadoop下載地址：/apache/hadoop/common/hadoop-1.2.1/hadoop-1.2.1-bin.tar.gz下載Eclipse hadoop的插件地址：hadoop-eclipse-plugin-1.2.1.jarLinux Hadoop下載地址：/apache/hadoop/common/hadoop-1.2.1/hadoop-1.2.1.tar.gz在linux服務(wù)器上創(chuàng)建用戶名為hadoop的用戶：rootlocalhost # useradd hadoop設(shè)置密碼：rootlocalhost # passwd hadoop添加hadoop用戶使用vim、vi等命令的權(quán)限：rootlocalhost # vim /etc/sudoers root ALL=(ALL) ALLhadoop ALL=(ALL) ALL此處保存是可能需要使用:wq!強(qiáng)制保存。以下文檔如無特殊說明均使用hadoop賬戶進(jìn)行操作1.Hadoop單機(jī)部署1. 下載hadoop-1.2.1.tar.gz文件。2. 運(yùn)行命令 tar zxvf hadoop-1.2.1.tar.gz將hadoop解壓到自己喜歡的目錄下（我的解壓在/usr/local/目錄下）3. 編輯hadoop-1.2.1目錄下的conf/hadoop-env.sh文件，將其中的JAVA_HOME配置為自己的jdk目錄（如我的為：JAVA_HOME=/usr/local/jdk1.7.0_60）4. 到此出Hadoop單機(jī)部署基本完成。5. 單機(jī)模式的操作方法默認(rèn)情況下，Hadoop被配置成以非分布式模式運(yùn)行的一個獨(dú)立Java進(jìn)程。這對調(diào)試非常有幫助。下面的實(shí)例將已解壓的conf目錄拷貝作為輸入，查找并顯示匹配給定正則表達(dá)式的條目。輸出寫入到指定的output目錄。hadooplocalhost hadoop-1.2.1$ mkdirinputhadooplocalhost hadoop-1.2.1$ cpconf/*.xmlinputhadooplocalhost hadoop-1.2.1$ bin/hadoopjarhadoop-examples-1.2.1.jar grepinputoutputdfsa-z.+$catoutput/*注：語法不理解沒關(guān)系看下面進(jìn)一步說明顯示結(jié)果1dfsadmin2.Hadoop偽分布式模式部署1.下載hadoop-1.2.1.tar.gz文件。2.運(yùn)行命令 tar zxvf hadoop-1.2.1.tar.gz將hadoop解壓到自己喜歡的目錄下（我的解壓在/usr/local/目錄下）3.編輯hadoop-1.2.1目錄下的conf/hadoop-env.sh文件，將其中的JAVA_HOME配置為自己的jdk目錄（如我的為：JAVA_HOME=/usr/local/jdk1.7.0_60）4.編輯配置文件注：以前的版本是hadoop-site.xml,可hadoop在0.20版本，配置文件由以前的hadoop-site.xml文件變成三個配置文件core-site.xml,hdfs-site.xml,mapred-site.xml.內(nèi)在的原因是因?yàn)閔adoop代碼量越來越寵大，拆解成三個大的分支進(jìn)行獨(dú)立開發(fā)，配置文件也獨(dú)立了hadooplocalhost hadoop-1.2.1$ vim conf/core-site.xml:hdfs:/localhost:9000hadooplocalhost hadoop-1.2.1$ vim conf/hdfs-site.xml:dfs.replication1hadooplocalhost hadoop-1.2.1$ vim conf/mapred-site.xml:mapred.job.trackerlocalhost:90015.免密碼ssh設(shè)置現(xiàn)在確認(rèn)能否不輸入口令就用ssh登錄localhost:hadooplocalhost hadoop-1.2.1$ sshlocalhost如果不輸入口令就無法用ssh登陸localhost，執(zhí)行下面的命令：hadooplocalhost hadoop-1.2.1$ ssh-keygen-tdsa-P-f/.ssh/id_dsahadooplocalhost hadoop-1.2.1$ cat/.ssh/id_dsa.pub/.ssh/authorized_keys6.執(zhí)行首先使用hadoop命令對HadoopFileSystem(HDFS)進(jìn)行格式化。首先，請求namenode對DFS文件系統(tǒng)進(jìn)行格式化。在安裝過程中完成了這個步驟，但是了解是否需要生成干凈的文件系統(tǒng)是有用的。hadooplocalhost hadoop-1.2.1$ bin/hadoopnamenode -format接下來，啟動Hadoop守護(hù)進(jìn)程。啟動Hadoop守護(hù)進(jìn)程：hadooplocalhost hadoop-1.2.1$ bin/start-all.sh注：1）Hadoop守護(hù)進(jìn)程的日志寫入到$HADOOP_LOG_DIR目錄(默認(rèn)是$HADOOP_HOME/logs)2）啟動hadoop，但ssh端口不是默認(rèn)的22怎么樣？好在它可以配置。在conf/hadoop-env.sh里改下。如：exportHADOOP_SSH_OPTS=-p1234瀏覽NameNode和JobTracker的網(wǎng)絡(luò)接口，它們的地址默認(rèn)為：NameNode-http:/localhost:50070/JobTracker-http:/localhost:50030/將輸入文件拷貝到分布式文件系統(tǒng)：hadooplocalhost hadoop-1.2.1$ bin/hadoopfs-putconfinput運(yùn)行發(fā)行版提供的示例程序：hadooplocalhost hadoop-1.2.1$ bin/hadoopjarhadoop-examples-1.2.1.jargrepinputoutputdfsa-z.+查看輸出文件：將輸出文件從分布式文件系統(tǒng)拷貝到本地文件系統(tǒng)查看：hadooplocalhost hadoop-1.2.1$ bin/hadoopfs-getoutputoutput$catoutput/*或者在分布式文件系統(tǒng)上查看輸出文件：hadooplocalhost hadoop-1.2.1$ bin/hadoopfs-catoutput/*完成全部操作后，停止守護(hù)進(jìn)程：hadooplocalhost hadoop-1.2.1$ bin/stop-all.shHadoop在這個偽分布式配置中啟動5個守護(hù)進(jìn)程：namenode、secondarynamenode、datanode、jobtracker和tasktracker。在啟動每個守護(hù)進(jìn)程時(shí)，會看到一些相關(guān)信息（指出存儲日志的位置）。每個守護(hù)進(jìn)程都在后臺運(yùn)行。圖1說明完成啟動之后偽分布式配置的架構(gòu)。圖1.偽分布式Hadoop配置3.Hadoop集群搭建用了三臺服務(wù)器作了hadoop集群的部署測試，服務(wù)器有29（下面簡稱129）30（下面簡稱130）31（下面簡稱131）架構(gòu)規(guī)化如下：1）129作為NameNode，SecondaryNameNode，JobTracker；2）130和131作為DataNode,TaskTracker1. 創(chuàng)建hadoop用戶在三臺linux服務(wù)器上分別創(chuàng)建用戶名為hadoop的用戶：rootlocalhost # useradd hadoop設(shè)置密碼：rootlocalhost # passwd hadoop添加hadoop用戶使用vim、vi等命令的權(quán)限：rootlocalhost # vim /etc/sudoers root ALL=(ALL) ALLhadoop ALL=(ALL) ALL此處保存是可能需要使用:wq!強(qiáng)制保存。2. 配置無密碼登錄以hadoop用戶名登陸名稱節(jié)點(diǎn)（129）執(zhí)行hadooplocalhost $ ssh-keygen -t rsa然后一路回車，完畢后生成文件.ssh/id_rsa.pub按以下步驟執(zhí)行命令：hadooplocalhost $ cd .ssh/hadooplocalhost .ssh$ cp id_rsa.pub authorized_keyshadooplocalhost .ssh$ ssh localhostLast login: Mon Nov 24 17:09:56 2014 from localhosthadooplocalhost $ 如果不需要密碼則直接登陸進(jìn)去的話，就達(dá)到要求；否則需檢查authorized_keys的權(quán)限，看是否為644（-rw-r-r-）。然后執(zhí)行命令hadooplocalhost $ ssh-copy-id -i hadoop30hadooplocalhost $ ssh 30如果不需要密碼能登陸成功30，則ssh的配置結(jié)束。同理執(zhí)行hadooplocalhost $ ssh-copy-id -i hadoop31hadooplocalhost $ ssh 31如果不需要密碼能登陸成功31，則ssh的配置結(jié)束。（免密碼登錄也可以按以下方式操作：以hadoop用戶登陸數(shù)據(jù)節(jié)點(diǎn)服務(wù)器（130，131），創(chuàng)建.ssh目錄，并給與600權(quán)限（chmod600.ssh）;再把名稱129服務(wù)器上的authorized_keys復(fù)制到目錄數(shù)據(jù)節(jié)點(diǎn)（130，131）./ssh,注意權(quán)限和目錄結(jié)構(gòu)跟名稱節(jié)點(diǎn)保持一致，然后再從名稱129節(jié)點(diǎn)用ssh登陸數(shù)據(jù)節(jié)點(diǎn)（130，131），如果不需要密碼能登陸成功，則ssh的配置結(jié)束。）3.hadoop軟件安裝及集群部署1）下載hadoop-1.2.1.tar.gz文件。2）運(yùn)行命令 tar zxvf hadoop-1.2.1.tar.gz將hadoop解壓到自己喜歡的目錄下（我的解壓在/usr/local/目錄下）3）編輯hadoop-1.2.1目錄下的conf/hadoop-env.sh文件，將其中的JAVA_HOME配置為自己的jdk目錄（如我的為：JAVA_HOME=/usr/local/jdk1.7.0_60）4)修改masters和slaves配置修改文件/usr/local/hadoop-1.2.1/conf/slaves及/usr/local/hadoop-1.2.1/conf/masters,把數(shù)據(jù)節(jié)點(diǎn)的主機(jī)名加到slaves、名稱節(jié)點(diǎn)主機(jī)名加到masters?？梢约佣鄠€，每行一個。注意主機(jī)名需要在每個服務(wù)器的/etc/hosts映射好。在129上執(zhí)行hadooplocalhost hadoop-1.2.1$ vi conf/slaves3031在130、131上執(zhí)行hadooplocalhost hadoop-1.2.1$ vi conf/masters 295)Master(129)配置129為master結(jié)點(diǎn)，則129的配置文件如下：hadoop在0.20版本，配置文件由以前的hadoop-site.xml文件變成三個配置文件core-site.xml，hdfs-site.xml，mapred-site.xml。內(nèi)在的原因是因?yàn)閔adoop代碼量越來越寵大，拆解成三個大的分支進(jìn)行獨(dú)立開發(fā)，配置文件也獨(dú)立了。下面是129三個配置文件示例：hadooplocalhost hadoop-1.2.1$ cat conf/core-site.xml hdfs:/29:9000hadoop.tmp.dir/data/hadoopData/root/tmp/hadoophadooplocalhost hadoop-1.2.1$ cat conf/hdfs-site.xml dfs.replication 3hadooplocalhost hadoop-1.2.1$ cat conf/mapred-site.xml mapred.job.tracker29:90016）Slave(130、131上)配置在Slave(130、131上)上的配置文件如下(hdfs-site.xml不需要配置)：conf/core-site.xml 、conf/mapred-site.xml文件與129上的相同 至此hadoop集群部署完成。4.初始化和啟動hadoop集群4.1初始化文件系統(tǒng)初始化namenode,為HDFS作第一次運(yùn)行的準(zhǔn)備。hadooplocalhost hadoop-1.2.1$ bin/hadoopnamenode -format注：一看到format就和磁盤格式化聯(lián)想到一起，然后這個format是對hdfs來說的，所以有些人害怕真的是格式化自己的文件系統(tǒng)了，其實(shí)大可不必?fù)?dān)心,namenodeformat只是初始化一些目錄和文件而已。4.2啟動Hadoop在master結(jié)點(diǎn)配置用戶環(huán)境變量,在master結(jié)點(diǎn)29上啟動hadoop集群程序，執(zhí)行bin目錄下的start-all.shhadooplocalhost hadoop-1.2.1$ bin/start-all.sh停止hadoophadooplocalhost hadoop-1.2.1$ bin/stop-all.sh4.3測試在hdfs上創(chuàng)建test1文件夾，上傳文件到此目錄下hadooplocalhost hadoop-1.2.1$ bin/hadoopfs-mkdirtest1hadooplocalhost hadoop-1.2.1$ bin/hadoopfs-put./README.txttest1hadooplocalhost hadoop-1.2.1$ bin/hadoop fs -lsFound1itemsdrwxr-xr-x-hadoopsupergroup02011-07-2119:58/user/hadoop/test1運(yùn)行一個map-reduce示例程序wordcount，運(yùn)行結(jié)果如下：hadooplocalhost hadoop-1.2.1$ hadoop jar hadoop-examples-1.2.1.jar wordcount/user/hadoop/test1/README.txtoutput1查看輸出結(jié)果文件，這個文件在hdfs上:hadooplocalhost hadoop-1.2.1$ bin/hadoop fs -ls output1hadooplocalhost hadoop-1.2.1$ bin/hadoop fs -cat output1/part-r-000004.4.管理界面與命令29:50070/dfshealth.jsp29:50030/jobtracker.jsp30:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=/5.win7本地Eclipse遠(yuǎn)程管理配置hadoop5.1將hadoop-eclipse-plugin-1.2.1.jar插件添加到本地eclipseplugins目錄下5.2將hadoop-1.2.1-bin.tar.gz解壓到本地目錄（我解壓在D:myworkToolshadoop-1.2.1）下5.3重啟Eclipse，通過Open Perspective菜單打開Map Reduce視圖，如下：選中大象圖標(biāo)，編輯Hadoop配置信息：hadoop.tmp.dir 中的值為hadoop服務(wù)器上 conf/core-site.xml中hadoop.tmp.dir的值通過 window-open prespective-resource打開Resource視圖，即可看到DFS：這樣可以正常的進(jìn)行HDFS分布式文件系統(tǒng)的管理：上傳，刪除等操作。為下面測試做準(zhǔn)備，需要先建了一個目錄 user/root/input2，然后上傳兩個txt文件到此目錄：intput1.txt 對應(yīng)內(nèi)容：Hello Hadoop Goodbye Hadoopintput2.txt 對應(yīng)內(nèi)容：Hello World Bye WorldHDFS的準(zhǔn)備工作好了，下面可以開始測試了。新建簡單Mapreduce項(xiàng)目通過向?qū)陆ㄒ粋€Map/Reduce項(xiàng)目。在此過程中，點(diǎn)擊配置Hadoop的安裝路徑。新建一個測試類WordCountTest：package com.hadoop.learn.test; import java.io.IOException;import java.util.StringTokenizer; import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.mapreduce.Mapper;import org.apache.hadoop.mapreduce.Reducer;import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;import org.apache.hadoop.util.GenericOptionsParser;import org.apache.log4j.Logger; /* * 運(yùn)行測試程序 * * author yongboy * date 2012-04-16 */public class WordCountTest private static final Logger log = Logger.getLogger(WordCountTest.class); public static class TokenizerMapper extendsMapper private final static IntWritable one = new IntWritable(1);private Text word = new Text(); public void map(Object key, Text value, Context context)throws IOException, InterruptedException (Map key : + key);(Map value : + value);StringTokenizer itr = new StringTokenizer(value.toString();while (itr.hasMoreTokens() String wordStr = itr.nextToken();word.set(wordStr);(Map word : + wordStr);context.write(word, one); public static class IntSumReducer extendsReducer private IntWritable result = new IntWritable(); public void reduce(Text key, Iterable values,Context context) throws IOException, InterruptedException (Reduce key : + key);(Reduce value : + values);int sum = 0;for (IntWritable val : values) sum += val.get();result.set(sum);(Reduce sum : + sum);context.write(key, result); public static void main(String args) throws Exception Configuration conf = new Configuration();String otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();if (otherArgs.length != 2) System.err.println(Usage: WordCountTest );System.exit(2); Job job = new Job(conf, word count);job.setJarByClass(WordCountTest.class); job.setMapperClass(TokenizerMapper.class);job.setCombinerClass(IntSumReducer.class);job.setReducerClass(IntSumReducer.class);job.setOutputKeyClass(Text.class);job.setOutputValueClass(IntWritable.class); FileInputFormat.addInputPath(job, new Path(otherArgs0);FileOutputFormat.setOutputPath(job, new Path(otherArgs1); System.exit(job.waitForCompletion(true) ? 0 : 1);右鍵，選擇“Run Configurations”,彈出窗口，點(diǎn)擊“Arguments”選項(xiàng)卡,在“Program argumetns”處預(yù)先輸入?yún)?shù):hdfs:/29:9000/user/hadoop/test1 hdfs:/29:9000/user/hadoop/output3“VM argumetns”處輸入?yún)?shù)-DHADOOP_USER_NAME=hadoop（VM argumetns如果不設(shè)置參數(shù)可能會有權(quán)限的問題Permission denied,如果你的hadoop賬號為hadooptest，或者xxx 則此處設(shè)置為-DHADOOP_USER_NAME=hadooptest、-DHADOOP_USER_NAME=xxx網(wǎng)上也有其他解決方案網(wǎng)上的解決方案：修改hdfs-core.xml配置文件，關(guān)閉權(quán)限驗(yàn)證。dfs.permissionsfalse）如圖：備注：參數(shù)為了在本地調(diào)試使用，而非真實(shí)環(huán)境。然后，點(diǎn)擊“Apply”，然后“Close”?，F(xiàn)在可以右鍵，選擇“Run on Hadoop”，運(yùn)行。但此時(shí)會出現(xiàn)類似異常信息：12/04/24 15:32:44 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform. using builtin-java classes where applicable12/04/24 15:32:44 ERROR security.UserGroupInformation: PriviledgedActionException as:Administrator cause:java.io.IOException: Failed to set permissions of path: tmphadoop-AdministratormapredstagingAdministrator-519341271.staging to 0700Exception in thread main java.io.IOException: Failed to set permissions of path: tmphadoop-AdministratormapredstagingAdministrator-519341271.staging to 0700 at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:682) at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:655) at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509) at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344) at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189) at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:116) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:856) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:850) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:850) at org.apache.hadoop.mapreduce.Job.submit(Job.java:500) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:530) at com.hadoop.learn.test.WordCountTest.main(WordCountTest.java:85)這個是Windows下文件權(quán)限問題，在Linux下可以正常運(yùn)行，不存在這樣的問題。解決方法是，修改/hadoop-1.0.2/src/core/org/apache/hadoop/fs/FileUtil.java里面的checkReturnValue，注釋掉即可（有些粗暴，在Window下，可以不用檢查）：/* * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file * distributed with this work for additional information * regarding copyright ownership. The ASF licenses this file * to you under the Apache License, Version 2.0 (the * License); you may not use this file except in compliance * with the License. You may obtain a copy of the License at * * /licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an AS IS BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */package org.apache.hadoop.fs;import java.io.*;import java.util.Enumeration;import java.util.zip.ZipEntry;import java.util.zip.ZipFile;import mons.logging.Log;import mons.logging.LogFactory;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.permission.FsAction;import org.apache.hadoop.fs.permission.FsPermission;import org.apache.hadoop.io.IOUtils;import org.apache.hadoop.io.nativeio.NativeIO;import org.apache.hadoop.util.StringUtils;import org.apache.hadoop.util.Shell;import org.apache.hadoop.util.Shell.ShellCommandExecutor;/* * A collection of file-processing util methods */public class FileUtil private static final Log LOG = LogFactory.getLog(FileUtil.class); /* * convert an array of FileStatus to an array of Path * * param stats * an array of FileStatus objects * return an array of paths corresponding to the input */ public static Path stat2Paths(FileStatus stats) if (stats = null) return null; Path ret = new Pathstats.length; for (int i = 0; i stats.length; +i) reti = statsi.getPath(); return ret; /* * convert an array of FileStatus to an array of Path. * If stats if null, return path * param stats * an array of FileStatus objects * param path * default path to return in stats is null * return an array of paths corresponding to the input */ public static Path stat2Paths(FileStatus stats, Path path) if (stats = null) return new Pathpath; else return stat2Paths(stats); /* * Delete a directory and all its contents. If * we return false, the directory may be partially-deleted. */ public static boolean fullyDelete(File dir) throws IOException if (!fullyDeleteContents(dir) return false; return dir.delete(); /* * Delete the contents of a directory, not the directory itself. If * we return false, the directory may be partially-deleted. */ public static boolean fullyDeleteContents(File dir) throws IOException boolean deletionSucceeded =