《大數(shù)據(jù)技術(shù)原理與應(yīng)用》實驗一WordCount

實驗一MapReduce--1--入門程序WordCount一、實驗?zāi)康腗apReduce界的helloworld程序就是WordCount程序。所謂WordCount，就是單詞計數(shù)，就是用來統(tǒng)計一篇或者一堆文本文件中的各單詞的出現(xiàn)次數(shù)。二、實驗原理按照我們普通的編寫代碼的邏輯，單詞計數(shù)程序應(yīng)該是這樣的：1、逐行讀取文本內(nèi)容2、把讀取到的一行文本內(nèi)容切割為一個一個的單詞3、把每個單詞出現(xiàn)一次的信息記錄為一個key-value，也就是“單詞-1”4、收集所有相同的單詞，然后統(tǒng)計value寫出的value值得總和，也就是key為同一個單詞的所有1的和。實驗過程首先在HDFS上創(chuàng)建輸入文件目錄test，命令如下：hadoop@ubuntu:~$hadoopfs-mkdirtest接著將本地的文件put到HDFS上test目錄下，并查看，命令如下：hadoop@ubuntu:~/txtfile$hadoopfs-put-f*.txttesthadoop@ubuntu:~$hadoopfs-lstestFound3items-rw-r--r--

1hadoopsupergroup

242016-10-2221:43test/file1.txt-rw-r--r--

1hadoopsupergroup

242016-10-2221:43test/file2.txt-rw-r--r--

1hadoopsupergroup

1149572016-10-2319:57test/news.txt注釋:-f是可以overwrite的意思使用InteliJ新建一個Javaproject,編寫wordcount程序,如下所示：packageexample;importorg.apache.hadoop.fs.Path;importorg.apache.hadoop.io.IntWritable;importorg.apache.hadoop.io.LongWritable;importorg.apache.hadoop.io.Text;importorg.apache.hadoop.mapred.*;importjava.io.IOException;importjava.util.Iterator;importjava.util.StringTokenizer;/**

*Createdbyhadoopon10/22/16.

*/publicclassWordCount{

///Mapper:<LongWritableText>projectto<TextIntWritable>

publicstaticclassMapextendsMapReduceBaseimplementsMapper<LongWritable,Text,Text,IntWritable>{

privatefinalstaticIntWritableone=newIntWritable(1);

privateTextword=newText();

///reporterreporttheprogressortheyarelive

publicvoidmap(LongWritablekey,Textvalue,OutputCollector<Text,IntWritable>output,Reporterreporter)throwsIOException{

Stringline=value.toString();

StringTokenizertokenizer=newStringTokenizer(line);

while(tokenizer.hasMoreTokens()){

//setchangestringtoText

word.set(tokenizer.nextToken());

output.collect(word,one);}}}

publicstaticclassReduceextendsMapReduceBaseimplementsReducer<Text,IntWritable,Text,IntWritable>{

///iscalledforeverykey

///IntWritable:youcanuseitasint

publicvoidreduce(Textkey,Iterator<IntWritable>values,OutputCollector<Text,IntWritable>output,Reporterreporter)throwsIOException{

intsum=0;

while(values.hasNext()){

sum+=values.next().get();

}

output.collect(key,newIntWritable(sum));

}

}

publicstaticvoidmain(String[]args)throwsException{

JobConfconf=newJobConf(WordCount.class);

conf.setJobName("wordcount");

conf.setOutputKeyClass(Text.class);

conf.setOutputValueClass(IntWritable.class);

conf.setMapperClass(Map.class);

//conf.setCombinerClass(Reduce.class);

conf.setReducerClass(Reduce.class);

//inputformat

conf.setInputFormat(TextInputFormat.class);

conf.setOutputFormat(TextOutputFormat.class);

FileInputFormat.setInputPaths(conf,newPath(args[0]));

FileOutputFormat.setOutputPath(conf,newPath(args[1]));

JobClient.runJob(conf);

}}導(dǎo)出jar包File->ProjectStructure->Artifacts->綠色“+”->JAR->Frommoduleswithdependcies->applyBuild->BuildArtifacts運行jar包hadoop@ubuntu:~/txtfile$hadoopjarWordCount.jartesttest-out最后查看HDFS上的結(jié)果并將結(jié)果get到本地hadoop@ubuntu:~$hadoopfs-lstest-outFound2items-rw-r--r--

1hadoopsupergroup

02016-10-2300:59test-out/_SUCCESS-rw-r--r--

1hadoopsupergroup

402016-10-2300:59te