技術(shù)學(xué)習(xí)adtec-hadoop培訓(xùn)第二周

1、HadoopHadoop【】【】Raid3Raid34556各各7數(shù)據(jù)塊的8數(shù)據(jù)塊的899建立pipeline依次將目標(biāo)數(shù)據(jù)塊寫入各個(gè)datanode建立pipeline依次將目標(biāo)數(shù)據(jù)塊寫入各個(gè)datanode可以在hdfs-site.xmlDatanodehdfs可以在hdfs-site.xmlDatanodehdfs#-*-coding:UTF-8-*- import sys #-*-coding:UTF-8-*- import sys if name = main /沒有按時(shí)發(fā)送心跳的datanodeI/O如果datanodenamenode會檢測沒有按時(shí)發(fā)送心跳的datanodeI/O

2、如果datanodenamenode會檢測Numberofminutesbet Numberofminutesbet ns.Ifzero, thetrash feature 在Hadoop2.x在Hadoop2.x注意，hadoop沒有當(dāng)前目錄的概念，也沒有cd注意，hadoop沒有當(dāng)前目錄的概念，也沒有cd上傳文件到上傳文件到設(shè)置sshdatanode和tasktracker（hadoop-daemon.sh設(shè)置sshdatanode和tasktracker（hadoop-daemon.shstart-all.shstart-all.shHDFSAPIHDFSAPIpublicRLCatsi

3、c publicicpublicRLCatsic publicicvoidmain(String args)throwsExceptionInputStreamin=try in = new URL(args0).openStream(); IOUtils.copyBytes(in,System.out,4096,false); finally 使到運(yùn)行javac，jps使到運(yùn)行javac，jps添加Java和Hadoopimport import添加Java和Hadoopimport import 編譯時(shí)指定編譯時(shí)指定build.xml propertyname=buildbuild.xml

4、 propertyname=buildlocation=buildproperty filesetincludename=*/*.jarfilesetincludename=*/*.jar續(xù) javacincludeantruntime=falsesrcdir=srcclasspath refid=hadoop-jarjarfile=續(xù) javacincludeantruntime=falsesrcdir=srcclasspath refid=hadoop-property name=classpath refid=hadoop-用ant用antpublicclassemo publicpub

5、licclassemo publicicvoidmain(Stringargs)throwsIOExceptionConfigurationconf=newFileSystemfs= Pathfile=續(xù)if(fs.exists(file)else/Writing toln(File 續(xù)if(fs.exists(file)else/Writing toln(File /Readingfrome toHDFSJava Stringdata=指南第3章“Theerface指南第3章“Theerface安裝yum-y installgccgcc-c+autoconf安裝yum-y installgc

6、cgcc-c+autoconf 測試HDFSCAPI：一段C測試HDFSCAPI：一段C/Followingis a libhdfsleadaptedfromthe src/c+/libhdfs/hdfs_write.cof the/Followingis a libhdfsleadaptedfromthe src/c+/libhdfs/hdfs_write.cof the Hadoopargc,char *argv)hdfsFSfs =if (!fs)f(stderr,Cannotconnectto char* fileName=char* message= size= eto HDFSC續(xù)

7、exists =hdfsExists(fs,if(exists-1)f(stdout,續(xù)exists =hdfsExists(fs,if(exists-1)f(stdout,File %sexists!n,/ Createand openfile forhdfsFile outFile = hdfsOpenFile(fs,fileName,O_WRONLY|O_CREAT,0, 0, if(!outFile)f(stderr,Failedto open%sforwriting!n,/writetohdfsWrite(fs,outFile,(void*)message,hdfsCloseFile

8、(fs,續(xù)/Openfile forhdfsFileinFile =hdfsOpenFile(fs,fileName,O_RDONLY,續(xù)/Openfile forhdfsFileinFile =hdfsOpenFile(fs,fileName,O_RDONLY,0, 0, if(!inFile)f(stderr,Failed to open%s for reading!n,char*data= malloc(sizeof(char)*/ Read fromtSize readSize= hdfsRead(fs,inFile, (void*)data,f(stdout,%sn,hdfsClos

9、eFile(fs,return 0;gcchdfs_cpp_demo.c-I$HADOOP_HOME/src/c+/libhdfsgcchdfs_cpp_demo.c-I$HADOOP_HOME/src/c+/libhdfs -L$HADOOP_HOME/c+/Linux-amd64-64/lib/-lhdfs-L$JAVA_HOME/jre/lib/amd64/server-ljvm-o利用ant利用ant64/lib:$JAVA_HOME/jre/lib/amd64/server./hdfs_cpp_demo64/lib:$JAVA_HOME/jre/lib/amd64/server./h

10、dfs_cpp_demoHadoop2.xHDFS Hadoop2.xHDFS BlockStorageService（1namenode服務(wù)）提供datanodeBlockStorageService（1namenode服務(wù)）提供datanode2：datanode1.xHDFS制在2.x1.xHDFS制在2.x多namenodes及datanodedatanode多namenodes及datanodedatanode塊池（Block每一個(gè)datanode為所有的block 塊池（Block每一個(gè)datanode為所有的block BlockIDblock一個(gè)Namenode datanod

11、e為其他Namenode1.x的Namenode2.xnamenode1.x的Namenode2.xnamenode Hash返一在分布式系統(tǒng)里常用采用ntSideMountHash返一在分布式系統(tǒng)里常用采用ntSideMountTablehdfs-c-hdfs-c-續(xù)nn-host2:rpc-續(xù)nn-host2:rpc-. Other commonconfigurationStep1clusterIdamenode -format -clusterId StepStep1clusterIdamenode -format -clusterId Step2clusterIdamenode-format -clusterIdhdfsdfsadmin-allowSnapshothdfsdfsadmin