實驗2-熟悉常用的HDFS操作

1、實驗2.熟悉常用的HDF蹶作實驗2熟悉常用的HDFS操作1 實驗目的1. 理解HDFS在Hadoop體系結(jié)構(gòu)中的角色;2. 熟練使用HDFS操作常用的Shell命令；3. 熟悉HDFS操作常用的JavaAPI。2 實驗平臺操作系統(tǒng)：LinuxHadoop版本：2.6.0或以上版本JDK版本：1.6或以上版本1. JavaIDE:Eclipse實驗內(nèi)容和要求編程實現(xiàn)以下指定功能，并利用Hadoop提供的Shell命令完成相同任務：向HDFS中上傳任意文本文件，如果指定的文件在HDFS中已經(jīng)存在，由用戶指定是追加到原有文件末尾還是覆蓋原有的文件；wenubuntu:/usr/local/hadoo

2、p/btn$./hdfsdfs-test-etext.txtwenaubuntui/usr/local/hadDop/btrSJhdfsdfsppendloFllelocal.txttext,txtappendToFile;/usr/local/hadDQp/bin/local.txt做有那個文件或目錄）Menubuntu:/usr/local/hadDop/btn$./hdfsdfs-cp*ffile:/home/h3doop/localttxttext,txtcp:1file:/hone/hadoop/local.txt1:NosuchfileordirectoryPrsbLemsJava

3、docDeclarationCon&ole'Sl'erminfll'riHDF5ApiJavaAplkatrnnjAjs(7lit)/j'yrravti-7-opFriidknfiiTidD4/biryjavfl(Cec15,?C164S4：?8AM)loq4；na圈hqappnderjcpLLdbefoundfurlogger(orgpapachedoop.conf-ConTiquration.depr«catlogj:WARNPleaseinitialisethelog4jsystenproperly,lc»q4；:warjjse

4、ehttp：/loqqinq1apa<lie1orq/l,og4j/rj.2/fraqhhtiTr#nQCflnflgforinfp./user/hadoop/texl.txt已存在，/hoiie/hacInnp/teKt.tMt至/user/hadDoa/'texttxt從HDFS中下載指定文件，如果本地文件與要下載的文件名稱相同，則自動對下載的文件重命名；hadoop世11勤11口史11£/11$./'10匚31./11眉"0。2/勤111$./hdf£dfs-tet-efile：/fwfteyhadoop/text.txthadoopQ

5、ubuntut/usr/loca'L/hadoop/bln$./hdf£df£-copyToLocaLtext.txt,/text2,txthadoopubuntu:/usr/Localhadocp；blnS，/hdfsdfs-copyToLocaltext*txt./text.txtk.rJncjau.Lrii/f/I1fJk.；史1fTu.Problems疝avadxE,DedarationCcnole網(wǎng)<temii-iflted>HDFSApiA|ppLicdtion/jsr/lib/jvm/jdd-?-operjdk-jmd54/biryjdVf

6、l(Dec15,20lcg4j:WARNPleaseinitializethelog4jsystenproperly.Iog4j:WARNSeehttp：/logging,/LQg4j/!.2/taq.htril#noconfigtorloreinf/kome/hadoop/te?t.txt已存在.(1) 將圭新名名為：/MrreFhadDDp/teid:.txtG下韓完成將HDFS中指定文件的內(nèi)容輸出到終端中;hadoopgubuntu:/uisr/local/iadoop/bln$./hdfsdfs-cattext*txthellohadoop土Protjlems蒞Ja

7、vadoc底Dedaraticn9ConsoleES<terminated>HDFSApiJavaApplication/usr/Lib/j¥rn/java-7-apenjdk-anid&4/bin/java(Dec侶駟Laq4j：WARNPleaseinitializetliclog4js/steaproperly.應取丈忡：/jser/hadoop/text.txtLog4j:WARNSeehttp:/L/Log4j/I.2/1aq.html#nDConfigformoreinfo.hfillihadetip建取完成顯示HDF

8、S中指定的文件的讀寫權(quán)限、大小、創(chuàng)建時間、路徑等信息;hdoap$uburtu：/u$rylocal/n$./hdfsdfs-Is-htext.t)ct-rw-r-r-1hddoopsupergroup132016-12-1504;54text.txtlag4j:WARNNoaoaenderscouLdbetcundforLogger(Drg.apache.hadcap.ccnf.Configuration,ilqg4j:WARNPleaseinitializethelcg4jsystempros&rly;WARNSechttp；/logging.apsche.argAogj/1.2/

9、faq.html/poconfigformoreinfo.(2) Wt:hdfs：/lacalhost：9O9&fu5er/hadaop/te;(i,txt枚珞：大?。篿s時間：2»1G-13-1564:54:37給定HDFS中某一個目錄，輸出該目錄下的所有文件的讀寫權(quán)限、大小、創(chuàng)建時間、路徑等信息，如果該文件是目錄,則遞歸輸出該目錄下所有文件相關信息;Rhadooputkuntu:/usr/locL/hadcopybtnJhdfsdfs-Ls-A-h/useryhadoop!*rw*r*-r*-1hadoopsupergroup132016-12*lb/s&r/ha

10、doop/texttIxtterminatedHDFSApiJdvAAppliticn/usr/lib/jvm/java-7-apenjdk-amd64/biryjava(Deci5r20155;D2;40AMloglj:WARNNoappenderscouldbeFoundfcrlogger(Qrg.apachE.hadQap.cDnf.ConfigUrati.Dn.depr:WARNPleaseiniticilizeth?Iog4jprojjerly.logj；WARNSethttp:/Logging,apacherorgi/lDg4j/L2/faq.htrrl+noconfigforno

11、reinfo.(a*3>iW曰染二用有文炸的樣息：/user/hadoopS®f?:hdt5：/localhost:900G/user/hadoop/text.txt枳匣:rw-r-r-E：11(3) 間間：3B»12IS64:54:37提供一個HDFS內(nèi)的文件的路徑，對該文件進行創(chuàng)建和刪除操作。如果文件所在目錄不存在，則自動創(chuàng)建目錄;hadoopubvntu:/u£r/local./h>doop/bin$>/hdfsdfs-rmdtirl/dtr2/filenamem:'dtrl/dirZ/rilerdne1:Nosuchfileord

12、irectoryProblemsJjdac®：DetkinHonQConsolefS<terminat.ed>HDFSApiJava山叫)氐3口0可，鵬1/1舊/叫1/"咨-70>如業(yè)岫mdg/bi凹池a(Decisr2016s?oe：z<Icgj:WARNMoappenderscouldbefoundfcrlegger(org.apache.hodoop.conf<ConfigorotionTJlog4j；WARMPleaseinitializethelng4jsysteiiproperly.Log4j；WAKNSeehttp：/loggin

13、/lcq4j/l.2/faq.htnl#ricconfigformoreinfo.(4) 創(chuàng)隆女件獎：/ser/htjdGop/l»ipuc創(chuàng)購徑：/user/hadoop/inpur/Text.t>r提供一個HDFS的目錄的路徑，對該目錄進行創(chuàng)建和刪除操作。創(chuàng)建目錄時，如果目錄文件所在目錄不存在則自動創(chuàng)建相應目錄；刪除目錄時，由用戶指定當該目錄不為空時是否還刪除該目錄；hddoopnbuntu:/usr/lacat/hadoap/blnS*/hdfsdTs-pikdtr-pdtri/dlrihdcopubuntu:/usr/locl/hdcp;btn$

14、*/hdfsdfs-rdtrdirl/dtr2hadoopubuntu：/usr/Local/hadoop/bLnS»/hdfSdfs-rn-Rdlrl/dtr2H：2WNomehflit?<jdLegtsv*_Prebkms伊JavadocDcclaraticriConsoleS3<termnaLed>HDFSApiJavaApplication)/usr/lit/jvm/java-7-openjdk-anid64/bin/java(Dec15,20165Loq4j:warnnoappenderscouldbefoundforlogger(org.apache.h

15、adoop.conf.corHquratlog4j：WARNPleaseinitializetheIog4s/stemprooerly,Log4j：WARNSeehttp：/1ogging.apachecrgog4j/1.2ffaq.html*noconfigformoreinfo.目歧禾為空,不攜:Aiser/hadcop/input(5) 向HDFS中指定的文件追加內(nèi)容，由用戶指定內(nèi)容追加到原有文件的開頭或結(jié)尾；hadpopAubuntu：/u!>r/lacdL7hdduup/bLii$«/hdfsdfs-ppendToFllelocal.txttexttxtppendTo

16、Fllet/usr/LocaL/hadoop/bln/Local.txt(Nosuchftleordtircctory)hadoopALihuntu：/usr'/lacdl.yhdduup/bln$+/hdf5dfs-gettLxt.txtgot:text*txt1:Fileexistshadoopiibuntu;/usr/lacdl./h<idoupybln$,/cattext+txt>>local-txt“cM:nosuchfileordirectoryhddo0>jubuntui/i>sr/la<dl/hdduQp/btii$./hdfsdfs

17、-copyFropiLocdl-ftext»txttext*txt?.ProWems回Javadoc凰DecLarabarJF導Console£3<terminatedHDFSApiJavaApplicationyusr/lib/jvm/java7-open.dkamd6/bin；jaa(Dec15,2016Log4:WAR.MHoappendscouldbefauridfarlogger(org.apache.hadoop.caul.ConfiguraLog4：WKRNPleaseiniildliztthelog4jsystemprouerly.Log4:WARNS

18、eehttp：/loggirgapache.crg/lag4j/1.2/faq.htmlnoconigformoreinfo.rliiHl?14/userfhadnop/tPxt.txt刪除HDFS中指定的文件;h3do«pvbuntu:/usr/lool/hodoop/bin$*/hdfsdfs-rntext»txt16/12/1505;14;51INFOfs+TrashPolicyDefault:Hanenodetrashconfiguration:OelettonintervdV=sMtnutes,tncrval=0minutes-Deletedtext*t>t

19、由于已經(jīng)刪除，所以會顯示操作失敗。grEiriacEghum。mAppMauDn/U5r7iiD，jm，j3vm-H)pew<jK-amQ4/Diwjavatlcg4j:WARNNoappendsrscouldbefcundforloggerorg.apache.hadoop.c&nf.ConfiguraLog4j:wakmPleaseinitializeTheRg4jsystemproperly.Iog4j:WARMSee/log4j/1.2/faq.htnloconfigft>rmoreinfo.操作關敝C女件不存在或制院

20、關毆)刪除HDFS中指定的目錄，由用戶指定目錄中如果存在文件時是否刪除目錄；由于前面已經(jīng)刪除，所以顯示無該文件hadoopuburtu:/usr/1.cc3L/hi(icop/btn$./hdfsdfs-rmdtrdtri/dir2mdtr:kdtri/dtrz':nosuchrileordirectoryIhadaopubuntu:/u£r/locsl/hidoop/biin$*/hdfsdfs*rn-Rdirl/dir2Irm:PLUdw':nd$uchfLLe心;dlrectorv%Problems板Javadoc忌DeclarationConsoleS3III

21、<terminated>HQF5ApiJavaApplication/u5r/Lib/jviTi/javd-7-cipenjdk-flind64/bin/javd(Dec15,20l65;1;2lag4jrWftHMNoappeuderscouldbefoundforlogger(org.aiache.hadcop.corf,confIguratIon.lac|4j:warmPleaseinitially?Thelog4jsysTenproperly,l.t)g4j：WARMhttp：/Ingging,apachenrg/1ogej/1r7/taq,fitnlJtnacnnfigf

22、ormoreInfn.M錄不為理，不僻-在HDFS中，將文件從源路徑移動到目的路徑。hadooputNintai/usr/locaI/hadoop/btn$/hdfsdfs-rvtext.txttext2.txt移動成功：terminated>HDrSApHJauiApplicaticn/usr/lb/wm/java-7'Cpenjdk-imd64/bin/.avii(Dec15,20155：2B：11AM)locjj:warnNoapppnder?rsuldbefounlfarlogger(org.apache,hriocp.conf.ccnfigurationdpprecal

23、ionLoq4:dAWJPleaseinitiatizethelag4steaprajerly.Iog4j:WARNSechttp:/lodging>apache.srg/j/1.2/fhtalfnotonffornoreInfo.2. 犒立信hdfs：/User/hdidOOp/ti!xl.txt精出#hdf$I/user/hdno|/ncwTxt編程實現(xiàn)一個類“MyFSDataInputStream”,該類侖也承"org.apache.hadoop.fs.FSDataInputStream"，要求如下：實現(xiàn)按行讀取HDFS中指定文件的方法“readLine()",如果讀到文件末尾，則返回空，否則返回文件一行的文本。進行文檔讀取如下：ProblemsJevodotDcclation2Console悝<cerminaed>mvfsDacampjestreatjavaAnpli匚就on/usr/lib；jvm；java-7-opendkamd04/bin/jevalog4j；WARNNoappenderscouldbefoundforlogger(ergnapache.hadoop.conf.Configura