畢業(yè)設(shè)計(jì)外文文獻(xiàn)-基于網(wǎng)絡(luò)爬蟲的信息挖掘系統(tǒng)設(shè)計(jì)與實(shí)現(xiàn)

附錄A外文翻譯—原文部分InformationMiningSystemDesignandmplementationBasedonWebCrawlerShanLin,You-mengLi,Qing-chengLiCollegeofInformationTechnicalScienceNankaiUniversityTianjin,300072,CHINAE-mail:lsskyshan@,solsikja@,liqch@Abstract–WiththeinformationexplosioncausingbytheWorldWideWebinrecentyears,theissueofhowtoexecutetheenormousinformationefficientlyatareasonablelosthasbecometheconcernofinformationproviders,serviceagenciesandendusers.Whenmanyresearchfocusonhowtodesignanefficientwebcrawler,wepayourattentiontohowtomakethebestoftheresultofwebcrawler.Inthispaper,wedescribethedesignandimplementationofaninformationminingsystemrunningontheresultsofwebcrawlertogainmoremetadatafromunstructureddocumentsforfocusedsearch(suchasRSSsearch).Wepresentthesoftwarearchitectureofthesystem,describeefficienttechniquesforachievinghighperformanceandreportpreliminaryexperimentalresultstoprovethatthissystemcanaddresstheissueofrobustness,flexibilityandaccuracyatalowcost.Keywords:Crawler,informationmining,RSS,lowcost.IntroductionTheexplosivegrowthoftheWorldWideWebgivespeopleamagicchangeoftheirlifestylesandworkingmanners.Astudyreleasedin2003[1]showedthatthevolumeofinformationontheWeb,whichisaccessibledirectly,isabout167terabytes,consistingabout2.5billionpages.Accordingtothelatestsurvey[2],byDecember2007,thetotalofnetizensintheworldhadincreasedto1,320million,withasharpincreaseof265.6%.Althoughexponentiallyincreasingamountsofmaterialareavailable,findingandmakingsenseofthismaterialispotentiallyuseful,butdifficultwithpresentsearchtechnology,HowtomakethebestofthehugedataandmanagethedocumentsontheInternetefficientlybecomeaveryimportanttasktoinformationprovidersandwebserviceagencies.Ouroverallaimistodesignafeasibleandflexibledistributedinformationminingsystem,whichcanmakethebestofthemetadataresultfromwebcrawlers,maximizethebenefitsobtainedperdownloadedpageandgetmoreby-productsatacomparativelylowcost.Weimplementthesystemarchitectureonthebasisofasimplebreadth-firstcrawlercalled‘WebSpider’,althoughthesystemcanbeadaptedtootherstrategies.Wereportpreliminaryexperimentalresultsinsection3,andtheconclusionanddirectionforfutureworkwillbepresentedattheendofthispaper.InformationMiningWebinformationminingtechniqueisaspecialexpandedapplicationofdataminingtechniquesonmanagingthehugeinformationontheInternet.WebinformationminingistheprocessofscratchingthemetadatafromtheInternet,analyzingfromdifferentperspectivesandsummarizingitintousefulinformation.Itincludesinformationextraction,informationretrieval,naturallanguageprocessinganddocumentsummarization.Informationminingcanadoptsomedataminingtechniques,buttherearesignificantdifferencesbetweenthem.Informationminingworkswithunstructureddata,suchasWebpagesandtextdocuments,incontrasttoDataMiningwhichisbasedonstructureddatalikerelationaldata.WebCrawlerThehugesizeofdataontheInternetgivethebirthofwebsearchengines,whicharebecomingmoreandmoreindispensableastheprimarymeansoflocatingrelevantinformation.Suchsearchenginesrelyonmassivecollectionsofwebpagesthatareacquiredbytheworkofwebcrawlers,alsoknownaswebrobotsorspiders.Awebcrawlerisaprogram,whichbrowsestheWorldWideWebinamethodical,automatedmanner.Webcrawlersaremainlyusedforautomatingmaintenancetasksbyscratchinginformationautomatically.Typically,acrawlerbeginswithasetofgivenWebpages,calledseeds,andfollowsallthehyperlinksitencountersalongtheway,toeventuallytraversetheentireWeb[3].GeneralcrawlersinserttheURLsintoatreediagramandvisittheminabreadth-firstmanner.Therehasbeensomerecentacademicinterestinnewtypesofcrawlingtechniques,suchasfocusedcrawlingbasedonsemanticweb[6,8],cooperativecrawling[10],distributedwebcrawler[7],andintelligentcrawling[9],andthesignificanceofsoftcomputingcomprisingfuzzylogic(FL),artificialneuralnetworks(ANNs),geneticalgorithms(GAs),androughsets(RSs)highlighted[11].Thebehaviorofawebcrawleristheoutcomeofacombinationofpolicies:1Aselectionpolicythatstatedwhichpagestodownload.2Are-visitpolicythatstateswhentocheck.3Apolitenesspolicythatstateshowtoavoidoverloadingwebsites.4Aparallelizationpolicythatstateshowtocoordinatedistributedwebcrawlers.[4]Intheridofrepeatedoperation,crawlersneedmakearecordofthewebpageswhichhavebeendownloadedbyHashTable.Thatmeansaftercrawlingsearchenginesstorenumerouspagesintheirdatabases.Thehardertaskisthatthecrawlingandstoringworkshouldrepeatinacertainperiod.TakingthemostpopularsearchengineGoogleasanexample,in2003,Google’scrawlercrawledineverymonth,butnow,crawlsevery2or3days.Socrawlingonthemassivepagesinsuchfrequency,thecostofnetresourceandstorageishuge.Itisexactlythemotivationofthispaperthatsincewehavetorunacrawlertofetchnumerouspagesofdataataenormouscostofmachinehourandstorage,whydon’twetakefulladvantageofitandtrytogetmoreusefulinformationintheformofmetadatawhichisdataaboutdata?RDFandRSSThispaperdescribesthedesignandimplementationofanoptimizeddistributedinformationminingsystem,takingtheapplicationofscratchingRSS(ReallySimpleSyndication)FeedfromnetasanexamplewhichisonthebasisofRDF.TheResourceDescriptionFramework(RDF)isageneral-purposelanguageforrepresentinginformationintheWeb.ThisdocumentdefinesanXML(ExtensibleMarkupLanguage)syntaxforRDFcalledRDF/XMLintermsofNamespacesinXML,theXMLinformationSetandXMLBase.[5]RDFallowsforrepresentationofrichmetadatarelationshipsbeyondwhatispossiblewithearlierflat-structuredRSS.TheReallySimpleSyndication(RSS)isastandardformattodescriptandsyndicatethewebinformation.ItisalightweightXMLformatdesignedforsharingheadlinesandhandingotherwebcontentsyndication,whichiswidelyusedinInternetnews,BlogandWiki.RSSisaformatusedtoindexinformationandmetadata.Forinstance,notalltheInternetnews’contentisalwaysfree.Butthemetadataofthearticlesisusuallyshared,suchastitle,author,linkandabstract.SoRSSbecometheinformationplatformofthesemetadata,andwecanregardRSSasanefficientwaytogetandsharewebinformation.Figure1asaboveshowsthemaintagsofstandardformatoftheRSS2.0document.BysubscribingtheRSSfeeds,Figure1.RSS2.0maintagtreerepresentation.youcanreceivethenewestinformationwithoutanyoperation.ThatisthemostimportantcharacterofRSS–SyndicationandAggregation.SoRSShasalreadybecomethemostpopularapplicationofXML.BecauseRSSfollowtheXMLstandardformat,wecanparseRSSSeeddocumentsbytheDOM(DocumentObjectModel).TheprocessofcertifyaRSSdocumentshouldbedividedintotwostepsasfollow:TheheadofthedocumentfollowtheRSSformat.ThedocumentcanbesetDOMandparsedsuccessfully.ThedetailedimplementationwillbepresentatSection3.DesignOverview3.1AssumptionsIndesigningawebinformationminingsystemforourneeds,wehavebeenguidedbyassumptionsthatofferbothchallengesandopportunities,whichareunderguidanceofsomepreliminaryobservation.1Theinformationminingsystemshouldstorehugedataandnumerousfilestemporarily.Asthelimitationofexperimentinstruments,weneednotconsiderthelimitofstorage.2Asthelimitationofbandwidth,wesetthelongestresponsetimefordownloadertoensurethesystemcanruncontinuallyandnormally.Buttheovertimewillreducethescratchingspeed.Sohighsustainedbandwidthismoreimportantthanlowlatency.3Thesystemshouldbebuiltfromseveralcomponents.Sinceitisnotthekeytosolveinthispaper,wedon’tconsidertheproblemoftoleratingandrecovering.3.2ArchitectureThisInformationMiningSystemconsistsoffourmajorkindsofcomponents–Crawler,InformationMiningMachine,FilterandDownloaderasshowninFigure2.Eachoftheseistypicallyacommoditycomputerrunningaser-levelserverprocess.Figure2.SystemarchitectureInthesystem,Crawlerisusedtoscratchingallkindsofwebpagessuchashtml,xml,asp,jspandsoonfromasetofseedpages.TheoutputofCrawlerisformattedintheattributesofnumber,URL,Text(abstractinformationaboutURL).Sincethecrawlerisnotessentialforourexperimentalsetup,wewon’tintroducethealgorithmanddetailedimplementationofcrawlerinthispaper.Notethatweonlyparseforhyperlinks,andnotforindexingtermsby‘WebSpider’,whichwouldsignificantlyslowdowntheapplication.ThenthedatawillbesendintoMiningMachinetoprocesswhichisthekeycomponentofthesystemwiththehelpofFilter.Thedetailedimplementationwillbedescribedinthefollowingsection.AtlastDownloadertakethechargeofdownloadingthewebpagesfollowingthelistfromInformationMiningMachine,scratchthemetadataandstoreintheserverdatabase.Inordertoachievehigh-performancewhichmeansdownloadhundredsoreventhousandsofpagespersecond,thedesignoftheclusterofDownloadersisquiteimportant.Forsystemflexibilityconsideration,thenumberoftheDownloaderisnotfixed.Thatmeanswecaninsertdownloadersintothesystemasweneedtoadapttodifferentexperimentconditionsandapplicationswithareasonableamountofwork.Beforedownloading,thesystemcandetectthenumberofthedownloadersautomatically,andtheitemsintheoutputlist.ToguaranteetheaccuracyofInformationMiningsystem,afterdownloadingthepagefilesuccessfully,theDownloaderchecksthefileagaintomakesurethatitisavalidRSSfeed.AsalltheworkofparsingaXMLfilecanbeimplementedbysetaDOM.SowecanjudgeaRSSfileinthemannerofcheckingwhetheritcanbestructuredasavalidDOMstructure.Atthesametime,thesystemscratchesthemetadatasuchastitle,link,dateandsoonfromDOMinterfacesandstoresinthedatabase.InformationMiningMachineTheminingmachinecomponenttraversestheitemslistedinthefile‘link.txt’inthedataflow,whichisimplementatedinC++.Itisconvenienttoscratchthelinkweneedbyregularexpression.Forexample,RSSisaspecialXMLfile,aXMLapplication,conformstotheW3C’sRDFSpecificationandisextensibleviaXMLnamespaceand/orRDFbasedmodularization[12].Sowedefinetheregularexpressionendingby‘.xml’atfirst:Exp(RSS)={,(.*)(?=\.xml),}(1)Aftersomeexperiments,wefindthat:1)Somewebpages(html,xml,asp,jsp,php…)aredirectedbytheirserverstojumpfromanon-RSSlinktoaRSSlinkautomatically.2)SomeURLdirectoryjumptoaRSSlinkdirectly.Forexample,theURLasfollowedactuallypointsataRSSfileaboutnews./rss2.aspAlthoughitseemstobeanaspwebpage,itisactuallydirectedtoaRSSfileacquiescently.SoifonlyscratchXMLfiles,wewillmissalotofRSSseeds.Thenweredefinetheregularexpressionasfollows:Exp(RSS)={,(.*)[(?=\.xml)|(?=\.asp)|(?=\.jsp)|(?=\.php)],}(2)IftheURL’sformatistallywiththeregularexpression(2)asabove,theinformationminingmachineinsertittothelistofpotentialhandlingtargets.Thenthishandling-listwillbesenttotheFilterthroughthedataflowsimultaneously.Experientially,executingtimealwaysinthelineargrowth,becausealltheworkshouldbedonebytraversingthewholedocument,andparsingitonthedifferentdetailedlevel.Here,thechallengeistoavoidtraversingandoverparsingasfaraspossible.Thusinoursystem,wedesignthecomponentcalledFiltertoco-operatewithinformationminingmachine,whichisinchargeofdealingwiththeproblem.Beforefetchingthevaluableinformationhiddeninunstructuredwebpages,theFilterofoursystemwillpreinspectthesedocuments,sendmetadatatotheInformationMiningMachinewhichismostpossiblytobeaRSSfile,andwhichisimpossible.Atfirst,theFilterdownloadfilestothesystemcacheandreadonly50bitsofeachpagerelatedtothelinkfromtheMiningMachine,thencheckoutwhetherthese50bitsdatafollowthestandardRSS1.0(moredetailsoftheRSS1.0referto[13]).InRSS1.0,alltheRSSfilesbeginwiththefollowingformat:<?Xmlversion="1.0"encoding="utf-8"?>Ofcourse,therearesomeothercodingstandardsuchasGB2312,UTF-16.Westilluseregularexpressiontocheckthebeginning50bitsofthefileswhetheritfollowsRSS1.0standard.IftheresultisTRUE,theFilterreturnsthelinkofthepagetotheInformationMiningMachine,ifnot,thislinkwillbeflittedoutwithoutnomoreunnecessaryoperation.ExperimentalResultandAnalysisWepresentthepreliminaryexperimentalresultsandexperiencehereanddosomebriefanalysisonit.Adetailedanalysisofperformancebottlenecksandscalingbehaviorisbeyondthescopeofthispaper,andwouldrequireafastsimulationtested,sinceitwouldnotbepossible(orappropriate)todosuchastudywithourcurrentInternet.ExperimentalResultonStep1SinceRSSiswidelyusedinwebnews,Blog,Wikiandsoon,ourexperimentalinitializingSeedLinkfortheCrawlershouldcoverasmanykindsofthsesaspectsaspossible.Becauseofourexperimentalcondition,thescopewecoveredontheInternetisverylimited.Soa‘right’seedlinkissignificantwhichcankeepthesystemrunningmoreefficiently.Asouranalysis,aseedlinkpagewhichisfulloflinkscanincreasethemininghitrate.OnStep1,wechoosethefollowingURLsastheseedlinkoftheCrawlerrunningrespectivelyincomparison:1B:ApopularBlogdiscoverysite.2Techcrunch:Oneofthemostfamousweblog.ExperimentalResultonStep2Wechoosethelink‘/p/articles/?sm=rss’ofBNETwhichispointedtothepageofaRSSresourcemapsiteandfullofInternetnews,bythestep1oftheexperiment.AfterthreeDownloadersrunning100hours,thenumberofhyperlinksin‘link.txt’requestlistis105025,including101872validURLs.ThetrendofthespeedofRSSinformationminingexecutedbyoneoftheDownloadersisshowninFigure3.ThegraphinFigure3revealsthatthenumberofvalidRSSSeedsscratchedbyInformationMiningMachineapproximatelypresentsalineargroethwiththeexcutingtime.Andtheflatpartofthetrendisrelatedtothelinkstructureofthewebsite.Atlastwescratch2312RSSFeeds,aftersendtoFilter,thereare2007validRSSFeeds.Theharvestrateisabout0.3345perminutewhichislimitedbythebrandwedth.FutureWorkWehavedescribedtheInformationMiningSystem,adistributedsystemforfindingoutvaluablestructuredmetadatahiddeninthethousandsofmillionsofunstructuredwebdocuments.Inaddition,wepresentpreliminaryexperimentsalongwithsomebriefanalysis.InthisInformationMiningSystem,thereareobviouslysomeimprovementscanbemade.Amajoropenissueforfutureworkisadetailedsolutiontoincreasetheharvestrateofourinformationminingsystem.Althoughtheharvestrateistightlyrelatedtothebrandwidth,wecanoptimizethesystemarchitecturetoimprove.Ascompletewebcrawlingcoveragecannotbeachieved,duetothevastsizeofthewholeinternetandtoresourceavailability,oursystemcan’tscratchalltheRSSFeeds.Sohowtoincreasethecoveragerateisanothertask.Forthefuturework,wewillmonitortheRSSSeeds,setmeasurementstandardssuchaslifecycleandfreshconditionwhichwerejustlikethemeasurementoftherealseedsinthenatureworld.ItwillbeacompletelynewideaaboutRSSSeeds,butabsolutelynecessarytohandlethemillionsofRSSSeeds.Inaddition,wewillimprovetheDownloaderconponentbythewayofsupervisedlearningtoincreasetheharvestrateofRSSscratching.Bysomeguidanceself-learedfromsampledata,thedownloadercanjudgeeasilytodownloadpagesselectively.Lastbutnottheleast,inordertomanagetheseRSSSeedswegetfromtheminingSystemefficiently,thewayofevaluatingshouldbeconsidered.Theimprovementsaboveallwillmakethissystemmorerealisticreliableandfriendliertousers.Figure3.Scratchingtrend

附錄B外文翻譯—譯文部分基于網(wǎng)絡(luò)爬蟲的信息挖掘系統(tǒng)設(shè)計(jì)與實(shí)現(xiàn)摘要-近年來，信息量突增。萬維網(wǎng)如何解決在一個(gè)點(diǎn)有效的執(zhí)行大量的信息，以及減少損失已經(jīng)成為提供者、服務(wù)機(jī)構(gòu)和用戶關(guān)注的焦點(diǎn)。當(dāng)許多研究的重點(diǎn)是如何設(shè)計(jì)一個(gè)高效的網(wǎng)絡(luò)爬蟲，而我們的研究重點(diǎn)是如何使爬蟲的結(jié)果是最好的。在下文中，我們描述了信息挖掘系統(tǒng)的設(shè)計(jì)與實(shí)現(xiàn)過程。在Web爬蟲的結(jié)果中獲得更多的元數(shù)據(jù)。用于集中搜索非結(jié)構(gòu)化文檔（如RSS搜索）。我們介紹了系統(tǒng)的軟件架構(gòu)，描述了如何實(shí)現(xiàn)高績(jī)效性能的有效技術(shù)。Keywords:爬蟲，信息挖掘，RSS，低成本。1介紹萬維網(wǎng)的爆炸式增長(zhǎng)使人們生活方式和工作方式都發(fā)生了很大變化。2003年發(fā)布的一項(xiàng)研究顯示，可直接訪問的網(wǎng)絡(luò)信息量，約167兆字節(jié)，約25億頁。根據(jù)最新調(diào)查，到2007年12月全，世界網(wǎng)民總數(shù)增加到13.2億，大幅增長(zhǎng)265.6%。雖然數(shù)量呈指數(shù)增長(zhǎng)。但發(fā)現(xiàn)和搜集這些有用的信息還是很難。如何充分利用龐大的數(shù)據(jù)并管理，成為了互聯(lián)網(wǎng)的一項(xiàng)重要任務(wù)。我們的總體目標(biāo)是設(shè)計(jì)一個(gè)靈活可行性高的分布式信息挖掘系統(tǒng)，使爬蟲抓取到的數(shù)據(jù)最優(yōu)化，并且使得每一頁的下載都能帶來最大化的信息。我們WebSpider系統(tǒng)基于廣度優(yōu)先，并且可以適應(yīng)其他策略。信息挖掘Web信息挖掘技術(shù)對(duì)于管理和拓展互聯(lián)網(wǎng)上的海量信息非常有效。信息挖掘是對(duì)互聯(lián)網(wǎng)的元數(shù)據(jù)進(jìn)行抓取的一個(gè)過程，分析不同來源的信息，提取其中的有效信息。它包括信息提取、信息檢索、自然語言處理和文檔摘要。信息挖掘和數(shù)據(jù)挖掘看起來相似，但兩者存在著顯著的差異。信息挖掘使用非結(jié)構(gòu)數(shù)據(jù)，而數(shù)據(jù)挖掘基于結(jié)構(gòu)化數(shù)據(jù)。1.2信息挖掘互聯(lián)網(wǎng)上的龐大數(shù)據(jù)量使得搜索引擎越來越多，數(shù)據(jù)定位成為不可或缺的手段。這些搜索引擎依賴于大規(guī)模的網(wǎng)絡(luò)爬蟲，也稱為網(wǎng)絡(luò)機(jī)器人或蜘蛛。在擺脫重復(fù)操作上，爬蟲需要做一個(gè)網(wǎng)頁去重記錄，也稱url去重，這意味著在抓取搜索引擎后，他們的數(shù)據(jù)庫有大量的頁面信息，更艱巨的任務(wù)是爬蟲爬取重復(fù)的信息，使得數(shù)據(jù)重復(fù)量過于龐大。以最受歡迎的谷歌搜索引擎來說，以前是一個(gè)月爬一次，現(xiàn)在是2-3天爬取一次。所以如此頻繁的爬取網(wǎng)頁，網(wǎng)絡(luò)成本和資源存儲(chǔ)是巨大的，所以我們必須運(yùn)行一個(gè)爬蟲來獲取其中的元數(shù)據(jù)而非整個(gè)頁面，并嘗試以新的形式去存儲(chǔ)這些元數(shù)據(jù)。2RDFandRSS本文介紹了以網(wǎng)絡(luò)為例分布式信息挖掘系統(tǒng)RSS的應(yīng)用（ReallySimpleSyndication）和RDF基礎(chǔ)的設(shè)計(jì)和優(yōu)化。資源描述框架（RDF）是一個(gè)用于表示信息的通用網(wǎng)絡(luò)語言。本文檔定義了一個(gè)XML（可擴(kuò)展的RDF的標(biāo)記語言）語法，稱為RDF/XML中的命名空間術(shù)語，XML信息集和XMLBase。真正簡(jiǎn)單的聚合（RSS）是一種用來描述和聯(lián)合Web信息的標(biāo)準(zhǔn)格式。它是一個(gè)輕量級(jí)XML，旨在分享標(biāo)題并聯(lián)合處理其他網(wǎng)絡(luò)內(nèi)容，廣泛應(yīng)用于互聯(lián)網(wǎng)新聞，博客和維基。RSS是一個(gè)用于索引信息和元數(shù)據(jù)的格式。并非所有互聯(lián)網(wǎng)新聞的內(nèi)容都是免費(fèi)的。但文章的元數(shù)據(jù)通常是共享的，例如標(biāo)題，作者，鏈接和摘要。所以RSS成了這些元數(shù)據(jù)的信息平臺(tái)，我們可以考慮RSS是獲取和共享Web信息的有效方式。通過查看RSS文檔，可以了解最新的信息操作。這是RSS最重要的特征-企業(yè)聯(lián)合組織和聚合。所以RSS已經(jīng)成為了最流行的XML應(yīng)用程序。因?yàn)镽SS遵循XML標(biāo)準(zhǔn)格式，我們可以通過DOM解析RSS種子文檔（文檔對(duì)象模型）。驗(yàn)證RSS文檔的過程應(yīng)分為以下兩個(gè)步驟：1)文檔的頭部遵循RSS格式。2）文檔可以轉(zhuǎn)換為DOM并解析成功。具體實(shí)施將在第3節(jié)中介紹。3設(shè)計(jì)概述3.1假設(shè)為了設(shè)計(jì)一個(gè)web信息挖掘系統(tǒng)的需求，我們了假設(shè)了一些虛擬環(huán)境。1.信息挖掘系統(tǒng)應(yīng)存儲(chǔ)大量數(shù)據(jù)和臨時(shí)的大量文件。由于實(shí)驗(yàn)儀器限制，我們不需要考慮存儲(chǔ)限制。2.由于帶寬的限制，我們?cè)O(shè)置了最長(zhǎng)的響應(yīng)時(shí)間，以確保系統(tǒng)可以持續(xù)正常運(yùn)行。但是響應(yīng)過長(zhǎng)會(huì)降低訪問頻率。因此高的持續(xù)性帶寬比低延遲更重要。3.這個(gè)系統(tǒng)應(yīng)該由幾個(gè)組件組成。由于這不是本文要解決的關(guān)鍵問題，所以我們沒有考慮容量和恢復(fù)的問題。3.2體系結(jié)構(gòu)該信息挖掘系統(tǒng)由四個(gè)主要部分組成各種組件-爬蟲，信息挖掘機(jī)器、過濾器和下載器，如圖所示。每一個(gè)都是典型的ser-level服務(wù)器運(yùn)行進(jìn)程。在該系統(tǒng)中，爬行器是用來爬取各種物體的從html、xml、asp、jsp等網(wǎng)頁中提取的種子頁集。格式化爬蟲程序的輸出屬性的數(shù)量，URL，文本(抽象信息關(guān)于URL)。因?yàn)榕佬衅鲗?duì)我們來說不是必需的實(shí)驗(yàn)設(shè)置，我們不介紹算法和爬蟲的實(shí)現(xiàn)。請(qǐng)注意,我們只解析超鏈接，而不是索引項(xiàng)，這將大大減緩應(yīng)用程序的能耗。然后數(shù)據(jù)將被發(fā)送到挖掘機(jī)械加工，這得于其中的關(guān)鍵部件系統(tǒng)與過濾器的幫助。詳細(xì)的實(shí)現(xiàn)將在下一節(jié)中描述。最后Downloader負(fù)責(zé)下載網(wǎng)頁，根據(jù)信息挖掘機(jī)的列表，擦除元數(shù)據(jù)并存儲(chǔ)在服務(wù)器數(shù)據(jù)庫中。在為了提高性能，這意味著下載每秒數(shù)百甚至數(shù)千頁的設(shè)計(jì)集群的下載器是非常重要的。為系統(tǒng)靈活性考慮，數(shù)量的下載程序不是固定的。這意味著我們可以插入下載到我們需要適應(yīng)的系統(tǒng)中，不同的實(shí)驗(yàn)條件和應(yīng)用合理工的作量。在下載之前,系統(tǒng)可以檢測(cè)下載的數(shù)量以及輸出列表中的項(xiàng)。系統(tǒng)保證信息挖掘的準(zhǔn)確性，成功下載頁面文件后，下載程序再次檢查文件以確保它是有效的RSS提要。因?yàn)榻馕鯴ML文件的所有工作都可以通過設(shè)置DOM來實(shí)現(xiàn)。所以我們可以判斷一個(gè)RSS文件以檢查它是否可以結(jié)構(gòu)化的方式有效的DOM結(jié)構(gòu)。同時(shí)，系統(tǒng)劃痕標(biāo)題，鏈接，日期等元數(shù)據(jù)來自DOM接口和數(shù)據(jù)庫中的存儲(chǔ)。3.3信息挖掘機(jī)的信息信息挖掘機(jī)組件遍歷項(xiàng)目列在”link.txt”的文件流中，用c++實(shí)現(xiàn)。提取鏈接很方便，我們只需要用正則表達(dá)式。例如，RSS是一個(gè)特殊XML文件，一個(gè)XML應(yīng)用程序，符合W3C的RDF規(guī)范，可通過XMLnamespace擴(kuò)展和/或基于RDF的模塊化。所以我們定義以”.xml”結(jié)尾的正則表達(dá)式:Exp(RSS)={,(.*)(?=\.xml),}如果URL的格式符合正則表達(dá)式,信息挖掘機(jī)將其插入到潛在處理目標(biāo)列表。那么這份清單就可以同時(shí)通過數(shù)據(jù)流發(fā)送到過濾器了。從經(jīng)驗(yàn)上看，執(zhí)行時(shí)間總是線性的增加，因?yàn)樗械墓ぷ鞫紤?yīng)該通過遍歷來完成整個(gè)文檔，并對(duì)其進(jìn)行了詳細(xì)的分析。這里的挑戰(zhàn)是盡可能地避免遍歷和過度解析。因此在系統(tǒng)中，我們進(jìn)行設(shè)計(jì)稱為Filter的組件用于與過濾協(xié)作信息挖掘機(jī)。在獲取隱藏的有價(jià)值的信息之前在非結(jié)構(gòu)化網(wǎng)頁中，系統(tǒng)的過濾器會(huì)進(jìn)行預(yù)檢查這些文檔向信息發(fā)送元數(shù)據(jù)，挖掘機(jī)器最有可能是一個(gè)RSS文件，這是不可能的。首先，過濾器下載文件到系統(tǒng)緩存中，每個(gè)頁面