大數(shù)據(jù)挖掘外文翻譯文獻

文獻信息：文獻標題：AStudyofDataMiningwithBigData（大數(shù)據(jù)挖掘研究）國外作者：VHShastri，VSreeprada文獻出處：《InternationalJournalofEmergingTrendsandTechnologyinComputerScience》,2016,38(2):99-103字數(shù)統(tǒng)計：英文2291單詞，12196字符；中文3868漢字外文文獻：AStudyofDataMiningwithBigDataAbstractDatahasbecomeanimportantpartofeveryeconomy,industry,organization,business,functionandindividual.BigDataisatermusedtoidentifylargedatasetstypicallywhosesizeislargerthanthetypicaldatabase.Bigdataintroducesuniquecomputationalandstatisticalchallenges.BigDataareatpresentexpandinginmostofthedomainsofengineeringandscience.Datamininghelpstoextractusefuldatafromthehugedatasetsduetoitsvolume,variabilityandvelocity.ThisarticlepresentsaHACEtheoremthatcharacterizesthefeaturesoftheBigDatarevolution,andproposesaBigDataprocessingmodel,fromthedataminingperspective.Keywords:BigData,DataMining,HACEtheorem,structuredandunstructured.I.IntroductionBigDatareferstoenormousamountofstructureddataandunstructureddatathatoverflowtheorganization.Ifthisdataisproperlyused,itcanleadtomeaningfulinformation.Bigdataincludesalargenumberofdatawhichrequiresalotofprocessinginrealtime.Itprovidesaroomtodiscovernewvalues,tounderstandin-depthknowledgefromhiddenvaluesandprovideaspacetomanagethedataeffectively.Adatabaseisanorganizedcollectionoflogicallyrelateddatawhichcanbeeasilymanaged,updatedandaccessed.Dataminingisaprocessdiscoveringinterestingknowledgesuchasassociations,patterns,changes,anomaliesandsignificantstructuresfromlargeamountofdatastoredinthedatabasesorotherrepositories.BigDataincludes3V’sasitscharacteristics.Theyarevolume,velocityandvariety.Volumemeanstheamountofdatageneratedeverysecond.Thedataisinstateofrest.Itisalsoknownforitsscalecharacteristics.Velocityisthespeedwithwhichthedataisgenerated.Itshouldhavehighspeeddata.Thedatageneratedfromsocialmediaisanexample.Varietymeansdifferenttypesofdatacanbetakensuchasaudio,videoordocuments.Itcanbenumerals,images,timeseries,arraysetc.DataMininganalysesthedatafromdifferentperspectivesandsummarizingitintousefulinformationthatcanbeusedforbusinesssolutionsandpredictingthefuturetrends.Datamining(DM),alsocalledKnowledgeDiscoveryinDatabases(KDD)orKnowledgeDiscoveryandDataMining,istheprocessofsearchinglargevolumesofdataautomaticallyforpatternssuchasassociationrules.Itappliesmanycomputationaltechniquesfromstatistics,informationretrieval,machinelearningandpatternrecognition.Dataminingextractonlyrequiredpatternsfromthedatabaseinashorttimespan.Basedonthetypeofpatternstobemined,dataminingtaskscanbeclassifiedintosummarization,classification,clustering,associationandtrendsanalysis.BigDataisexpandinginalldomainsincludingscienceandengineeringfieldsincludingphysical,biologicalandbiomedicalsciences.II.BIGDATAwithDATAMININGGenerallybigdatareferstoacollectionoflargevolumesofdataandthesedataaregeneratedfromvarioussourceslikeinternet,social-media,businessorganization,sensorsetc.WecanextractsomeusefulinformationwiththehelpofDataMining.Itisatechniquefordiscoveringpatternsaswellasdescriptive,understandable,modelsfromalargescaleofdata.Volumeisthesizeofthedatawhichislargerthanpetabytesandterabytes.Thescaleandriseofsizemakesitdifficulttostoreandanalyseusingtraditionaltools.BigDatashouldbeusedtominelargeamountsofdatawithinthepredefinedperiodoftime.Traditionaldatabasesystemsweredesignedtoaddresssmallamountsofdatawhichwerestructuredandconsistent,whereasBigDataincludeswidevarietyofdatasuchasgeospatialdata,audio,video,unstructuredtextandsoon.BigDataminingreferstotheactivityofgoingthroughbigdatasetstolookforrelevantinformation.Toprocesslargevolumesofdatafromdifferentsourcesquickly,Hadoopisused.Hadoopisafree,Java-basedprogrammingframeworkthatsupportstheprocessingoflargedatasetsinadistributedcomputingenvironment.Itsdistributedfilesystemsupportsfastdatatransferratesamongnodesandallowsthesystemtocontinueoperatinguninterruptedattimesofnodefailure.ItrunsMapReducefordistributeddataprocessingandisworkswithstructuredandunstructureddata.III.BIGDATAcharacteristics-HACETHEOREM.Wehavelargevolumeofheterogeneousdata.Thereexistsacomplexrelationshipamongthedata.Weneedtodiscoverusefulinformationfromthisvoluminousdata.Letusimagineascenarioinwhichtheblindpeopleareaskedtodrawelephant.Theinformationcollectedbyeachblindpeoplemaythinkthetrunkaswall,legastree,bodyaswallandtailasrope.Theblindmencanexchangeinformationwitheachother.Figure1:BlindmenandthegiantelephantSomeofthecharacteristicsthatincludeare:i.Vastdatawithheterogeneousanddiversesources:Oneofthefundamentalcharacteristicsofbigdataisthelargevolumeofdatarepresentedbyheterogeneousanddiversedimensions.Forexampleinthebiomedicalworld,asinglehumanbeingisrepresentedasname,age,gender,familyhistoryetc.,ForX-rayandCTscanimagesandvideosareused.Heterogeneityreferstothedifferenttypesofrepresentationsofsameindividualanddiversereferstothevarietyoffeaturestorepresentsingleinformation.ii.Autonomouswithdistributedandde-centralizedcontrol:thesourcesareautonomous,i.e.,automaticallygenerated;itgeneratesinformationwithoutanycentralizedcontrol.WecancompareitwithWorldWideWeb(WWW)whereeachserverprovidesacertainamountofinformationwithoutdependingonotherservers.iii.Complexandevolvingrelationships:Asthesizeofthedatabecomesinfinitelylarge,therelationshipthatexistsisalsolarge.Inearlystages,whendataissmall,thereisnocomplexityinrelationshipsamongthedata.Datageneratedfromsocialmediaandothersourceshavecomplexrelationships.IV.TOOLS: OPENSOURCEREVOLUTIONLargecompaniessuchasFacebook,Yahoo,Twitter,LinkedInbenefitandcontributeworkonopensourceprojects.InBigDataMining,therearemanyopensourceinitiatives.Themostpopularofthemare:ApacheMahout:ScalablemachinelearninganddataminingopensourcesoftwarebasedmainlyinHadoop.Ithasimplementationsofawiderangeofmachinelearninganddataminingalgorithms:clustering,classification,collaborativefilteringandfrequentpatternmining.R:opensourceprogramminglanguageandsoftwareenvironmentdesignedforstatisticalcomputingandvisualization.RwasdesignedbyRossIhakaandRobertGentlemanattheUniversityofAuckland,NewZealandbeginningin1993andisusedforstatisticalanalysisofverylargedatasets.MOA:Streamdataminingopensourcesoftwaretoperformdatamininginrealtime.Ithasimplementationsofclassification,regression;clusteringandfrequentitemsetminingandfrequentgraphmining.ItstartedasaprojectoftheMachineLearninggroupofUniversityofWaikato,NewZealand,famousfortheWEKAsoftware.ThestreamsframeworkprovidesanenvironmentfordefiningandrunningstreamprocessesusingsimpleXMLbaseddefinitionsandisabletouseMOA,AndroidandStorm.SAMOA:ItisanewupcomingsoftwareprojectfordistributedstreamminingthatwillcombineS4andStormwithMOA.VowpalWabbit:opensourceprojectstartedatYahoo!ResearchandcontinuingatMicrosoftResearchtodesignafast,scalable,usefullearningalgorithm.VWisabletolearnfromterafeaturedatasets.Itcanexceedthethroughputofanysinglemachinenetworkinterfacewhendoinglinearlearning,viaparallellearning.V.DATAMININGforBIGDATADataminingistheprocessbywhichdataisanalysedcomingfromdifferentsourcesdiscoversusefulinformation.DataMiningcontainsseveralalgorithmswhichfallinto4categories.Theyare:1.AssociationRule2.Clustering3.Classification4.RegressionAssociationisusedtosearchrelationshipbetweenvariables.Itisappliedinsearchingforfrequentlyvisiteditems.Inshortitestablishesrelationshipamongobjects.Clusteringdiscoversgroupsandstructuresinthedata.Classificationdealswithassociatinganunknownstructuretoaknownstructure.Regressionfindsafunctiontomodelthedata.Thedifferentdataminingalgorithmsare:CategoryAlgorithmAssociationApriori,FPgrowthClusteringK-Means,Expectation.ClassificationDecisiontrees,SVMRegressionMultivariatelinearregressionTable1.ClassificationofAlgorithmsDataMiningalgorithmscanbeconvertedintobigmapreducealgorithmbasedonparallelcomputingbasis.BigDataDataMiningItiseverythingintheworldnow.ItistheoldBigData.Sizeofthedataislarger.Sizeofthedataissmaller.Involvesstorageandprocessingoflargedatasets.Interestingpatternscanbefound.BigDataisthetermforlargedataset.Dataminingreferstotheactivityofgoingthroughbigdatasettolookforrelevantinformation.Bigdataistheasset.Dataminingisthehandlerwhichprovidebeneficialresult.Bigdata"variesdependingonthecapabilitiesoftheorganizationmanagingtheset,andonthecapabilitiesoftheapplicationsthataretraditionallyusedtoprocessandanalysethedata.Dataminingreferstotheoperationthatinvolverelativelysophisticatedsearchoperation.Table2.DifferencesbetweenDataMiningandBigDataVI.ChallengesinBIGDATAMeetingthechallengeswithBIGDataisdifficult.Thevolumeisincreasingeveryday.Thevelocityisincreasingbytheinternetconnecteddevices.Thevarietyisalsoexpandingandtheorganizations’capabilitytocaptureandprocessthedataislimited.ThefollowingarethechallengesinareaofBigDatawhenitishandled:1.Datacaptureandstorage2.Datatransmission3.Datacuration4.Dataanalysis5.DatavisualizationAccordingto,challengesofbigdataminingaredividedinto3tiers.Thefirsttieristhesetupofdataminingalgorithms.Thesecondtierincludes1.InformationsharingandDataPrivacy.2.DomainandApplicationKnowledge.Thethirdoneincludeslocallearningandmodelfusionformultipleinformationsources.3.Miningfromsparse,uncertainandincompletedata.4.Miningcomplexanddynamicdata.Figure2:PhasesofBigDataChallengesGenerallyminingofdatafromdifferentdatasourcesistediousassizeofdataislarger.Bigdataisstoredatdifferentplacesandcollectingthosedatawillbeatedioustaskandapplyingbasicdataminingalgorithmswillbeanobstacleforit.Nextweneedtoconsidertheprivacyofdata.Thethirdcaseisminingalgorithms.Whenweareapplyingdataminingalgorithmstothesesubsetsofdatatheresultmaynotbethatmuchaccurate.VII.ForecastofthefutureTherearesomechallengesthatresearchersandpractitionerswillhavetodealduringthenextyears:AnalyticsArchitecture:Itisnotclearyethowanoptimalarchitectureofanalyticssystemsshouldbetodealwithhistoricdataandwithreal-timedataatthesametime.AninterestingproposalistheLambdaarchitectureofNathanMarz.TheLambdaArchitecturesolvestheproblemofcomputingarbitraryfunctionsonarbitrarydatainrealtimebydecomposingtheproblemintothreelayers:thebatchlayer,theservinglayer,andthespeedlayer.ItcombinesinthesamesystemHadoopforthebatchlayer,andStormforthespeedlayer.Thepropertiesofthesystemare:robustandfaulttolerant,scalable,general,andextensible,allowsadhocqueries,minimalmaintenance,anddebuggable.Statisticalsignificance:Itisimportanttoachievesignificantstatisticalresults,andnotbefooledbyrandomness.AsEfronexplainsinhisbookaboutLargeScaleInference,itiseasytogowrongwithhugedatasetsandthousandsofquestionstoansweratonce.Distributedmining:Manydataminingtechniquesarenottrivialtoparalyze.Tohavedistributedversionsofsomemethods,alotofresearchisneededwithpracticalandtheoreticalanalysistoprovidenewmethods.Timeevolvingdata:Datamaybeevolvingovertime,soitisimportantthattheBigDataminingtechniquesshouldbeabletoadaptandinsomecasestodetectchangefirst.Forexample,thedatastreamminingfieldhasverypowerfultechniquesforthistask.Compression:DealingwithBigData,thequantityofspaceneededtostoreitisveryrelevant.Therearetwomainapproaches:compressionwherewedon’tlooseanything,orsamplingwherewechoosewhatisthedatathatismorerepresentative.Usingcompression,wemaytakemoretimeandlessspace,sowecanconsideritasatransformationfromtimetospace.Usingsampling,weareloosinginformation,butthegainsinspacemaybeinordersofmagnitude.ForexampleFeldmanetalusecoresetstoreducethecomplexityofBigDataproblems.Coresetsaresmallsetsthatprovablyapproximatetheoriginaldataforagivenproblem.Usingmerge-reducethesmallsetscanthenbeusedforsolvinghardmachinelearningproblemsinparallel.Visualization:AmaintaskofBigDataanalysisishowtovisualizetheresults.Asthedataissobig,itisverydifficulttofinduser-friendlyvisualizations.Newtechniques,andframeworkstotellandshowstorieswillbeneeded,asforexamplethephotographs,infographicsandessaysinthebeautifulbook”TheHumanFaceofBigData”.HiddenBigData:Largequantitiesofusefuldataaregettinglostsincenewdataislargelyuntaggedfilebasedandunstructureddata.The2012IDCstudyonBigDataexplainsthatin2012,23%(643exabytes)ofthedigitaluniversewouldbeusefulforBigDataiftaggedandanalyzed.However,currentlyonly3%ofthepotentiallyusefuldataistagged,andevenlessisanalyzed.VIII.CONCLUSIONTheamountsofdataisgrowingexponentiallyduetosocialnetworkingsites,searchandretrievalengines,mediasharingsites,stocktradingsites,newssourcesandsoon.BigDataisbecomingthenewareaforscientificdataresearchandforbusinessapplications.Dataminingtechniquescanbeappliedonbigdatatoacquiresomeusefulinformationfromlargedatasets.Theycanbeusedtogethertoacquiresomeusefulpicturefromthedata.BigDataanalysistoolslikeMapReduceoverHadoopandHDFShelpsorganization.中文譯文：大數(shù)據(jù)挖掘研究摘要數(shù)據(jù)已經(jīng)成為各個經(jīng)濟、行業(yè)、組織、企業(yè)、職能和個人的重要組成部分。大數(shù)據(jù)是用于識別大型數(shù)據(jù)集的一個術語，通常其大小比典型的數(shù)據(jù)庫要大。大數(shù)據(jù)引入了獨特的計算和統(tǒng)計挑戰(zhàn)。在工程和科學的大部分領域，大數(shù)據(jù)目前都有延伸。由于大數(shù)據(jù)的數(shù)量之多、速度之快、種類之繁，所以可以使用數(shù)據(jù)挖掘，有助于從龐大的數(shù)據(jù)集中提取有用的數(shù)據(jù)。本文介紹了HACE定理，它描述了大數(shù)據(jù)革命的特征，并從數(shù)據(jù)挖掘角度提出了一個大數(shù)據(jù)處理模型。關鍵詞：大數(shù)據(jù)，數(shù)據(jù)挖掘，HACE定理，結構化和非結構化。一、簡介大數(shù)據(jù)指的是大量的結構化數(shù)據(jù)和非結構化數(shù)據(jù)，這些數(shù)據(jù)遍布了整個組織。如果這些數(shù)據(jù)被正確使用，將會產(chǎn)生有意義的信息。大數(shù)據(jù)包括大量的數(shù)據(jù)，需要大量的實時處理。它提供了兩個空間，一個用于發(fā)現(xiàn)新價值，并從隱藏的價值中了解深入的知識，另一個用于有效管理數(shù)據(jù)。數(shù)據(jù)庫是一個與數(shù)據(jù)相關的邏輯上有組織的集合，可以方便地管理、更新和訪問。數(shù)據(jù)挖掘是從數(shù)據(jù)庫或其他存儲庫中存儲的大量數(shù)據(jù)中發(fā)現(xiàn)有趣的知識(如關聯(lián)、模式、更改、異常和重要結構)的過程。大數(shù)據(jù)包括3V的特征。它們是大量（volume）、高速（velocity）和多樣（variety）。大量意味著每秒生成的數(shù)據(jù)量。數(shù)據(jù)是靜態(tài)的，它的規(guī)模特征也是眾所周知的。高速是數(shù)據(jù)生成的速度。大數(shù)據(jù)應該有高速數(shù)據(jù)，社交媒體產(chǎn)生的數(shù)據(jù)就是一個例子。多樣意味著可以采取不同類型的數(shù)據(jù)，例如音頻、視頻或文檔。它可以是數(shù)字、圖像、時間序列、數(shù)組等。數(shù)據(jù)挖掘從不同的角度分析數(shù)據(jù)，并將其匯總為有用的信息，可用于商業(yè)解決方案和預測未來趨勢。數(shù)據(jù)挖掘（DM）也稱為數(shù)據(jù)庫中的知識發(fā)現(xiàn)（KDD），或者知識發(fā)現(xiàn)和數(shù)據(jù)挖掘，是為關聯(lián)規(guī)則等模式自動搜索大量數(shù)據(jù)的過程。它應用了統(tǒng)計學、信息檢索、機器學習和模式識別等方面的許多計算技術。數(shù)據(jù)挖掘僅在短時間內(nèi)從數(shù)據(jù)庫中提取所需的模式。根據(jù)要挖掘的模式類型，可以將數(shù)據(jù)挖掘任務分為匯總、分類、聚類、關聯(lián)和趨勢分析。在包括物理、生物和生物醫(yī)學等科學和工程領域在內(nèi)的所有領域，大數(shù)據(jù)都有延伸。二、大數(shù)據(jù)挖掘一般而言，大數(shù)據(jù)是指大量數(shù)據(jù)的集合，這些數(shù)據(jù)來自互聯(lián)網(wǎng)、社交媒體、商業(yè)組織、傳感器等各種來源。我們可以借助數(shù)據(jù)挖掘技術來提取一些有用的信息。這是一種從大量數(shù)據(jù)中發(fā)現(xiàn)模式以及描述性、可理解的模型的技術。容量是數(shù)據(jù)的大小，大于PB和TB。規(guī)模和容量的增加使得傳統(tǒng)的工具難以存儲和分析。在預定的時間段內(nèi)，應該使用大數(shù)據(jù)挖掘大量數(shù)據(jù)。傳統(tǒng)的數(shù)據(jù)庫系統(tǒng)旨在解決少量的結構化和一致性的數(shù)據(jù)，而大數(shù)據(jù)包括各種數(shù)據(jù)，如地理空間數(shù)據(jù)、音頻、視頻、非結構化文本等。大數(shù)據(jù)挖掘是指通過大數(shù)據(jù)集來查找相關信息的活動。為了快速處理不同來源的大量數(shù)據(jù)，使用了Hadoop。Hadoop是一個免費的基于Java的編程框架，支持在分布式計算環(huán)境中處理大型數(shù)據(jù)集。其分布式文件系統(tǒng)支持節(jié)點之間的快速數(shù)據(jù)傳輸速率，并允許系統(tǒng)在發(fā)生節(jié)點故障時不中斷運行。它為分布式數(shù)據(jù)處理進行MapReduce，用于結構化和非結構化數(shù)據(jù)。三、大數(shù)據(jù)特征——HACE定理我們有大量的異構數(shù)據(jù)。數(shù)據(jù)之間存在復雜的關系。我們需要從這些龐大的數(shù)據(jù)中發(fā)現(xiàn)有用的信息。讓我們想象一下，一個盲人被要求畫大象的場景。每個盲人收集到的信息可能會認為軀干像墻，腿像樹，身體像墻，尾巴像繩子。盲人們可以相互交換信息。圖1：盲人和大象其中的一些特征包括：1.具有異構及不同來源的海量數(shù)據(jù)：大數(shù)據(jù)的基本特征之一是大量的異構數(shù)據(jù)和多樣數(shù)據(jù)。例如，在生物醫(yī)學世界中，個人用姓名、年齡、性別、家族病史等來表示，用于X射線和CT掃描圖像和視頻。異構是指同一個體的不同表現(xiàn)形式，多樣是指用各種特征來表示單一信息。2.具有分布式和非集中式控制的自治：來源是自治的，即自動生成；它在沒有任何集中控制的情況下生成信息。我們可以將它與萬維網(wǎng)（WWW）進行比較，其中每臺服務器都提供一定數(shù)量的信息，而不依賴于其他服務器。3.復雜且不斷演化的關系：隨著數(shù)據(jù)量變得無限大，存在的關系也很大。在早期階段，當數(shù)據(jù)很小時，數(shù)據(jù)之間的關系并不復雜。社交媒體和其他來源生成的數(shù)據(jù)具有復雜的關系。四.工具：開放源碼革命Facebook、雅虎、Twitter、LinkedIn等大公司受益于開源項目，并為之做出貢獻。在大數(shù)據(jù)挖掘中，有許多開源計劃。其中最受歡迎的是：ApacheMahout：主要基于Hadoop的可擴展機器學習和數(shù)據(jù)挖掘的開源軟件。它實現(xiàn)了廣泛的機器學習和數(shù)據(jù)挖掘算法：聚類、分類、協(xié)同過濾和頻繁模式。R：為統(tǒng)計計算和可視化設計的開源編程語言和軟件環(huán)境。R是由在新西蘭奧克蘭大學的RossIhaka和RobertGentleman在1993年開始設計的，用于統(tǒng)計分析超大型數(shù)據(jù)集。MOA：流數(shù)據(jù)挖掘開源軟件，可以實時進行數(shù)據(jù)挖掘。它具有分類、回歸、聚類和頻繁項集挖掘和頻繁圖挖掘等實現(xiàn)。它始于新西蘭懷卡托大學機器學習小組的一個項目，以WEKA軟件著稱。流框架為使用簡單的根據(jù)XML來定義和運行流過程提供了一個環(huán)境，并能夠使用MOA、Android和StormSAMOA：這是一個新的即將推出的分布式流挖掘軟件項目，它將S4和Storm與MOA結合在一起。VowpalWabbit：在雅虎啟動的開源項目。研究并繼續(xù)在微軟研究院設計一個快速的、可擴展的、有用的學習算法。VW能夠從大量特征數(shù)據(jù)集中學習。在進行線性學習、通過并行學習時，它可以超過任何單機網(wǎng)絡接口的吞吐量。五、大數(shù)據(jù)的數(shù)據(jù)挖掘數(shù)