K-means聚類(lèi)試驗(yàn)

1、實(shí)驗(yàn)算法K-means聚類(lèi)實(shí)驗(yàn)【實(shí)驗(yàn)名稱(chēng)】K-means聚類(lèi)實(shí)驗(yàn)【實(shí)驗(yàn)要求】掌握K-means模型應(yīng)用過(guò)程，根據(jù)模型要求進(jìn)行數(shù)據(jù)預(yù)處理，建模，評(píng)價(jià)與應(yīng)用；【背景描述】聚類(lèi)算法是一種典型的無(wú)監(jiān)督學(xué)習(xí)算法，在聚類(lèi)算法中根據(jù)樣本之間的相似性，將樣本劃分到不同的類(lèi)別中，對(duì)于不同的相似度計(jì)算方法，會(huì)得到不同的聚類(lèi)結(jié)果，常用的相似度計(jì)算方法有歐式距離法。【知識(shí)準(zhǔn)備】了解K-means模型的使用場(chǎng)景，數(shù)據(jù)標(biāo)準(zhǔn)。了解Python/Spark數(shù)據(jù)處理一般方法。了解spark模型調(diào)用，訓(xùn)練以及應(yīng)用方法【實(shí)驗(yàn)設(shè)備】Windows或Linux操作系統(tǒng)的計(jì)算機(jī)。部署Spark,Python,本實(shí)驗(yàn)提供centos6.8

2、環(huán)境?！緦?shí)驗(yàn)說(shuō)明】采用UCI機(jī)器學(xué)習(xí)庫(kù)中的wine數(shù)據(jù)集作為算法數(shù)據(jù)，除去原來(lái)的類(lèi)別號(hào)，把數(shù)據(jù)看做沒(méi)有類(lèi)別的樣本，訓(xùn)練K-means模型，對(duì)樣本進(jìn)行聚類(lèi)?！緦?shí)驗(yàn)環(huán)境】Spark2.3.1,Pyrhon3.X,實(shí)驗(yàn)在命令行pyspark中進(jìn)行，或者把代碼寫(xiě)在py腳本，由于本次為實(shí)驗(yàn)，以學(xué)習(xí)模型為主，所以在命令行中逐步執(zhí)行代碼，以便更加清晰地了解整個(gè)建模流程?！緦?shí)驗(yàn)步驟】第一步：?jiǎn)?dòng)pyspark命令行中鍵入pyspark-masterlocal4,本地模式啟動(dòng)spark與pythonGCC4.4.720129313作小Hat4.4.7-1)CftlihmTyp*p*.copyrighrt.cre

3、ditsorlicsnstformensiafanutinn.cUtepMhsruLnsmultipleSLFMJbindings.SLFU：FoundBindinginjdn/tpark-2412116.jdrifofo4j/ivDl/suticljOM0rttMtdM-.clKSSL=-4J：Foindtindingin|ar：fila：/o3：/hadQdp-j.l.i3/sharfi.-,hado；p/c&wndn/ib/sl.l4j-lojLi-i./.25.iriIcLoggerfilfider.cLassJSLT4LSettittp;/A*Mh5U4xrg/c&d

4、t5.hlml#aultiplc_birdrngi+crancxpUriation,SIF4T-ArtjaIbin-inqisrtft/peong.ilfdj.irpl.oj4jloggprfactor/2019-04-751S；41；16WMiNativeCodeLDader；62-Lhabletoloadnative-hadooplibraryfaryourplatfom.usingbuiltim-jdvaclassesiereapplicableS&itinqd#Fdult1唱levelto.TuJdjutU用irqlv#1u u館”.、父1_叫1_門(mén)“皿4-白/).FflfEp

5、“4Rn、皿5吳“mwLmvfU蚓嘰U/_/_f/_/_/_f/_yf_ryf_r_j_jjj/_/rsioi2.3UsUsinzPythianversionversion3.S.2(detault(detault, ,JulJul2201617:17:53:061 1SparkSess-ian-dvliable曰9spark.*.4 4I I第二步：導(dǎo)入用到的包，并讀取數(shù)據(jù)：(1) .導(dǎo)入所需的包frompysparkimportSparkContext,SQLContext,SparkConffrommathimportsqrt(2) .讀取數(shù)據(jù)源，數(shù)據(jù)源地址為：/opt/algorith

6、m/kmeans/wine.txtdf_wine=sc.textFile(ufile:/opt/algorithm/kmeans/wine.txt).map(lambdax:str(x).split(,).map(lambdax:float(z)forzinx)frompysparkuiportSparkContextrStJLCont日?d,SparkConffirofnmthimpartsqrtdfjwijie=sc.textFile(ufile：/opt/alganthffl/kmeans/wine,txt).niap(，lambdax：str(x).map(lambdax：z)forz

7、inx)a|(3) .數(shù)據(jù)轉(zhuǎn)換為DataFramedf_wine_rdd=sqlContext.createDataFrame(df_wine)a，adfjrfinc_rdd-sqlContext.ersatftDatarramstdfinc)17rlS:?fiWARNflijortSto-Filtrg&t叮1獨(dú)刮/4(1.returningNaSuchCbjtFxroption|(4) .展示數(shù)據(jù)df_wine_rdd.show()(ifjwine_r(ld.showOT-_Tr-r-L-r-4-l-4-4-T44=T-FIII2|314|5|6|7|8|9|LSIJl|INI13|

8、Ml+十一一 TH+T+*r+-+第三步：數(shù)據(jù)預(yù)處理(1) .去掉類(lèi)別標(biāo)號(hào)那一類(lèi)df_wine_rdd=df_wine_rdd.drop(_1)(2) .構(gòu)建向量importpyspark.ml.featureasml_featurecols=df_wine_rdd.columnsvectorAssembler=ml_feature.VectorAssembler().setInputCols(cols).setOutputCol(cols)|1*0|14,23111.013.2|1 。13.16|1.0114.2711.9113.24|1.9|14-2|1-0|14.3911.9114.06

9、I7I14.S3|1.013.|1.0|13.75I1.9IU.75|1.0|14,30111.013,63111.0143|1.013.831.712.43 口 5,51.732.1411173352,&718.61.952.516,8127,02.81W.S265101102.8113.03.353.062.763,243.490,Z80,260.30.242.592.&7I21.0I1L8,Q2.82.69|0.391.762-45|15.21112.33.；7|3.39|0.24l.ff7|2.45|14P6|96,012,5|2,52|0.321512.61|L7.6

10、|121,0|22.510JIl.42.17|14MQ1.352.27I1B.0中 02498.02.983295,54L043.921065-0128%領(lǐng)1.053.41050,02.815.6B|1.S33.171185.0|2AS|7.80.963.45L480.0I1.8214.32|1,Q4|2.93|735.0|1.97&.75|1.05|2.951450.0|1.我13,25l.Q2|3.5S|L290.0|1.2515,05|1.96|3,58|1295,G|2.161.4S1.731.711,87|1.81L921.572.318.02E16.82.41116.D2a

11、lidfwinerdd-dfwinerdd.drop()importpyspark,ml.featureasml_featurec&ls-dtwine_rdd,calumnsvectorAssembler-(nl_featdrerVectorAssmbl.er(TsetInputCols(c&ls).setOutputColfcols)wine_Vc=vectorAssenbler.transfom(df_wine_rdd I I(3).對(duì)數(shù)據(jù)進(jìn)行標(biāo)準(zhǔn)化standardScaler=ml_feature.StandardScaler().setInputCol(cols).se

12、tOutputCol(cols_st).setWithMean(True).setWithStd(True).fit(wine_Vc)wine_Vc_St=standardScaler.transform(wine_Vc)nnd式口rfl_fmtu明文;)ndirdsc.sorinputcol(calsJ.otoutpuicoHJc.a工竹M刖1ru*Lt&irrtittiSTdIvc)wineVtSt=群taftd3rdscaTL&r.tm6年仃12楠白u(yù)c)!-一第四步：構(gòu)建模型并應(yīng)用，輸出聚類(lèi)后的分類(lèi).Kmeans模型，設(shè)置輸入列，迭代次數(shù)，輸出列，聚類(lèi)數(shù)importpy

13、spark.ml.clusteringasml_clusteringclusters=ml_clustering.KMeans(featuresCol=cols_st,predictionCol=Pred,initMode=k-means|,initSteps=5,tol=1e-4,maxIter=20,seed=None).fit(wine_Vc_St)wine_Vc_St_clusters=clusters.transform(wine_Vc_St),5in|rtpyEfari.nl.tlMtaringKUjcluitarifq八、clusters gclueinq.Kjieint(tut

14、urco15O.-predictionCflil=Pre(l.initn)lelL-iadr)g|inLLSlyJ5.-5.-tol=lv-4,，MJuIgES,swdlitin?，).liTftfuie/c_St)17t22r03riAATdBLAS：61-Fdn回loICJJupl倒胡LhcinfJoniCDT.gj.thub.fonnil.rClibLlUtiueSyilMBl_AS2019-G4-2517t22;C3RAHM6L.SSLdFdil旬laIgJijipl#nldtiQAfFoniLQT.yjtKubonniJJ昭L4SwiiwVcSt_cltf8trs-cluitert.

15、trirnfoHI(mintmintvcSt)m mI I(2),打印模型wine_Vc_St_clusters.show()wwrke_/c_St_cluitenihMI2019-0i-3517：j3：0ffWAPW-TrunotMth*、tnn?ionVaplnline*it+aoTh*b備匚立nheAdjustedbysettingspjrkd-burniaxTcStringFipldl!i1inSpairkFnvfonif第五步：構(gòu)建模型評(píng)估方法(1),輸出聚類(lèi)中心并合并到數(shù)據(jù)表importpyspark.mllib.linalgaslinalgcenter=zip(0,1,2,lin

16、alg.Vectors.dense(z)forzinclusters.clusterCenters()centers=sqlContext.createDataFrame(center).toDF(pred,center)wine_Vc_St_clusters_centers=wine_Vc_St_clusters.join(centers,on=pred)b“unpcrt分pdrk.nll必liklg曲liridlg4小center=zip(GGt t1,22t tliiulg.VectOE.densM/fori iinclusters.cljsterCenters)aacenters=5q

17、KontextcresteDataFrameicenter1,t(jDF(,prtd,fcenter)wineVcStclust&n_centers=winej/cStcljaters.joincenters,on=3r?dwineVcStclust&n_centers=winej/cStcljaters.joincenters,on=3r?d ) )4 4I I(2),計(jì)算出誤差平方和WSSSE=wineVcStclusterscenters.select(center,colsst).rdd.map(lambdapl.2pl.216163724?3724?第曲的從仃13.1

18、4.13.14.0M0M1J.1J.1J.1J.14.14.1.7112.431.7112.431.7912.141.7912.142.3612.671 1.951.9512,5?2,5?W W|3B7l.7B|3B7l.7B|2.d51.0712.d51.0713.452.1513.452.1513.61l.M123.61l.M12.17.17SIT|,S|U7.9|2.813.16.28we.DRrbfi|2.7610.26JGLDI2.S3.2410.2113.DI3.BSI3.4910.24HDD|2E)|LS|0.M113D|3.Z7|3喇機(jī)3*90,9|25|J-52|0.JULa

19、I2e|J.51|e.3tVJ.VJ.0|3.BU.OS0.20叫112.2915.M|L94i.2Bi4aei，g2.9115.6SIL.Qr2.1617,0|0,95l.EDK32|1.0Al.V7|fi.75|1.051.58|5.35|1.031.25|5.96|1.001.965.211.081 114|COIE|工 用5內(nèi)|1%2工1,口 工 .4次 . lC5O+0lll3,2J,7e.2.U,L+.lU0S.GI(13,Uf2.31r2,S7V.I1460,01114.37.1,95.2.5,1.I735dlim:比2,強(qiáng).工仃一jlJ5).n|14.2.17fi,1羋1.|1

20、2M*i(14.33.J,ff7,2.|12Q5.a|：li.05,2.15,2.1,.IlC4S,0HU.&3Jl.Mr2.17f.|必更55%心731川10.19bJ2521lfal/(H.L5M791309B4215.1.4773000031713J51.394936427767112,25341490679220.BJIn45S!BJIn45S!都安界8585x:sqrt(linalg.Vectors.squared_distance(linalg.Vectors.dense(list(x.asDict()center),linalg.Vectors.

21、dense(list(x.asDict()cols_st).sum()/wine_Vc_St_clusters_centers.count()WMSlfc-wnw_7c_5t_clu9ters_centers.wltccenter*-1lcaU_at.rdd.iap(|lbdaKrSqrtnuialgnVMtQrarSquardjdistijricedinalg.VtoFSHden5c(lL5tXHa5Dict(H-centcruIJm虱9,丫箕如1評(píng)神*1心1以+口sOict：!.*cols_st-lH11.smI)wine_Vc_St_clu5Jter5_centers.cointlAAA

22、|第六步：輸出模型效果(1).打印結(jié)果print(誤差平方和=+str(WSSSE)intr 誤若平方和=+strWSSSE)誤爰 u 方和=2.921651410515S335aI第七步：可以通過(guò)以下命令執(zhí)行python文件,查看最終結(jié)果spark-submit/opt/algorithm/kmeans/K-means.py【問(wèn)題與回答】1、Q:K值怎么確定？A：對(duì)于K值的確定，是Kmean的一個(gè)最大缺點(diǎn)，往往需要經(jīng)驗(yàn)判斷，統(tǒng)計(jì)學(xué)上，遍歷k1,樣本量，step,刻畫(huà)不同k的類(lèi)樣本與質(zhì)心平均距離曲線確定k的取值。2013-M-2517:30:13IkFO2819-04-2517:30:13IN

23、FOriver(1/1EJCEIZLIIQIF154qFintshiedtdkGflin31D(T2D429)_1739bytesresultw-tlw-tliodriverT*k5*tNiiPdgeir：54-Finishi0.0inn胖第.0(TI042glin39monlothast(executordlWHB必？517:30:13INFO2019-04-2517:36:13INFO201S-a4-3517:36:13IWO唉蘭干工和:2.525fl248BB642175TakScliPclLlerTii3l：54，RmovedT.skSei:31.0AwhoptakshaveallromipletedsftSSiheduler：54-ResultStage31(cDuntatNativEthcdAccessorImpI.java：Q：DJKJSihffduler;S4-Jo-b51finished:onuritfinished:onuritata