python爬蟲(chóng)實(shí)戰(zhàn)

1、python爬蟲(chóng)實(shí)戰(zhàn)，多線程爬取京東jd html頁(yè)面：無(wú)需登錄的網(wǎng)站的爬蟲(chóng)實(shí)戰(zhàn)2014-12-02 20:04:31標(biāo)簽：網(wǎng)站爬蟲(chóng) Python import版權(quán)聲明：原創(chuàng)作品，如需轉(zhuǎn)載，請(qǐng)與作者聯(lián)系。否則將追究法律責(zé)任?！厩把浴?2#本腳本僅用于技術(shù)交流，請(qǐng)3勿用于其他用途4#byRiver5#Date:2014-12-0219:00:006【需求說(shuō)明】以京東為示例，爬取頁(yè)面的，獲取頁(yè)面中得數(shù)據(jù)：記錄到data.txt；獲取頁(yè)面中得圖片,保存下來(lái)。1、list的url如下2、商品詳情頁(yè)的url如下:【技術(shù)說(shuō)明】1使用了Python的以下庫(kù)2importos#檢查文件是否存在等34fromH

2、TML Parseri mp ortHTML Parser#5用于解析html的庫(kù)，有坑：如果2.66的Python，可能悲劇7importhttplib,re#發(fā)起http請(qǐng)求8im portsysjs on ,datetime,bisect#使用了二分快速查找fromurl parsei mp orturl parse#解析url，分析出url的各部分功能使用fromthreadi ngim po rtThread#多線程importsocket#設(shè)置httplib間超時(shí)時(shí)【代碼邏輯說(shuō)明】1、run（獲取最終要的結(jié)果）2、ParseListpageurl:返回list的總共的頁(yè)面數(shù)量3、j

3、udgelist:判斷該list是否已經(jīng)爬取完畢了，第一個(gè)list中的所有url、最后list的所有url都爬取完畢了，那么久說(shuō)明list的所有Page爬取完畢了（實(shí)際上是一種弱校驗(yàn)）4、getfinalurl_content:如果list沒(méi)爬取完畢，每個(gè)list爬取，解析list中得每個(gè)html（判斷html是否爬取過(guò)），獲得內(nèi)容和img【坑說(shuō)明】1、需要設(shè)置超時(shí)時(shí)間，和重試，否則爬取一個(gè)url卡住的時(shí)候，整個(gè)線程都悲劇了。2、有編碼的坑，如果頁(yè)面是gb2312的編碼，需要轉(zhuǎn)換為utf-8的編碼:htt prest mp. decode(gbk).e ncode(utf-8)3、parser

4、.feed的內(nèi)容，如果存在一些特殊字符，可能需要替換，否則解析出來(lái)會(huì)莫名不4、圖片保存，根據(jù)url獲取前面兩個(gè)數(shù)字，保存。以免一個(gè)目錄下保存了過(guò)多的圖片?！緢?zhí)行結(jié)果】1、con sole輸出2、data.txt存儲(chǔ)解析出來(lái)的內(nèi)容3、judegurl.txt（保存已經(jīng)爬取過(guò)的url）4、圖片（下載的圖片）【代碼詳情】1#-*-codi ng:utf-8-*-3_author_-River4#本腳本僅用于技術(shù)交流，請(qǐng)勿用于其他用途5#byRiver6#Date:2014-12-0219:00:0078importos#倉(cāng)J建文件9fromHTMLParserimportHTMLParser#用于解

5、析html的庫(kù)，有坑：如果2.6的1python，可能悲劇0importhttplib,re#發(fā)起http請(qǐng)求11im portsysjs on ,datetime,bisect#使用了二分快速查找1fromurlparseimporturlparse#解析url，分析出url的各部分功能2fromthread ingimpo rtThread#使用多線程1importsocket#設(shè)置httplib超時(shí)時(shí)間31#htmlparser的使用簡(jiǎn)介4#定義intt方法：需要使用到得屬性#定義handle_starttag，處理你想分析的tag的具體操作5#定義handle data，遇到你定義的情

6、況，獲取相應(yīng)標(biāo)簽的data1 -6#定義你獲取最終返回的各種數(shù)據(jù)1ClassList Page Parser（HTML Parser）:7def_init_(self):1oSeIf.ha ndledtags=a81self. process ing=None9self.flag=2self.li nk=”02self.setlinks=set()#該list頁(yè)面中包含的每個(gè)商品的url，定義為set，1主要是為了使用其特性：去重2self. pageNo=12self.alldata=23HTMLParser._init_(self)2defha ndle_starttag(self,tag

7、,attrs):4p attern=re.co mp ile(rA0-92,)2#attrs是屬性的list，每個(gè)屬性(包含key，value)又是一個(gè)元組52#已上為例子：判斷了該list的長(zhǎng)度為3(其他的a標(biāo)簽就被過(guò)濾了)6iftagi nseif.ha ndledtagsa ndle n(attrs)=3:#非常關(guān)鍵的是，找出你想的2url和不想要的url的區(qū)別7#prin tdebug:attrs,attrs28self.flag=2self.data=9self. process in g=tag3fortarget,hrefi nattrs:#03區(qū)別1ifpattern2.mat

8、ch(href):#3們想要的url2else:33P ass3#1、長(zhǎng)度為14#2,href是由規(guī)則的：3所以，以下代碼就出來(lái)了53iftag in self.ha ndledtagsa ndle n( attrs)=1:6self.flag=3self.data=非常關(guān)鍵的是，找出你想的url和不想要的url的再加一層判斷，如果匹配上pattern2，說(shuō)明是我cat=737%2C794%2C798&p age=10&amp ;JL=6_0_O7self. process in g=tag3_forhref,urlinattrs:#非常關(guān)鍵的是，找出你想的url和不想

9、要的url的區(qū)別83#prin tdebug:attrs,attrs9ifpattem3.match(url):40#pnn tdebug:url,url4self.lasturl=urlelse:4p ass24defha ndle_data(self,data):3ifself. processi ng:#去掉空格4pass#其實(shí)這里我們根本沒(méi)使用獲取到得data，就pass把4else:4匸P ass54defha ndle_e ndtag(self,tag):6iftag=self. process ing:4self. process ing=None74defgetl in ks(

10、self):8retu rn self.setl inks4defgetlasturl(self):9retu rn self.lasturl50#Fi nall Page Parser的定義過(guò)程參考上個(gè)p arser，關(guān)鍵是怎樣分析頁(yè)面，最終5寫出代碼，并且驗(yàn)證，這里就不詳細(xì)說(shuō)了1classFi nallP age Parser(HTML Parser):5def_init_(self):2 5self.ha ndledtags=div,h1,stro ng,a,del,div,img,li,s3pan ,tbody,tr,th,td,i5self. process ing=None4sel

11、f.title=55self.jd price=5self.refprice=6self. partimgs_show=set()#展示圖片5self. partimgs=set()#75self. partdetail=#8self.s pecificati on=#5self.ty peOrsize=set()#9self.div=60self.flag=詳情圖片商品詳情，參數(shù)等規(guī)格參數(shù) 尺碼和類型6self.flagrefprice=1self.flagtitle=62self.flagjd price=6self.flagt yp eOrsize=3self.flag partimgs

12、二6self.flag partdetail=”46self.flags pecificati on =5self.flagt yp eOrsize=6self.li nk=”6self. partsli nks=67HTMLParser._init_(self)6defha ndle_starttag(self,tag,attrs):8self.titleflag=6self.flagrefprice=97self.flagtitle=oself.flagjd price=7self.flagty peOrsize=”1self.flag partimgs二72self.flag partd

13、etail=”7self.flags pecificati on =3self.flagty peOrsize=”7iftag in self.ha ndledtags:47self.data=5self. process in g=tag7iftag=div:6forkey,valuei nattrs:77self.div=value#取出div的name判斷是否是所需要的圖片等元素7iftag=i:8self.flagty peOrsize-match7iftag=aa ndle n( attrs)=2:98tm pflag=0forkey,valuei nattrs:8ifkey=hre

14、fa ndtmp flag=first82ifkey=titlea ndvalue!=:8tmp flag=t mp flag+sec ond3ift mp flag=firstsec on d:84self.flagty peOrsize-match8iftag=h1:5self.flagtitle=match8iftag=stro nga ndle n( attrs)=2:6Jort mp class,idi nattrs:87ifid=jd-p rice:8self.flagjd price=match8iftag=del:8cSelf.flagrefpnce-match99iftag=

15、li:0self.flag partdetail=match9iftag=thortag=trortag=td:#+#19#879498.htmltd中有br的只取到第一個(gè)，需要把92喜歡為“”9self.flags pecificati on =match3iftag=img:94imgtmp_flag=9imgt mp二5forkey,valuei nattrs:9if(key=srcorkey=data-lazyload):69imgt mp=value7ifkey=width:#可能還有l(wèi)ogo9ifre.search(rAd1,9$,value):8ifin t(value)=160

16、:99imgtmp_flag= no1break0ifself.div=s pec-itemsa ndimgtm p!-:0imgt mp=re.co mp ile/n5/).subC7 n1/,imgt mp)10self. partimgs_show.add(imgt mp)1elifimgtmp_flag!= no a ndimgt mp!-:1defha ndle_data(self,data):0ifself. process ing:21self.data+=data獲取成功0ifself.flagtitle=match:#3self.title=data10ifself.flag

17、jd pnce=-match:4self.jd pnce=data.stn p()1ifself.flagty peOrsize=-match:0ifself.flagrefpnce=-match:51self.refprice=data.stri p()oifself.flag partdetail=matcha ndre.search(r6keytmp=data.split(”:)0.strip()1valuetmp=data.split(”:)1.strip()07self. partdetailkeyt mp =valuet mp1ifself.flags pecificatio n

18、=-matcha nddata.stri p()!-a nddata.str0ip ()!=主體:8else:10p ass9defha ndle_e ndtag(self,tag):1iftag=self. process ing:1self. process ing=None01defgetdata(self):1retu rn title:self.title, partimgs_show:self. partimgs_show,jd pri1ce:self.jd price,refprice:self.refprice, partimgs:self. partimgs,hp artde

19、tail:self. partdetail,s pecificatio n:self.s pecificatio n,ty12P eOrsize:self.t yp eOrsize1#定義方法httpread，用于發(fā)起http的get請(qǐng)求，返回http的獲取內(nèi)容1#這也是代碼抽象的結(jié)果，如若不抽象這塊代碼出來(lái)，后續(xù)你回發(fā)現(xiàn)很多重復(fù)3的寫這塊代碼11defhtt pread(host,url,headers):4htt prest mp=”1try:1conn=htt plib.HTT PConn ectio n(host)5Conn .request(GET,url,N on e,header

20、s)1htt pres=c onn. getres pon se()6htt prest mp=htt pres.read()1exce ptExce pti on,e:17conn=htt plib.HTT PConn ectio n(host):,data):#獲取成功Conn .request(GET,url,N on e,headers)1htt pres=c onn. getres pon se()81 htt prest mp=htt pres.read()1printe9fin ally:1ifco nn:2oConn. close()1returnhtt prest mp2#定

21、義方法sendhttp，調(diào)用httpread，獲取結(jié)果并替換編碼(gbk換為utf-8),1并保存到文件中(以免下次再去下載頁(yè)面，這樣就節(jié)省了時(shí)間)12#2defse ndhtt p(u rl,host,savefile):#定義htt p頭部，很多網(wǎng)站對(duì)于你不攜帶User-Age nt及Referer等情況，是2不允許你爬取。3#具體的httP的頭部有些啥信息，你可以看chrome,右鍵審查元素，點(diǎn)擊2network，點(diǎn)擊其中一個(gè)鏈接，查看requestheader4headers=Host:host,1Conten t-t yp e:a pp licatio n/x-www-form-ur

22、le ncoded;charset=UTF-8,2Acce pt:text/html;q=0.9,image/web p,*/*;q=0.8,512htt prest mp=”6try:12htt prest mp=htt pread(host,url,headers)7ifhtt prest mp=:#1htt prest mp=htt pread(host,url,headers)2ifhttprestmp=”:#重試2次8htt prest mp=htt pread(host,url,headers)2exce ptExce pti on,e:9try:1htt prest mp=htt

23、 pread(host,url,headers)30ifhtt prest mp=”:#1htt prest mp=htt pread(host,url,headers)3ifhttprestmp=”:#重試2次1htt prest mp=htt pread(host,url,headers)13exce ptExce pti on,e:2printe1printe3ifre.search(rcharset=gb2312,htt prestm p):#31轉(zhuǎn)碼為utf-8(因?yàn)槿侄际褂昧藆tf-8)3htt prestm p.rep lace(charset=gb2312,charset=u

24、tf-8)4try:1htt prest mp=htt prest mp .decode(gbk).e ncode(utf-8)#35所以要加上try1exceptException,e:#如果html編碼本來(lái)就是utf8或者轉(zhuǎn)換編碼出錯(cuò)的時(shí)候,3就啥都不做，就用原始內(nèi)容61printe3try:7witho pen (savefile,w)asfile_object:1file_object.write(htt prest mp)2-ofile_object.flush()81exce ptExce pti on,e:3printe9returnhtt prest mp1#list的頁(yè)面的解

25、析方法40def parseList pageurl(list pageurl):1urlobj=url parse(list pageurl)4ifurlobj.query:11geturl=urlobj. path+urlobj.query4else:2geturl=urlobj .p ath1htmlfile=html/list+geturl43ifnot1htt presult=se ndhtt p( geturl,urlobj.host name,htmlfile)4witho pen( htmlfile)asfile:4htmlc on te nt=file.read()14p a

26、rser=List Page Parser。#聲明一個(gè)解析對(duì)象5Parser.feed(htmlcontent.replace(amp;,”)#將html的內(nèi)容feed進(jìn)去呦和tdebug:htmlc on te nt,htmlc on te nt4finalparseurl=parser.getlinks()#然后get數(shù)據(jù)即可如果是gb2312得編碼，就要有可能轉(zhuǎn)碼失敗,6last pageurl=p arser.getlasturl()4urlobj_last pageurl=url parse(last pageurl)7#pnn tdebug:urlobj_last pageurl,

27、urlobj_last pageurl14total PageNo=08#pnn turlobj1ifre.search(r & ,urlobj_last pageurl.query):4try:91exce ptExce pti on,e:5pnn tlast pageurl:+str(last pageurl)Oprinte1p arseList pageurl_rult=fi nalp arseurls:fi nalp arseurl,tota IP ageNo51:total PageNo1if parseList pageurl_rultfi nalp arseurls!=a

28、 ndp arseList pageurl_ru5lttotal PageNo!i：2print1else:53pnn t-%m-%d%H:%M:%S)+ ,p arselist pageurlfail:+list pageurl1retu rnp arseList pageurl_rult5#最終的html頁(yè)面的解析方法：會(huì)使用到html得解析器Fi nail Page Parser4defparseFi nallyurl(fi nallyurl):5urlobj=url parse(fi nallyurl)5geturl=urlobj .p ath1htmlfiledir=html/fi

29、nally/+geturl.s plit(7)10:256ifnot1try:5os.makedirs(htmlfiledir)7exce ptExce pti on,e:15printe8htmlfile=htmlfiledi葉geturl1ifnot5htt presult=se ndhtt p( geturl,urlobj.host name,htmlfile)91ifhtt presuit:6printOelse:1print6Witho pen( htmlfile)asfile:1htmlc onten t=file.read()6p arser= Fin all Page Pars

30、er()21#htmmparser遇到/就表示tag結(jié)尾，所以必須替換，遇到br/替換為BRBR6否則會(huì)解析失敗3htmlco nten t=re.co mp ile(vbr/).sub(BRBR,htmlco nte nt)1p arser.feed(htmlc ontent)64finalp arseurl=p arser.getdata()1iffi nalp arseurl:6print5else:1ePrint6retu rnfinalp arseurl#獲取圖片的方法6defgetimg(imgdir,imgurl):71imgobj=url parse(imgurl)6getimgurl=imgobj .p ath8imgt mpp athlist=getimgurl.s plit(7)1img name=imgt mpp athlistle n(imgt mpp athlist)-169i