《數(shù)據(jù)采集技術(shù)》課件-Middlewares中間件

Middlewares——中間件Middlewares—中間件scrapy中間件Middlewares—中間件根據(jù)其程序執(zhí)行流程位置不同分為兩類(lèi)：(介于引擎與下載器之間)下載中間件(介于引擎與爬蟲(chóng)組件之間)爬蟲(chóng)中間件Middlewares—中間件中間件位于同一個(gè)middlewares.py文件之中：下載中間件下載中間件作用：對(duì)request對(duì)象進(jìn)行設(shè)置隨機(jī)請(qǐng)求頭、IP代理、Cookie等1判斷response對(duì)象的狀態(tài)碼，來(lái)決定是否將response提交給引擎2下載中間件process_request方法用來(lái)處理正常的請(qǐng)求對(duì)象的數(shù)據(jù)，每個(gè)request通過(guò)下載中間件時(shí)被調(diào)用process_request(self,request,spider)request要處理的Request對(duì)象spider該request對(duì)應(yīng)的Spider對(duì)象參數(shù)如下：下載中間件process_response方法用于對(duì)response進(jìn)行處理，下載器完成http請(qǐng)求，傳遞響應(yīng)給引擎的時(shí)候調(diào)用process_response(self,request,response,spider)request表示response所對(duì)應(yīng)的requestresponse表示被處理的response參數(shù)如下：spider表示response所對(duì)應(yīng)的spider下載中間件process_exception方法用來(lái)處理拋異常的請(qǐng)求對(duì)象的數(shù)據(jù)process_exception(self,request,exception,spider)request表示response所對(duì)應(yīng)的requestexception表示拋出的異常參數(shù)如下：spider表示response所對(duì)應(yīng)的spider下載中間件案例：middlewares.py，代碼如下：classRandomUserAgent(object):defprocess_request(self,request,spider):useragent=random.choice(USER_AGENTS)request.headers.setdefault("User-Agent",useragent)下載中間件案例：settings.py，代碼如下：USER_AGENTS=["Mozilla/4.0(compatible;MSIE6.0;WindowsNT5.1;SV1;AcooBrowser;.NETCLR1.1.4322;.NETCLR2.0.50727)","Mozilla/4.0(compatible;MSIE7.0;WindowsNT6.0;AcooBrowser;SLCC1;.NETCLR2.0.50727;MediaCenterPC5.0;.NETCLR3.0.04506)","Mozilla/4.0(compatible;MSIE7.0;AOL9.5;AOLBuild4337.35;WindowsNT5.1;.NETCLR1.1.4322;.NETCLR2.0.50727)","Mozilla/5.0(Windows;U;MSIE9.0;WindowsNT9.0;en-US)","Mozilla/5.0(compatible;MSIE9.0;WindowsNT6.1;Win64;x64;Trident/5.0;.NETCLR3.5.30729;.NETCLR3.0.30729;.NETCLR2.0.50727;MediaCenterPC6.0)","Mozilla/5.0(compatible;MSIE8.0;WindowsNT6.0;Trident/4.0;WOW64;Trident/4.0;SLCC2;.NETCLR2.0.50727;.NETCLR3.5.30729;.NETCLR3.0.30729;.NETCLR1.0.3705;.NETCLR1.1.4322)","Mozilla/4.0(compatible;MSIE7.0b;WindowsNT5.2;.NETCLR1.1.4322;.NETCLR2.0.50727;InfoPath.2;.NETCLR3.0.04506.30)","Mozilla/5.0(Windows;U;WindowsNT5.1;zh-CN)AppleWebKit/523.15(KHTML,likeGecko,Safari/419.3)Arora/0.3(Change:287c9dfb30)","Mozilla/5.0(X11;U;Linux;en-US)AppleWebKit/527+(KHTML,likeGecko,Safari/419.3)Arora/0.6“]下載中間件設(shè)置settings.py中的DOWNLOADER_MIDDLEWARES，取消注釋?zhuān)篋OWNLOADER_MIDDLEWARES={'mySpider.middlewares.MyspiderDownloaderMiddleware':543,}COOKIES_ENABLED=False案例：禁用cookies以防止某些網(wǎng)站根據(jù)Cookie來(lái)封鎖爬蟲(chóng)下載中間件設(shè)置成功后，運(yùn)行爬蟲(chóng)就會(huì)發(fā)現(xiàn)請(qǐng)求頭設(shè)置成功請(qǐng)求發(fā)送給服務(wù)器之前下載中間件隨機(jī)選擇一個(gè)請(qǐng)求頭，就可以避免總使用一個(gè)請(qǐng)求頭防止爬蟲(chóng)被服務(wù)器發(fā)現(xiàn)爬蟲(chóng)中間件爬蟲(chóng)中間件爬蟲(chóng)中間件作用：1在Response發(fā)送給Spider之前，對(duì)Response進(jìn)行處理2在Request發(fā)送給Scheduler之前，對(duì)Request進(jìn)行處理3在Item發(fā)送給ItemPipeline之前，對(duì)Item進(jìn)行處理爬蟲(chóng)中間件自帶了許多SpiderMiddleware爬蟲(chóng)中間件SPIDER_MIDDLEWARES_BASE變量的內(nèi)容如下：'scrapy.spidermiddlewares.httperror.HttpErrorMiddleware':50,'scrapy.spidermiddlewares.offsite.OffsiteMiddleware':500,'scrapy.spidermiddlewares.referer.RefererMiddleware':700,'scrapy.spidermiddlewares.urllength.UrllengthMiddleware':800,'scrapy.spidermiddlewares.depth.DepthMiddleware':900,SPIDER_MIDDLEWARES={'mySpider.middlewares.MyspiderSpiderMiddleware':543,}爬蟲(chóng)中間件自帶了許多SpiderMiddleware爬蟲(chóng)中間件process_spider_input方法當(dāng)Response被爬蟲(chóng)中間件處理時(shí)被調(diào)用。返回結(jié)果是：None或者拋出一個(gè)異常process_spider_input(response,spider)response被處理的Response對(duì)象spider該Response對(duì)應(yīng)的Spider參數(shù)如下：爬蟲(chóng)中間件process_spider_output方法當(dāng)Spider處理Response返回結(jié)果時(shí)，該方法被調(diào)用process_spider_output(response,result,spider)response生成該輸出的ResponseresultSpider返回的結(jié)果參數(shù)如下：spider其結(jié)果對(duì)應(yīng)的Spider爬蟲(chóng)中間件process_spider_exception方法當(dāng)process_spider_input方法拋出異常時(shí)被調(diào)用process_spider_exception(response,exception,spider)爬蟲(chóng)中間件process_start_r