爬蟲基礎(chǔ)-塔卡庫urllib和urllib30301

1、Python爬蟲基礎(chǔ)課教室講師：塔卡 時間：2018年4月教學(xué)監(jiān)督熱線 教學(xué)監(jiān)督熱線 01了解fiddler抓包工具的基本使用知識點(diǎn)回顧本節(jié)知識點(diǎn)01. 掌握urllib模塊02. 了解urllib3模塊教學(xué)監(jiān)督熱線 urllib知識點(diǎn)一教學(xué)監(jiān)督熱線 urllib庫中的4大模塊01020304urllib.requests 請求模塊urllib.parse 解析模塊urllib.error 異常處理模塊urllib.robotparse robots.txt解析模塊urllib.request內(nèi)容urlopen基本請求添加header頭信息使用cookie設(shè)置代理發(fā)出請求: urllib.re

2、quest.urlopenfrom urllib import requesturl = rs = request.urlopen(url)print(rs)# print(rs.read()print(rs.read().decode(utf-8) # 讀取網(wǎng)頁源代碼print(rs.getcode() # 狀態(tài)碼print(rs.headers) # 和info一樣的,返回的頭信息print()print(rs.geturl() # 返回請求的url例:使用Request，添加heardersfrom urllib import requesturl = headers = User-Ag

3、ent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36,req = request.Request(url,headers=headers) #添加headers頭信息rs1 = request.urlopen(req)print(rs1)例:from urllib import parsex = parse.quote(潭州) # url 編碼print(x)y = parse.unquote(x) # ur

4、l解碼print(y)url3 = .format(x)rs = request.urlopen(url3)urllib.parse.quote/urllib.parse.unquote例:d1 = a:1,b:2x = parse.urlencode(d1) # 轉(zhuǎn)成字符串print(x)url3 = y = /?wd=pythonz = parse.urljoin(url3,y)print(z)urllib.parse.urlencode/urllib.parse.urljoin例:urllib.parse需掌握的方法:urllib.parse.quote() # url 中文編碼urll

5、ib.parse.unquote() # url中文解碼urllib.parse.urlencode() # 轉(zhuǎn)成&拼接的字符串urllib.parse.urljoin() # url拼接get/post方法請求urllib庫，默認(rèn)是get方式提交，urlopen的data參數(shù)默認(rèn)為None，當(dāng)data參數(shù)不為空的時候，urlopen提交方式為Post（request的get_method）,如果要使用其他方式必須Request(method=options)或req.get_method = lambda : options。urllib.request.Request(url, data=

6、None, headers=, method=None)get方法請求data = urllib.parse.urlencode(data) #對data進(jìn)行轉(zhuǎn)碼，轉(zhuǎn)化成str類型new_url = url+?+data # URL拼接result = urllib.request.urlopen(new_url) response = result.read() print(response.decode(utf8) url = 請求的地址/URL” data = 提交的數(shù)據(jù)post方法請求data = urllib.parse.urlencode(data) # 對data進(jìn)行轉(zhuǎn)碼，轉(zhuǎn)化成

7、str類型data = data.encode(utf-8) #進(jìn)行編碼,post請求只支持byte類型new_url = urllib.request.Request(url,data) #對url和參數(shù)進(jìn)行包裝response = urllib.request.urlopen(new_url) response = result.read() print(response.decode(utf8) url = 請求的地址/URL data = 提交的數(shù)據(jù)cookie的使用from http import cookiejarurl = # 創(chuàng)建cookiejar對象cookie = cook

8、iejar.CookieJar()# 使用HTTPCookieProcessor創(chuàng)建cookie處理器，并以它為參數(shù)構(gòu)建opener對象opener = request.build_opener(request.HTTPCookieProcessor(cookie)# 把opner變成urlopenrequest.install_opener(opener)rs = request.urlopen(url)print(cookie)例:設(shè)置代理:url5 = # 測試ip地址的網(wǎng)站proxy = http:ip:port,https:ip:port# 創(chuàng)建代理處理器proxies = requ

9、est.ProxyHandler(proxy)# 創(chuàng)建opener對象opener = request.build_opener(proxies,request.HTTPHandler )request.install_opener(opener)data = request.urlopen(url5)print(data.read().decode()例:# error異常處理 URLError是HTTPError的父類 只有HTTPError有code屬性url1 = url2 = try: request.urlopen(url1)except error.URLError as e:

10、if hasattr(e,code): print(HTTPError) elif hasattr(e,reason): print(URLError)error異常處理:例:urllib3知識點(diǎn)二教學(xué)監(jiān)督熱線 PoolManager:# PoolManager實(shí)例來生成請求,由該實(shí)例對象處理與線程池的連接以及線程安全的所有細(xì)節(jié)，不需要任何人為操作：http = urllib3.PoolManager()# 通過request()方法創(chuàng)建一個請求resp = http.request(get,url)html = resp.dataprint(html.decode()HTTPConnecti

11、onPool:url = http = urllib3.HTTPConnectionPool( )resp = http.request(get,url)print(resp.data.decode(gbk)設(shè)置代理:import urllib3proxy = urllib3.ProxyManager( , headers=connection: keep-alive)r = proxy.request(GET, )print(r.status)print(r.data)操作cookie:沒有找到相關(guān)的操作，只能通過headers設(shè)置的方式實(shí)現(xiàn)http訪問庫的需求總結(jié)- 正確發(fā)出http、https請求-