大數(shù)據(jù)實(shí)時(shí)計(jì)算spark-2爬蟲

urllib2import

urllib2response

=

urllib2.urlopen("htprint

response.read()")urllib2.urlopen(url,data,timeout)爬蟲構(gòu)造請(qǐng)求對(duì)象req

=

urlllib2.request("resp

=

urllib2.urlopen(r)resp.read()")爬蟲POSTimport

urllibimport

urllib2values

=

{"username":"tom","password":"123456"}url

=

"

";resp

=

urllib2.urlopen(url,data)print

resp.read()爬蟲GETimport

urllibimport

urllib2values

=

{}values["username"]="tom"values["password"]="123456"data

=

urllib.urlencode(values);url

=

" ?"

+data;req=urllib2.request(url)resp

=

urllib2.urlopen(req)print

resp.read()爬蟲設(shè)置Headerimport

urllib

;

import

urllib2url

=

"

";user_agent="Mozilla/4.0

(compatible;

MSIE

5.5;Windows

NT)"values

=

{'username'

:

tom', 'password':

123'}headers={'User-Agent':user_agent}data=urllib.urlencode(values)req=urllib2.request(url,data,headers)resp=urllib2.urlopen(req)爬蟲防盜鏈headers={...,'Referer':'ht/articles'};爬蟲使用加入的header，。請(qǐng)求來自于header信息。有些服務(wù)器會(huì)

import

urllib2enable_proxy=Trueproxy_handler

=

urllib2.ProxyHandler({"http":'htt:8080'})爬蟲使用加入的header，有些服務(wù)器會(huì)請(qǐng)求來自于

。header信息。null_proxy_handler=urllib2.ProxyHandler({})if

enable_proxy:opener

=urllib2.build_opener(proxy_handler)else:opener

=

urllib2.build_opener(null_proxy_handler)urllib2.install_opener(opener)爬蟲超時(shí)urllib2.urlopen(url,timeout=10)爬蟲URLErrorimport

urllib2req=urllb2.Request("try:urllib2.urlopen(request)exception

urllib2.URLError

,e:print

e.reason")爬蟲')HTTPErrorimport

urllib2req

=

urllib2.Request('try:urllib2.urlopen(req)except

urllib2.HTTPError,

e:print

e.codeprint

e.reason爬蟲')多錯(cuò)誤處理import

urllib2req

=

urllib2.Request('try:urllib2.urlopen(req)except

urllib2.HTTPError,

e:print

e.codeexcept

urllib2.URLError,e:print

e.reasonelse

:print

'ok'爬蟲應(yīng)用獲取importliblib.=handler=urllib2.HTTPJar()Processor()opener

=urllib2.build_opener(handler)response

=

opener.open('ht')for

item

in

:print

'Name

=

'+print

'Value

=

'+item.value爬蟲應(yīng)用保存import到文件libimport

urllib2.txt'filename

=

'd:///Mozilla不好使,LWP=

lib.LWPJarokJar(filename))handler

=

urllib2.HTTP

Processor(opener

=

urllib2.build_opener(handler)response

=

opener.open("ht.save(ignore_discard=True,ignore_expires=True))")爬蟲應(yīng)用從文件import lib;import

urllib2.txt'lib.Mozilla

Jar()filename

=

'd:/=.load('.txt',

ignore_discard=True,ignore_expires=True)req

=

urllib2.Request("ht")爬蟲應(yīng)用從文件Processor())opener

=

urllib2.build_opener(urllib2.HTTPresponse

=

opener.open(req)print

response.read()爬蟲使用應(yīng)用模擬 登錄。Processor())opener

=

urllib2.build_opener(urllib2.HTTPresponse

=

opener.open(req)print

response.read()爬蟲正則表達(dá)式貪婪模式ab*查找abbbbc得到abbbb非貪婪模式ab*?查找abbbbc得到ab.*?表示使用非貪婪模式大部分語言都是貪婪模式，python也是。使用的話使用非貪婪模式爬蟲使用re.findall方法返回匹配子串列表。

import

reptn=ur'<a\s*href="([\u0000-\uffff&&^"]*?)"'url='htres=re.finditer(ptn,url)for

r

in

res

:print

r.group(1)'爬蟲提取系統(tǒng)時(shí)間進(jìn)行文件保存import

timemillis=

int(round(time.time()

*1000))print

millis爬蟲-時(shí)間單位取值import

osurl

=

'

...'finename

=

os.path.join('d:/a.jpg')urllib.urlretrieve(url,

finename)爬蟲-文件到本地二進(jìn)制文件到本地磁盤上文件

import

osimport

shutilsrcFile

=

os.path.basename(srcPath)dstFile

=

os.path.basename(srcPath)shutil.copyFile(src,dst);爬蟲-文件hbase的thrifthbase是java編寫，對(duì)于第語言，可以通過thrift1和thrift2進(jìn)行通信，生產(chǎn)環(huán)境通常都是thrift2。啟動(dòng)hbase的thrift2服務(wù)器$>hbase-deamon.sh

start

thrift2驗(yàn)證是否啟動(dòng)成功thrift2啟動(dòng)在9090端口。$>netstat-anop

|

grep

9090使用Thrift寫入hbaseThrift

python庫0.9.2版本編譯安裝thrift$>./configure$>make$>make

install生成python代碼$>thrift

-o

outdir

-gen

py

hbase\thrift2python中使用thrift2(Linux)Thrift

windows編譯器和thrift

python模塊thrift

python模塊thrift

windows編譯器/apache/thrift/0.10.0/thrift-0.10.0.exepython中使用thrift2(Windows)安裝thrift

python模塊[注意先安裝python]cmd>解壓thrift-0.9.0.tar.gzcmd>cd

\thrift-0.9.0\lib\pycmd>setup.py

install找到hbase.thrift文件[thrift1]D:\downloads\bigdata\hbase-1.2.4\hbase-thrift\src\main\resources\

\apache\hadoop\hbase\thrift\hbase.thriftpython中使用thrift2(Windows)找到hbase.thrift文件[thrift2]D:\downloads\bigdata\hbase-1.2.4\hbase-thrift\src\main\resources\

\apache\hadoop\hbase\thrift2\hbase.thrift使用編譯器編譯hbase的thrfit代碼cmd>cp

hbase.thrift

d:/thrift/hbase.thriftcmd>cd

d:/thriftcmd>mkdir

outcmd>thrift

-o

./outdir

-gen

py

./hbase.thriftpython中使用thrift2(Windows)使用 令如下python中使用thrift2(Windows)生成代碼如下python中使用thrift2(Windows)將生成的py庫文件 到IDEA的項(xiàng)目庫中python中使用thrift2(Windows)編寫python

，

hbase[PythonHbaseDemo.py]導(dǎo)入相應(yīng)模塊#

-*-

coding=utf-8

-*-import

sys#導(dǎo)入安裝thrift模塊from

thrift

import

Thriftfrom

thrift.transport

import

TSocketfrom

thrift.transport

import

TTransportfrom

tocol

import

TBinaryProtocol#導(dǎo)入編譯生成的hbase

模塊from

mythrfit.hbase

import

THBaseServicefrom

mythrfit.hbase.ttypes

import

*python中使用thrift2(Windows)編寫python

，

hbase[PythonHbaseDemo.py]連接hbase

thrift2server#hbase的thrift服務(wù)器地址s201:9090transport=TSocket.TSocket('s201',9090)transport

=

TTransport.TBufferedTransport(transport)protocol

=

TBinaryProtocol.TBinaryProtocol(transport)client

=

THBaseService.Client(protocol)#開啟連接，準(zhǔn)備傳輸數(shù)據(jù)

transport.open()python中使用thrift2(Windows)編寫python

，

hbase[PythonHbaseDemo.py]get查詢#get查詢table

=

'ns1:t1'row='row1'get

=

TGet()#定義表名

#定義rowkey#創(chuàng)建get對(duì)象get.row=row

#設(shè)置rowkeyr=client.get(table,get)python中使用thrift2(Windows)編寫python

，

hbase[PythonHbaseDemo.py]put#puttable='ns1:t1'row='row2'v1=TColumnValue('f1','id','101')v2=TColumnValue('f1','name','tomas')v3=TColumnValue('f1','age','12')vals=[v1,v2,v3]put

=

TPut(row,vals)client.put(table,put)#指定行和列

#調(diào)用put執(zhí)行python中使用thrift2(Windows)編寫python

，

hbase[PythonHbaseDemo.py]put#deletedelete

=

TDelete('row1',

[v1])client.deleteSingle(table,

delete)#關(guān)閉傳輸

transport.close();python中使用thrift2(Windows)編寫python

，

hbase[PythonHbaseDemo.py]批量#putrowkey=1

;table='ns1:t1'while

rowkey

<

10000

:srowkey

=

'row'

+

str(rowkey);v1=TColumnValue('f1','id',''

+

str(rowkey))v2=TColumnValue('f1','name','tom'

+

str(rowkey))v3=TColumnValue('f1','age',''

+

str(rowkey

%

100))vals=[v1,v2,v3]#