4.2.3大數(shù)據(jù)方法的驅(qū)動(dòng)力-數(shù)據(jù)獲取和數(shù)據(jù)來(lái)源

《大數(shù)據(jù)導(dǎo)論》數(shù)據(jù)獲取和數(shù)據(jù)來(lái)源數(shù)據(jù)獲取和數(shù)據(jù)來(lái)源日志傳感器Web數(shù)據(jù)采集IP骨干傳輸數(shù)據(jù)中心傳輸數(shù)據(jù)傳輸數(shù)據(jù)集成數(shù)據(jù)清洗冗余消除數(shù)據(jù)預(yù)處理數(shù)據(jù)獲取和數(shù)據(jù)來(lái)源數(shù)據(jù)采集數(shù)據(jù)獲取和數(shù)據(jù)來(lái)源傳感器數(shù)據(jù)獲取和數(shù)據(jù)來(lái)源日志文件數(shù)據(jù)獲取和數(shù)據(jù)來(lái)源Web爬蟲數(shù)據(jù)獲取和數(shù)據(jù)來(lái)源采集方法模式數(shù)據(jù)結(jié)構(gòu)數(shù)據(jù)規(guī)模復(fù)雜性應(yīng)用程序傳感器pull結(jié)構(gòu)化或非結(jié)構(gòu)化中復(fù)雜的視頻監(jiān)控，庫(kù)存管理日志文件push結(jié)構(gòu)化或半結(jié)構(gòu)化小容易Web日志，點(diǎn)擊流Web爬蟲pull混合大中等搜索，社交網(wǎng)絡(luò)分析根據(jù)數(shù)據(jù)采集方式的不同,數(shù)據(jù)采集方法又可以大致分為以下兩類:?基于拉(pull-based)的方法,數(shù)據(jù)由集中式或分布式的代理主動(dòng)收集。?基于推(push-based)的方法,數(shù)據(jù)由源或第三方推向數(shù)據(jù)匯聚點(diǎn)。數(shù)據(jù)獲取和數(shù)據(jù)來(lái)源大數(shù)據(jù)傳輸過程數(shù)據(jù)獲取和數(shù)據(jù)來(lái)源IP骨干傳輸數(shù)據(jù)獲取和數(shù)據(jù)來(lái)源數(shù)據(jù)中心傳輸——中心體系架構(gòu)數(shù)據(jù)獲取和數(shù)據(jù)來(lái)源TCP協(xié)議DCTCPVamananUDP協(xié)議擁塞控制機(jī)制數(shù)據(jù)中心傳輸——傳輸協(xié)議數(shù)據(jù)獲取和數(shù)據(jù)來(lái)源數(shù)據(jù)預(yù)處理數(shù)據(jù)集成數(shù)據(jù)集成技術(shù)在邏輯上和物理上把來(lái)自不同數(shù)據(jù)源的數(shù)據(jù)進(jìn)行集中,為用戶提供一個(gè)統(tǒng)一的視圖數(shù)據(jù)清洗數(shù)據(jù)清洗是指在數(shù)據(jù)集中發(fā)現(xiàn)不準(zhǔn)確、不完整或不合理數(shù)據(jù),并對(duì)這些數(shù)據(jù)進(jìn)行修補(bǔ)或移除以提高數(shù)據(jù)質(zhì)量的過程冗余消除去重技術(shù)能夠顯著地減少存儲(chǔ)空間,