商業(yè)智能（BI）介紹

BI/DWIntroductionJamesChenPSOSr.Consultant程序=算法+數(shù)據(jù)結(jié)構(gòu)--NiklausWirth數(shù)據(jù)結(jié)構(gòu)離散結(jié)構(gòu)線形結(jié)構(gòu)-數(shù)組鏈表?xiàng)ｊ?duì)列樹(shù)圖算法空間復(fù)雜度時(shí)間復(fù)雜度用戶內(nèi)存中數(shù)據(jù)結(jié)構(gòu)物理數(shù)據(jù)結(jié)構(gòu)E-R業(yè)務(wù)流程二者相輔相成，不同應(yīng)用側(cè)重不同經(jīng)典業(yè)務(wù)系統(tǒng)模型Skillset數(shù)據(jù)庫(kù)C++/J2EE/.netPowerDesignerER_WINOOUML面向過(guò)程OODesignWorkflowDevelopBusinessTypeModelIdentifyBusinessInterfacesIdentifySystemInterfaces&OpsCreateInitialCompSpecs&ArchitectureDiscoverBusinessOperationsRefineInterfaces&OpsRefineComponentSpecs&ArchitectureDefineInterfaceInformationModelsSpecifyOperationPre/PostconditionsSpecifyComponentInterfaceConstraintsBusinessConceptModelUseCaseModelBusinessInterfacesSystemInterfacesInterfacesInterfacesComponentSpecs&ArchitectureComponentSpecs&ArchitectureComponentSpecs&ArchitectureBusinessTypeModelArchitecturePatternsExistingInterfacesExistingAssetsComponentIdentification

ComponentInteractionComponentSpecification笑一笑數(shù)據(jù)倉(cāng)庫(kù)初學(xué)者：數(shù)據(jù)倉(cāng)庫(kù)與數(shù)據(jù)庫(kù)有什么區(qū)別？

某專家：一個(gè)有倉(cāng)一個(gè)沒(méi)倉(cāng)！

BI/DW的引出統(tǒng)計(jì)報(bào)表需求數(shù)字越來(lái)越重要報(bào)表越來(lái)越多效率越來(lái)越差數(shù)據(jù)倉(cāng)庫(kù)的定義“數(shù)據(jù)倉(cāng)庫(kù)是在企業(yè)管理和決策中面向主題的,集成的,與時(shí)間相關(guān)的和不可修改的數(shù)據(jù)集合”

BillInmon數(shù)據(jù)倉(cāng)庫(kù)(DataWarehouse)是一種專為聯(lián)機(jī)分析應(yīng)用和決策支持系統(tǒng)提供數(shù)據(jù)源的結(jié)構(gòu)化的數(shù)據(jù)環(huán)境數(shù)據(jù)倉(cāng)庫(kù)要解決的問(wèn)題是從數(shù)據(jù)庫(kù)中獲取信息的問(wèn)題。數(shù)據(jù)信息INFORMATIONINFORMATIONINFORMATIONINFORMATION影響數(shù)據(jù)倉(cāng)庫(kù)的幾個(gè)人物BillInmon-《數(shù)據(jù)倉(cāng)庫(kù)》RalphKimball-《數(shù)據(jù)倉(cāng)庫(kù)工具箱》BernardLiautaud-《商業(yè)智能》OLTP與數(shù)據(jù)倉(cāng)庫(kù)系統(tǒng)RDBMS關(guān)系數(shù)據(jù)庫(kù)SAP/ERPVSAM主機(jī)系統(tǒng)EXCELWeb/XMLOLTP業(yè)務(wù)系統(tǒng)數(shù)據(jù)倉(cāng)庫(kù)系統(tǒng)

數(shù)據(jù)是當(dāng)前的數(shù)據(jù)總在實(shí)時(shí)變化中存儲(chǔ)明細(xì)數(shù)據(jù)優(yōu)化是針對(duì)即時(shí)更新的事務(wù)處理支持日常的業(yè)務(wù)支持辦事人員或行政人員

數(shù)據(jù)是歷史的數(shù)據(jù)是靜態(tài)的，除數(shù)據(jù)刷新外數(shù)據(jù)是匯總的優(yōu)化是針對(duì)大批量查詢而不是更新支持長(zhǎng)遠(yuǎn)的業(yè)務(wù)戰(zhàn)略決策支持決策人員和管理人員面向事務(wù)面向決策5-10年數(shù)據(jù)集市分析型CRM業(yè)務(wù)指標(biāo)分析數(shù)據(jù)倉(cāng)庫(kù)BI系統(tǒng)建設(shè)方法（1）直接建立BI應(yīng)用OLTP數(shù)據(jù)源各種前端應(yīng)用KPI指標(biāo)報(bào)表多維分析即席查詢數(shù)據(jù)挖掘數(shù)據(jù)倉(cāng)庫(kù)環(huán)境業(yè)務(wù)智能層中央數(shù)據(jù)倉(cāng)庫(kù)數(shù)據(jù)倉(cāng)庫(kù)數(shù)據(jù)集結(jié)數(shù)據(jù)源公司數(shù)據(jù)源外部數(shù)據(jù)源BIW數(shù)據(jù)集市數(shù)據(jù)倉(cāng)庫(kù)管理子系統(tǒng)數(shù)據(jù)倉(cāng)庫(kù)元數(shù)據(jù)目錄信息訪問(wèn)和引用部件BI系統(tǒng)建設(shè)方法（2）企業(yè)數(shù)據(jù)倉(cāng)庫(kù)體系架構(gòu)RelationalPackageLegacyExternalsourceDataCleanToolSourceDataDataStagingWareHouseAdmin.ToolsEnterpriseDataWarehouseDataExtraction,TransformationandloadDatamartDatamartEnterprise/CentralDataWarehouseRDBMSROLAPRDBMSRDBMSArchitectedDatamartsCentralMetadataDataModelingToolEnd-UserToolEnd-UserToolMDBEnd-UserToolEnd-UserToolLocalMetadataLocalMetadata典型物理架構(gòu)ETL_DM1ETL_DM2BAS_DW1BAS_DW2BAS_OP1BAS_OP2Bas_sw01Bas_sw02Dx_sp01Dx_sp02Dx_sp03元數(shù)據(jù)管理服務(wù)器BAS_SM1BAS_SM270T!企業(yè)數(shù)據(jù)倉(cāng)庫(kù)的目的數(shù)據(jù)整合--建立一個(gè)企業(yè)統(tǒng)一的信息平臺(tái)建立統(tǒng)一的業(yè)務(wù)數(shù)據(jù)定義建立唯一的統(tǒng)計(jì)分析數(shù)據(jù)源快速訪問(wèn)—采用適合企業(yè)BI應(yīng)用的數(shù)據(jù)組織方式新需求的開(kāi)發(fā)快數(shù)據(jù)獲取的效率快業(yè)務(wù)管理層業(yè)務(wù)操作層戰(zhàn)略層底層數(shù)據(jù)低粒度匯總高粒度匯總建設(shè)數(shù)據(jù)倉(cāng)庫(kù)的主要任務(wù)數(shù)據(jù)倉(cāng)庫(kù)建模ETL前端展現(xiàn)組數(shù)據(jù)挖掘元數(shù)據(jù)數(shù)據(jù)倉(cāng)庫(kù)引擎NCRTeredataORACLE9以上DB2EEESybaseIQMSSQLServer維模型的設(shè)計(jì)每個(gè)維可以由一個(gè)或多個(gè)維層次或聚合路徑組成Dimension1事實(shí)Dimension2Dimension4measure1measure2.....supportiveattributes....Dimension3aggregationpathordimensionhierarchydimensionkey1dimensionkey2.....aggregationlevelsIWS模型的特點(diǎn)簡(jiǎn)單視圖—實(shí)例倉(cāng)庫(kù)底層設(shè)計(jì)考量（一）慢速變化維問(wèn)題快速變化和慢速變化倉(cāng)庫(kù)底層設(shè)計(jì)考量（二）粒度問(wèn)題一天變化多次入庫(kù)時(shí)間倉(cāng)庫(kù)中間層設(shè)計(jì)考量基于用戶標(biāo)識(shí)多個(gè)粒度匯總綜合各個(gè)數(shù)據(jù)集市效率考慮倉(cāng)庫(kù)集市層設(shè)計(jì)考量面向應(yīng)用報(bào)表集市挖掘集市主題集市KPI集市維模型以最細(xì)粒度數(shù)據(jù)為主表加物化視圖數(shù)據(jù)倉(cāng)庫(kù)與數(shù)據(jù)集市數(shù)據(jù)源CUBE報(bào)表事實(shí)表維表維表維表維表數(shù)據(jù)倉(cāng)庫(kù)事實(shí)視圖維視圖維視圖維視圖維視圖業(yè)務(wù)驅(qū)動(dòng)數(shù)據(jù)驅(qū)動(dòng)數(shù)據(jù)集市設(shè)計(jì)技巧星型還是雪花？組合代理鍵慢速變化維維度的不同層次處理聚合表的設(shè)計(jì)….ETL過(guò)程源表結(jié)構(gòu)--目標(biāo)結(jié)構(gòu)工具？自己編碼？--業(yè)務(wù)邏輯的任務(wù)已經(jīng)很大實(shí)時(shí)任務(wù)和批量任務(wù)CDC的處理ODS的處理？BODateIntegratorInformaticaPowerCenterDatastageSagent數(shù)據(jù)源大數(shù)據(jù)量變化數(shù)據(jù)關(guān)聯(lián)數(shù)據(jù)手工數(shù)據(jù)抽取時(shí)機(jī)和最遲到達(dá)時(shí)間！數(shù)據(jù)源考量-大數(shù)據(jù)量二十余種清單上萬(wàn)個(gè)文件，大小不一一天幾十GB的數(shù)據(jù)數(shù)據(jù)源考量-變化數(shù)據(jù)實(shí)時(shí)變化未存歷史數(shù)據(jù)源考量-關(guān)聯(lián)數(shù)據(jù)沒(méi)有transaction幾個(gè)文件不一致怎么辦？數(shù)據(jù)源考量-手工數(shù)據(jù)外部錄入？OLTP系統(tǒng)？天氣社會(huì)指標(biāo)集團(tuán)客戶信息操作型CRM數(shù)據(jù)源考量-歷史數(shù)據(jù)更新修改昨日數(shù)據(jù)批錯(cuò)價(jià)/延遲話單等ETL抽取/轉(zhuǎn)換/加載源數(shù)據(jù)的理解加載時(shí)間4小時(shí)ETL實(shí)現(xiàn)工具Loader匯總（數(shù)據(jù)庫(kù)匯總還是文件匯總）調(diào)度ETL-調(diào)度一天幾十GB的數(shù)據(jù)并行度/任務(wù)大小/資源/優(yōu)先級(jí)調(diào)度清單類（大數(shù)據(jù)量）-文件的拆分和合并/實(shí)時(shí)處理業(yè)務(wù)類（實(shí)體之間的關(guān)系復(fù)雜）-稽合帳務(wù)類（數(shù)據(jù)準(zhǔn)確）-驗(yàn)證參數(shù)類（變化影響其他實(shí)體）-告警/事先處理其他任務(wù)月末處理前端展現(xiàn)固定報(bào)表多維分析即席查詢儀表盤數(shù)據(jù)挖據(jù)BO(CR)Hyperion(Brio)CognosMicrostrategy儀表盤OLAP分析報(bào)表，即席查詢固定報(bào)表格式靈活多變CrystalReportMSTRBrioReportNet多維分析MOLAP--CognosPowerPlay,HyperionEssbaseROLAP--BOE,MSTRHOLAP--BO--MSOLAP切片旋轉(zhuǎn)切塊鉆取OLAP分析多角度、多層次對(duì)同一主題進(jìn)行分析通過(guò)旋轉(zhuǎn)、鉆取、切片等技術(shù)探察數(shù)據(jù)旋轉(zhuǎn)：按不同順序組織各個(gè)維，對(duì)結(jié)果進(jìn)行考察鉆?。涸谝粋€(gè)維內(nèi)部沿著從高到低或從低到高的方向考察數(shù)據(jù)(有上鉆和下鉆兩種)切片：在確定某些維數(shù)據(jù)的情況下對(duì)其他維進(jìn)行觀察有MOLAP、ROLAP、HOLAP三種實(shí)現(xiàn)方式OLAP設(shè)計(jì)-指標(biāo)還是維度性別年齡長(zhǎng)途類型通話時(shí)段客戶數(shù)通話時(shí)長(zhǎng)性別年齡客戶數(shù)通話時(shí)長(zhǎng)長(zhǎng)途類型通話時(shí)長(zhǎng)通話時(shí)段通話時(shí)長(zhǎng)

1-2點(diǎn)通話時(shí)長(zhǎng)即席查詢語(yǔ)義層效率很關(guān)鍵BO語(yǔ)義層儀表盤儀表盤、KPI---EPM整合了企業(yè)的整個(gè)績(jī)效、把告警、報(bào)表、分析穿在了一起B(yǎng)OEPMHyperionBPM數(shù)據(jù)挖掘機(jī)器學(xué)習(xí)數(shù)理統(tǒng)計(jì)數(shù)據(jù)庫(kù)預(yù)測(cè)、特征分析、客戶分類SAS\SPSS數(shù)據(jù)挖掘?qū)嵤┑捏w系結(jié)構(gòu)數(shù)據(jù)倉(cāng)庫(kù)數(shù)據(jù)挖掘引擎算法數(shù)據(jù)選擇知識(shí)庫(kù)評(píng)價(jià)修改知識(shí)建模分析員領(lǐng)域知識(shí)元數(shù)據(jù)元數(shù)據(jù)管理什么是元數(shù)據(jù)？元數(shù)據(jù)和開(kāi)發(fā)文檔有何區(qū)別？為什么單單在數(shù)據(jù)倉(cāng)庫(kù)中如此重視元數(shù)據(jù)？文檔語(yǔ)音計(jì)費(fèi)GPRS計(jì)費(fèi)文檔帳務(wù)文檔結(jié)算營(yíng)業(yè)文檔文檔客服文檔MIS網(wǎng)管HR文檔文檔文檔元數(shù)據(jù)管理(續(xù))業(yè)務(wù)元數(shù)據(jù)業(yè)務(wù)元數(shù)據(jù)用業(yè)務(wù)名稱、定義、描述和別名來(lái)表示數(shù)據(jù)倉(cāng)庫(kù)和業(yè)務(wù)系統(tǒng)中的各種屬性，直接供業(yè)務(wù)分析人員使用。技術(shù)元數(shù)據(jù)技術(shù)元數(shù)據(jù)描述了數(shù)據(jù)源、數(shù)據(jù)轉(zhuǎn)換、抽取過(guò)程、加載策略以及目標(biāo)數(shù)據(jù)庫(kù)的定義等。元數(shù)據(jù)管理(續(xù))CWM1.0數(shù)據(jù)源ETL數(shù)據(jù)倉(cāng)庫(kù)數(shù)據(jù)集市OLAPCUBE元數(shù)據(jù)管理(續(xù))能分析什么？WhereUsedDependsOn事先還是事后？自動(dòng)化修改？典型的項(xiàng)目組織架構(gòu)

項(xiàng)目經(jīng)理首席業(yè)務(wù)分析師ETL組組長(zhǎng)；設(shè)計(jì)師、軟件工程師項(xiàng)目管理組開(kāi)發(fā)組首席架構(gòu)設(shè)計(jì)師項(xiàng)目經(jīng)理助理項(xiàng)目管理數(shù)據(jù)倉(cāng)庫(kù)組組長(zhǎng)；設(shè)計(jì)師、軟件工程師數(shù)據(jù)展現(xiàn)組組長(zhǎng)；設(shè)計(jì)師、軟件工程師數(shù)據(jù)挖掘庫(kù)組組長(zhǎng)；設(shè)計(jì)師、軟件工程師項(xiàng)目領(lǐng)導(dǎo)小組客戶代表公司領(lǐng)導(dǎo)質(zhì)量經(jīng)理支持組系統(tǒng)組培訓(xùn)組客戶代表測(cè)試組組長(zhǎng)；測(cè)試工程師項(xiàng)目組織架構(gòu)配置管理組謝謝演講完畢，謝謝觀看！內(nèi)容總結(jié)BI/DWIntroduct