從統(tǒng)一數(shù)據(jù)治理到下一代企業(yè)級+Data+Agent+的實踐與創(chuàng)新

......2018年的33ZB，增長到2025年的175ZB。Q數(shù)據(jù)貫穿于AI應用的整個鏈路數(shù)據(jù)治理–AI應用中不可或缺的部分https://www.llamaindex.ai/blog/introducing-llamacloud-and-llamaparse-af8cedf9006b?GDPR、CCPA等要求CONFLUENTCONFLUENT"----------h、---------"新業(yè)務和LLM需要一種統(tǒng)一的方式來訪問所有數(shù)據(jù)。Technical&BusinessDataConnecttotheDataThatMattersMost.ConnecttotheDataThatMattersMost.Automaticallyfind,classify,andmapallofyourdata-everywhere.allofyourdata-everywhere.Automateend-to-enddatarightsrequestsandreporting.Automateend-to-enddatarightsrequestsandreporting.Enrichtechnicalmetadatawithbusinessandoperationalmetadataforfullvisibility.AutomaticallyclassifymoretypesofdatainAutomaticallyclassifymoretypesofdatainmoreplaces.IPDataSimplifyandautomatedatalifecyclemanagementfromcollectiontodestruction.DataScience/AIScience/AIStreamingAnalyticsWarehouseBusinessIntelligenDataOrchestrationDataOrchestrationAAB組織壁壘技術壁壘安全壁壘管理壁壘trino鄉(xiāng)stor數(shù)據(jù)智能調優(yōu)統(tǒng)一權限管控統(tǒng)一數(shù)據(jù)管控面，旨在幫助用戶快速構建和管理湖倉架構，實現(xiàn)全方位湖倉管理數(shù)據(jù)智能調優(yōu)統(tǒng)一權限管控統(tǒng)一數(shù)據(jù)/元數(shù)據(jù)管理視圖從元數(shù)據(jù)層面上達到SSOT↓頻等Metastore數(shù)據(jù)倉庫實時消息數(shù)據(jù)湖Next-GenDataCatalogistheCoreinNewOpenDataArchitecture/ Tabulardata…}….}Non-tabulardata…}….}TechnologyTechnologyCommunicationCommunicationDataDataternetofternetofthingsAutomatioAutomationNetworkingNetworkingDataDataEngineerDataIngestionDataIngestionSparkSparkHDFSHDFSClientS3S3SDKutomatioutomationCopyCopyStructuredUnstructuredTraining/Checkpoint/Training/Checkpoint/Metrics/ModelData統(tǒng)一元數(shù)據(jù)治理前統(tǒng)一元數(shù)據(jù)治理后數(shù)據(jù)導出容易造成信息泄漏；數(shù)據(jù)使用無法跟蹤統(tǒng)一數(shù)據(jù)查找和訪問；統(tǒng)一安全模型和數(shù)據(jù)訪問DataScientistAIEngineerDataScientistAIEngineerTechnologyDataEngineerDataIngestionTensorflowCommunicationSparkTensorflowPytorchPytorchHDFSClientETLETLDataS3SDKternetofthingsGravitino!WriteDataReadDataAWriteDataGravitinoACLUnifiedAccessControlGravitinoDatasetUnstructuredDataScientistAIEngineerDataScientistAIEngineerTechnologyDataEngineerDataIngestionTensorflowCommunicationSparkTensorflowPytorchPytorchHDFSClientETLETLDataS3SDKternetofthingsGravitino!WriteDataReadDataAWriteDataGravitinoACLUnifiedAccessControlGravitinoDatasetUnstructuredDataUnstructuredDataStructuredDataDataDataNetworking生成式大語言模型（LLM）的功能相結合。由決策引擎選擇RAG方法；使用多種數(shù)據(jù)源/知識庫結合關鍵字和語義檢索的結果；檢索前后進行一定預處理；基于向量的檢索；易于實現(xiàn)，有限的數(shù)據(jù)，缺少對查詢和結果進行校驗KB等）?框架（LangChain，LlamaIndex?基礎模型/Embedding模型?不能提供針對某個用戶的準確信息在Retrieval前：在Retrieval前：-TextToSQL（或其它查詢語法）-改寫問題（通過LLM）-對查詢結果進行re-ranking--提供更多的上下文信息和一定的推理??更進一步，在Retrieval前進行查詢路由：-在不同KB之間選擇-在不同retrieve方式之間選擇-使用LLM判斷在RAG框架中為每個數(shù)據(jù)庫、每種數(shù)據(jù)源開發(fā)連接器/reader、獲取其描述信息、獲得prompt模版、進行NL2SQL/QL、查詢數(shù)據(jù)、輸入給LLMn每個數(shù)據(jù)源/存儲都需流程打通、產(chǎn)品打通非結構化數(shù)據(jù)非結構化數(shù)據(jù)資產(chǎn)地圖與資產(chǎn)治理傳統(tǒng)機器學習大模型預訓練/微調支持設置TTL與TTV數(shù)據(jù)處理流程A流程數(shù)據(jù)加工spark/Flink數(shù)據(jù)分析OLAP數(shù)據(jù)探查數(shù)據(jù)加工spark/Flink數(shù)據(jù)分析OLAP數(shù)據(jù)探查Notebook模型訓練LLM模型部署數(shù)據(jù)集成集成引擎根據(jù)血緣推薦TTL、TTV管理前降本40%管理后!降本40%管理后特征分析特征分析pytorchpandas模型訓練pytorchpandas模型訓練模型部署提取特征sparkModel數(shù)據(jù)落倉模型部署提取特征sparkModel數(shù)據(jù)落倉TaloslcebergFileset模型校驗TaloslcebergFileset模型校驗數(shù)據(jù)分析數(shù)據(jù)分析python微調/預訓練Dockersparkpython用戶問答語料數(shù)據(jù)選取微調數(shù)據(jù)微調/預訓練Dockersparkpython用戶問答語料數(shù)據(jù)選取微調數(shù)據(jù)FilesetlcebergHive/lceberglcebergpythonpython部署/在線推理校驗/離線推理Docker測試數(shù)據(jù)部署/在線推理校驗/離線推理Docker測試數(shù)據(jù)Fileset實踐2:某互聯(lián)網(wǎng)社交平臺業(yè)務側耦合度高：元數(shù)據(jù)使用方調用異構數(shù)據(jù)源方式多種多樣數(shù)據(jù)治理能力有限：無法提供統(tǒng)一的審計、權限管理、TTL能力半結構化/非結構化數(shù)據(jù)源缺乏管理跨源數(shù)據(jù)Schema維護成本高OneMeta：集成Gravitino提供定制化接口：dropPartitiosByFilter/loadFileDetail/loadFiles......提供定制化catalog實現(xiàn)：BiliIcebergCatalog/BiliKafkaCatalog/BiliDatabusCatalog......降低代碼的侵入性，便于同步社區(qū)最新代碼解耦業(yè)務方復雜依賴，降低元數(shù)據(jù)使用成本解決由于引擎間差異、數(shù)據(jù)源差異造成的元數(shù)據(jù)不一致問題解決由于HiveMetaStore造成的性能瓶頸Fileset文件治理主要流程：1.數(shù)據(jù)治理平臺制定治理策略TTL