小米融合云架構(gòu)

1、小米融合云架構(gòu)技術(shù)創(chuàng)新，變革未來融合云介紹業(yè)務(wù)快速發(fā)展管理成本接入復(fù)雜隔離需求生態(tài)鏈需求公網(wǎng)訪問融合云動(dòng)機(jī)小米內(nèi)部業(yè)務(wù)生態(tài)鏈業(yè)務(wù)融合云介紹服務(wù)化統(tǒng)身份認(rèn)證權(quán)限管理多租戶Web console基于開源功能擴(kuò)展基于HBase做二級(jí)索引系統(tǒng)擴(kuò)展基于HBase和HDFS做對(duì)象存儲(chǔ)公網(wǎng)調(diào)用支持Http請(qǐng)求/無狀態(tài)融合云設(shè)計(jì)融合云介紹融合云架構(gòu)Web ConsoleSDK團(tuán)隊(duì)管理結(jié)構(gòu) 化存 儲(chǔ)對(duì)象 存儲(chǔ)消息 隊(duì)列流式 消息離線深度分析學(xué)習(xí)自動(dòng) 部署Docker鏡像HBase/HDFS/Kudu/TensorFlow/Spark/Docker團(tuán)隊(duì)管理Organization(Org)資源隸屬Team用戶身份

2、隸屬于OrgAdmin/MemberOrg admin Team認(rèn)證CAS簽名認(rèn)證生成秘鑰秘鑰輪換權(quán)限管理團(tuán)隊(duì)管理(Cloud-Manager)Cloud-ManagerServiceWeb consoleSDKCAS公司賬號(hào)user: teamList resource: org簽名簽名認(rèn)證Org(團(tuán)隊(duì)): notebookAdmin TeamDev_TeamTablesBucket/ ObjectsqueuesReport TeamAdminAdminR團(tuán)隊(duì)管理示例cc融合云存儲(chǔ)/計(jì)算數(shù)據(jù)流結(jié)構(gòu)化存儲(chǔ)(SDS)離線分析消息隊(duì)列(EMQ)對(duì)象存儲(chǔ)(FDS)消息隊(duì)列(EMQ)Snapshot

3、Export流式消息服務(wù)SDS StreamMR/ Spark結(jié)構(gòu)化存儲(chǔ)(SDS)分布式數(shù)據(jù)庫基于HBase數(shù)據(jù)模型擴(kuò)展支持EntityGroup/Primary Key支持?jǐn)?shù)據(jù)類型多語言SDK支持ACL擴(kuò)展功能擴(kuò)展局部二級(jí)索引全局二級(jí)索引Stream實(shí)時(shí)恢復(fù)Structure Data Store(SDS)結(jié)構(gòu)化存儲(chǔ)(SDS)SDS架構(gòu)ACLQuotaHBaseSDS CoreSDSServiceObject MappingFrontEndMulti-platform SDKThrift ProcessorHttpServer (TTransport)Thrift ProcessorHttp

4、Client (TTransport)flush/HttpRequest read/HttpResponseSDS Service認(rèn)證與Cloud-Manager交互元信息管理UserTable = TableId = HBase TableSDKJava/php/python/go/c+/node.jsSDKService結(jié)構(gòu)化存儲(chǔ)(SDS)SDS CoreSDS Request HBase RequestObject MappingJava Object SDS RequestACL表級(jí)別權(quán)限控制EntityGroup間權(quán)限隔離QuotaTokenBucket算法Soft Limit引入集

5、群過載quota集群過載前允許超發(fā)SDS架構(gòu)ACLQuotaHBaseSDS CoreSDSServiceObject MappingFrontEndMulti-platform SDK結(jié)構(gòu)化存儲(chǔ)(SDS)實(shí)現(xiàn)Entity Group(EG)功能業(yè)務(wù)特性: 以用戶為中心定義Entity Group KeysEG在同個(gè)regionEG內(nèi)跨行寫原子性EG內(nèi)局部二級(jí)索引強(qiáng)致Schema管理EntityGroup/Primary Index/Secondary Index/Column數(shù)據(jù)類型Bool/Int8/Int16/Float/Double/String/Binary, Set使用Ordere

6、dBinary編碼確保順序條件查詢SDS Core主要功能結(jié)構(gòu)化存儲(chǔ)(SDS)局部二級(jí)索引數(shù)據(jù)Family/索引FamilyHBase coprocessor中更新索引讀取主記錄構(gòu)造索引rowkey更新主記錄/刪除舊索引/寫入新索引支持Eager/Lazy/Immutable IndexLazy invalid index定期清理Eager/Immutable Index支持屬性投影全局二級(jí)索引使用單獨(dú)的HBase table存儲(chǔ)索引使用Themis提供HBase跨表更新原子性/xiaomi/themis全局單調(diào)遞增時(shí)間戳: /xiaomi/chronos強(qiáng)致性二級(jí)索引EGRow01main_

7、row01Index_rowData FamilyIndex Family結(jié)構(gòu)化存儲(chǔ)(SDS)使用Java annotation和反射實(shí)現(xiàn)toRecord: Object = MapfromRecord: Map = ObjectObject Mapping對(duì)象映射SDS Object Mapping例Table Schema定義(SQL)SDS Object Mapping例Table Schema定義SDS Object Mapping例查詢SDS Object Mapping例查詢結(jié)構(gòu)化存儲(chǔ)(SDS)場(chǎng)景OLAP增量數(shù)據(jù)處理原理Stream Family存儲(chǔ)image使用hbase co

8、processor build images 推送image到流式消息服務(wù)實(shí)時(shí)恢復(fù)StreamF1F2MRSnapshoterDataImagesnapshotsRecoverySnapshot just before Ts流式消息服務(wù)TsWeb console例結(jié)構(gòu)化存儲(chǔ)(SDS)SDS rest server 24 Intel(R) Xeon(R) E5620 2.4 GHz, 64G, 萬兆網(wǎng)HBase region server Cpu/內(nèi)存/網(wǎng)絡(luò)同上，8 * 800G ssd，5臺(tái)YCSB client性能測(cè)試Put Througput/Latency記錄大小ThrougputAvg

9、(ms)P95(ms)P99(ms)0.1k337380.52790.5k300470.466111k256520.46610Get Througput/Latency記錄大小ThrougputAvg(ms)P95(ms)P99(ms)0.1k302130.26470.5k274570.28391k269320.29312對(duì)象存儲(chǔ)(FDS)KB到TB級(jí)別的文件基于HBase & HDFSBucket & ObjectRestful API多語言SDK權(quán)限管理事件通知Quota管理30+業(yè)務(wù)使用File Data Store(FDS)FrontEndMeta ManagerBlob StoreH

10、BaseHDFSCDN對(duì)象存儲(chǔ)(FDS)Bucket TableBucket基本信息/ACL/QuotaObject TableObject基本信息/ACL/meta/通知信息Object name = blobUriBlobInfo TableBlobUri = FileId抽象Blob write/read/get接口，隱藏具體實(shí)現(xiàn)FileInfo TableFileId = Path, basicInfoMultiPart Upload TableuploadId = partInfo元數(shù)據(jù)管理對(duì)象存儲(chǔ)(FDS)BlobManager/FileManagerBlob存儲(chǔ)magic/dat

11、a/md5offset管理打開文件MaxOpenNum/MaxFileSizeIdle Files Busy FilesBlobClient創(chuàng)建Blob: BlobWriterWrapped OutputStream關(guān)閉Blob獲取Blob: BlobReaderWrapped InputStreamBlobStoreBlobManager BlobInfo(BlobId-FileId)FileManager FileInfo(FileId-FilePath)Busy FilesIdle FilesHDFS FilesBlobClientcreateBlob/ getBlob closeBlo

12、b對(duì)象存儲(chǔ)(FDS)創(chuàng)建對(duì)象BlobClient: createBlob將object信息寫入hbase object table讀取對(duì)象從hbase object table獲取blobUriBlobClient: getBlob對(duì)象列表t=苗hbase object table刪除對(duì)象刪除hbase object table中對(duì)應(yīng)記錄對(duì)象橾作對(duì)象存儲(chǔ)(FDS)Object Cleaner 民據(jù)object table刪除blob tableFile Cleaner & Compactor基于BlobInfo Table和FileInfo Table民據(jù)BlobInfo Table統(tǒng)計(jì)Fil

13、e使用率歸并低使用率文件Archive無Blob引用文件刪除過期Archive文件Cleaner & CompactorobjNameblobUriblobUriFileIdFileIdFileInfoFDS性能消息隊(duì)列(EMQ)應(yīng)用間異步通信支持P2P, Pub/Sub兩種模式基于HBase提供分布式能力At least once語義多讀者消息優(yōu)先級(jí)死信隊(duì)列Short/Long PoolingElastic Message Queue(EMQ)EMQsend messagereceive messageack message消息隊(duì)列(EMQ)QueueHBase TablePartition

14、sPartitionIdPartition分布致性hashEMQ架構(gòu)Partition_0Partition_nRegion_0Region_nQueueHBase TableHBasePartition0Partition1Partition2Partition3Partition4Partition5EMQ Server消息隊(duì)列(EMQ)Message MessageIdmessageId = hbaseRow = Timestamp = currentTime + delayTime Message BodyMessage狀態(tài)消息狀態(tài)NewInvisibleReadDeadDeleteRECVTIMEOUTRECVACKRECV LIMITCHANGE VISIBILITYSEND消息隊(duì)列(EMQ)內(nèi)存信息 Available set/Received set/Delete set 多讀者Partition內(nèi)存狀態(tài)Available Set(Sort by MessageId)Received Set(Sort byT