高性能+Kubernetes+元數(shù)據(jù)存儲(chǔ)+KubeBrain+的設(shè)計(jì)思路和落地效果-字節(jié)跳動(dòng)+許辰

InfoQ1InfoQ1ArchSummit全球架構(gòu)師峰會(huì)KubeBrain字節(jié)跳動(dòng)高性能K8s元信息存儲(chǔ)許辰字節(jié)跳動(dòng)資深研發(fā)工程師

許辰字節(jié)跳動(dòng)基礎(chǔ)架構(gòu)工程師本科和碩士畢業(yè)于北京大學(xué)計(jì)算機(jī)系負(fù)責(zé)大規(guī)模Kubernetes系統(tǒng)的構(gòu)建和優(yōu)化KubeBrain/KubeGateway/KubeZoo等多個(gè)項(xiàng)目的發(fā)起人ArchSu^nitI全球架構(gòu)師峰會(huì)仁InfoGArchSu^nitArchSu^nit全球架構(gòu)師峰會(huì)ArchSu^nitArchSu^nit全球架構(gòu)師峰會(huì)大綱-背景介紹-設(shè)計(jì)思路-性能優(yōu)化-落地效果?未來演進(jìn)

北m冃景?Kubernetes規(guī)模增大10倍以上＞公司業(yè)務(wù)快速發(fā)展＞存儲(chǔ)、大數(shù)據(jù)、機(jī)器學(xué)習(xí)等場景云原生化?新場景對(duì)Kubernetes性能要求更高. ByteDancelu字節(jié)別阮/]Ci＞離線場景，. ByteDancelu字節(jié)別阮/]CiInfoQ如何擴(kuò)展Kubernetes集群多個(gè)集群橫向擴(kuò)展＞降低運(yùn)維管理成本＞減少資源碎片＞提高資源利用率ArchSKubernetes的架構(gòu)特點(diǎn)apiserverapiserver所有組件通過apisever交互etcd存在性能問題apiserverapiserver所有組件通過apisever交互etcd存在性能問題cdetththArchSu^iit全球架構(gòu)師峰會(huì)

元信息存儲(chǔ)etcdCloudNativeLandscape2018-11-27T01:14:16Z8707947Scf.ioDatabaseStreaming&MessagingApplicationDefinition&Image

BuildContinuousIntegration&DeliveryLUo>Ed$>eapueuo一一c%a&<l^onojnifi*Scheduling&OrchestrationCoordination&Service

DiscoveryRemoteProcedureCallServiceProxyAPIGatewayServiceMesh一￡￡額壽芝2UO-.2±:SMP』0CNCfmoXmgCloud-NativeStorage0>E-cn(x(loarttM?tcAutomation&Configuration5Mcos>o」dpnooPublic回回cf.ioArchSummit全球架構(gòu)師峰會(huì)ContainerRuntimeContainerRegistryThislandscapeisintendedasamapthroughthepreviouslyunchartedterrainofcloudnativetechnologies.Therearemanyroutestodeployingacloudnativeapplication,withCNCFProjectsrepresentingaparticularlywell-traveledpathAllCloudNative

Infrastructure.?IcmoAOuttyerCloud-NaliveNetworkKeyManagementSecurity&Compliance■(SSLUr"ICLOUDNATIVEMLandscape「rCLOUDNATIVELJKubernetesCertifiedServiceProviderfluentdetcdQtatroniHIFTKubernetesTrainingPartnerInfoQetcd存在的問題限流能力弱，雪崩情況下自愈能力差ExpensiveRead存在大雖內(nèi)存拷貝容易00M分頁讀導(dǎo)致延退增高選主walsnap單單raftgroup,leader有單點(diǎn)瓶頸BoltDB存儲(chǔ)引擎寫寫互斥，限制性能高負(fù)載下讀寫延退增高線上無法defrag減少碎片ArchSummitiInfoQArchSummitArchSummit全球架構(gòu)師峰會(huì)ArchSummitArchSummit全球架構(gòu)師峰會(huì)大腦諧音科比大腦諧音科比KobeBryantInfoQ自研元信息存儲(chǔ)如何解決存儲(chǔ)瓶頸?ArchSu^nitArchSu^nit全球架構(gòu)師峰會(huì)大綱-背景介紹-設(shè)計(jì)思路-性能優(yōu)化-落地效果?未來演進(jìn)

K8s元信息存儲(chǔ)的需求(1)-單Key讀，提供線性一致性-Range掃描讀，支持快照讀，支持分頁?K8s樂觀鎖resourceversion-單KeyCAS/Watch-Kuberneteslist-watch的底層依賴ArchSu^nit構(gòu)師峰InfoQInfoQArchSu^nit全球架構(gòu)師峰會(huì)InfoQArchSu^nit全球架構(gòu)師峰會(huì)InfoQArchSu^nit全球架構(gòu)師峰會(huì)InfoQArchSu^nit全球架構(gòu)師峰會(huì)ArchSummitArchSummit? IllfoCt^全球架構(gòu)師峰會(huì).K8s.元信息存儲(chǔ)的需求(2)

K8s.元信息存儲(chǔ)的需求(3)所以etcd為目前K8s唯一支持的存儲(chǔ)ArchSu^nit全球架構(gòu)師峰會(huì)ArchSu^nit全球架構(gòu)師峰會(huì)ArchSu^nit全球架構(gòu)師峰會(huì)ArchSu^nit全球架構(gòu)師峰會(huì)KubeBrain架構(gòu)InfoGInfoQInfoQInfoQInfoQKubeBrain架構(gòu)apiserverTSOKube

BrainKube

BrainKube

BrainArchSummitiWatchCacheLeaderFollowerFollower事務(wù)寫withCAS\7快照讀ByteKVmulti-raft/強(qiáng)一致性KV存儲(chǔ)-主負(fù)責(zé)寫和事件分發(fā),從負(fù)責(zé)讀-CAS事務(wù)寫-快照讀實(shí)現(xiàn)架構(gòu)圖客戶端SDK接入層接入層邏輯層寫■Watch讀

一致性協(xié)議選主TSO磁盤存儲(chǔ)層分布式KVStoreArchSu^nit師峰InfoGArchSu^nitArchSu^nit全球架構(gòu)師峰會(huì)ArchSu^nitArchSu^nit全球架構(gòu)師峰會(huì)客戶端接入層邏輯層存儲(chǔ)層寫■WatchSDK接入層讀

一致性協(xié)議選主TSO分布式KVStoreInfoGInfoQArchSu^nitInfoQArchSu^nit全球架構(gòu)師ArchSu^nit全球架構(gòu)師峰會(huì)ArchSu^nit全球架構(gòu)師峰會(huì)存儲(chǔ)層一分布式KVStoreIByteKV?MultiRaftGoup-全局有序Range分區(qū)?強(qiáng)一致性?支持多key事務(wù)?支持CAS?支持快照讀?高性能sqlproxysqlproxy.IsqlproxykvclientKVMasterserversR/Wleaderleaderleaderfollower?MultiRaftGoup-全局有序Range分區(qū)?強(qiáng)一致性?支持多key事務(wù)?支持CAS?支持快照讀?高性能sqlproxysqlproxy.IsqlproxykvclientKVMasterserversR/WleaderleaderleaderfollowerfollowerfollowerRaftfollowerVfollowerfollowergetpartitionlocations...getglobaltimestampgetpartitionlocationsgetglobaltimestamploadbalanceR/WRaftkvproxykvproxykvproxykvclientR/WArchSummitiInfoQ1

ArchSummiti存儲(chǔ)層-數(shù)據(jù)格式etcdetcd存儲(chǔ)格式莊Keyrevisionnkey1valuelrevisionn+1key2value2revisionn+2key1value3revisionn+3key1value4內(nèi)存Btree索引維護(hù)key和莊Keyrevisionnkey1valuelrevisionn+1key2value2revisionn+2key1value3revisionn+3key1value4內(nèi)存Btree索引維護(hù)key和revision的映射關(guān)系revisionn+4key2value.底層存儲(chǔ)引擎全局有序，有寫熱點(diǎn)那問題BoltDB的存儲(chǔ)引擎為B+tree,key單調(diào)遞增減少樹的重平衡以revision為基礎(chǔ)的mvcc實(shí)現(xiàn)并發(fā)控制,達(dá)到事務(wù)隔離的效果，代替鎖取得更好的讀寫性能revision單調(diào)遞増，對(duì)watch友好

存儲(chǔ)層-數(shù)據(jù)格式KubeBrainKubeBrain存儲(chǔ)格式；三Key索弓key1revision2數(shù)據(jù)key1key1revision2revisionlvaluelvalue2magicrawkeysplitkeyrevisionkey2revisionskey2revisionsvalue3固定值用戶key固定值指導(dǎo)range分片版本號(hào)；三Key索弓key1revision2數(shù)據(jù)key1key1revision2revisionlvaluelvalue2magicrawkeysplitkeyrevisionkey2revisionskey2revisionsvalue3固定值用戶key固定值指導(dǎo)range分片版本號(hào)對(duì)于索引值為01.底層ByteKV無論是點(diǎn)查還是范圍查詢，均非常友好避免了熱點(diǎn)key的產(chǎn)生對(duì)用戶key迸行編碼，保證編碼前后順序不a.InfoQ1客戶端接入層磁盤存儲(chǔ)層ArchSu^iit全球架構(gòu)師峰會(huì)邏輯層寫■WatchSDK接入層讀

一致性協(xié)議分布式KVStore選主TSOInfoGArchSu^nitArchSu^nit全球架構(gòu)師峰會(huì)ArchSu^nitArchSu^nit全球架構(gòu)師峰會(huì)邏輯層一寫開始事務(wù)數(shù)據(jù)操作事務(wù)提交txn.Startputifkeynotexistsputifvalueequalsdeleteifvalueequals索引CAS操作分布式存儲(chǔ)CASCompact刪除歷史版本InfoGInfoQArchSummitInfoQArchSummit全球架構(gòu)師InfoQArchSummitInfoQArchSummit全球架構(gòu)師ArchSu^nit全球架構(gòu)師峰會(huì)ArchSu^nit全球架構(gòu)師峰會(huì)邏輯層一Watch(1)可靠性-不重復(fù)、不丟失順序性-保證最終狀態(tài)的一致性實(shí)時(shí)性-高性能—定有一個(gè)單點(diǎn)對(duì)消息進(jìn)行排序采用主從架構(gòu)MathiasVerraes ( ―IFollow丿X/@mathiasverraes \ 丿Thereareonlytwohardproblemsindistributedsystems:2.Exactly-oncedelivery1.Guaranteedorderofmessages2.Exactly-oncedelivery11:40AM-14Aug20157,233Retweets5,793Likes。練包g.,Q71口7.2KD5.8KInfoQ

邏輯層一Watch(2)Leaderapiserverswrite&watchwrite&watchFollowerFollowerforwardforward僅主節(jié)點(diǎn)負(fù)責(zé)寫入和事件生成Leaderapiserverswrite&watchwrite&watchFollowerFollowerforwardforward從節(jié)點(diǎn)只讀InfoQArchSu^nitInfoQArchSu^nit全球架構(gòu)師峰InfoQArchSu^nitInfoQArchSu^nit全球架構(gòu)師峰邏輯層一Watch(3)-寫入滑動(dòng)窗口記錄并發(fā)寫操作的結(jié)果?消費(fèi)滑動(dòng)窗口中的數(shù)據(jù)實(shí)現(xiàn)有序的Event推送?當(dāng)前消費(fèi)的最大位置為Brain層的CommittedIndex，與快照讀有關(guān)ArchSu^nit全球架構(gòu)師峰會(huì)ArchSu^nit全球架構(gòu)師峰會(huì)ArchSu^nit全球架構(gòu)師峰會(huì)ArchSu^nit全球架構(gòu)師峰會(huì)邏輯層一單Key讀revision遞增方向A遍歷方向獲取到的第一個(gè)keyINTERNALKEYVALUErevisionkeyrevisionobjectkeyivaluei??????objectkeyjvaluej分布式存儲(chǔ)遍歷范圍InfoGInfoQArchSu^nit全球架構(gòu)師峰會(huì)InfoQArchSu^nit全球架構(gòu)師峰會(huì)InfoQArchSu^nit全球架構(gòu)師峰會(huì)InfoQArchSu^nit全球架構(gòu)師峰會(huì)邏輯層一Range讀分布式存儲(chǔ)InfoQInfoQInfoQInfoQ邏輯層一Range讀一致性KubeBrainClientFollowerLeader,Range1,Get<SlidingIndex Return.SlidingIndex根據(jù)SlidingIndex進(jìn)行過濾?SnapshotScan,Return■RangeResponse*ByteKV-Range從Leader獲取滑動(dòng)窗

口當(dāng)前CommittedIndex序?根據(jù)當(dāng)前序號(hào)進(jìn)行快照讀-Range后Client通過Watch從leaderRingBuffer中獲取增量事件,達(dá)到ArchSummit全球架構(gòu)師峰ArchSu^nitArchSu^nit全球架構(gòu)師峰會(huì)ArchSu^nitArchSu^nit全球架構(gòu)師峰會(huì).4.邏輯層一選主不基于raft,而是基于lease>cas使用client-goleader-election包在同一時(shí)刻只會(huì)有一個(gè)eader,不會(huì)產(chǎn)生腦裂InfoG邏輯層一tso客戶端邏輯層磁盤存儲(chǔ)層ArchSu^iit全球架構(gòu)師峰會(huì)接入層寫■WatchSDK接入層讀

一致性協(xié)議分布式KVStore選主TSOInfoG接入層?Stream代替Unary避免OOM?多纟隹度限流，防止雪崩 ?降低接入成本ArchSu^iit全球架構(gòu)師峰會(huì)InfoQ1接入層邏輯層磁盤存儲(chǔ)層ArchSu^iit全球架構(gòu)師峰會(huì)客戶端WatchSDK接入層讀

一致性協(xié)議分布式KVStore選主TSOInfoGInfoQArchSu^nit全球架構(gòu)師峰會(huì)InfoQArchSu^nit全球架構(gòu)師峰會(huì)客戶端?主從檢測，寫和watch發(fā)往主節(jié)點(diǎn)，讀發(fā)往從節(jié)點(diǎn) ?錯(cuò)誤重試?讀負(fù)載均衡，roundrobin?健康監(jiān)測?増強(qiáng)可觀測性ArchSu^iit全球架構(gòu)師峰會(huì)InfoQ1K8s.元信息存儲(chǔ)的需求功能讀寫API接口單key讀取range快照讀單keycreateifnotexits單keyupdatecas單keydeletecasetcdWatch事件保序、不重、不丟KubeBrain數(shù)據(jù)庫KubeBrain功能讀寫API接口單key讀取range快照讀單keycreateifnotexits單keyupdatecas單keydeletecasetcdWatch事件保序、不重、不丟KubeBrain數(shù)據(jù)庫KubeBrainetcd高可用—致性QQQ性能性能性能功能大綱-背景介紹-設(shè)計(jì)思路?性能優(yōu)化-落地效果?未來演進(jìn)ArchSummit?全球架構(gòu)師峰會(huì)InfoQInfoQInfoQInfoQ性能優(yōu)化寫優(yōu)化-寫優(yōu)化-update方法中，PreKV字段apiserver不會(huì)使用,PrevKv*/ArchSu^iit全球架構(gòu)師峰會(huì)InfoQ1InfoQArchSu^nit全球架構(gòu)師峰會(huì)InfoQArchSu^nit全球架構(gòu)師峰會(huì)InfoQArchSu^nit全球架構(gòu)師峰會(huì)InfoQArchSu^nit全球架構(gòu)師峰會(huì)壓測數(shù)據(jù)大綱大綱大綱大綱-背景介紹-設(shè)計(jì)思路-性能優(yōu)化-落地效果?未來演進(jìn)ArchSummit?全球架構(gòu)師峰會(huì)InfoQArchSummitInfoQArchSummit全球架構(gòu)師峰會(huì)InfoQArchSummitInfoQArchSummit全球架構(gòu)師峰會(huì)落地效果?壓測環(huán)境-配合apiserver優(yōu)化手段，規(guī)模達(dá)5w節(jié)點(diǎn)200wPod?生產(chǎn)環(huán)境2W節(jié)點(diǎn)100wPod超大集群，有效降低資源碎片率Totalobjectcount8000007000006000005000004000002022*042403:35:00400000brair>lf(resource=pods): 708602一brain-lf(resource=nodes}: 1310730000020000010000019:0020:0021:0022:0023:0000:0001:0002:0003:0004:0005:0006:0007:0008:0009:0010:0011:0012:0ArchSu^nit全球架構(gòu)師峰會(huì)ArchSu^nit全球架構(gòu)師峰會(huì)ArchSu^nit全球架構(gòu)師峰會(huì)ArchSu^nit全球架構(gòu)師峰會(huì)2022-04-2403:01:0020:0022: