IBM馬達(dá)：Kubernetes 中基于策略的資源分配

Kuberentes#269:BatchJobAdmissionandflexibleresourceallocation(基于策略的資源共享)

@k82cn,馬達(dá).IBM

UserCases:RunmultipletypeofworkloadsinDataCenter

UserCases:RunmultipletypeofworkloadsinDataCenter

●Longrunningservice(apparea)&bigdata(bigdataarea)canshareresources:

oSupportdefineresourceusageofeacharea,e.g.50%resourcestoapparea,50%tobigdataarea.

oSupportborrow/lendingprotocol:iftheresourcesisidleinonearea,itcanbelendoutandbepreemptedbackwhenlaunchmoretasks

●Runmultipleclusterinbigdataarea,e.g.Hadoop&Spark:

oSupportdefineresourcesusageofeachclusterwithinbigdataarea

oSupportsharingresourcesbetweenthoseclusters

Kubernetesfeatures&gaps

●AdmissionController+Quota:staticplan/allocation

●MultipleScheduler:NoQoS

●Auto-Scaling&Node-levelQoS:nocluster-levelQoS

●Re-scheduling&Preemption/Eviction

●Workload-specificcontroller&ThridPartyResources

ArchitectOverview

1.BatchJobControllerwillcalculatedeservedresourcebasedonScheduler’s

configuration,arbitrator’spolicy,e.g.DRF,andnamespace’srequest

2.BatchJobControllerwillevictPodsofoverusednamespace

3.SchedulerdispatchtasksbasedonQuota(#ofdeserved),PodsandNodes

attributes

4.BatchJobAdmissionmakesure“namespacewillnotbeoverused”

ArchitectOverview

Quota&BatcJobQuota(??)

Hard(Quota)

Deserved(BatchJob)

Reserved

1.Thereservedsectiondefinestheresourcesthat

reservedforthenamespace.Thetotalreserved

resourcescannotexceedclusterresources

2.ThedeservedisupdatedbyBatchJobController

insteadofuser,itdefinesthetotalresourcesthat

allocatedtoanamespace;thedeservedresourcescannotexceedQuota.hardandcannotlessthan

Reserved(exceptionexcluded,e.g.Nodefailed)

1

scheduler

BatchJobQuotaAdmission

2

Updatedeserved

1.5

5

ArchitectOverview

4

3

Calculatedeservedresourcesofeachnamespace(??)basedon:

1.TotalresourcesfromNodes

Pods

Nodes

SchedulerselectsaNodeforPodbasedoncurrentalgorithm,e.gPodAffinity.

2.ResourcerequirementfrompendingPods3.ResourcerequirementfromrunningPods

4.ReservedresourceofQuota

BatchJobController

BatchJob

Eviction

UpdateEvictionforoverusednamespaceaftergraceperiod

kubelet

kubelet

BatchJobisawrapperofAPIObject,

e.g.RC/RS;it’llcreatePodswhichis

ownedbyit.

Beforelaunching,kubeletcheckscheduler’srequest:reject

overusedscheduler’srequest.ifnodeisexhaust,evictmost

overusedscheduler’srequest

BatchJobController

Quota-1CPU:1Mem:1

Quota-2CPU:0Mem:0

ns-1

ns-2

ThecontrollerwillselectaPodtoevict.After

eviction,assign

resourcestounderused

ns-2

namespace

Pre-emption&ReclaiminBatchJobController

BatchJobController

Quota-1CPU:2Mem:2

ns-1

Thecontrollerre-calculateresourceallocationforeachnamespace,shrinktheoverused

namespace

BatchJobController

Quota1CPU:1Mem:1

Quota-2CPU:1Mem:1

ns-1

Evictionpolicyispluggale

e.g.laststartfirstevict,

ResourceRequirementsofBatchJob

?Case1:Theresourcerequirementis

sumofeachexecutor(thenumberoftotaltasksisunknown)

?Case2:Theresourcerequirementsissumofnon-terminaltasks(pending+running),butthetotalnumbertaskisunknown

?Case3:Theresourcerequirementissumoftotaltasks

Backlog

?BatchJobcontrollerpolicy&configuration(e.g.DRF??,weightofnamespace)

?HAofBatchJobController

?Hierarchicalnamespace

?BatchJobcontrollerpolicyforlimits(ResourceQoS)

?Hostnameindeserved??

Reference

?DesignDoc:

/a//document/d/1

-

H2h