K8S高可用集群部署手冊(cè)

K8S高可用集群部署手冊(cè)系統(tǒng)環(huán)境準(zhǔn)備1.1K8S多主高可用集群架構(gòu)圖1.2資源準(zhǔn)備清單K8S集群角色I(xiàn)P地址主機(jī)名資源規(guī)格操作系統(tǒng)安裝組件masterK8S-master014C4G/100GCentos7.9x64kube-apiserver

kube-schedule

controller-manager

kube-proxy，kube-dnskeepalived+haproxy

kubelet，container

calico，etcd，

metrics-serverdashboardK8S-master024C4G/100GCentos7.9x64K8S-master034C4G/100GCentos7.9x640VIPmaster高可用的浮動(dòng)IP，kubeadm初始化masternodeK8S-node014C4G/100GCentos7.9x64kube-proxy

kubelet，kube-dns

calico，container

ingress-controllerK8S-node024C4G/100GCentos7.9x64K8S-node034C4G/100GCentos7.9x64所有節(jié)點(diǎn)安裝前初始化配置所有節(jié)點(diǎn)(master+node)都需要做安裝前的初始化配置#1.關(guān)閉防火墻systemctlstopfirewalldsystemctldisablefirewalld#2.關(guān)閉selinuxsed-i's/SELINUX=enforcing/SELINUX=disabled/'/etc/selinux/config#永久setenforce0#臨時(shí)#3.設(shè)置宿主機(jī)名稱vim/etc/hosts0k8s.cluster.localK8S-master01K8S-master02K8S-master03K8S-node01K8S-node02K8S-node03#4.將橋接的IPv4流量傳遞到iptables的鏈，修改內(nèi)核參數(shù)：vim/etc/sysctl.d/99-sysctl.confkernel.sysrq=0紅框部分是需要增加及修改的內(nèi)容net.bridge.bridge-nf-call-ip6tables=紅框部分是需要增加及修改的內(nèi)容net.bridge.bridge-nf-call-iptables=1user.max_user_namespaces=28633net.ipv4.ip_forward=1net.ipv4.conf.all.send_redirects=0net.ipv4.conf.default.send_redirects=0net.ipv4.conf.all.accept_source_route=0net.ipv4.conf.default.accept_source_route=0net.ipv4.conf.all.accept_redirects=0net.ipv4.conf.default.accept_redirects=0net.ipv4.conf.all.secure_redirects=0net.ipv4.conf.default.secure_redirects=0net.ipv4.icmp_echo_ignore_broadcasts=1net.ipv4.icmp_ignore_bogus_error_responses=1net.ipv4.conf.all.rp_filter=1net.ipv4.conf.default.rp_filter=1net.ipv4.tcp_syncookies=1kernel.dmesg_restrict=1net.ipv6.conf.all.accept_redirects=0net.ipv6.conf.default.accept_redirects=0#5.使修改的內(nèi)核參數(shù)生效并加載br_netfilter模塊sysctl-p/etc/sysctl.d/99-sysctl.confmodprobebr_netfilterlsmod|grepbr_netfilter#6.配置時(shí)間同步y(tǒng)uminstall-yntpdatentpdate#7.配置免密登錄ssh-keygen-trsassh-copy-id-i/root/.ssh/id_rsa.pubroot@ssh-copy-id-i/root/.ssh/id_rsa.pubroot@ssh-copy-id-i/root/.ssh/id_rsa.pubroot@ssh-copy-id-i/root/.ssh/id_rsa.pubroot@ssh-copy-id-i/root/.ssh/id_rsa.pubroot@ssh-copy-id-i/root/.ssh/id_rsa.pubroot@所有節(jié)點(diǎn)升級(jí)系統(tǒng)內(nèi)核版本CentOS系統(tǒng)默認(rèn)的內(nèi)核版本是3.10，對(duì)于k8s-v1.24及以后版本來說，在生產(chǎn)環(huán)境上能部署，能運(yùn)行，但是在使用k8s期間會(huì)出現(xiàn)很多問題，即不穩(wěn)定因數(shù)。為了能在生產(chǎn)環(huán)境中穩(wěn)定運(yùn)行，對(duì)于1.24及以上版本而言，對(duì)于CentOS系統(tǒng)而言，需要升級(jí)系統(tǒng)內(nèi)核。而elrepo內(nèi)核目前已經(jīng)更新到了5版本。所以本次就用它來升級(jí)系統(tǒng)內(nèi)核在/etc/yum.repos.d/下創(chuàng)建一個(gè)repo文件：[elrepo]name=elrepobaseurl=/elrepo/archive/kernel/el7/x86_64gpgcheck=0enabled=1清空和刷新yum源元數(shù)據(jù)緩存yumcleanallyummakecache查鏡像倉庫中內(nèi)核包yumlist--showduplicatekernel*安裝內(nèi)核yuminstall-ykernel-lt-5.4.247yuminstall-ykernel-lt-devel-5.4.247kernel-ml：ml是mainlinestable的縮寫，elrepo-kernel中羅列出來的是最新的穩(wěn)定主線版本。kernel-lt：lt是longtermsupport的縮寫，elrepo-kernel中羅列出來的長期穩(wěn)定支持版本。查看內(nèi)核編碼awk-F\''$1=="menuentry"{printi++":"$2}'/etc/grub2.cfg設(shè)置啟動(dòng)的內(nèi)核grub2-set-default0重啟操作系統(tǒng)reboot查看生效版本內(nèi)核uname-a移除舊版本內(nèi)核軟件包查看已經(jīng)安裝的內(nèi)核版本：rpm-qa|grepkernel刪除指定包：yumremove-y包名稱所有節(jié)點(diǎn)安裝并配置ipvsyum-yinstallipsetipvsadm------安裝ipvsadm軟件vim/etc/sysconfig/modules/ipvs.modules-----修改ipvs配置文件#!/bin/bashmodprobe--ip_vsmodprobe--ip_vs_lcmodprobe--ip_vs_lblcmodprobe--ip_vs_lblcrmodprobe--ip_vs_rrmodprobe--ip_vs_wrrmodprobe--ip_vs_shmodprobe--ip_vs_dhmodprobe--ip_vs_nqmodprobe--ip_vs_sedmodprobe--ip_vs_ftpmodprobe--nf_conntrackmodprobe--ip_tablesmodprobe--ip_setmodprobe--ipt_setmodprobe--ipt_rpfiltermodprobe--ipt_REJECTmodprobe--ipipmodprobe--xt_set注意：Linux系統(tǒng)升級(jí)過內(nèi)核版本后nf_conntrack_ipv4需要修改為：nf_conntrackchmod755/etc/sysconfig/modules/ipvs.modules----增加執(zhí)行權(quán)限/bin/bash/etc/sysconfig/modules/ipvs.modules----執(zhí)行腳本lsmod|grep-eipvs-enf_conntrack----查看對(duì)應(yīng)模塊是否加載成功所有Master節(jié)點(diǎn)安裝并配置keepalived+haproxy高可用集群1.在所有master節(jié)點(diǎn)安裝haproxy和keepalived軟件yum-yinstallhaproxykeepalived1.6.1配置keepalived組件2.配置keepalived主節(jié)點(diǎn)(k8s-master1節(jié)點(diǎn)操作)vim/etc/keepalived/keepalived.conf------編輯配置文件將全局配置里面的vrrp_strict注釋掉router_idLVS_DEVEL#每個(gè)keepalived主機(jī)唯一標(biāo)識(shí)，建議使用當(dāng)前主機(jī)名，但多節(jié)點(diǎn)重名不影響vrrp_skip_check_adv_addr#對(duì)所有通告報(bào)文都檢查，會(huì)比較消耗性能，啟用此配置后，如果收到的通告報(bào)文和上一個(gè)報(bào)文是同一個(gè)路由器，則跳過檢查，默認(rèn)值為全檢查vrrp_strict#嚴(yán)格遵守VRRP協(xié)議,禁止以下狀況:1.無VIP地址2.配置了單播鄰居3.在VRRP版本2中有IPv6地址，開啟動(dòng)此項(xiàng)會(huì)自動(dòng)開啟iptables防火墻規(guī)則，建議關(guān)閉此項(xiàng)配置vrrp_garp_interval0#gratuitousARPmessages報(bào)文發(fā)送延遲，0表示不延遲vrrp_gna_interval0#unsolicitedNAmessages（不請(qǐng)自來）消息發(fā)送延遲添加如下配置：vrrp_scriptcheck_haproxy{script"/etc/haproxy/check_haproxy.sh"#檢測腳本路徑interval5#腳本運(yùn)行間隔時(shí)間5stimeout60#超時(shí)60返回失敗fall3#3次檢測失敗，開始降低優(yōu)先級(jí)rise2#2次檢測成功，表示成功weight-20#優(yōu)先級(jí)每次降低20userroot}vrrp_instanceVI_1{stateMASTERinterfaceens32#實(shí)際網(wǎng)卡名稱virtual_router_id100#路由IDpriority100#優(yōu)先級(jí)配置，優(yōu)先級(jí)越高越優(yōu)先advert_int1#心跳檢查時(shí)間1sauthentication{auth_typePASSauth_pass1111}virtual_ipaddress{0/24devens32labelens32:0#配置vip地址}track_script{check_haproxy}}3.配置keepalived備節(jié)點(diǎn)(k8s-master2節(jié)點(diǎn)操作)vim/etc/keepalived/keepalived.confvrrp_scriptcheck_haproxy{script"/etc/haproxy/check_haproxy.sh"#檢測腳本路徑interval5#腳本運(yùn)行間隔時(shí)間5stimeout60#超時(shí)60返回失敗fall3#3次檢測失敗，開始降低優(yōu)先級(jí)rise2#2次檢測成功，表示成功weight-20#優(yōu)先級(jí)每次降低20userroot}vrrp_instanceVI_1{stateBACKUPinterfaceens32#實(shí)際網(wǎng)卡名稱virtual_router_id100#路由IDpriority90#優(yōu)先級(jí)配置，越高越優(yōu)先，備節(jié)點(diǎn)優(yōu)先級(jí)設(shè)置比主節(jié)點(diǎn)小advert_int1#心跳檢查時(shí)間1sauthentication{auth_typePASSauth_pass1111}virtual_ipaddress{0/24devens32labelens32:0#配置vip地址，指定實(shí)際網(wǎng)卡接口}track_script{check_haproxy}}4.配置keepalived備節(jié)點(diǎn)(k8s-master3節(jié)點(diǎn)操作)vim/etc/keepalived/keepalived.confvrrp_scriptcheck_haproxy{script"/etc/haproxy/check_haproxy.sh"#檢測腳本路徑interval5#腳本運(yùn)行間隔時(shí)間5stimeout60#超時(shí)60返回失敗fall3#3次檢測失敗，開始降低優(yōu)先級(jí)rise2#2次檢測成功，表示成功weight-20#優(yōu)先級(jí)每次降低20userroot}vrrp_instanceVI_1{stateBACKUPinterfaceens32#實(shí)際網(wǎng)卡名稱virtual_router_id100#路由IDpriority80#優(yōu)先級(jí)配置，越高越優(yōu)先，備節(jié)點(diǎn)優(yōu)先級(jí)設(shè)置比主節(jié)點(diǎn)小advert_int1#心跳檢查時(shí)間1sauthentication{auth_typePASSauth_pass1111}virtual_ipaddress{0/24devens32labelens32:0#配置vip地址，指定實(shí)際網(wǎng)卡接口}track_script{check_haproxy}}5.在3臺(tái)master節(jié)點(diǎn)配置檢查腳本/etc/haproxy/check_haproxy.shvim/etc/haproxy/check_haproxy.sh#!/bin/basherr=0forkin$(seq15)docheck_code=$(pgrephaproxy)if[[$check_code==""]];thenerr=$(expr$err+1)sleep5continueelseerr=0breakfidoneif[[$err!="0"]];thenecho"systemctlstopkeepalived"/usr/bin/systemctlstopkeepalivedexit1elseexit0fichmoda+x/etc/haproxy/check_haproxy.sh-----賦予檢查腳本執(zhí)行權(quán)限systemctlenablekeepalived#設(shè)置keepalived開機(jī)自啟動(dòng)systemctlrestartkeepalived#啟動(dòng)keepalivedsystemctlstatuskeepalived#查看keepalived狀態(tài)1.6.2配置haproxy組件6.在3臺(tái)master節(jié)點(diǎn)上配置haproxy組件(三臺(tái)master節(jié)點(diǎn)配置都一致)vim/etc/haproxy/haproxy.cfg====編輯haproxy配置文件defaults#全局配置modehttp#默認(rèn)模式(tcp/udp/http/https)logglobaloptionhttplog#日志采集采用httplogoptiondontlognull#不記錄健康檢查日志optionhttp-server-close#每次請(qǐng)求完畢后關(guān)閉http通道optionforwardforexcept/8optionredispatch#當(dāng)請(qǐng)求得服務(wù)器down之后切換至健康服務(wù)器retries3#3次連接失敗確定服務(wù)器不可用timeouthttp-request10s#默認(rèn)http超時(shí)時(shí)間timeoutqueue1m#默認(rèn)隊(duì)列超時(shí)時(shí)間timeoutconnect10s#默認(rèn)連接超時(shí)時(shí)間timeoutclient1m#默認(rèn)客戶端超時(shí)時(shí)間timeoutserver1m#默認(rèn)服務(wù)器超時(shí)時(shí)間timeouthttp-keep-alive10s#默認(rèn)持久連接超時(shí)時(shí)間timeoutcheck10s#默認(rèn)心跳檢查超時(shí)時(shí)間maxconn3000#默認(rèn)最大連接數(shù)-----------------------------------------------------------------frontendmonitor-inbind*:33305#設(shè)置haproxy組件的端口狀態(tài)監(jiān)控modehttpoptionhttplogmonitor-uri/monitor#設(shè)置haproxy組件的監(jiān)控路徑listenstats#設(shè)置一個(gè)名為：stats的haproxy監(jiān)控頁面實(shí)列bind*:8006#設(shè)置haproxy統(tǒng)計(jì)監(jiān)控頁面的端口modehttpstatsenablestatshide-version#隱藏監(jiān)控統(tǒng)計(jì)頁面的haproxy版本號(hào)statsuri/stats#設(shè)置haproxy監(jiān)控統(tǒng)計(jì)頁面的URL路徑statsrefresh30s#設(shè)置haproxy監(jiān)控統(tǒng)計(jì)頁面的自動(dòng)刷新時(shí)間statsrealmHaproxy\Statisticsstatsauthadmin:admin#設(shè)置haproxy監(jiān)控統(tǒng)計(jì)頁面的賬號(hào)-----------------------------------------------------------------#K8S-master高可用負(fù)載配置frontendK8S-masterbind*:9443#設(shè)置k8s-master的前端監(jiān)聽端口為：9443modetcp#設(shè)置監(jiān)聽模式tcpoptiontcplog#設(shè)置日志采集模式tcplogtcp-requestinspect-delay30s#設(shè)置等待數(shù)據(jù)傳輸?shù)淖畲蟪瑫r(shí)時(shí)間default_backendK8S-master#設(shè)置轉(zhuǎn)發(fā)的后端服務(wù)器組-----------------------------------------------------------------------------------------------backendK8S-mastermodetcp#設(shè)置監(jiān)聽模式tcpoptiontcplog#設(shè)置日志采集模式tcplogoptiontcp-check#設(shè)置健康檢查模式tcp-checkbalanceroundrobin#設(shè)置后端服務(wù)器組負(fù)載均衡模式：輪詢default-serverinter10sdowninter5srise2fall2slowstart60smaxconn250maxqueue256weight100serverK8S-master01:6443checkinter2000fall2rise2weight100#配置真實(shí)后端服務(wù)器及端口serverK8S-master02:6443checkinter2000fall2rise2weight100serverK8S-master03:6443checkinter2000fall2rise2weight100systemctlenablehaproxy#設(shè)置haproxy開機(jī)自啟動(dòng)systemctlrestarthaproxy#啟動(dòng)haproxysystemctlstatushaproxy#查看haproxy狀態(tài)1.6.3測試keepalived+haproxy高可用集群的HA在主節(jié)點(diǎn)K8S-master01上面必須有VIP地址，如圖：3臺(tái)K8S-master節(jié)點(diǎn)都可以ping通VIP地址，如圖：斷開K8S-master01主機(jī)網(wǎng)卡，VIP漂移至K8S-master02節(jié)點(diǎn)，如圖：重新打開K8S-master01網(wǎng)卡，VIP從K8S-master02又重新回到K8S-master01，如圖：所有節(jié)點(diǎn)安裝docker并進(jìn)行配置1.安裝dockerwget/docker-ce/linux/centos/docker-ce.repo-O/etc/yum.repos.d/docker-ce.repo-----下載docker-ce的安裝yum源文件yum-yinstalldocker-ce*-------安裝docker2.啟動(dòng)docker服務(wù)systemctlenabledockersystemctlrestartdocker3.創(chuàng)建docker默認(rèn)配置文件daemon.josn，修改docker鏡像倉庫與驅(qū)動(dòng)模式cd/etc/docker----切換到docker目錄vim/etc/docker/daemon.josn----修改daemon.josn文件，添加如下內(nèi)容{"exec-opts":["native.cgroupdriver=systemd"],"registry-mirrors":[""]}4.修改docker.services服務(wù)cd/usr/lib/systemd/systemvimdocker.service--exec-optnative.cgroupdriver=systemd----添加此配置，修改docker驅(qū)動(dòng)模式5.重新啟動(dòng)dockersystemctldaemon-reloadsystemctlrestartdocker所有節(jié)點(diǎn)安裝containerd并進(jìn)行配置安裝containerdwget/docker-ce/linux/centos/docker-ce.repo-O/etc/yum.repos.d/docker-ce.repo-----下載docker-ce的安裝yum源文件yum-yinstallcontainerd-------安裝containerdcontainerd--version----查看當(dāng)前的containerd版本啟動(dòng)containerd服務(wù)systemctldaemon-reloadsystemctlenablecontainerdsystemctlrestartcontainerd生成containerd默認(rèn)配置文件containerdconfigdefault>/etc/containerd/config.tomlvim/etc/containerd/config.toml-------修改默認(rèn)containerd配置文件sandbox_image="/google_containers/pause:3.9"SystemdCgroup=true[plugins."io.containerd.grpc.v1.cri".registry.mirrors][plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]endpoint=["","",""][plugins."io.containerd.grpc.v1.cri".registry.mirrors."k8s.gcr.io"]endpoint=["/google_containers"]創(chuàng)建containerd.conf配置文件vim/etc/modules-load.d/containerd.confoverlaybr_netfilter重新啟動(dòng)containerdsystemctldaemon-reloadsystemctlrestartcontainerdmodprobeoverlaymodprobebr_netfilterlsmod|grepoverlaylsmod|grepbr_netfilter檢查containerd.service配置文件，如圖：cat/etc/systemd/system/multi-user.target.wants/containerd.serviceDelegate：選項(xiàng)允許containerd以及運(yùn)行時(shí)自己管理自己創(chuàng)建容器的cgroups。不設(shè)置這個(gè)選項(xiàng)，systemd就會(huì)將進(jìn)程移到自己的cgroups中，從而導(dǎo)致containerd無法正確獲取容器的資源使用情況。KillMode：這個(gè)選項(xiàng)用來處理containerd進(jìn)程被殺死的方式。默認(rèn)情況下，systemd會(huì)在進(jìn)程的cgroup中查找并殺死containerd的所有子進(jìn)程。將KillMode的值設(shè)置為process，這樣可以確保升級(jí)或重啟containerd時(shí)不殺死現(xiàn)有的容器所有containerd節(jié)點(diǎn)安裝并配置crictl工具1.下載安裝包：wget/kubernetes-sigs/cri-tools/releases/download/crictl-v1.26.0-linux-amd64.tar.gz2.解壓安裝包：tarzxvfcrictl-v1.26.0-linux-amd64.tar.gz-C/usr/local/bin3.配置crictl.yaml文件vim/etc/crictl.yamlruntime-endpoint:unix:///run/containerd/containerd.sockimage-endpoint:unix:///run/containerd/containerd.socktimeout:2debug:falsepull-image-on-create:false4.配置環(huán)境變量：exportPATH=$PATH:/usr/local/bin:/usr/local/sbin#命令追加到PATH環(huán)境變量

source~/.bashrc#使變量立即生效5.更新runc，需要下載安裝包：wget/opencontainers/runc/releases/download/runc.amd64mvrunc.amd64/usr/local/sbin/runc

chmod755/usr/local/sbin/runc6.重新啟動(dòng)containerdsystemctldaemon-reloadsystemctlrestartcontainerd所有containerd節(jié)點(diǎn)安裝并配置nerdctl工具1.下載安裝包：wget/containerd/nerdctl/releases/download/v1.2.0/nerdctl-1.2.0-linux-amd64.tar.gz2.解壓安裝包并配置環(huán)境變量：tar-zxvfnerdctl-1.2.0-linux-amd64.tar.gz-C/usr/local/binexportPATH=$PATH:/usr/local/bin:/usr/local/sbin#命令追加到PATH環(huán)境變量

source~/.bashrc#使變量立即生效所有containerd節(jié)點(diǎn)配置kubernetes安裝yum源文件vim/etc/yum.repos.d/kubernetes.repo----編輯一個(gè)repo文件，如下：[Kubernetes]name=Kubernetesbaseurl=/kubernetes/yum/repos/kubernetes-el7-x86_64/enabled=1gpgcheck=1repo_gpgcheck=1gpgkey=/kubernetes/yum/doc/yum-key.gpg/kubernetes/yum/doc/rpm-package-key.gpg1.10所有containerd節(jié)點(diǎn)安裝并配置kubeadm/kubectl/kubeletyum-yinstallkubeadm*kubelet*kubectl*----安裝kubeadm/kubelet/kubectl修改kubelet配置文件vim/etc/sysconfig/kubeletKUBELET_CGROUP_ARGS="--cgroup-driver=systemd"KUBELET_EXTRA_ARGS="--cgroup-driver=systemd"KUBELET_EXTRA_ARGS="--fail-swap-on=false"KUBE_PROXY_MODE="ipvs"systemctlenablekubeletsystemctlrestartkubeletsystemctlstatuskubelet注意：kubelet服務(wù)安裝完畢后，K8S集群沒有初始化時(shí)服務(wù)屬于未運(yùn)行狀態(tài)是正?，F(xiàn)象1.11部署K8S高可用群集1.11.1K8S-master主節(jié)點(diǎn)使用kubeadminit初始化1.初始化K8S-master01節(jié)點(diǎn)kubeadmconfigprintinit-defaults--component-configsKubeProxyConfiguration,KubeletConfiguration>kubeadm-init.yaml2.編輯kubeadm-init.yamlvimkubeadm-init.yaml3.驗(yàn)證語法是否正確kubeadminit--configkubeadm-init.yaml--dry-run4.預(yù)拉取鏡像kubeadmconfigimagespull--configkubeadm-init.yaml5.在K8S-master01上初始化集群kubeadminit--configkubeadm-init.yaml--upload-certs--ignore-preflight-errors=Swap記錄下在主K8S-master節(jié)點(diǎn)上面生成的kubeadmjoin信息，備節(jié)點(diǎn)加入需要使用：kubeadmjoin0:9443--tokenabcdef.0123456789abcdef\--discovery-token-ca-cert-hashsha256:fb2f038a1701156595ecd0b15e6d3a89506e646b1aebeba9f60e9e2b475f89d2\--control-plane--certificate-key3841b50a220510d0686b07b4148d3cf54b8f970c8867d11d9c65905989cabf6f主master節(jié)點(diǎn)配置環(huán)境變量：mkdir-p$HOME/.kubesudocp-i/etc/kubernetes/admin.conf$HOME/.kube/configsudochown$(id-u):$(id-g)$HOME/.kube/configexportKUBECONFIG=/etc/kubernetes/admin.confK8S首個(gè)master節(jié)點(diǎn)初始化完成，下一步進(jìn)行其他master節(jié)點(diǎn)和node節(jié)點(diǎn)加入集群并進(jìn)行初始化配置1.11.2K8S-master備節(jié)點(diǎn)使用kubeadmjoin加入集群完成初始化1.備節(jié)點(diǎn)master02，master03復(fù)制主節(jié)點(diǎn)相關(guān)證書mkdir-p/etc/kubernetes/pki/mkdir-p/etc/kubernetes/pki/etcd----備節(jié)點(diǎn)創(chuàng)建pki，etcd目錄scp-r:/etc/kubernetes/pki/ca.*/etc/kubernetes/pki/scp-r:/etc/kubernetes/pki/sa.*/etc/kubernetes/pki/scp-r:/etc/kubernetes/pki/front-proxy-ca.*/etc/kubernetes/pki/scp-r:/etc/kubernetes/pki/etcd/ca.*/etc/kubernetes/pki/etcd/scp-r:/etc/kubernetes/admin.conf/etc/kubernetes/2.主節(jié)點(diǎn)master01將kubeadm-init.yaml上傳備節(jié)點(diǎn)至master02，master03scp–rkubeadm-init.yamlroot@:/rootscp–rkubeadm-init.yamlroot@:/root3.備節(jié)點(diǎn)master02，master03使用kubeadmjoin加入主節(jié)點(diǎn)集群kubeadmjoin0:9443--tokenabcdef.0123456789abcdef\--discovery-token-ca-cert-hashsha256:fb2f038a1701156595ecd0b15e6d3a89506e646b1aebeba9f60e9e2b475f89d2\--control-plane--certificate-key3841b50a220510d0686b07b4148d3cf54b8f970c8867d11d9c65905989cabf6f--ignore-preflight-errors=Swap注意：--certificate-key3841b50a220510d0686b07b4148d3cf54b8f970c8867d11d9c65905989cabf6f一般2小時(shí)過期，過期了需要在主master節(jié)點(diǎn)重新生成一個(gè)，然后備節(jié)點(diǎn)使用新的證書密鑰進(jìn)行kubeadmjoin加入，生成命令如下：完成備節(jié)點(diǎn)master加入主節(jié)點(diǎn)集群后，需要將node節(jié)點(diǎn)加入1.11.3K8S-node節(jié)點(diǎn)使用kubeadmjoin加入集群完成初始化kubeadmjoin0:9443--tokenabcdef.0123456789abcdef\--discovery-token-ca-cert-hashsha256:fb2f038a1701156595ecd0b15e6d3a89506e646b1aebeba9f60e9e2b475f89d2--ignore-preflight-errors=Swap完成所以node節(jié)點(diǎn)加入K8S集群在主節(jié)點(diǎn)上面查詢是否已經(jīng)加入成功，如圖：所以節(jié)點(diǎn)已經(jīng)全部加入，NotReady表示沒有安裝網(wǎng)絡(luò)組件到此，所有node節(jié)點(diǎn)全部加入到了K8S集群中了。默認(rèn)在node節(jié)點(diǎn)上無法運(yùn)行kubectl命令，如圖：可以將主節(jié)點(diǎn)K8S-master01上面的/root/.kube目錄整體復(fù)制到3臺(tái)node節(jié)點(diǎn)上去scp-r.kuberoot@:/root/scp-r.kuberoot@:/root/scp-r.kuberoot@:/root/1.12部署K8S群集網(wǎng)絡(luò)組件calico1.主節(jié)點(diǎn)下載calico.yaml文件curl-O/projectcalico/calico/v3.25.0/manifests/calico-typha.yamlmvcalico-typha.yamlcalico.yamlvimcalico.yaml-----修改配置文件中，如圖：2.將calico.yaml文件中所使用的默認(rèn)鏡像源docker.io修改為國內(nèi)源地址，操作如下：未修改前的：修改后的：sed-i's#docker.io/##g'calico.yaml原先在conatinerd配置文件里面配置了鏡像倉庫地址，故需要將calico.yaml文件里面的docker.io前綴去掉2.安裝calico組件，執(zhí)行如下命令：kubectlapply-fcalico.yaml安裝完成后，需要等待10-20分鐘左右，所有calicopod運(yùn)行正常，如圖：2.安裝calicoctl網(wǎng)絡(luò)組件命令工具下載安裝工具，命令如下：curl-L/projectcalico/calico/releases/download/v3.25.0/calicoctl-linux-amd64-ocalicoctlchmod755calicoctl-----------給工具賦予執(zhí)行權(quán)限mvcalicoctl/usr/local/bin-------移動(dòng)有執(zhí)行權(quán)限的命令至/usr/local/bin目錄下exportPATH=$PATH:/usr/local/bin:/usr/local/sbin#命令追加到PATH環(huán)境變量

source~/.bashrc#使變量立即生效calicoctlnodestatus-----查詢所有節(jié)點(diǎn)運(yùn)行的的calico狀態(tài)calicoctlgetIPPool-oyaml------查詢當(dāng)前yaml配置文件的CIDR地址池1.13部署K8S群集包管理工具h(yuǎn)elm組件1.下載heml-v3wgethttps://get.helm.sh/helm-v3.11.0-linux-amd64.tar.gz2.解壓并配置helm軟件包tar-zxvfhelm-v3.11.0-linux-amd64.tar.gzmvlinux-amd64/helm/usr/local/bin/exportKUBECONFIG=/root/.kube/config2.添加常用的chart倉庫helmrepoaddazure/kubernetes/charts/helmrepoaddkaiyuanshe/kubernetes/charts/helmrepoaddbitnami/bitnami3.查看chart倉庫：helmrepolist3.更新chart倉庫:helmrepoupdate4.查看阿里云chart倉庫中的memcached5.移除chart倉庫1.14部署K8S群集企業(yè)級(jí)私有倉庫Harbor組件Harbor的高可用方案大致可以分為下面兩種，一種依賴共享存儲(chǔ)來保存鏡像數(shù)據(jù)，另一種基于不同Harbor服務(wù)器間的鏡像復(fù)制實(shí)現(xiàn)。如下圖所示：共享存儲(chǔ)的選取，Harbor的后端存儲(chǔ)目前支持本地文件系統(tǒng)、NFS、CephFS、azure、gcs、AWSs3,、swift以及阿里云oss共享存儲(chǔ)的選取，Harbor的后端存儲(chǔ)目前支持本地文件系統(tǒng)、NFS、CephFS、azure、gcs、AWSs3,、swift以及阿里云oss雙主復(fù)制架構(gòu)在遇到大鏡像時(shí)有同步延遲，并且一個(gè)實(shí)例故障后需要手動(dòng)重新開啟復(fù)制策略才能再次同步雙主復(fù)制架構(gòu)在遇到大鏡像時(shí)有同步延遲，并且一個(gè)實(shí)例故障后需要手動(dòng)重新開啟復(fù)制策略才能再次同步本文采用方式二：Habor節(jié)點(diǎn)之間復(fù)制的高可用方式部署高可用habor組件Habor組件安裝需要依賴docker，docker-compose組件，前面已經(jīng)將docker組件安裝完畢了，現(xiàn)在需要安裝docker-compose組件yum-yinstallepel-releaseyum-yinstallpython-pipyum-yinstalldocker-compose下載habor組件軟件并上傳至K8S-master01和K8S-master02這個(gè)節(jié)點(diǎn)wget/goharbor/harbor/releases/download/v2.7.0/harbor-offline-installer-v2.7.0.tgztar-zxvfharbor-offline-installer-v2.7.0.tgz-C/usr/local/---解壓下載軟件在2臺(tái)Harbor節(jié)點(diǎn)修改habor配置文件cd/usr/local/harborcpharbor.yml.tmplharbor.ymlvimharbor.yml-------修改配置文件在2臺(tái)Harbor節(jié)點(diǎn)安裝harbor組件./install.sh5.配置Harbor倉庫的鏡像復(fù)制關(guān)系登錄節(jié)點(diǎn)1的Harbor控制臺(tái)界面進(jìn)行如下操作：倉庫管理-新建目標(biāo)點(diǎn)擊：復(fù)制管理---新建規(guī)則：登錄節(jié)點(diǎn)2的Harbor控制臺(tái)界面進(jìn)行如下操作：倉庫管理-新建目標(biāo)6.安裝keepalived+nginx實(shí)現(xiàn)Harbor倉庫組件主備節(jié)點(diǎn)的前端負(fù)載均衡yum-yinstallkeepalivedyum-yinstallnginxsystemctlenablekeepalivedsystemctlenablenginxsystemctlrestartkeepalivedsystemctlrestartnginxvim/etc/keepalived/keepalived.conf----修改keepalived配置文件主節(jié)點(diǎn)keepalived配置如下：備節(jié)點(diǎn)配置如下：備節(jié)點(diǎn)優(yōu)先級(jí)和state與主節(jié)點(diǎn)配置不一樣，其他都一致主備節(jié)點(diǎn)修改nginx配置文件vim/etc/nginx/nginx.conf-----修改nginx配置文件，主備節(jié)點(diǎn)配置一致配置nginx的健康檢查腳本，配置如下：cd/etc/nginxvimcheck_nginx.shchmod755check_nginx.sh腳本內(nèi)容