使用Static Pod 創建高可用外部etcd叢集：詳解步驟

從官網可以知道，我們在建立一組高可用的Kubernetes Cluster時，針對ETCD有二種做法，一種是與Master節點共用，另一種則是將ETCD再獨立出來三個節點成為一組Cluster來進行運作。

本文將說明如何建置出外部的ETCD Cluster，並以Static Pod的方式運行與管理，最後再與Kubernetes cluster整合。

本文將說明內容如下述：

1. 為什麼要建立外部ETCD?
2. 為什麼用Static Pod方式?
3. 實作
4. Kubernetes cluster建置整合
5. 結論

那就開始吧!!

1.為什麼要建立外部ETCD?

如果把ETCD獨立出來，架構就會變成如下圖所示：

kubeadm HA topology — external ETCD

這種架構是將Master與ETCD成員關係進行解構(decouples)，優點是確保就算有任一master成員出現異常或是任一ETCD成員出現異常，這種架構可以有效降低上述任一種情況出現後的所造成的系統衝擊。

但這種架構的缺點是需要更多的節點(master+etcd=6)來實現，對於資源的要求較高，另外ETCD與Master之間是靠網路來通訊也可能會有風險。在規劃時需要將其考慮進去。

2.為什麼用Static Pod方式?

Static Pod指的是在Pod在指定的節點上運行，而且由Kubelet直接管理，不會透過apiserver，與一般的Pod(ex. deployment)管理方式不同。對於Kubernetes apiserver來說，static pod看的到但管不了。

那kubelet是怎麼管理Static Pod呢?

• 負責在Static Pod崩潰後重啟
• 重啟後的Static Pod會綁定到指定節點的Kubelet
• kubelet會透過Kubernetes apiserver幫每個static pod自動再建立一個mirror pod

那為什麼我會用Static Pod的方式來建立external ETCD呢?

答案是為了符合ETCD原本在Kubernetes內的類型，怎麼說呢，以下是Kubernetes內常見的Static Pod：

• etcd
• kube-apiserver
• kube-controller-manager
• kube-scheduler

如同上述，如果使用的架構是堆疊型，那也就意味ETCD在Kubernetes內也是以Static Pod的方式運行，那今天將它們獨立出來，如果還是使用原本的型態運行，那對於之後的整合與管理就會比較一致與容易。

接下來就進行實作部分，以下說明如何在三台獨立的節點上以static pod的方式完成ETCD的建置。

3. 實作

(Step0). 開始前準備

#----------------------------------------------
# S3-1. 關閉swap (all nodes)
#----------------------------------------------
[etcdX]# swapoff -a && sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab/fstab

#---------------------------------------------
# S3-2. 安裝iproute-tc (all nodes)
#---------------------------------------------
[etcdX]# yum install iproute-tc
[etcdX]# vim /etc/modules-load.d/k8s.conf
overlay
br_netfilter

[etcdX]# modprobe overlay
[etcdX]# modprobe br_netfilter

[etcdX]# vim /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1

[etcdX]# sysctl --system

(Step1). 基本軟體安裝

#---------------------------------------------
# S3-3. 安裝runtime (all nodes)
#---------------------------------------------
[etcdX]# export VERSION=1.27
[etcdX]# curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/CentOS_8/devel:kubic:libcontainers:stable.repo
[etcdX]# curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable:cri-o:$VERSION.repo https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:$VERSION/CentOS_8/devel:kubic:libcontainers:stable:cri-o:$VERSION.repo
[etcdX]# yum install cri-o -y
[etcdX]# systemctl enable --now crio
[etcdX]# crio version
[etcdX]# yum list cri-o --showduplicates|sort -r > crio.version

#---------------------------------------------
# S3-4. 安裝kubelet, kubeadm, kubectl (all nodes)
#---------------------------------------------
[etcdX]# vim /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg

[etcdX]# yum clean all ; yum repolist

[etcdX]# yum list kubelet --showduplicates|sort -r > kubelet.version
[etcdX]# yum list kubeadm --showduplicates|sort -r > kubeadm.version
[etcdX]# yum list kubectl --showduplicates|sort -r > kubectl.version

[etcdX]# yum install kubelet-1.27.6-0 kubeadm-1.27.6-0 kubectl-1.27.6-0

#---------------------------------------------
# S3-5. 建立systemd service (all nodes)
# 尚未要建立k8s cluster，用原本的kubelet.conf會無法啟動，建立新的服務啟動檔
#---------------------------------------------
[etcdX]# vim /etc/crio/crio.conf
=> cgroup_manager = "systemd"
[etcdX]# cat /etc/crio/crio.conf | grep cgroup_manager
[etcdX]# systemctl restart crio

[etcdX]# cat << EOF > /usr/lib/systemd/system/kubelet.service.d/20-etcd-service-manager.conf
[Service]
ExecStart=
ExecStart=/usr/bin/kubelet --address=127.0.0.1 --pod-manifest-path=/etc/kubernetes/manifests --cgroup-driver=systemd --runtime-request-timeout=15m --container-runtime-endpoint=unix:///var/run/crio/crio.sock
Restart=always
EOF

[etcdX]# systemctl daemon-reload
[etcdX]# systemctl restart kubelet

※ 此處要注意：

(1) 官網上的路徑已修改到/usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf => (kubeadm 1.13.5之後)(依照官網的方法將無法啟動kubelet)

(2) systemd要換成crio用的cgroup(kubelet預設值為cgroupfs) (/etc/crio/crio.conf)

#---------------------------------------------
# S3-6. 建立kubeadm config (all nodes)
#---------------------------------------------
[etcdX]# vim /root/kubeadm_setup.sh
#!/bin/bash
export HOST0=10.107.88.15
export HOST1=10.107.88.16
export HOST2=10.107.88.17

export NAME0="etcd01"
export NAME1="etcd02"
export NAME2="etcd03"

mkdir -p /tmp/${HOST0}/ /tmp/${HOST1}/ /tmp/${HOST2}/

HOSTS=(${HOST0} ${HOST1} ${HOST2})
NAMES=(${NAME0} ${NAME1} ${NAME2})

for i in "${!HOSTS[@]}"; do
HOST=${HOSTS[$i]}
NAME=${NAMES[$i]}

cat << EOF > /tmp/${HOST}/kubeadmcfg.yaml
---
apiVersion: "kubeadm.k8s.io/v1beta3"
kind: InitConfiguration
nodeRegistration:
 name: ${NAME}
localAPIEndpoint:
 advertiseAddress: ${HOST}
---
apiVersion: "kubeadm.k8s.io/v1beta3"
kind: ClusterConfiguration
etcd:
 local:
     serverCertSANs:
     - "${HOST}"
     peerCertSANs:
     - "${HOST}"
     extraArgs:
         initial-cluster: ${NAMES[0]}=https://${HOSTS[0]}:2380,${NAMES[1]}=https://${HOSTS[1]}:2380,${NAMES[2]}=https://${HOSTS[2]}:2380
         initial-cluster-state: new
         name: ${NAME}
         listen-peer-urls: https://${HOST}:2380
         listen-client-urls: https://${HOST}:2379
         advertise-client-urls: https://${HOST}:2379
         initial-advertise-peer-urls: https://${HOST}:2380
EOF
done

[etcdX]# ./kubeadm_setup.sh
[etcdX]# tree /tmp/<etcd_ip>

(Step2). 建立Certificates

#------------------------------------------------
# S3-7. 產生CA (etcd01)
#------------------------------------------------
[etcdX]# kubeadm init phase certs etcd-ca
[etcdX]# tree /etc/kubernetes/pki/etcd/
ca.crt
ca.key

#------------------------------------------------
# S3-8. 為每個成員建立CA (etcd01)
#------------------------------------------------
[etcdX]# export HOST0=10.107.88.15
[etcdX]# export HOST1=10.107.88.16
[etcdX]# export HOST2=10.107.88.17
[etcdX]# kubeadm init phase certs etcd-server --config=/tmp/${HOST2}/kubeadmcfg.yaml
[etcdX]# kubeadm init phase certs etcd-peer --config=/tmp/${HOST2}/kubeadmcfg.yaml
[etcdX]# kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${HOST2}/kubeadmcfg.yaml
[etcdX]# kubeadm init phase certs apiserver-etcd-client --config=/tmp/${HOST2}/kubeadmcfg.yaml
[etcdX]# cp -R /etc/kubernetes/pki /tmp/${HOST2}/

# 刪除不可重複使用的CA
[etcdX]# find /etc/kubernetes/pki -not -name ca.crt -not -name ca.key -type f -delete
[etcdX]# tree /etc/kubernetes/pki/etcd
/etc/kubernetes/pki/etcd
├── ca.crt
└── ca.key
(只剩下最早產生的CA)

[etcdX]# kubeadm init phase certs etcd-server --config=/tmp/${HOST1}/kubeadmcfg.yaml
[etcdX]# kubeadm init phase certs etcd-peer --config=/tmp/${HOST1}/kubeadmcfg.yaml
[etcdX]# kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${HOST1}/kubeadmcfg.yaml
[etcdX]# kubeadm init phase certs apiserver-etcd-client --config=/tmp/${HOST1}/kubeadmcfg.yaml
[etcdX]# cp -R /etc/kubernetes/pki /tmp/${HOST1}/
[etcdX]# find /etc/kubernetes/pki -not -name ca.crt -not -name ca.key -type f -delete

[etcdX]# kubeadm init phase certs etcd-server --config=/tmp/${HOST0}/kubeadmcfg.yaml
[etcdX]# kubeadm init phase certs etcd-peer --config=/tmp/${HOST0}/kubeadmcfg.yaml
[etcdX]# kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${HOST0}/kubeadmcfg.yaml
[etcdX]# kubeadm init phase certs apiserver-etcd-client --config=/tmp/${HOST0}/kubeadmcfg.yaml
=> HOST0因為是本機，不用cp

# 刪除不應該從HOST0複製的CA
[etcdX]# find /tmp/${HOST2} -name ca.key -type f -delete
[etcdX]# find /tmp/${HOST1} -name ca.key -type f -delete

#------------------------------------------------
# S3-9. copy 到各主機 (etcd01)
#------------------------------------------------
[etcdX]# USER=root
[etcdX]# HOST=${HOST1}
[etcdX]# scp -r /tmp/${HOST}/* ${USER}@${HOST}:
[etcdX]# ssh ${USER}@${HOST}
[etcdX]# chown -R root:root pki
[etcdX]# mv pki /etc/kubernetes/

#------------------------------------------------
# S3-10. 確認 (在每台主機上確認路徑相同)
#------------------------------------------------
[etcdX]# tree /root
[etcdX]# tree /etc/kubernetes/pki

(Step3). 建立ETCD Cluster

#------------------------------------------------
# S3-11. 產生static pod manifest (etcd01/02/03)
#------------------------------------------------
[etcd01]# kubeadm init phase etcd local --config=/root/kubeadmcfg.yaml
[etcd02]# kubeadm init phase etcd local --config=/root/kubeadmcfg.yaml
[etcd03]# kubeadm init phase etcd local --config=/root/kubeadmcfg.yaml

#-------------------------------------------------
# S3-12. etcdctl installation
#-------------------------------------------------
[etcdX]# ETCD_RELEASE=$(curl -s https://api.github.com/repos/etcd-io/etcd/releases/latest|grep tag_name | cut -d '"' -f 4)
[etcdX]# echo $ETCD_RELEASE
v3.5.9

[etcdX]# wget https://github.com/etcd-io/etcd/releases/download/${ETCD_RELEASE}/etcd-${ETCD_RELEASE}-linux-amd64.tar.gz
[etcdX]# tar zxvf etcd-v3.5.9-linux-amd64.tar.gz
[etcdX]# cd etcd-v3.5.9-linux-amd64
[etcdX]# ls -al
[etcdX]# cp -rp etcdctl /usr/local/bin

#------------------------------------------------
# S3-13. 確認etcd cluster
#------------------------------------------------
[etcdX]# ETCDCTL_API=3 etcdctl \
--cert /etc/kubernetes/pki/etcd/peer.crt \
--key /etc/kubernetes/pki/etcd/peer.key \
--cacert /etc/kubernetes/pki/etcd/ca.crt \
--endpoints https://10.107.88.15:2379 endpoint health

https://10.107.88.15:2379 is healthy: successfully committed proposal: took = 10.08693ms
https://10.107.88.16:2379 is healthy: successfully committed proposal: took = 10.912799ms
https://10.107.88.17:2379 is healthy: successfully committed proposal: took = 10.461484ms

4. Kubernetes cluster建置整合

#------------------------------------------
# S4-1. 複製任一台etcd的ca到master01
#------------------------------------------
[etcd]# scp -rp /etc/kubernetes/pki/etcd/ca.crt root@master01:/root/etcd-ca
[etcd]# scp -rp /etc/kubernetes/pki/apiserver-etcd-client.crt root@master01:/root/etcd-ca
[etcd]# scp -rp /etc/kubernetes/pki/apiserver-etcd-client.key root@master01:/root/etcd-ca

#------------------------------------------
# S4-2. 設定master01的kubeadm-config.yaml
#------------------------------------------
[root]# vim kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
kubernetesVersion: stable
apiServer:
   certSANs:
   - "test.example.poc"
controlPlaneEndpoint: "10.107.88.9:6443"
etcd:
       external:
              endpoints:
              - https://10.107.88.15:2379
              - https://10.107.88.16:2379
              - https://10.107.88.17:2379
              caFile: /etc/kubernetes/pki/etcd/ca.crt
              certFile: /etc/kubernetes/pki/apiserver-etcd-client.crt
              keyFile: /etc/kubernetes/pki/apiserver-etcd-client.key
networking:
    podSubnet: "10.244.0.0/16"

#------------------------------------------
# S4-3. master01初始化
#------------------------------------------
[root]# kubeadm init --config kubeadm-config.yaml --upload-certs

5. 結論

以上就是將ETCD從Control-Plane獨立出來的做法，一般來說，大部分的情境較少用這種架構，大部分還是使用與Control-Plane整合在一起的堆疊型的架構，包含企業商用的解決方式(ex. OpenShift)也是使用堆疊型的ETCD，以下提供幾點堆疊vs獨立的比較如下：

• [Stacked]: 由於在同一個節點，所以apiserver與etcd之間只要用loopback就可以溝通，Read操作就可以不用再透過master，也不用透過TCP/IP網路就可以讀取到資料，速度與穩定度都可以滿足。
• [External]: 必須透過TCP/IP網路溝通
• [Stacked]: ETCD比較重視Disk i/o，如果在同一個節點運行，網路就不會變成傳輸的瓶頸。
• [External]: 網路頻寬會直接影響整體服務效能。
• [Stacked]: 當Control-Plane成員壞了一台，代表在同一台的ETCD也會無法提供服務
• [External]: 就算Master/ETCD任一個成員壞了，也不會直接造成立即性的系統崩壞發生。
• [Stacked]: 部署簡單，不需要額外資源.
• [External]: 要另外部署，也有額外資源需求。

本文說明了如何建立External ETCD的做法並且整合進Kubernetes Cluster部署，並且簡單說明了二者的實用性上的差異提供給大家參考。

本文有點長，感謝看到現在的你。

你的鼓勵是我繼續分享的動力，下篇文章再見。

※ References:

• https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/ha-topology/
• https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/
• https://www.jianshu.com/p/c481543002c8