ETCD還原實戰指南：Cluster問題輕鬆解決

超健忘閒人

發佈於kubernetes怕忘記

2024/03/07閱讀時間約 15 分鐘

最近碰到K8S Cluster出現問題，在不使用其他快照功能的前提之下，透過平常ETCD的備份來還原Cluster，本篇就將這個還原的做法記錄下來，下次若再次發生(希望不要!)，就可以作為參考文件。

同樣地，以下是本篇將會提到的幾個重點：

ETCD的資料類型
ETCD 備份
ETCD 還原
結論

1. ETCD的資料類型

ETCD的資料預設會分成二種資料類型：

snap(Snapshot)：儲存快照資料，預防WAL檔案過多而設計的快照，存放ETCD資料狀態。檔案附檔名為.snap。如果打開一個.snap檔案，可以發現除了開頭的幾個字是看不到的字元外，之後都是可見字元，因為實際上就是直接將儲存的JSON格式化後的字串。
wal(Write Ahead Log)：儲存預寫式日誌，最大的功能是記錄整個資料變化的全部過程，在ETCD中，所有資料的修改在提交之前，都先寫到WAL中。每個wal都是由一條條的記錄所構成。

2. ETCD 備份

#------------------------------------------------------
# S2-1. 前置作業
#------------------------------------------------------
[master]# mkdir -p /root/backup_$(date +%Y%m%d)
[master]# cp -r /etc/kubernetes /root/backup_$(date +%Y%m%d)/
[master]# cp -r /var/lib/etcd/ /root/backup_$(date +%Y%m%d)/
[master]# cp -r /var/lib/kubelet/ /root/backup_$(date +%Y%m%d)/
[master]# ls -al /root/backup_$(date +%Y%m%d)/

#------------------------------------------------------
# S2-2. 備份ETCD
#------------------------------------------------------
[master]# ETCDCTL_API=3 etcdctl --endpoints="https://127.0.0.1:2379" \
--cert="/etc/kubernetes/pki/etcd/server.crt" \
--key="/etc/kubernetes/pki/etcd/server.key" \
--cacert="/etc/kubernetes/pki/etcd/ca.crt" \
snapshot save /root/backup_$(date +%Y%m%d)/snap-$(date +%Y%m%d).db

[master]# ls -alh
drwx------   3 root root   20 Feb  5 15:52 etcd
drwx------   8 root root 4.0K Feb  5 15:52 kubelet
drwxr-xr-x   4 root root  125 Feb  5 15:52 kubernetes
-rw-------   1 root root 169M Feb  5 15:53 snap-20240205.db

3. ETCD 還原

#------------------------------------------------------
# S3-1. 確認member名稱
#------------------------------------------------------
[master]# ETCDCTL_API=3 etcdctl --endpoints 10.107.88.12:2379,10.107.88.13:2379,10.107.88.14:2379 \
--cert="/etc/kubernetes/pki/etcd/server.crt" \
--key="/etc/kubernetes/pki/etcd/server.key" \
--cacert="/etc/kubernetes/pki/etcd/ca.crt" \
member list --write-out=table

#------------------------------------------------------
# S3-1. 還原前準備 (master01,master02, master03)
# 停掉所有master節點上的kube-apiserver, etcd
#------------------------------------------------------
[master]# cd /etc/kubernetes
[master]# mv manifests manifests.bak
[master01]# scp -rp /etc/kubernetes/manifests.bak.master01 root@lb01:/root/backup_20240205/
[master02]# scp -rp /etc/kubernetes/manifests.bak.master02 root@lb01:/root/backup_20240205/
[master03]# scp -rp /etc/kubernetes/manifests.bak.master03 root@lb01:/root/backup_20240205/
[master]# crictl ps
(確認相對應的Pod都已經完全停止)

[master]# mv /var/lib/etcd /var/lib/etcd.bak
[master01]# scp -rp /var/lib/etcd.bak.master01 root@lb01:/root/backup_20240205/
[master02]# scp -rp /var/lib/etcd.bak.master02 root@lb01:/root/backup_20240205/
[master03]# scp -rp /var/lib/etcd.bak.master03 root@lb01:/root/backup_20240205/
=> 執行`etcdctl member list`會出現錯誤訊息

※ 停止前

※ 停止後

#------------------------------------------------------
# S3-3. 還原ETCD (master01,master02, master03)
# 要用同一份snapshot還原
#------------------------------------------------------
[master]# scp -rp /root/backup_20240205 root@master02.test.example.poc:/root/
[master]# scp -rp /root/backup_20240205 root@master03.test.example.poc:/root/

#------------------------------------------------------
# S3-4. 執行還原 (master01)
#------------------------------------------------------
[master01]# ETCDCTL_API=3 etcdctl snapshot restore /root/backup_20240205/snap-20240205.db \
--endpoints=10.107.88.12:2379 \
--name=master01.test.example.poc \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--initial-advertise-peer-urls=https://10.107.88.12:2380 \
--initial-cluster-token=etcd-cluster-0 \
--initial-cluster=master01.test.example.poc=https://10.107.88.12:2380,master02.test.example.poc=https://10.107.88.13:2380,master03.test.example.poc=https://10.107.88.14:2380 \
--data-dir=/var/lib/etcd

[master01]# mv /etc/kubernetes/manifests.bak /etc/kubernetes/manifests

#------------------------------------------------------
# S3-5. 執行還原 (master02)
#------------------------------------------------------
[master]# ETCDCTL_API=3 etcdctl snapshot restore /root/backup_20240205/snap-20240205.db \
--endpoints=10.107.88.13:2379 \
--name=master02.test.example.poc \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--initial-advertise-peer-urls=https://10.107.88.13:2380 \
--initial-cluster-token=etcd-cluster-0 \
--initial-cluster=master01.test.example.poc=https://10.107.88.12:2380,master02.test.example.poc=https://10.107.88.13:2380,master03.test.example.poc=https://10.107.88.14:2380 \
--data-dir=/var/lib/etcd

[master02]# mv /etc/kubernetes/manifests.bak /etc/kubernetes/manifests

#------------------------------------------------------
# S3-6. 執行還原 (master03)
#------------------------------------------------------
[master]# ETCDCTL_API=3 etcdctl snapshot restore /root/backup_20240205/snap-20240205.db \
--endpoints=10.107.88.14:2379 \
--name=master03.test.example.poc \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--initial-advertise-peer-urls=https://10.107.88.14:2380 \
--initial-cluster-token=etcd-cluster-0 \
--initial-cluster=master01.test.example.poc=https://10.107.88.12:2380,master02.test.example.poc=https://10.107.88.13:2380,master03.test.example.poc=https://10.107.88.14:2380 \
--data-dir=/var/lib/etcd

[master03]# mv /etc/kubernetes/manifests.bak /etc/kubernetes/manifests

#------------------------------------------------------
# S3-6. 確認
#------------------------------------------------------
[master]# crictl ps
[master]# kubectl get podn -n kube-system
[master]# ETCDCTL_API=3 etcdctl --endpoints 10.107.88.12:2379,10.107.88.13:2379,10.107.88.14:2379 \
--cert="/etc/kubernetes/pki/etcd/server.crt" \
--key="/etc/kubernetes/pki/etcd/server.key" \
--cacert="/etc/kubernetes/pki/etcd/ca.crt" \
member list --write-out=table