👨💻簡介
今天早上在下kubectl get pods時，突然跳出了以下錯誤
Unable to connect to the server: x509: certificate has expired or is not yet valid下了kubeadm alpha certs check-expiration之後才發現原來是憑證過期
因此紀錄一下解決過程
Information
- environment
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
- docker version
Client:
Version: 1.13.1
API version: 1.26
Package version: docker-1.13.1-162.git64e9980.el7.centos.x86_64
Go version: go1.10.3
Git commit: 64e9980/1.13.1
Built: Wed Jul 1 14:56:42 2020
OS/Arch: linux/amd64
Server:
Version: 1.13.1
API version: 1.26 (minimum version 1.12)
Package version: docker-1.13.1-162.git64e9980.el7.centos.x86_64
Go version: go1.10.3
Git commit: 64e9980/1.13.1
Built: Wed Jul 1 14:56:42 2020
OS/Arch: linux/amd64
Experimental: false
- kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-15T16:56:34Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
- k8s version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-15T16:58:53Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-15T16:51:04Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
💡解決步驟
1. 更新憑證
- 執行以下命令備份現有k8s 憑證
$ mkdir -p $HOME/k8s-old-certs/pki
$ /bin/cp -p /etc/kubernetes/pki/*.* $HOME/k8s-old-certs/pki $ ls -l $HOME/k8s-old-certs/pki/
輸出類似以下內容:
total 56
-rw-r--r-- 1 root root 1090 Jul 27 2022 apiserver-etcd-client.crt
-rw------- 1 root root 1679 Jul 27 2022 apiserver-etcd-client.key
-rw-r--r-- 1 root root 1099 Jul 27 2022 apiserver-kubelet-client.crt
-rw------- 1 root root 1679 Jul 27 2022 apiserver-kubelet-client.key
-rw-r--r-- 1 root root 1229 Jul 27 2022 apiserver.crt
-rw------- 1 root root 1675 Jul 27 2022 apiserver.key
-rw-r--r-- 1 root root 1025 Jul 25 2020 ca.crt
-rw------- 1 root root 1675 Jul 25 2020 ca.key
-rw-r--r-- 1 root root 1038 Jul 27 2022 front-proxy-ca.crt
-rw------- 1 root root 1675 Jul 27 2022 front-proxy-ca.key
-rw-r--r-- 1 root root 1058 Jul 27 2022 front-proxy-client.crt
-rw------- 1 root root 1679 Jul 27 2022 front-proxy-client.key
-rw------- 1 root root 1679 Jul 25 2020 sa.key
-rw------- 1 root root 451 Jul 25 2020 sa.pub
2. 執行以下命令備份現有k8s設定檔
/bin/cp -p /etc/kubernetes/*.conf $HOME/k8s-old-certs ls -ltr $HOME/k8s-old-certs
輸出類似以下內容:
total 36
-rw------- 1 root root 5506 Jul 27 2022 kubelet.conf
-rw------- 1 root root 5453 Jul 27 2022 admin.conf
-rw------- 1 root root 5489 Jul 27 2022 controller-manager.conf
-rw------- 1 root root 5437 Jul 27 2022 scheduler.conf
drwxrwxr-x 2 root root 4096 Jul 27 12:05 pki
3. 執行以下命令備份家目錄設定:
mkdir -p $HOME/k8s-old-certs/.kube
/bin/cp -p ~/.kube/config $HOME/k8s-old-certs/.kube/.
ls -l $HOME/k8s-old-certs/.kube/.
輸出類似以下內容:
total 8
-rw------- 1 root root 5454 Jul 27 2022 config
4. 執行以下命令更新所有k8s憑證
kubeadm alpha certs renew all輸出類似以下內容:
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed
5. 執行以下命令確認憑證是否更新並顯示364天後過期:
kubeadm alpha certs check-expiration
輸出類似以下內容:
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Jul 26, 2024 04:14 UTC 364d no
apiserver Jul 26, 2024 04:15 UTC 364d ca no
apiserver-etcd-client Jul 26, 2024 04:15 UTC 364d etcd-ca no
apiserver-kubelet-client Jul 26, 2024 04:15 UTC 364d ca no
controller-manager.conf Jul 26, 2024 04:15 UTC 364d no
etcd-healthcheck-client Jul 26, 2024 04:15 UTC 364d etcd-ca no
etcd-peer Jul 26, 2024 04:15 UTC 364d etcd-ca no
etcd-server Jul 26, 2024 04:15 UTC 364d etcd-ca no
front-proxy-client Jul 26, 2024 04:15 UTC 364d front-proxy-ca no
scheduler.conf Jul 26, 2024 04:16 UTC 364d no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Jul 23, 2030 03:21 UTC 6y no
etcd-ca Jul 23, 2030 03:21 UTC 6y no
front-proxy-ca Jul 24, 2032 03:09 UTC 8y no
2. 確認kubelet設定
這時候下 kubectl get pods可能會噴以下錯誤
error: You must be logged in to the server (Unauthorized)
此時可以使用指令比對一下家目錄的設定檔是否更新
diff $HOME/.kube/config/ /etc/kubernetes/admin.conf
如果沒顯示任何輸出，原因是家目錄的設定檔沒更新，因此必須複製一份新的設定檔到家目錄
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
接著需要重起kubelet以及相關的docker服務(master與node都要重啟)
systemctl restart docker
systemctl restart kubelet
如果以上指令沒有輸出表示成功，最後確認是否新的設定檔成功
kubectl get node
NAME STATUS ROLES AGE VERSION
master Ready <none> 3y2d v1.18.6
worker1 Ready <none> 3y2d v1.18.6
worker2 Ready <none> 3y2d v1.18.6
以上就是這次更新憑證的過程。
📚Reference
- Certificate Management with kubeadm | Kubernetes
- Kubernetes “x509: certificate has expired or is not yet valid” error | by Guilherme Pereira | Medium
- K8S: Unable To Connect To The Server: X509: Certificate Has Expired Or Is Not Yet Valid | IvoNet
- k8s自签证书过期x509: certificate has expired or is not yet valid报错 — cerberus43 — 博客园 (cnblogs.com)
