K8S憑證過期：X509: Certificate Has Expired Or Is Not Yet Valid

更新於 2023/07/29發佈於 2023/07/29閱讀時間約 17 分鐘

👨‍💻簡介

今天早上在下kubectl get pods時，突然跳出了以下錯誤

Unable to connect to the server: x509: certificate has expired or is not yet valid

下了kubeadm alpha certs check-expiration之後才發現原來是憑證過期

因此紀錄一下解決過程

Information

environment

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

docker version

Client:
 Version:         1.13.1
 API version:     1.26
 Package version: docker-1.13.1-162.git64e9980.el7.centos.x86_64
 Go version:      go1.10.3
 Git commit:      64e9980/1.13.1
 Built:           Wed Jul  1 14:56:42 2020
 OS/Arch:         linux/amd64

Server:
 Version:         1.13.1
 API version:     1.26 (minimum version 1.12)
 Package version: docker-1.13.1-162.git64e9980.el7.centos.x86_64
 Go version:      go1.10.3
 Git commit:      64e9980/1.13.1
 Built:           Wed Jul  1 14:56:42 2020
 OS/Arch:         linux/amd64
 Experimental:    false

kubeadm version

kubeadm version: &version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-15T16:56:34Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

k8s version

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-15T16:58:53Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-15T16:51:04Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

💡解決步驟

1. 更新憑證

執行以下命令備份現有k8s 憑證

$ mkdir -p $HOME/k8s-old-certs/pki 
$ /bin/cp -p /etc/kubernetes/pki/*.* $HOME/k8s-old-certs/pki 
$ ls -l $HOME/k8s-old-certs/pki/

輸出類似以下內容:

total 56
-rw-r--r-- 1 root root 1090 Jul 27  2022 apiserver-etcd-client.crt
-rw------- 1 root root 1679 Jul 27  2022 apiserver-etcd-client.key
-rw-r--r-- 1 root root 1099 Jul 27  2022 apiserver-kubelet-client.crt
-rw------- 1 root root 1679 Jul 27  2022 apiserver-kubelet-client.key
-rw-r--r-- 1 root root 1229 Jul 27  2022 apiserver.crt
-rw------- 1 root root 1675 Jul 27  2022 apiserver.key
-rw-r--r-- 1 root root 1025 Jul 25  2020 ca.crt
-rw------- 1 root root 1675 Jul 25  2020 ca.key
-rw-r--r-- 1 root root 1038 Jul 27  2022 front-proxy-ca.crt
-rw------- 1 root root 1675 Jul 27  2022 front-proxy-ca.key
-rw-r--r-- 1 root root 1058 Jul 27  2022 front-proxy-client.crt
-rw------- 1 root root 1679 Jul 27  2022 front-proxy-client.key
-rw------- 1 root root 1679 Jul 25  2020 sa.key
-rw------- 1 root root  451 Jul 25  2020 sa.pub

2. 執行以下命令備份現有k8s設定檔

/bin/cp -p /etc/kubernetes/*.conf $HOME/k8s-old-certs 
ls -ltr $HOME/k8s-old-certs

輸出類似以下內容:

total 36
-rw------- 1 root root 5506 Jul 27  2022 kubelet.conf
-rw------- 1 root root 5453 Jul 27  2022 admin.conf
-rw------- 1 root root 5489 Jul 27  2022 controller-manager.conf
-rw------- 1 root root 5437 Jul 27  2022 scheduler.conf
drwxrwxr-x 2 root root 4096 Jul 27 12:05 pki

3. 執行以下命令備份家目錄設定:

mkdir -p $HOME/k8s-old-certs/.kube 
/bin/cp -p ~/.kube/config $HOME/k8s-old-certs/.kube/. 
ls -l $HOME/k8s-old-certs/.kube/.

輸出類似以下內容:

total 8
-rw------- 1 root root 5454 Jul 27  2022 config

4. 執行以下命令更新所有k8s憑證

kubeadm alpha certs renew all

輸出類似以下內容:

[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'

certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed

5. 執行以下命令確認憑證是否更新並顯示364天後過期:

kubeadm alpha certs check-expiration

輸出類似以下內容:

[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Jul 26, 2024 04:14 UTC   364d                                    no
apiserver                  Jul 26, 2024 04:15 UTC   364d            ca                      no
apiserver-etcd-client      Jul 26, 2024 04:15 UTC   364d            etcd-ca                 no
apiserver-kubelet-client   Jul 26, 2024 04:15 UTC   364d            ca                      no
controller-manager.conf    Jul 26, 2024 04:15 UTC   364d                                    no
etcd-healthcheck-client    Jul 26, 2024 04:15 UTC   364d            etcd-ca                 no
etcd-peer                  Jul 26, 2024 04:15 UTC   364d            etcd-ca                 no
etcd-server                Jul 26, 2024 04:15 UTC   364d            etcd-ca                 no
front-proxy-client         Jul 26, 2024 04:15 UTC   364d            front-proxy-ca          no
scheduler.conf             Jul 26, 2024 04:16 UTC   364d                                    no
CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Jul 23, 2030 03:21 UTC   6y              no
etcd-ca                 Jul 23, 2030 03:21 UTC   6y              no
front-proxy-ca          Jul 24, 2032 03:09 UTC   8y              no

2. 確認kubelet設定

這時候下 kubectl get pods可能會噴以下錯誤

error: You must be logged in to the server (Unauthorized)

此時可以使用指令比對一下家目錄的設定檔是否更新

diff $HOME/.kube/config/  /etc/kubernetes/admin.conf

如果沒顯示任何輸出，原因是家目錄的設定檔沒更新，因此必須複製一份新的設定檔到家目錄

cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

接著需要重起kubelet以及相關的docker服務(master與node都要重啟)

systemctl restart docker 
systemctl restart kubelet

如果以上指令沒有輸出表示成功，最後確認是否新的設定檔成功

kubectl get node

NAME      STATUS   ROLES    AGE      VERSION
master    Ready    <none>   3y2d     v1.18.6
worker1   Ready    <none>   3y2d     v1.18.6
worker2   Ready    <none>   3y2d     v1.18.6

以上就是這次更新憑證的過程。

📚Reference

17會員

83內容數

golang

留言0

查看全部

發表第一個留言支持創作者！

Alan的開發者天地的其他內容

WSL環境下使用websocket連線被拒

👨‍💻簡介因在wsl環境下使用websocket通訊協議，並在windows使用postman發生連線被拒嘗試了localhost與127.0.0.1都無效，爬文後找到了一些解決辦法，這邊簡單紀錄一下

#Windows #wsl

《Zeabur - 部署服務從未如此簡單》

👨‍💻簡介從來沒想過部署可以如此的方便快速，第一次接觸到Zeabur的時候覺得他跟一般的雲端服務商差不多，架設網站用個vm之類的，但仔細去摸索後才發現他是個想讓開發人員專注在寫扣這件事上，不需去管任何infra相關事項的一個服務，像是架設wordpress需要sql，就簡單的點兩下即可完

#部署 #專案 #部落格

Kubernetes Deployment：Overview

什麼是 Kubernetes Deployment? 一樣先來個官網解說 A Deployment provides declarative updates for Pods and ReplicaSets. You describe a desired state in a Deployment,

#Kubernetes #devops #Deployment

vscode在wsl環境下連接ssh

👨‍💻Intro 為了實現vscode在wsl環境下使用ssh連接到gce的需求，在爬文後找到一篇解法，因此記錄下來，方便日後查詢 🎯setup 建立ssh.bat檔案，並放在windows下任意位置 2. vscode設定ssh.bat路徑打開vscode的settings -> exte

#VSCode #ssh #wsl

使用Docker安裝GitLab

GitLab不僅是一個源代碼管理工具，它還提供了一個統一的平台，將開發、運營和安全等流程整合在一起。通過GitLab，可以在單一的應用程序中進行專案計劃、源代碼管理、分支控制、CI/CD等操作。不論角色為何，GitLab提供了一個統一的資訊來源，讓你輕鬆地管理和追蹤整個開發流程。它消除了工具鏈的複雜

#Docker #GitLab

Kubernetes Service：Overview

什麼是Kubernetes Service？先來個官網的解說 A Kubernetes Service is an abstraction which defines a logical set of Pods and a policy by which to access them. 白話文就是

#Kubernetes #devops

WSL環境下使用websocket連線被拒

#Windows #wsl

《Zeabur - 部署服務從未如此簡單》

#部署 #專案 #部落格

Kubernetes Deployment：Overview

什麼是 Kubernetes Deployment? 一樣先來個官網解說 A Deployment provides declarative updates for Pods and ReplicaSets. You describe a desired state in a Deployment,

#Kubernetes #devops #Deployment

Kubernetes Service：Overview

什麼是Kubernetes Service？先來個官網的解說 A Kubernetes Service is an abstraction which defines a logical set of Pods and a policy by which to access them. 白話文就是

#Kubernetes #devops

你可能也想看

Google News 追蹤