kubernetes 인증서 만료되어 kubectl 실행 안될 때

Unable to connect to the server: x509: certificate has expired or is not yet valid: current time 2023-10-18T14:59:15+09:00 is after 2023-10-17T01:26:03Z

위와 같은 에러 발생하며 kubectl 명령을 사용할 수 없는 상황이 발생했습니다. 영문 경고메세지를 대충 해석해보면 느낌이 오시겠지만, kubernetes 인증서 기간이 만료되었다는 내용의 경고입니다.

[root@Minhang02 pki]# kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[check-expiration] Error reading configuration from the Cluster. Falling back to default configuration

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Oct 17, 2023 01:26 UTC   <invalid>       ca                      no      
apiserver                  Oct 17, 2023 01:26 UTC   <invalid>       ca                      no      
apiserver-etcd-client      Oct 17, 2023 01:26 UTC   <invalid>       etcd-ca                 no      
apiserver-kubelet-client   Oct 17, 2023 01:26 UTC   <invalid>       ca                      no      
controller-manager.conf    Oct 17, 2023 01:26 UTC   <invalid>       ca                      no      
etcd-healthcheck-client    Oct 17, 2023 01:26 UTC   <invalid>       etcd-ca                 no      
etcd-peer                  Oct 17, 2023 01:26 UTC   <invalid>       etcd-ca                 no      
etcd-server                Oct 17, 2023 01:26 UTC   <invalid>       etcd-ca                 no      
front-proxy-client         Oct 17, 2023 01:26 UTC   <invalid>       front-proxy-ca          no      
scheduler.conf             Oct 17, 2023 01:26 UTC   <invalid>       ca                      no      

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Oct 14, 2032 01:26 UTC   8y              no      
etcd-ca                 Oct 14, 2032 01:26 UTC   8y              no      
front-proxy-ca          Oct 14, 2032 01:26 UTC   8y              no

-> kubeadm certs check-expiration 명령어를 통해 kubernetes 인증서 만료 일자를 확인해볼 수 있습니다. 확인해보니 이미 기간이 만료되어 있는 상태였습니다. (개인적으로 테스트용으로 구성해 놓았던 서버라 이런 사태가…)

[root@Minhang02 pki]# kubeadm certs renew all
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[renew] Error reading configuration from the Cluster. Falling back to default configuration

certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed

Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.

-> kubeadm certs renew all 명령어를 통해 kubernetes 인증서를 갱신할 수 있습니다.

갱신하고 다시 kubernetes 인증서 정보를 확인해보시면,

[root@Minhang02 pki]# kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[check-expiration] Error reading configuration from the Cluster. Falling back to default configuration

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Oct 17, 2024 06:21 UTC   364d            ca                      no      
apiserver                  Oct 17, 2024 06:21 UTC   364d            ca                      no      
apiserver-etcd-client      Oct 17, 2024 06:21 UTC   364d            etcd-ca                 no      
apiserver-kubelet-client   Oct 17, 2024 06:21 UTC   364d            ca                      no      
controller-manager.conf    Oct 17, 2024 06:21 UTC   364d            ca                      no      
etcd-healthcheck-client    Oct 17, 2024 06:21 UTC   364d            etcd-ca                 no      
etcd-peer                  Oct 17, 2024 06:21 UTC   364d            etcd-ca                 no      
etcd-server                Oct 17, 2024 06:21 UTC   364d            etcd-ca                 no      
front-proxy-client         Oct 17, 2024 06:21 UTC   364d            front-proxy-ca          no      
scheduler.conf             Oct 17, 2024 06:21 UTC   364d            ca                      no      

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Oct 14, 2032 01:26 UTC   8y              no      
etcd-ca                 Oct 14, 2032 01:26 UTC   8y              no      
front-proxy-ca          Oct 14, 2032 01:26 UTC   8y              no

-> 인증서가 갱신되어 이제 364일 남았다고 뜨는 것을 확인 가능합니다.

[root@Minhang02 pki]# kubectl get pod -o wide
NAME                        READY   STATUS    RESTARTS   AGE    IP            NODE        NOMINATED NODE   READINESS GATES
bookstack-77df4879c-6z2v2   1/1     Running   0          28d    10.244.1.42   minhang03   <none>           <none>
bookstack-77df4879c-hqwgp   1/1     Running   0          28d    10.244.2.45   minhang04   <none>           <none>
jupyter-66477949-jvvqk      1/1     Running   0          307d   10.244.2.31   minhang04   <none>           <none>