kubernetes 인증서 만료되어 kubectl 실
Unable to connect to the server: x509: certificate has ...
쿠버네티스 공식 다큐멘테이션 내의 kubeadm으로 설치하는 문서를 기반으로 작성하였습니다.
(참조 문서: https://kubernetes.io/ko/docs/setup/production-environment/tools/kubeadm/install-kubeadm/)
- OS는 Amazon Linux 2를 사용하였습니다.
[root@minhangk8s-01 ~]# cat /etc/*release
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"
SUPPORT_END="2025-06-30"
Amazon Linux release 2 (Karoo)- 설치 요구 조건으로 2 CPU, 2GB 메모리가 있습니다. 2 CPU, 4GB 메모리 환경에서 진행했습니다. 총 3대로 구성할 것입니다.
- Swap을 끄는 것과, selinux를 끄라는 조건이 있는데, Amazon Linux 2에는 이미 이 설정이 되어있는 상태라 진행할 필요가 없었습니다. selinux를 끄는 것은, 매뉴얼에 정확히 selinux가 언급된 것은 아니지만, 전체 클러스터간에 전체 네트워크 통신이 가능해야 한다는 표현이 있습니다. selinux를 정교하게 관리할 수 있다면 켜두어도 설치가 가능할 수 있습니다.
- 그 외 세부적인 설치 요구 조건은 공식 다큐멘테이션 참고바랍니다.
kubeadm을 통한 kubernetes 설치
Amazon Linux는 Redhat Linux 계열 OS입니다. Red Hat-based distributions를 따라 설치 진행합니다.
- OS 기본 설정들
- 패키지 최신화를 위해 yum update를 한번 해줍니다.
[root@minhangk8s-01 ~]# yum -y update
Loaded plugins: langpacks, priorities, update-motd
amzn2-core | 3.6 kB 00:00:00
amzn2extra-docker | 2.9 kB 00:00:00
(1/4): amzn2extra-docker/2/x86_64/updateinfo | 13 kB 00:00:00
(2/4): amzn2extra-docker/2/x86_64/primary_db
...
Replaced:
grub2.x86_64 1:2.06-14.amzn2.0.1 grub2-tools.x86_64 1:2.06-14.amzn2.0.1
Complete!- KST 사용과 한글 지원을 위한 locale 설정을 진행합니다.
[root@minhangk8s-01 ~]# rm -f /etc/localtime
[root@minhangk8s-01 ~]# ln -s /usr/share/zoneinfo/Asia/Seoul /etc/localtime
[root@minhangk8s-01 ~]# localedef -v -c -i en_US -f UTF-8 en_US.UTF-8
/usr/share/i18n/locales/en_US:15: non-symbolic character value should not be used
/usr/share/i18n/locales/en_US:16: non-symbolic character value should not be used
/usr/share/i18n/locales/en_US:17: non-symbolic character value should not be used
...
LC_CTYPE: table for map "tolower": 139850954557471 bytes
LC_CTYPE: table for map "totitle": 0 bytes
LC_CTYPE: table for width: 0 bytes
[root@minhangk8s-01 ~]# localedef -v -c -i ko_KR -f UTF-8 ko_KR.UTF-8
/usr/share/i18n/locales/ko_KR:48: non-symbolic character value should not be used
/usr/share/i18n/locales/ko_KR:52: non-symbolic character value should not be used
/usr/share/i18n/locales/ko_KR:55: non-symbolic character value should not be used
...
LC_CTYPE: table for map "tolower": 140086909933599 bytes
LC_CTYPE: table for map "totitle": 0 bytes
LC_CTYPE: table for width: 0 bytes- SELinux를 Permissive 모드로 변경
상술하였듯이, Amazon Linux는 SELinux가 애초에 Disabled 상태입니다. 확인만 한번 하고 넘어갑니다.
[root@minhangk8s-01 ~]# getenforce
Disabled굳이 더 보안 등급이 높은 Permissive 모드로 올리지 않고, disabled 상태로 두고 진행하면 됩니다.
- /etc/hosts 파일 수정
-> /etc/hosts 파일에 hostname 추가합니다.
[root@minhangk8s-01 ~]# vi /etc/hosts
[root@minhangk8s-01 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost6 localhost6.localdomain6
192.168.100.161 MinhangK8S-01
192.168.100.162 MinhangK8S-02
192.168.100.163 MinhangK8S-03
192.168.100.164 MinhangK8S-04
192.168.100.165 MinhangK8S-05-> vi 편집기를 통해 위와 같이 k8s 설치에 사용할 호스트명들을 집어 넣었습니다.
- Container Runtime 설치
Container Runtime으로 Docker를 설치합니다. (나중에 kubernetes 초기화시 필요한 tc 패키지도 같이 설치해줍니다.)
[root@minhangk8s-01 ~]# yum -y install docker tc
Loaded plugins: langpacks, priorities, update-motd
Resolving Dependencies
...
======================================================================================================================================================================================
Package Arch Version Repository Size
======================================================================================================================================================================================
Installing:
docker x86_64 20.10.25-1.amzn2.0.3 amzn2extra-docker 43 M
iproute-tc x86_64 5.10.0-2.amzn2.0.3 amzn2-core 432 k
Installing for dependencies:
containerd x86_64 1.6.19-1.amzn2.0.5 amzn2extra-docker 28 M
libcgroup x86_64 0.41-21.amzn2 amzn2-core 66 k
pigz x86_64 2.3.4-1.amzn2.0.1 amzn2-core 81 k
runc x86_64 1.1.7-4.amzn2 amzn2extra-docker 3.0 M
Transaction Summary
======================================================================================================================================================================================
Install 2 Packages (+4 Dependent packages)
...
Installed:
docker.x86_64 0:20.10.25-1.amzn2.0.3 iproute-tc.x86_64 0:5.10.0-2.amzn2.0.3
Dependency Installed:
containerd.x86_64 0:1.6.19-1.amzn2.0.5 libcgroup.x86_64 0:0.41-21.amzn2 pigz.x86_64 0:2.3.4-1.amzn2.0.1 runc.x86_64 0:1.1.7-4.amzn2
Complete!
[root@minhangk8s-01 ~]# systemctl enable --now docker
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.
-> 도커 구동
[root@minhangk8s-01 ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
-> 도커 명령어 동작하는 것 확인
- cri-dockerd 설치
Docker를 Kubernetes의 CRI(Container Runtime Interface)로 사용하기 위해 cri-dockerd를 추가로 설치해줍니다.
이 부분은 공식 매뉴얼에도 언급이 되어 있는데, Docker가 Kubernetes와 호환되기 위한 CRI 요구 사항을 만족하지 못하므로, Kubernetes 1.24부터 Docker 기본 상태로는 사용이 불가능하고, cri-dockerd라는 별도의 프로젝트를 통해 이 부분이 구현되어 있다는 것입니다.
따라서 cri-dockerd를 추가로 설치해주어야 docker 기반으로 kubernetes 사용이 가능합니다.
- git, go 설치
git clone으로 소스 파일을 가져올 것이며, 컴파일 과정중에 go가 필요하므로 git과 go를 설치합니다.
[root@minhangk8s-01 ~]# yum -y install git go
Loaded plugins: langpacks, priorities, update-motd
amzn2-core | 3.6 kB 00:00:00
Resolving Dependencies
--> Running transaction check
...
======================================================================================================================================================================================
Package Arch Version Repository Size
======================================================================================================================================================================================
Installing:
git x86_64 2.40.1-1.amzn2.0.1 amzn2-core 54 k
golang x86_64 1.20.10-1.amzn2.0.1 amzn2-core 682 k
...
Complete!- 소스파일 내려 받기
[root@minhangk8s-01 ~]# git clone https://github.com/Mirantis/cri-dockerd.git
Cloning into 'cri-dockerd'...
remote: Enumerating objects: 18364, done.
remote: Counting objects: 100% (2923/2923), done.
remote: Compressing objects: 100% (1100/1100), done.
remote: Total 18364 (delta 2062), reused 1904 (delta 1810), pack-reused 15441
Receiving objects: 100% (18364/18364), 42.91 MiB | 20.22 MiB/s, done.
Resolving deltas: 100% (9251/9251), done.- 소스 컴파일
[root@minhangk8s-01 cri-dockerd]# make cri-dockerd
GOARCH= go build -trimpath -ldflags " -s -w -buildid=`git log -1 --pretty='%h'` -X github.com/Mirantis/cri-dockerd/cmd/version.Version=0.3.7 -X github.com/Mirantis/cri-dockerd/cmd/version.PreRelease=`grep -q dev <<< "0.3.7" && echo "pre" || echo ""` -X github.com/Mirantis/cri-dockerd/cmd/version.GitCommit=`git log -1 --pretty='%h'`" -o cri-dockerd
[root@minhangk8s-01 cri-dockerd]# install -o root -g root -m 0755 cri-dockerd /usr/local/bin/cri-dockerd
[root@minhangk8s-01 cri-dockerd]# install packaging/systemd/* /etc/systemd/system
[root@minhangk8s-01 cri-dockerd]# sed -i -e 's,/usr/bin/cri-dockerd,/usr/local/bin/cri-dockerd,' /etc/systemd/system/cri-docker.service
[root@minhangk8s-01 cri-dockerd]# systemctl daemon-reload
[root@minhangk8s-01 cri-dockerd]# systemctl enable --now cri-docker.socket
Created symlink from /etc/systemd/system/sockets.target.wants/cri-docker.socket to /etc/systemd/system/cri-docker.socket.
[root@minhangk8s-01 cri-dockerd]# cd
[root@minhangk8s-01 ~]# - 설치 상태 확인
[root@minhangk8s-01 ~]# systemctl status cri-docker.socket
● cri-docker.socket - CRI Docker Socket for the API
Loaded: loaded (/etc/systemd/system/cri-docker.socket; enabled; vendor preset: disabled)
Active: active (listening) since Wed 2023-11-08 12:26:58 KST; 22s ago
Listen: /run/cri-dockerd.sock (Stream)
Nov 08 12:26:58 minhangk8s-01 systemd[1]: Starting CRI Docker Socket for the API.
Nov 08 12:26:58 minhangk8s-01 systemd[1]: Listening on CRI Docker Socket for the API.
-> 잘 떠있고,
[root@minhangk8s-01 ~]# ls -l /run/cri-dockerd.sock
srw-rw---- 1 root docker 0 Nov 8 12:26 /run/cri-dockerd.sock
-> 소켓 파일도 생겼음
- 쿠버네티스 yum repository 추가
[root@minhangk8s-01 ~]# cat <<EOF | tee /etc/yum.repos.d/kubernetes.repo
> [kubernetes]
> name=Kubernetes
> baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
> enabled=1
> gpgcheck=1
> gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
> exclude=kubelet kubeadm kubectl
> EOF
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-$basearch
enabled=1
gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
-> yum 명령어 사용을 위한 repo 파일 생성. EOF까지가 쭉 명령어임. root 계정으로 진행하고 있어서, 매뉴얼상의 명령어중 sudo는 뺐음
[root@minhangk8s-01 ~]# ls -l /etc/yum.repos.d/kubernetes.repo
-rw-r--r-- 1 root root 282 Nov 8 12:11 /etc/yum.repos.d/kubernetes.repo
-> repo 파일 생성된 것 확인
[root@minhangk8s-01 ~]# cat /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-$basearch
enabled=1
gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
-> 상단에 실행한데로 파일 내용 들어간 것 확인- 이 부분에서 영문 원본 매뉴얼(https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/)과 한글 번역 매뉴얼(https://kubernetes.io/ko/docs/setup/production-environment/tools/kubeadm/install-kubeadm/)간의 차이가 있습니다.
영문 매뉴얼은 yum repository가 pkgs.k8s.io를 가리키고 있고, 한글 매뉴얼은 packages.cloud.google.com을 가리키고 있습니다.
문제는 pkgs.k8s.io의 경우 연관 패키지가 다양하게 구비되어 있지 않은지, 제가 설치를 진행한 환경에서는, 이후 yum install 명령어 수행 과정중에 아래와 같은 에러가 발생했었습니다.
Error: Package: kubeadm-1.28.3-150500.1.1.x86_64 (kubernetes)
Requires: cri-tools >= 1.28.0
Available: cri-tools-1.25.0-1.amzn2.0.1.x86_64 (amzn2-core)
cri-tools = 1.25.0-1.amzn2.0.1
Available: cri-tools-1.26.1-1.amzn2.0.1.x86_64 (amzn2-core)
cri-tools = 1.26.1-1.amzn2.0.1
Available: cri-tools-1.26.1-1.amzn2.0.2.x86_64 (amzn2-core)
cri-tools = 1.26.1-1.amzn2.0.2
Available: cri-tools-1.26.1-1.amzn2.0.3.x86_64 (amzn2-core)
cri-tools = 1.26.1-1.amzn2.0.3
You could try using --skip-broken to work around the problem
You could try running: rpm -Va --nofiles --nodigest- 같은 에러가 발생하는 분이 있으시다면, yum repository를 packages.cloud.google.com으로 변경해보시기 바랍니다.
- kubelet, kubeadm, kubectl설치
[root@minhangk8s-01 ~]# yum -y install kubelet kubeadm kubectl --disableexclude=kubernetes
Loaded plugins: langpacks, priorities, update-motd
kubernetes | 1.4 kB 00:00:00
kubernetes/x86_64/primary | 137 kB 00:00:00
kubernetes 1022/1022
11 packages excluded due to repository priority protections
Resolving Dependencies
--> Running transaction check
---> Package kubeadm.x86_64 0:1.28.2-0 will be installed
--> Processing Dependency: kubernetes-cni >= 0.8.6 for package: kubeadm-1.28.2-0.x86_64
--> Processing Dependency: cri-tools >= 1.19.0 for package: kubeadm-1.28.2-0.x86_64
---> Package kubectl.x86_64 0:1.28.2-0 will be installed
---> Package kubelet.x86_64 0:1.28.2-0 will be installed
--> Processing Dependency: socat for package: kubelet-1.28.2-0.x86_64
--> Processing Dependency: ebtables for package: kubelet-1.28.2-0.x86_64
--> Processing Dependency: conntrack for package: kubelet-1.28.2-0.x86_64
--> Running transaction check
---> Package conntrack-tools.x86_64 0:1.4.4-5.amzn2.2 will be installed
--> Processing Dependency: libnetfilter_cthelper.so.0(LIBNETFILTER_CTHELPER_1.0)(64bit) for package: conntrack-tools-1.4.4-5.amzn2.2.x86_64
--> Processing Dependency: libnetfilter_cttimeout.so.1(LIBNETFILTER_CTTIMEOUT_1.0)(64bit) for package: conntrack-tools-1.4.4-5.amzn2.2.x86_64
--> Processing Dependency: libnetfilter_cttimeout.so.1(LIBNETFILTER_CTTIMEOUT_1.1)(64bit) for package: conntrack-tools-1.4.4-5.amzn2.2.x86_64
--> Processing Dependency: libnetfilter_cthelper.so.0()(64bit) for package: conntrack-tools-1.4.4-5.amzn2.2.x86_64
--> Processing Dependency: libnetfilter_cttimeout.so.1()(64bit) for package: conntrack-tools-1.4.4-5.amzn2.2.x86_64
--> Processing Dependency: libnetfilter_queue.so.1()(64bit) for package: conntrack-tools-1.4.4-5.amzn2.2.x86_64
---> Package cri-tools.x86_64 0:1.26.1-1.amzn2.0.3 will be installed
---> Package ebtables.x86_64 0:2.0.10-16.amzn2.0.1 will be installed
---> Package kubernetes-cni.x86_64 0:1.2.0-0 will be installed
---> Package socat.x86_64 0:1.7.3.2-2.amzn2.0.1 will be installed
--> Running transaction check
---> Package libnetfilter_cthelper.x86_64 0:1.0.0-10.amzn2.1 will be installed
---> Package libnetfilter_cttimeout.x86_64 0:1.0.0-6.amzn2.1 will be installed
---> Package libnetfilter_queue.x86_64 0:1.0.2-2.amzn2.0.2 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
======================================================================================================================================================================================
Package Arch Version Repository Size
======================================================================================================================================================================================
Installing:
kubeadm x86_64 1.28.2-0 kubernetes 11 M
kubectl x86_64 1.28.2-0 kubernetes 11 M
kubelet x86_64 1.28.2-0 kubernetes 21 M
Installing for dependencies:
conntrack-tools x86_64 1.4.4-5.amzn2.2 amzn2-core 186 k
cri-tools x86_64 1.26.1-1.amzn2.0.3 amzn2-core 18 M
ebtables x86_64 2.0.10-16.amzn2.0.1 amzn2-core 122 k
kubernetes-cni x86_64 1.2.0-0 kubernetes 17 M
libnetfilter_cthelper x86_64 1.0.0-10.amzn2.1 amzn2-core 18 k
libnetfilter_cttimeout x86_64 1.0.0-6.amzn2.1 amzn2-core 18 k
libnetfilter_queue x86_64 1.0.2-2.amzn2.0.2 amzn2-core 24 k
socat x86_64 1.7.3.2-2.amzn2.0.1 amzn2-core 291 k
Transaction Summary
======================================================================================================================================================================================
Install 3 Packages (+8 Dependent packages)
Total download size: 78 M
Installed size: 326 M
Downloading packages:
(1/11): conntrack-tools-1.4.4-5.amzn2.2.x86_64.rpm | 186 kB 00:00:00
(2/11): ebtables-2.0.10-16.amzn2.0.1.x86_64.rpm | 122 kB 00:00:00
(3/11): cri-tools-1.26.1-1.amzn2.0.3.x86_64.rpm | 18 MB 00:00:02
warning: /var/cache/yum/x86_64/2/kubernetes/packages/cee73f8035d734e86f722f77f1bf4e7d643e78d36646fd000148deb8af98b61c-kubeadm-1.28.2-0.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID 3e1ba8d5: NOKEY
Public key for cee73f8035d734e86f722f77f1bf4e7d643e78d36646fd000148deb8af98b61c-kubeadm-1.28.2-0.x86_64.rpm is not installed
(4/11): cee73f8035d734e86f722f77f1bf4e7d643e78d36646fd000148deb8af98b61c-kubeadm-1.28.2-0.x86_64.rpm | 11 MB 00:00:03
(5/11): a24e42254b5a14b67b58c4633d29c27370c28ed6796a80c455a65acc813ff374-kubectl-1.28.2-0.x86_64.rpm | 11 MB 00:00:03
(6/11): libnetfilter_cthelper-1.0.0-10.amzn2.1.x86_64.rpm | 18 kB 00:00:00
(7/11): libnetfilter_cttimeout-1.0.0-6.amzn2.1.x86_64.rpm | 18 kB 00:00:00
(8/11): socat-1.7.3.2-2.amzn2.0.1.x86_64.rpm | 291 kB 00:00:00
(9/11): libnetfilter_queue-1.0.2-2.amzn2.0.2.x86_64.rpm | 24 kB 00:00:00
(10/11): e1cae938e231bffa3618f5934a096bd85372ee9b1293081f5682a22fe873add8-kubelet-1.28.2-0.x86_64.rpm | 21 MB 00:00:09
(11/11): 0f2a2afd740d476ad77c508847bad1f559afc2425816c1f2ce4432a62dfe0b9d-kubernetes-cni-1.2.0-0.x86_64.rpm | 17 MB 00:00:09
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 4.6 MB/s | 78 MB 00:00:16
Retrieving key from https://packages.cloud.google.com/yum/doc/yum-key.gpg
Importing GPG key 0x13EDEF05:
Userid : "Rapture Automatic Signing Key (cloud-rapture-signing-key-2022-03-07-08_01_01.pub)"
Fingerprint: a362 b822 f6de dc65 2817 ea46 b53d c80d 13ed ef05
From : https://packages.cloud.google.com/yum/doc/yum-key.gpg
Retrieving key from https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
Importing GPG key 0x3E1BA8D5:
Userid : "Google Cloud Packages RPM Signing Key <gc-team@google.com>"
Fingerprint: 3749 e1ba 95a8 6ce0 5454 6ed2 f09c 394c 3e1b a8d5
From : https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : libnetfilter_cthelper-1.0.0-10.amzn2.1.x86_64 1/11
Installing : libnetfilter_cttimeout-1.0.0-6.amzn2.1.x86_64 2/11
Installing : libnetfilter_queue-1.0.2-2.amzn2.0.2.x86_64 3/11
Installing : conntrack-tools-1.4.4-5.amzn2.2.x86_64 4/11
Installing : ebtables-2.0.10-16.amzn2.0.1.x86_64 5/11
Installing : cri-tools-1.26.1-1.amzn2.0.3.x86_64 6/11
Installing : socat-1.7.3.2-2.amzn2.0.1.x86_64 7/11
Installing : kubernetes-cni-1.2.0-0.x86_64 8/11
Installing : kubelet-1.28.2-0.x86_64 9/11
Installing : kubectl-1.28.2-0.x86_64 10/11
Installing : kubeadm-1.28.2-0.x86_64 11/11
Verifying : kubectl-1.28.2-0.x86_64 1/11
Verifying : socat-1.7.3.2-2.amzn2.0.1.x86_64 2/11
Verifying : kubernetes-cni-1.2.0-0.x86_64 3/11
Verifying : cri-tools-1.26.1-1.amzn2.0.3.x86_64 4/11
Verifying : ebtables-2.0.10-16.amzn2.0.1.x86_64 5/11
Verifying : kubelet-1.28.2-0.x86_64 6/11
Verifying : libnetfilter_queue-1.0.2-2.amzn2.0.2.x86_64 7/11
Verifying : conntrack-tools-1.4.4-5.amzn2.2.x86_64 8/11
Verifying : libnetfilter_cttimeout-1.0.0-6.amzn2.1.x86_64 9/11
Verifying : libnetfilter_cthelper-1.0.0-10.amzn2.1.x86_64 10/11
Verifying : kubeadm-1.28.2-0.x86_64 11/11
Installed:
kubeadm.x86_64 0:1.28.2-0 kubectl.x86_64 0:1.28.2-0 kubelet.x86_64 0:1.28.2-0
Dependency Installed:
conntrack-tools.x86_64 0:1.4.4-5.amzn2.2 cri-tools.x86_64 0:1.26.1-1.amzn2.0.3 ebtables.x86_64 0:2.0.10-16.amzn2.0.1 kubernetes-cni.x86_64 0:1.2.0-0
libnetfilter_cthelper.x86_64 0:1.0.0-10.amzn2.1 libnetfilter_cttimeout.x86_64 0:1.0.0-6.amzn2.1 libnetfilter_queue.x86_64 0:1.0.2-2.amzn2.0.2 socat.x86_64 0:1.7.3.2-2.amzn2.0.1
Complete!
- 설치된 것 확인
[root@minhangk8s-01 ~]# kubelet --version
Kubernetes v1.28.2
[root@minhangk8s-01 ~]# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"28", GitVersion:"v1.28.2", GitCommit:"89a4ea3e1e4ddd7f7572286090359983e0387b2f", GitTreeState:"clean", BuildDate:"2023-09-13T09:34:32Z", GoVersion:"go1.20.8", Compiler:"gc", Platform:"linux/amd64"}
[root@minhangk8s-01 ~]# kubectl version
Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
The connection to the server localhost:8080 was refused - did you specify the right host or port?- 아직 설정을 완료한 것은 아니라 추가적인 메세지가 발생하기는 하지만, 모든 명령어가 먹히고 있고 버전 정보도 확인이 됩니다.
- kubelet 구동
[root@minhangk8s-01 ~]# systemctl enable --now kubelet
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.
-> systemctl에 enable 시켜두어 매번 재부팅 때마다 자동으로 띄우도록 합니다.
[root@minhangk8s-01 ~]# ps -ef | grep kubelet
root 51718 35655 0 12:15 pts/1 00:00:00 grep --color=auto kubelet
-> 헌데 확인해보면 프로세스가 떠있지 않는 상태입니다.
[root@minhangk8s-01 ~]# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Drop-In: /usr/lib/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: activating (auto-restart) (Result: exit-code) since Wed 2023-11-08 12:15:56 KST; 325ms ago
Docs: https://kubernetes.io/docs/
Process: 51719 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=1/FAILURE)
Main PID: 51719 (code=exited, status=1/FAILURE)
Nov 08 12:15:56 minhangk8s-01 systemd[1]: kubelet.service: main process exited, code=exited, status=1/FAILURE
Nov 08 12:15:56 minhangk8s-01 systemd[1]: Unit kubelet.service entered failed state.
Nov 08 12:15:56 minhangk8s-01 systemd[1]: kubelet.service failed.
-> status 확인해보면 뭔가 에러 발생해서 뜨지 않고 있다는 것을 알 수 있습니다. 아직초기화를 진행하지 않아서 그렇습니다.kubeadm을 통한 kubernetes 초기화
(참조 문서: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/ : 이 매뉴얼은 공식 다큐멘테이션에 따로 번역본이 없습니다.)
- kubeadm으로 init 명령 실행하기
[root@minhangk8s-01 ~]# kubeadm init --cri-socket=/run/cri-dockerd.sock --pod-network-cidr=10.244.0.0/16
W1108 12:29:05.512108 51329 initconfiguration.go:120] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/run/cri-dockerd.sock". Please update your configuration!
[init] Using Kubernetes version: v1.28.3
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
W1108 12:29:44.150350 51329 checks.go:835] detected that the sandbox image "registry.k8s.io/pause:3.6" of the container runtime is inconsistent with that used by kubeadm. It is recommended that using "registry.k8s.io/pause:3.9" as the CRI sandbox image.
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local minhangk8s-01] and IPs [10.96.0.1 192.168.100.61]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost minhangk8s-01] and IPs [192.168.100.61 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost minhangk8s-01] and IPs [192.168.100.61 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 9.002834 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node minhangk8s-01 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node minhangk8s-01 as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: u8tzqa.cmcz3h25j7p71nop
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.100.61:6443 --token u8tzqa.cmcz3h25j7p71nop \
--discovery-token-ca-cert-hash sha256:6706ae21c7bf93aad4bc3dae8194d5e4a845bf0b5f2fbce03b5666b68f7a3d86
-> 정상적으로 초기화 완료
-> 설치 옵션으로 넣어준 cri-socket은 Kubernetes가 컨테이너와 통신하는데 사용할 CRI를 지정하는 옵션인데, 도커를 런타임으로 사용하기 위해 위에 추가로 설치한 cri-dockerd를 설정한 것입니다.
-> pod-network-cidr은 이름 그대로 파드간의 통신에 사용될 CIDR설정값인데, flannel을 사용할 예정이기에, 10.244.0.0/16으로 설정했습니다. 이 값을 다르게 주면 flannel 설정시 에러 생깁니다.
-> 최하단의 discovery-token-ca-cert-hash 값은 다른 노드 가입시 사용해야 하므로 잘 저장해두어야 합니다.- kubelet 구동 확인
초기화 전에는 떠있지 않았던 kubelet이 정상적으로 떠있는 것 확인합니다.
[root@minhangk8s-01 ~]# ps -ef | grep kubelet
root 52526 52464 3 12:30 ? 00:00:07 kube-apiserver --advertise-address=192.168.100.61 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-account-signing-key-file=/etc/kubernetes/pki/sa.key --service-cluster-ip-range=10.96.0.0/12 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
root 52710 1 1 12:30 ? 00:00:03 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime-endpoint=unix:///run/cri-dockerd.sock --pod-infra-container-image=registry.k8s.io/pause:3.9
root 53198 35655 0 12:33 pts/1 00:00:00 grep --color=auto kubelet
-> 프로세스 떠있는 것 확인
[root@minhangk8s-01 ~]# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Drop-In: /usr/lib/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Wed 2023-11-08 12:30:18 KST; 3min 41s ago
Docs: https://kubernetes.io/docs/
Main PID: 52710 (kubelet)
Tasks: 11
Memory: 32.8M
CGroup: /system.slice/kubelet.service
└─52710 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml -...
Nov 08 12:33:11 minhangk8s-01 kubelet[52710]: E1108 12:33:11.964001 52710 kubelet.go:2855] "Container runtime network not ready" networkReady="NetworkReady=false reas...nitialized"
Nov 08 12:33:18 minhangk8s-01 kubelet[52710]: E1108 12:33:18.264261 52710 kubelet.go:2855] "Container runtime network not ready" networkReady="NetworkReady=false reas...nitialized"
Nov 08 12:33:23 minhangk8s-01 kubelet[52710]: E1108 12:33:23.266017 52710 kubelet.go:2855] "Container runtime network not ready" networkReady="NetworkReady=false reas...nitialized"
Nov 08 12:33:28 minhangk8s-01 kubelet[52710]: E1108 12:33:28.267073 52710 kubelet.go:2855] "Container runtime network not ready" networkReady="NetworkReady=false reas...nitialized"
Nov 08 12:33:33 minhangk8s-01 kubelet[52710]: E1108 12:33:33.268087 52710 kubelet.go:2855] "Container runtime network not ready" networkReady="NetworkReady=false reas...nitialized"
Nov 08 12:33:38 minhangk8s-01 kubelet[52710]: E1108 12:33:38.269858 52710 kubelet.go:2855] "Container runtime network not ready" networkReady="NetworkReady=false reas...nitialized"
Nov 08 12:33:43 minhangk8s-01 kubelet[52710]: E1108 12:33:43.270501 52710 kubelet.go:2855] "Container runtime network not ready" networkReady="NetworkReady=false reas...nitialized"
Nov 08 12:33:48 minhangk8s-01 kubelet[52710]: E1108 12:33:48.271156 52710 kubelet.go:2855] "Container runtime network not ready" networkReady="NetworkReady=false reas...nitialized"
Nov 08 12:33:53 minhangk8s-01 kubelet[52710]: E1108 12:33:53.271692 52710 kubelet.go:2855] "Container runtime network not ready" networkReady="NetworkReady=false reas...nitialized"
Nov 08 12:33:58 minhangk8s-01 kubelet[52710]: E1108 12:33:58.273299 52710 kubelet.go:2855] "Container runtime network not ready" networkReady="NetworkReady=false reas...nitialized"
Hint: Some lines were ellipsized, use -l to show in full.
-> kubelet 서비스 정상 구동중인 것 확인- KUBECONFIG 설정
[root@minhangk8s-01 ~]# kubectl get all
E1108 12:35:40.903079 53277 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E1108 12:35:40.903403 53277 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E1108 12:35:40.904862 53277 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E1108 12:35:40.906244 53277 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E1108 12:35:40.907781 53277 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
The connection to the server localhost:8080 was refused - did you specify the right host or port?
-> 설치가 완료되었음에도 kubectl 명령 결과가 이상하게 출력되는 상태일 것입니다.
[root@minhangk8s-01 ~]# echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile
[root@minhangk8s-01 ~]# source ~/.bash_profile
-> .bash_profile에 KUBECONFIG 환경변수를 추가해줍니다.
[root@minhangk8s-01 ~]# kubectl get all
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 5m45s
-> 이제 kubectl 명령어가 정상적으로 출력됩니다.- 파드와 컨테이너 확인
[root@minhangk8s-01 ~]# kubectl get pod --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-5dd5756b68-gw4q4 0/1 Pending 0 7m35s
kube-system coredns-5dd5756b68-ks2c4 0/1 Pending 0 7m35s
kube-system etcd-minhangk8s-01 1/1 Running 0 7m48s
kube-system kube-apiserver-minhangk8s-01 1/1 Running 0 7m50s
kube-system kube-controller-manager-minhangk8s-01 1/1 Running 0 7m48s
kube-system kube-proxy-bwrjc 1/1 Running 0 7m35s
kube-system kube-scheduler-minhangk8s-01 1/1 Running 0 7m48s
-> kubectl 명령어로 확인한 pod
[root@minhangk8s-01 ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
0c0fe10ee8f0 bfc896cf80fb "/usr/local/bin/kube…" 7 minutes ago Up 7 minutes k8s_kube-proxy_kube-proxy-bwrjc_kube-system_1c4bfd30-7db7-4dc0-89ba-803bb857091e_0
7523f9af8f34 registry.k8s.io/pause:3.6 "/pause" 7 minutes ago Up 7 minutes k8s_POD_kube-proxy-bwrjc_kube-system_1c4bfd30-7db7-4dc0-89ba-803bb857091e_0
c55dc870c16f 6d1b4fd1b182 "kube-scheduler --au…" 7 minutes ago Up 7 minutes k8s_kube-scheduler_kube-scheduler-minhangk8s-01_kube-system_0f5de60f9484ad67f17ef418d6d64bb5_0
d9e662dd268b 73deb9a3f702 "etcd --advertise-cl…" 7 minutes ago Up 7 minutes k8s_etcd_etcd-minhangk8s-01_kube-system_de3eb28788e39fb0f86973a987689bd9_0
348b6daee560 537434729123 "kube-apiserver --ad…" 7 minutes ago Up 7 minutes k8s_kube-apiserver_kube-apiserver-minhangk8s-01_kube-system_2bceb6d4ba4fe82879c476f08a0e8490_0
362ace28655f 10baa1ca1706 "kube-controller-man…" 7 minutes ago Up 7 minutes k8s_kube-controller-manager_kube-controller-manager-minhangk8s-01_kube-system_a6c1dd9fb363d3db6a4f846b945974d8_0
2ee236d3fec5 registry.k8s.io/pause:3.6 "/pause" 7 minutes ago Up 7 minutes k8s_POD_kube-apiserver-minhangk8s-01_kube-system_2bceb6d4ba4fe82879c476f08a0e8490_0
965514b764cf registry.k8s.io/pause:3.6 "/pause" 7 minutes ago Up 7 minutes k8s_POD_kube-scheduler-minhangk8s-01_kube-system_0f5de60f9484ad67f17ef418d6d64bb5_0
f67e70a9d9c6 registry.k8s.io/pause:3.6 "/pause" 7 minutes ago Up 7 minutes k8s_POD_etcd-minhangk8s-01_kube-system_de3eb28788e39fb0f86973a987689bd9_0
a8ae04fa7359 registry.k8s.io/pause:3.6 "/pause" 7 minutes ago Up 7 minutes k8s_POD_kube-controller-manager-minhangk8s-01_kube-system_a6c1dd9fb363d3db6a4f846b945974d8_0
-> docker 명령어로 확인한 container. 도커 기반으로 잘 돌아가고 있음을 알 수 있습니다.복사 붙여넣기를 위한 명령어 모음
위 과정을 따라해보시기 위한 분들을 위해, 편하게 복사&붙여넣기 할 수 있도록, 사용되었던 명령어들만 모아봤습니다.
yum -y update
rm -f /etc/localtime
ln -s /usr/share/zoneinfo/Asia/Seoul /etc/localtime
localedef -v -c -i en_US -f UTF-8 en_US.UTF-8
localedef -v -c -i ko_KR -f UTF-8 ko_KR.UTF-8
getenforce
vi /etc/hosts
cat /etc/hosts
yum -y install docker tc
systemctl enable --now docker
docker ps
yum -y install git go
git clone https://github.com/Mirantis/cri-dockerd.git
cd cri-dockerd
make cri-dockerd
install -o root -g root -m 0755 cri-dockerd /usr/local/bin/cri-dockerd
install packaging/systemd/* /etc/systemd/system
sed -i -e 's,/usr/bin/cri-dockerd,/usr/local/bin/cri-dockerd,' /etc/systemd/system/cri-docker.service
systemctl daemon-reload
systemctl enable --now cri-docker.socket
cd
systemctl status cri-docker.socket
ls -l /run/cri-dockerd.sock
cat <<EOF | tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
enabled=1
gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
EOF
ls -l /etc/yum.repos.d/kubernetes.repo
cat /etc/yum.repos.d/kubernetes.repo
yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
kubelet --version
kubeadm version
kubectl version
systemctl enable --now kubelet
ps -ef | grep kubelet
systemctl status kubelet
kubeadm init --cri-socket=/run/cri-dockerd.sock --pod-network-cidr=10.244.0.0/16
ps -ef | grep kubelet
systemctl status kubelet
echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile
source ~/.bash_profile
kubectl get all
kubectl get pod --all-namespaces
docker ps
