Sodaflow4 K8s 인증서 갱신 방법

kubectl version --short
Client Version: v1.18.19
Server Version: v1.18.19

K8s 갱신 인증서 대상

• apiserver.crt 

• apiserver-etcd-client.crt 

• apiserver-kubelet-client.crt 

• front-proxy-client.crt 

• etcd/healthcheck-client.crt 

• etcd/peer.crt 

• etcd/server.crt 

K8s 인증서 저장위치

cd /etc/kubernetes/pki
total 56
-rw-r--r--. 1 root root 1245 Nov 29 09:51 apiserver.crt
-rw-r--r--. 1 root root 1090 Nov 29 09:51 apiserver-etcd-client.crt
-rw-------. 1 root root 1679 Nov 29 09:51 apiserver-etcd-client.key
-rw-------. 1 root root 1675 Nov 29 09:51 apiserver.key
-rw-r--r--. 1 root root 1099 Nov 29 09:51 apiserver-kubelet-client.crt
-rw-------. 1 root root 1675 Nov 29 09:51 apiserver-kubelet-client.key
-rw-r--r--. 1 root root 1025 Nov 29 09:51 ca.crt
-rw-------. 1 root root 1675 Nov 29 09:51 ca.key
drwxr-xr-x. 2 root root  162 Nov 29 09:51 etcd
-rw-r--r--. 1 root root 1038 Nov 29 09:51 front-proxy-ca.crt
-rw-------. 1 root root 1675 Nov 29 09:51 front-proxy-ca.key
-rw-r--r--. 1 root root 1058 Nov 29 09:51 front-proxy-client.crt
-rw-------. 1 root root 1679 Nov 29 09:51 front-proxy-client.key
-rw-------. 1 root root 1675 Nov 29 09:51 sa.key
-rw-------. 1 root root  451 Nov 29 09:51 sa.pub

root CA 인증서인, ca.crt , front-proxy-ca.crt , etcd/ca.crt는 인증 기간이 10년이기 때문에 갱신 대상이 아닙니다.

1. root 권한으로 모든 MASTER node 에서 실행

kubectl 인증서를 갱신하기 위해서는 root 계정이 필수입니다. 

root 계정이 없다면 /etc/kubernetes 등 갱신에 필요한 디렉토리에 접근이 불가합니다.

또한 chown, chmod 등 user 권한을 등록하기 위해 필요합니다. 

openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text | grep 'Not'

mkdir $HOME/kubernetes-backup
cp -r /etc/kubernetes $HOME/kubernetes-backup/kubernetes-dir-$(date +"%Y%m%d")

kubeadm config view > $HOME/kubernetes-backup/kubeadm.yaml

cd $HOME/kubernetes-backup/
kubeadm alpha certs renew apiserver --config kubeadm.yaml
kubeadm alpha certs renew apiserver-kubelet-client --config kubeadm.yaml
kubeadm alpha certs renew apiserver-etcd-client --config kubeadm.yaml
kubeadm alpha certs renew front-proxy-client --config kubeadm.yaml
kubeadm alpha certs renew etcd-healthcheck-client --config kubeadm.yaml
kubeadm alpha certs renew etcd-peer --config kubeadm.yaml
kubeadm alpha certs renew etcd-server --config kubeadm.yaml

위의 명령어를 실행하면 /etc/kubernetes/pki 하위의 *.crt 파일들이 갱신됩니다. 

rm -f /etc/kubernetes/admin.conf
rm -f /etc/kubernetes/controller-manager.conf
rm -f /etc/kubernetes/kubelet.conf
rm -f /etc/kubernetes/scheduler.conf

ls -al /etc/kubernetes
drwxr-xr-x.  4 root root  125 Dec 28 10:18 .
drwxr-xr-x. 82 root root 8192 Nov 30 11:20 ..
drwxr-xr-x.  2 root root  113 Nov 29 09:51 manifests
drwxr-xr-x.  3 root root 4096 Nov 29 09:51 pki

kubeadm init phase kubeconfig all --config kubeadm.yaml

# 이중화의 경우 etcd 를 생성합니다. 
kubeadm init phase etcd local --config kubeadm.yaml

위 명령어를 실행하면, /etc/kubernetes 디렉터리 하위에 *.conf 파일이 생성됩니다.

ls -al /etc/kubernetes
total 44
drwxr-xr-x.  4 root root  125 Dec 28 10:18 .
drwxr-xr-x. 82 root root 8192 Nov 30 11:20 ..
-rw-------.  1 root root 5454 Nov 29 09:51 admin.conf
-rw-------.  1 root root 5486 Nov 29 09:51 controller-manager.conf
-rw-------.  1 root root 1854 Nov 29 09:51 kubelet.conf
drwxr-xr-x.  2 root root  113 Nov 29 09:51 manifests
drwxr-xr-x.  3 root root 4096 Nov 29 09:51 pki
-rw-------.  1 root root 5438 Nov 29 09:51 scheduler.conf

# 1. root 에게 kubectl 권한을 부여 (선택)
cp /etc/kubernetes/admin.conf /root/.kube/config
chmod -R 777 /root/.kube/config

# 2. user 에게 kubectl 권한을 부여 
cp /etc/kubernetes/admin.conf /home/<user>/.kube/config
chown -R [user:group] /home/<user>/.kube/config
chmod -R 777 /home/<user>/.kube/config

# 2-1. 혹은 <user> 계정으로 로그인 후
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

kubelet pki 재생성 

kubelet 을 재시작하면 다시 생성되는 것을 확인할 수 있습니다.

cd /var/lib/kubelet/
# pki 백업 및 삭제
cp pki pki-backup-$(date +"%Y%m%d")
rm -r pki

sudo systemctl stop kubelet
sudo systemctl stop docker
sudo systemctl daemon-reload
sudo systemctl start docker
sudo systemctl start kubelet

Kubernetes 동작 확인 

running 상태인 것을 확인합니다. 

systemctl status kubelet -l
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since Tue 2022-11-29 13:40:34 KST; 1 months 10 days ago
     Docs: https://kubernetes.io/docs/
 Main PID: 505 (kubelet)
... 중략 ...

2. 대표 MASTER node 에서 실행 

kubeadm token create --print-join-command > <NAS ROOT>/install-worker-join-$(date +"%Y%m%d").sh

cat <NAS ROOT>/install-worker-join-$(date +"%Y%m%d").sh
kubeadm join 192.168.50.221:6443 --token wecr3l.xyfl4d1ogr7avvnj     --discovery-token-ca-cert-hash sha256:d5216e39d9f022945fb6493024e78ae05ab59fd23e7a3919a949e42e766d83a6 

mkdir $HOME/kubernetes-backup
cp -r /etc/kubernetes $HOME/kubernetes-backup/kubernetes-backup-$(date +"%Y%m%d")

systemctl stop kubelet
systemctl daemon-reload

kubelet pki 재생성 

kubelet 을 재시작하면 다시 생성되는 것을 확인할 수 있습니다.

cd /var/lib/kubelet/
# pki 백업 및 삭제
cp pki pki-backup-$(date +"%Y%m%d")
rm -r pki

rm /etc/kubernetes/kubelet.conf
rm /etc/kubernetes/pki/ca.crt

bash <NAS ROOT>/install-worker-join-$(date +"%Y%m%d").sh

pki/ca.crt 는 join 실행 시 master의 ca.crt와 동일한 값으로 생성됩니다.

ls -al /etc/kubernetes
total 4
-rw-------  1 root root 1855 Dec 30 14:26 kubelet.conf
drwxr-xr-x. 2 root root    6 Oct 25 10:39 manifests
drwxr-xr-x. 2 root root   20 Dec 30 14:26 pki

ls -al /etc/kubernetes/pki/
total 4
-rw-r--r-- 1 root root 1025 Dec 30 14:26 ca.crt

Set namespace

모든 node 에 sodaflow 라는 namespace 가 있는지 확인합니다. 

해당 과정은 root 와 user 계정별로 각각 진행합니다. 

kubectl get ns | grep sodaflow

# 없다면 생성
kubectl create namespace sodaflow
kubectl config set-context sodaflow --namespace=sodaflow --cluster=kubernetes --user=kubernetes-admin
kubectl config use-context sodaflow

kubectl delete cm kubectl-config -n sodaflow
kubectl create cm kubectl-config --from-file=/root/.kube/config -n sodaflow

kubectl-config 가 정상적으로 생성된 것을 확인합니다. 

kubectl get cm --all-namespaces |grep kubectl-config
sodaflow      kubectl-config                                            1      7m8s

cd <SETUP ROOT>/sodaflow_app/yamls
kubectl delete -f sodaflow-app.yaml

# pod 가 내려간 것을 확인한 후
kubectl apply -f sodaflow-app.yaml

diff $HOME/kubernetes-backup-$(date +"%Y%m%d")/admin.conf /etc/kubernetes/admin.conf

본 문서는 Kubernetes 1.18.19 버전 기준의 템플릿 입니다.

# 1. 버전 확인 (all node)
kubectl version --short

# 2. 인증서 만료 날짜 확인 (all master node and root)
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text | grep 'Not'

# 모든 인증서 만료 날짜 확인 (all master node and root)
find /etc/kubernetes/pki/ -type f -name "*.crt" -print | xargs -L 1 -t  -i bash -c 'openssl x509  -noout -text -in {} | grep Not'

# 3. k8s conf 파일 및 인증서 파일 모두 백업 (all master node and root)
cp -r /etc/kubernetes $HOME/kubernetes-backup/kubernetes-backup-$(date +"%Y%m%d")

# 4. kubeadm.yaml 파일 생성 (all master node and root)
# 고객사별로 kubeadm.yaml 파일이 이미 생성 되어 있을 경우 해당 파일 사용 및 경로 입력
kubeadm config view > $HOME/kubernetes-backup/kubeadm.yaml

# 5. kubeadm.yaml 파일이 있는 경로로 이동 (all master node and root)
cd $HOME/kubernetes-backup

# 6. 인증서 갱신 (all master node and root)
kubeadm alpha certs renew apiserver --config kubeadm.yaml
kubeadm alpha certs renew apiserver-kubelet-client --config kubeadm.yaml
kubeadm alpha certs renew apiserver-etcd-client --config kubeadm.yaml
kubeadm alpha certs renew front-proxy-client --config kubeadm.yaml
kubeadm alpha certs renew etcd-healthcheck-client --config kubeadm.yaml
kubeadm alpha certs renew etcd-peer --config kubeadm.yaml
kubeadm alpha certs renew etcd-server --config kubeadm.yaml

# 7. Kubernetes 폴더의 conf 파일 삭제 (all master node and root)
rm /etc/kubernetes/admin.conf
rm /etc/kubernetes/controller-manager.conf
rm /etc/kubernetes/kubelet.conf
rm /etc/kubernetes/scheduler.conf

# 8. Kubernetes conf 파일 생성 (all master node and root)
kubeadm init phase kubeconfig all --config kubeadm.yaml
kubeadm init phase etcd local --config kubeadm.yaml

# 9. kubectl 명령서 실행을 위한 admin.conf 파일 복사 (all master node)
cp /etc/kubernetes/admin.conf /root/.kube/config
cp /etc/kubernetes/admin.conf /home/[user]/.kube/config
chown -R [user:group] /home/[user]/.kube/config
chmod -R 777 /home/[user]/.kube/config
# or user 계정으로 로그인 후
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

# 10. kubelet, docker 재시작 (all master node)
sudo systemctl stop kubelet
sudo systemctl stop docker
sudo systemctl daemon-reload
sudo systemctl start docker
sudo systemctl start kubelet

# 11. worker join을 위한 join 명령어 생성 (대표 master node)
kubeadm token create --print-join-command > <NAS ROOT>/install-worker-join-$(date +"%Y%m%d").sh

# Worker Node 작업
# 12. 인증서 백업 (all worker node)
cp -r /etc/kubernetes $HOME/kubernetes-backup/kubernetes-backup-$(date +"%Y%m%d")

# 13. Stop kublet and deamon-reload (all worker node)
systemctl stop kubelet
systemctl daemon-reload

# 14. Kubenetes conf 파일 및 인증서 삭제 (all worker node)
rm /etc/kubernetes/kubelet.conf
rm /etc/kubernetes/pki/ca.crt

# 15. Join worker (all worker node)
bash <NAS ROOT>/install-worker-join-$(date +"%Y%m%d").sh'

# 16. kubectl-config 파일 삭제 및 생성 (대표 master node)
kubectl delete cm kubectl-config -n sodaflow
kubectl create cm kubectl-config --from-file=/root/.kube/config -n sodaflow

# 17. Set namespace (all node)
kubectl create namespace sodaflow
kubectl config set-context sodaflow --namespace=sodaflow --cluster=kubernetes --user=kubernetes-admin
kubectl config use-context sodaflow

# 18. kubectl-config 참조하는 Deployment 삭제 (대표 master node)
cd <SETUP ROOT>/sodaflow_app/yamls
kubectl delete -f sodaflow-app.yaml

# pod 가 내려간 것을 확인한 후 (대표 master node)
# 19. Deployment 실행
kubectl apply -f sodaflow-app.yaml

# 20. admin.conf 인증서 비교
diff $HOME/kubernetes-backup/kubernetes-backup-$(date +"%Y%m%d")/admin.conf /etc/kubernetes/admin.conf