k8s 憑證更新 (kubeadm)

每年一次的 kubeadm 憑證更新，紀錄一下 ヾ(•ω•`)o

Photo by FlyD on Unsplash

Certificate Management with kubeadm

關於 k8s 憑證

• 由 kubeadm 產生的 client certificates 會在一年後過期
• 當 control-plane 更新時，kubeadm 會自動更新所有的 certificates，所以你可以定期更新 (小於1年)就可以不用擔心過期惹 (๑•̀ㅂ•́)و✧

確認 client certificates

certificates 相關的 path

• 在 /etc/kubernetes/pki 資料夾下
• 嵌入在 kubeconfig 中的檔案 (admin.conf, controller-manager.conf, scheduler.conf)

從檔案位置

像是下面去年的 certificates，將會在 2024/2/7 過期

/etc/kubernetes/pki

kubeconfig

(推薦) 透過 kubeadm 確認

kubeadm certs check-expiration

手動更新 certificates

A. renew certs

kubeadm certs renew all

因為我們的 etcd 是用 systemctl 啟動的，所以不用重啟

B. 重啟 static pods

因為 static pod 不是由 apiserver，由 kubelet 所控制，因此 kubectl 沒辦法 delete/restart 它們，所以需要手動重啟，讓他們得以被 apiserver 有權限可以操作

1. 將 static pods 的 manifests 都移走
2. 等待 kubeletConfiguration 中的 fileCheckFrequency 時間，讓 kubelet 清掉 static pod (官方說大概 20 秒，我自己測試是接近 30 秒)
3. (optional) 可以透過觀察 apiserver 是否被關掉來確定 static pod 已經被偵測且移除
4. 將 manifests 重新搬回來，基本上應該是不到五秒 static pod 就長出來了!
5. (optional) 可以透過觀察 apiserver 是否被成功恢復來確定 static pod 已經被偵測且長回來

# stop static pod (apiserver/contoller-manager/scheduler)
mkdir -p /tmp/manifests
mv /etc/kubernetes/manifests/*yaml /tmp/manifests

# wait fileCheckFrequency time (nearly 20s). Wait for disapper
watch "nerdctl ps | grep apiserver"

# start static pod
cp /tmp/manifests/*.yaml /etc/kubernetes/manifests

# monitor Wait for appear
watch "nerdctl ps | grep apiserver"

C. 更新 kubeconfig

拿 admin.conf 更新 kubeconfig

sudo cp /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

D. DONE & verify

kubeadm certs check-expiration

Additional: 自己寫的 shell script，可以一次跑完上面的所有步驟

# certs renew
echo "== certs renew"
kubeadm certs renew all
 
# stop static pod (apiserver/contoller-manager/scheduler)
echo "== stop static pods"
mkdir -p /tmp/manifests
mv /etc/kubernetes/manifests/*yaml /tmp/manifests
 
# wait fileCheckFrequency time (nearly 20s). Wait for disapper
echo "Wait for static pods closed (watch kube-apiserver container)..."
while true; do
  numLines=$(nerdctl ps | grep "kube-apiserver" | wc -l)
  if [ "$numLines" = "0" ]; then
    break
  fi
done
 
echo "!!static pod has been removed"
 
# start static pod
echo "== Start static pod"
cp /tmp/manifests/*.yaml /etc/kubernetes/manifests
 
# montior static pod
echo "== Initial static pod"
echo "Wait for static pods start(watch kube-apiserver container)..."
while true; do
  numLines=$(nerdctl ps | grep "kube-apiserver" | wc -l)
  if [ "$numLines" != "0" ]; then
    break
  fi
done
echo "!! static pod has started"
 
# update kubeconfig
echo "== update kubeconfig"
sudo cp /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
 
 
# display certs
kubeadm certs check-expiration