master上啟動的容器一直pending

文章目錄

• 【報錯】
• 【分析】
• 【解決】

【報錯】

• 隔壁部門研發環境k8s叢集（單master）的master上 kube-flannel 和 kube-proxy 一直pending，找我們幫忙排查。
• describe 檢視pod 報錯說挂載問題，一個節點上所有pod挂載不上，大機率是kubelet會報錯（當然也可以從journalctl指令過濾錯誤日志）。
• kubelet 報錯如下：

Redirecting to /bin/systemctl status  -l kubelet.service
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since 五 2021-10-08 12:35:54 CST; 53min ago
     Docs: https://kubernetes.io/docs/
 Main PID: 51270 (kubelet)
    Tasks: 16
   Memory: 48.3M
   CGroup: /system.slice/kubelet.service
           └─51270 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.1

10月 08 13:29:51 XL-master kubelet[51270]: E1008 13:29:51.379424   51270 nestedpendingoperations.go:270] Operation for "\"kubernetes.io/secret/e2f8f8ed-6386-4f0b-b7ef-ebde580ba880-flannel-token-bfxdm\" (\"e2f8f8ed-6386-4f0b-b7ef-ebde580ba880\")" failed. No retries permitted until 2021-10-08 13:31:53.37940227 +0800 CST m=+3359.052988748 (durationBeforeRetry 2m2s). Error: "MountVolume.SetUp failed for volume \"flannel-token-bfxdm\" (UniqueName: \"kubernetes.io/secret/e2f8f8ed-6386-4f0b-b7ef-ebde580ba880-flannel-token-bfxdm\") pod \"kube-flannel-ds-amd64-wh59m\" (UID: \"e2f8f8ed-6386-4f0b-b7ef-ebde580ba880\") : couldn't propagate object cache: timed out waiting for the condition"
10月 08 13:29:51 XL-master kubelet[51270]: E1008 13:29:51.382424   51270 reflector.go:125] object-"ingress-nginx"/"ingress-nginx-token-pnlcd": Failed to list *v1.Secret: secrets "ingress-nginx-token-pnlcd" is forbidden: User "system:node:XL-master" cannot list resource "secrets" in API group "" in the namespace "ingress-nginx": no relationship found between node "XL-master" and this object
10月 08 13:29:51 XL-master kubelet[51270]: E1008 13:29:51.580367   51270 reflector.go:125] object-"ingress-nginx"/"ingress-nginx-admission": Failed to list *v1.Secret: secrets "ingress-nginx-admission" is forbidden: User "system:node:XL-master" cannot list resource "secrets" in API group "" in the namespace "ingress-nginx": no relationship found between node "XL-master" and this object
10月 08 13:29:51 XL-master kubelet[51270]: E1008 13:29:51.780370   51270 reflector.go:125] object-"kube-system"/"flannel-token-bfxdm": Failed to list *v1.Secret: secrets "flannel-token-bfxdm" is forbidden: User "system:node:XL-master" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node "XL-master" and this object
10月 08 13:29:51 XL-master kubelet[51270]: I1008 13:29:51.980323   51270 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach
10月 08 13:29:51 XL-master kubelet[51270]: E1008 13:29:51.980714   51270 reflector.go:125] object-"kube-system"/"kube-flannel-cfg": Failed to list *v1.ConfigMap: configmaps "kube-flannel-cfg" is forbidden: User "system:node:XL-master" cannot list resource "configmaps" in API group "" in the namespace "kube-system": no relationship found between node "XL-master" and this object
10月 08 13:29:51 XL-master kubelet[51270]: I1008 13:29:51.986133   51270 kubelet_node_status.go:72] Attempting to register node xl-master
10月 08 13:29:52 XL-master kubelet[51270]: E1008 13:29:52.380341   51270 reflector.go:125] object-"kube-system"/"kube-proxy-token-74txd": Failed to list *v1.Secret: secrets "kube-proxy-token-74txd" is forbidden: User "system:node:XL-master" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node "XL-master" and this object
10月 08 13:29:52 XL-master kubelet[51270]: E1008 13:29:52.580340   51270 reflector.go:125] object-"kube-system"/"kube-proxy": Failed to list *v1.ConfigMap: configmaps "kube-proxy" is forbidden: User "system:node:XL-master" cannot list resource "configmaps" in API group "" in the namespace "kube-system": no relationship found between node "XL-master" and this object
10月 08 13:29:52 XL-master kubelet[51270]: E1008 13:29:52.780279   51270 reflector.go:125] object-"kubesphere-monitoring-system"/"qingcloud": Failed to list *v1.Secret: secrets "qingcloud" is forbidden: User "system:node:XL-master" cannot list resource "secrets" in API group "" in the namespace "kubesphere-monitoring-system": no relationship found between node "XL-master" and this object

           

【分析】

• 重點是這裡

User "system:node:XL-master" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node "XL-master" and this object
           

如上可知

system:node:XL-master

使用者沒有該命名空間的相關權限

• 配置檔案

  檢視

  /etc/kubernetes/kubelet.conf

  檔案，kubelet 确實使用的是

  system:node:XL-master

  使用者。

  具當事部門運維人員說并沒有更換過使用者，權限綁定也沒有人删除過。好吧我們是友軍，不深究了。

【解決】

建立一個

clusterrolebinding

給

system:node:XL-master

使用者綁定

cluster-admin

權限

kubectl create clusterrolebinding 10-18-XL-master --clusterrole=cluster-admin --user=system:node:XL-master
           

說明：10-18-XL-master是 clusterrolebinding 的名字，随便起的。