k8s之配置CNI網絡

k8s-cni-flannel

先介紹一下CNI 相關工作

1. k8s 的 kubelet 在啟動一個容器之前，會先做一些預先檢查以及 csi 的操作
2. 然後 kubelet 調用 CRI 的接口，通過 grpc 的方式和 CRI runtime 通信，告知 CRI 要建立 pod 了
3. 随後 CRI 的 Server 端收到通知後調用 OCI 的接口去真正的執行拉起 Pod 的操作
4. 不過在真的拉起 pod 之前，會先給 pod 建立一個 pause 容器，這個 pause 容器是一個特别小又特别穩定的程序，主要用來挂載網絡命名空間和存儲資源
5. 然後 CRI 調用 CNI 提供的接口，先在/etc/cni/net.d 目錄中擷取網絡插件配置(這個配置由每個插件自己通過 daemonset 的方式拷貝到主機上)，然後把插件的配置作為标準輸入，再把容器的運作時資訊作為環境變量，最後執行插件
6. CNI 插件執行完畢後，把執行結果(結果要包含一些關鍵資訊比如 Pod IP 等)直接幹到标準輸出上
7. CRI 從标準輸出上讀取插件執行結果再做後續操作，後續操作就是拉起真正的容器等

安裝cni插件

每個worker節點上安裝CNI插件驅動,先安裝cni驅動，否則node一直是no ready狀态

#CNI插件下載下傳位址
https://github.com/containernetworking/plugins
# cni插件flannel 會去預設目錄裡找 cni
mkdir /opt/cni/bin /etc/cni/net.d
tar -zxvf cni-xxx.tgz -C /opt/cni/bin

/etc/cni/net.d 是配置目錄           

我這裡用的是docker作為k8s的容器運作時是以插件是cri-docker

cri-docker 的配置裡必須要配置上--network-plugin=cni

下面部署flannel網絡

下載下傳位址：
https://github.com/flannel-io/flannel/blob/master/Documentation/kube-flannel.yml

kubectl apply -f kube-flannel.yaml
kubectl get pods -n kube-system           

#檢視日志沒有權限
kubectl logs -f kube-flannel-ds-s8rfn -n kube-system
Error from server (Forbidden): Forbidden (user=kubernetes, verb=get, resource=nodes, subresource=proxy) ( pods/log kube-flannel-ds-s8rfn)           

授權Apiserver通路kubelet

cat >apiserver-to-kubelet-rbac.yaml<<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:kube-apiserver-to-kubelet
rules:
  - apiGroups:
      - ""
    resources:
      - nodes/proxy
      - nodes/stats
      - nodes/log
      - nodes/spec
      - nodes/metrics
      - pods/log
    verbs:
      - "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:kube-apiserver
  namespace: ""
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-apiserver-to-kubelet
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: kubernetes
EOF

kubectl apply -f apiserver-to-kubelet-rbac.yaml           

上面的flannel可能會啟動失敗

需要注意以下配置

#伺服器上配置好hosts，要不然會啟動失敗
192.168.56.101  master
192.168.56.102  node1
192.168.56.103  node2           

網卡配置

#網關也要配置好，要不然提示找不到預設路由
GATEWAY=192.168.56.1           

準許node節點加入叢集

#檢視csr請求
kubectl get csr --kubeconfig=/opt/kubernetes/conf/kube-controller-manager.kubeconfig  -A -o wide
NAME                                                   AGE   SIGNERNAME                                    REQUESTOR           REQUESTEDDURATION   CONDITION
node-csr-XLjK0kUVExr3qp_Gon55QkKu2zjCLT_yJXi1OUQOMR4   10m   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   <none>              Pending
#準許csr請求，允許node加入k8s叢集
kubectl certificate approve node-csr-XLjK0kUVExr3qp_Gon55QkKu2zjCLT_yJXi1OUQOMR4 --kubeconfig=/opt/kubernetes/conf/kube-controller-manager.kubeconfig
certificatesigningrequest.certificates.k8s.io/node-csr-XLjK0kUVExr3qp_Gon55QkKu2zjCLT_yJXi1OUQOMR4 approved

#檢視node狀态
kubectl get node --kubeconfig=/opt/kubernetes/conf/kube-controller-manager.kubeconfig -o wide -n kube-flannel
NAME     STATUS     ROLES    AGE    VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION           CONTAINER-RUNTIME
master   Ready      <none>   2d1h   v1.24.1   192.168.56.101   <none>        CentOS Linux 7 (Core)   3.10.0-1160.el7.x86_64   docker://20.10.21
node1    Ready      <none>   95m    v1.24.1   192.168.56.102   <none>        CentOS Linux 7 (Core)   3.10.0-1160.el7.x86_64   docker://20.10.21
node2    NotReady   <none>   18s    v1.24.1   192.168.56.103   <none>        CentOS Linux 7 (Core)   3.10.0-1160.el7.x86_64   docker://20.10.21

#檢視 node2上面的flannel日志，發現一直在 PodInitializing
kubectl logs kube-flannel-ds-zwwlk -n kube-flannel --kubeconfig=/opt/kubernetes/conf/kube-controller-manager.kubeconfig 
Defaulted container "kube-flannel" out of: kube-flannel, install-cni-plugin (init), install-cni (init)
Error from server (BadRequest): container "kube-flannel" in pod "kube-flannel-ds-zwwlk" is waiting to start: PodInitializing

#顯示特定pod資源的詳細資訊，我發現pull 鏡像失敗
kubectl describe pod kube-flannel-ds-zwwlk -n kube-flannel --kubeconfig=/opt/kubernetes/conf/kube-controller-manager.kubeconfig
  Warning  FailedCreatePodSandBox  117s                kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed pulling image "registry.k8s.io/pause:3.6": Error response from daemon: Get "https://registry.k8s.io/v2/": dial tcp: lookup registry.k8s.io on [::1]:53: read udp [::1]:38258->[::1]:53: read: connection refused
           

遇到上不去網或者伺服器離線的情況

#先把鏡像下載下傳到本地，通過save把鏡像打包
docker save > xxx.tar
#上傳到伺服器，再load到伺服器上
docker load < xxx.tar           

當然建個鏡像倉庫私服最好不過了