當k8s叢集開啟了TLS認證後,每個節點的kubelet元件都要使用由kube-apiserver的CA簽發的有效證書才能與kube-apiserver通信;當節點非常多的時候,為每個節點都單獨簽署證書是一件非常繁瑣而又耗時的事情。此時k8s TLS bootstrap功能應運而生。k8s TLS bootstrap功能就是讓kubelet先使用一個預先商定好的低權限的bootstrap token連接配接到kube-apiserver,向kube-apiserver申請證書,然後kube-controller-manager給kubelet動态簽署證書,後續kubelet都将通過動态簽署的證書與kube-apiserver通信。
k8s TLS bootstrap解析-k8s TLS bootstrap流程分析
概述
當k8s叢集開啟了TLS認證後,每個節點的kubelet元件都要使用由kube-apiserver的CA簽發的有效證書才能與kube-apiserver通信;當節點非常多的時候,為每個節點都單獨簽署證書是一件非常繁瑣而又耗時的事情。
此時k8s TLS bootstrap功能應運而生。
k8s TLS bootstrap功能就是讓kubelet先使用一個預先商定好的低權限的bootstrap token連接配接到kube-apiserver,向kube-apiserver申請證書,然後kube-controller-manager給kubelet動态簽署證書,後續kubelet都将通過動态簽署的證書與kube-apiserver通信。
TLS bootstrap涉及元件相關參數
1.kube-apiserver
(1)
--client-ca-file
:認證用戶端證書的CA憑證;
(2)
--enable-bootstrap-token-auth
:設定為true則代表開啟TLS bootstrap特性;
2.kube-controller-manager
(1)
--cluster-signing-cert-file
、
--cluster-signing-key-file
:用來簽發kubelet證書的CA憑證和私鑰,這裡的kubelet證書指的是用來跟kube-apiserver通信,kube-apiserver認證kubelet身份的證書,是以--cluster-signing-cert-file指定的值與kube-apiserver的--client-ca-file指定值一緻,而私鑰則也是對應的私鑰;
(2)
--cluster-signing-duration
:簽發給kubelet的證書有效期;
3.kubelet
(1)
--bootstrap-kubeconfig
:TLS bootstrap的配置檔案,檔案中一般包含bootstrap token和master url等資訊;
(2)
--kubeconfig
:在kubelet的CSR被批複并被kubelet取回時,一個引用所生成的密鑰和所獲得證書的kubeconfig檔案會被寫入到通過 --kubeconfig所指定的檔案路徑下,而證書和密鑰檔案會被放到--cert-dir所指定的目錄中;
(3)
--rotate-certificates
:開啟證書輪換,kubelet在其現有證書即将過期時通過建立新的CSR來輪換其用戶端證書。
詳細流程解析
下面以kubeadm使用k8s TLS bootstrap将一個node節點加入已有的master為例,對TLS bootstrap部分進行詳細流程解析。
1.RBAC相關操作
(1)生成bootstrap token,建立bootstrap token secret;
bootstrap token secret模闆:
apiVersion: v1
data:
auth-extra-groups: system:bootstrappers:kubeadm:default-node-token
expiration: 2022-04-03T11:13:09+08:00
token-id: {token-id}
token-secret: {token-secret}
usage-bootstrap-authentication: "true"
usage-bootstrap-signing: "true"
kind: Secret
metadata:
name: bootstrap-token-{token-id}
namespace: kube-system
type: bootstrap.kubernetes.io/token
關于bootstrap token secret相關的格式說明:
secret的name格式為
bootstrap-token-{token-id}
的格式;
secret的type固定為
bootstrap.kubernetes.io/token
;
secret data中的token-id為6位數字字母組合字元串,token-secret為16位數字字母組合字元串;
secret data中的
auth-extra-groups
定義了bootstrap token所代表使用者所屬的的group,kubeadm使用了
system:bootstrappers:kubeadm:default-node-token
;
secret所對應的bootstrap token為
{token-id}.{token-secret}
;
bootstrap token secret示例:
apiVersion: v1
data:
auth-extra-groups: system:bootstrappers:kubeadm:default-node-token
expiration: 2022-04-03T11:13:09+08:00
token-id: abcdef
token-secret: 0123456789abcdef
usage-bootstrap-authentication: "true"
usage-bootstrap-signing: "true"
kind: Secret
metadata:
name: bootstrap-token-abcdef
namespace: kube-system
type: bootstrap.kubernetes.io/token
上述secret示例中,kubeadm生成的bootstrap token為
abcdef.0123456789abcdef
,其代表的使用者所在使用者組為
system:bootstrappers:kubeadm:default-node-token
;
(2)授予bootstrap token建立CSR證書簽名請求的權限,即授予kubelet建立CSR證書簽名請求的權限;
即建立ClusterRoleBinding -- kubeadm:kubelet-bootstrap
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: kubeadm:kubelet-bootstrap
...
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:node-bootstrapper
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:bootstrappers:kubeadm:default-node-token
kubeadm生成的bootstrap token所代表的使用者所在使用者組為
system:bootstrappers:kubeadm:default-node-token
,是以這裡綁定權限的時候将權限綁定給了使用者組
system:bootstrappers:kubeadm:default-node-token
;
接下來看下被授予的權限ClusterRole -- system:node-bootstrapper
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: system:node-bootstrapper
...
rules:
- apiGroups:
- certificates.k8s.io
resources:
- certificatesigningrequests
verbs:
- create
- get
- list
- watch
(3)授予bootstrap token權限,讓kube-controller-manager可以自動審批其發起的CSR;
即建立ClusterRoleBinding -- kubeadm:node-autoapprove-bootstrap
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: kubeadm:node-autoapprove-bootstrap
...
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:bootstrappers:kubeadm:default-node-token
kubeadm生成的bootstrap token所代表的使用者所在使用者組為
system:bootstrappers:kubeadm:default-node-token
,是以這裡綁定權限的時候将權限綁定給了使用者組
system:bootstrappers:kubeadm:default-node-token
;
接下來看下被授予的權限ClusterRole -- system:certificates.k8s.io:certificatesigningrequests:nodeclient
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
...
rules:
- apiGroups:
- certificates.k8s.io
resources:
- certificatesigningrequests/nodeclient
verbs:
- create
(4)授予kubelet權限,讓kube-controller-manager自動批複kubelet的證書輪換請求;
即建立ClusterRoleBinding -- kubeadm:node-autoapprove-certificate-rotation
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: kubeadm:node-autoapprove-certificate-rotation
...
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:nodes
kubelet建立的CSR使用者名格式為
system:node:<name>
,使用者組為
system:nodes
,是以kube-controller-manager為kubelet生成的證書所代表的使用者所在使用者組為
system:nodes
,是以這裡綁定權限的時候将權限綁定給了使用者組
system:nodes
;
接下來看下被授予的權限ClusterRole -- system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
...
rules:
- apiGroups:
- certificates.k8s.io
resources:
- certificatesigningrequests/selfnodeclient
verbs:
- create
2.啟動kubelet,開始TLS bootstrap
(0)根據bootstrap token以及master url等資訊生成bootstrap-kubeconfig檔案;
(1)啟動kubelet,配置了kubeconfig檔案目錄,但kubeconfig檔案為空,再指定bootstrap-kubeconfig檔案為上述步驟生成的bootstrap-kubeconfig檔案;
(2)kubelet發現配置的kubeconfig檔案為空,則加載bootstrap-kubeconfig檔案,讀取其中的bootstrap token以及master url;
(3)kubelet使用bootstrap token與apiserver通信,建立CSR證書簽名請求;
(4)kube-controller-manager批複CSR證書簽名請求,為其簽發相關證書;
(5)kubelet取回kube-controller-manager生成的相關證書,預設存放在/var/lib/kubelet/pki 目錄下,然後根據證書生成kubeconfig檔案,後續kubelet将使用該kubeconfig檔案與kube-apiserver進行認證通信;
# ls /var/lib/kubelet/pki/
kubelet-client-2022-03-18-14-29-00.pem kubelet-client-current.pem kubelet.crt kubelet.key
kubelet-client-current.pem是個軟鍊,指向kubelet-client-2022-03-18-14-29-00.pem檔案,kubelet-client-2022-03-18-14-29-00.pem檔案名記錄的是證書建立時間,後續kubelet将會根據證書過期時間,在證書臨過期前向kube-apiserver重新申請證書,然後自動輪換該證書;
# cat /etc/kubernetes/kubelet.conf
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0F...
server: https://192.168.1.10:6443
name: test-cluster
contexts:
- context:
cluster: test-cluster
user: system:node:test-cluster-node-1
name: system:node:test-cluster-node-1
current-context: system:node:test-cluster-node-1
kind: Config
preferences: {}
users:
- name: system:node:test-cluster-node-1
user:
client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
client-key: /var/lib/kubelet/pki/kubelet-client-current.pem
3.kubelet自動輪換證書
(1)kubelet在證書接近于過期時自動向kube-apiserver請求更新證書;
(2)kube-controller-manager自動批複,為其簽發新的證書;
(3)kubelet取回kube-controller-manager生成的相關證書,替換掉本地的舊證書,後續kubelet将使用新證書來與kube-apiserver進行認證通信;
總結
最後以一幅圖來總結一下k8s TLS bootstrap的整個流程。