本文創作與2017年4月，文中有很多錯誤和模糊的地方在很長的時間内沒有在此修正。以下内容都已遷移到了 kubernete-handbook( http://jimmysong.io/kubernetes-handbook )的“最佳實踐”章節中，請以 kubernetes-handbook 中的内容為準，本 書托管在 GitHub 上，位址 https://github.com/rootsongjc/kubernetes-handbook 。
和我一步步部署 kubernetes 叢集
本系列文檔介紹使用二進制部署
kubernetes
叢集的所有步驟，而不是使用
kubeadm
等自動化方式來部署叢集，同時開啟了叢集的TLS安全認證；
在部署的過程中，将詳細列出各元件的啟動參數，給出配置檔案，詳解它們的含義和可能遇到的問題。
部署完成後，你将了解系統各元件的互動原理，進而能快速解決實際問題。
是以本文檔主要适合于那些有一定 kubernetes 基礎，想通過一步步部署的方式來學習和了解系統配置、運作原理的人。
項目代碼中提供了彙總後的markdon和pdf格式的安裝文檔，pdf版本文檔
下載下傳注：本文檔中不包括docker和私有鏡像倉庫的安裝。
提供所有的配置檔案
叢集安裝時所有元件用到的配置檔案，包含在以下目錄中：
- etc： service的環境變量配置檔案
- manifest： kubernetes應用的yaml檔案
- systemd ：systemd serivce配置檔案
叢集詳情
- Kubernetes 1.6.0
- Docker 1.12.5（使用yum安裝）
- Etcd 3.1.5
- Flanneld 0.7 vxlan 網絡
- TLS 認證通信 (所有元件，如 etcd、kubernetes master 和 node)
- RBAC 授權
- kublet TLS BootStrapping
- kubedns、dashboard、heapster(influxdb、grafana)、EFK(elasticsearch、fluentd、kibana) 叢集插件
- 私有docker鏡像倉庫 harbor （請自行部署，harbor提供離線安裝包，直接使用docker-compose啟動即可）
步驟介紹
- 建立 TLS 通信所需的證書和秘鑰
- 建立 kubeconfig 檔案
- 建立三節點的高可用 etcd 叢集
- kubectl指令行工具
- 部署高可用 master 叢集
- 部署 node 節點
- kubedns 插件
- Dashboard 插件
- Heapster 插件
- EFK 插件
一、建立 kubernetes 各元件 TLS 加密通信的證書和秘鑰
kubernetes
系統的各元件需要使用
TLS
證書對通信進行加密，本文檔使用
CloudFlare
的 PKI 工具集
cfssl來生成 Certificate Authority (CA) 和其它證書；
生成的 CA 證書和秘鑰檔案如下：
- ca-key.pem
- ca.pem
- kubernetes-key.pem
- kubernetes.pem
- kube-proxy.pem
- kube-proxy-key.pem
- admin.pem
- admin-key.pem
使用證書的元件如下：
- etcd：使用 ca.pem、kubernetes-key.pem、kubernetes.pem；
- kube-apiserver：使用 ca.pem、kubernetes-key.pem、kubernetes.pem；
- kubelet：使用 ca.pem；
- kube-proxy：使用 ca.pem、kube-proxy-key.pem、kube-proxy.pem；
- kubectl：使用 ca.pem、admin-key.pem、admin.pem；
kube-controller
、
kube-scheduler
目前需要和
kube-apiserver
部署在同一台機器上且使用非安全端口通信，故不需要證書。
安裝 CFSSL
CFSSL
方式一：直接使用二進制源碼包安裝
$ wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
$ chmod +x cfssl_linux-amd64
$ sudo mv cfssl_linux-amd64 /root/local/bin/cfssl
$ wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
$ chmod +x cfssljson_linux-amd64
$ sudo mv cfssljson_linux-amd64 /root/local/bin/cfssljson
$ wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
$ chmod +x cfssl-certinfo_linux-amd64
$ sudo mv cfssl-certinfo_linux-amd64 /root/local/bin/cfssl-certinfo
$ export PATH=/root/local/bin:$PATH 方式二：使用go指令安裝
我們的系統中安裝了Go1.7.5，使用以下指令安裝更快捷：
$go get -u github.com/cloudflare/cfssl/cmd/...
$echo $GOPATH
/usr/local
$ls /usr/local/bin/cfssl*
cfssl cfssl-bundle cfssl-certinfo cfssljson cfssl-newkey cfssl-scan 在
$GOPATH/bin
目錄下得到以cfssl開頭的幾個指令。
建立 CA (Certificate Authority)
建立 CA 配置檔案
$ mkdir /root/ssl
$ cd /root/ssl
$ cfssl print-defaults config > config.json
$ cfssl print-defaults csr > csr.json
$ cat ca-config.json
{ "signing": { "default": { "expiry": "8760h" }, "profiles": { "kubernetes": { "usages": [ "signing", "key encipherment", "server auth", "client auth" ], "expiry": "8760h" } } } } 字段說明
-
ca-config.json
-
signing
CA=TRUE
-
server auth
-
client auth
建立 CA 證書簽名請求
$ cat ca-csr.json
{ "CN": "kubernetes", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "k8s", "OU": "System" } ] } - “CN”：
Common Name
- “O”：
Organization
生成 CA 證書和私鑰
$ cfssl gencert -initca ca-csr.json | cfssljson -bare ca
$ ls ca*
ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem 建立 kubernetes 證書
建立 kubernetes 證書簽名請求
$ cat kubernetes-csr.json
{ "CN": "kubernetes", "hosts": [ "127.0.0.1", "172.20.0.112", "172.20.0.113", "172.20.0.114", "172.20.0.115", "10.254.0.1", "kubernetes", "kubernetes.default", "kubernetes.default.svc", "kubernetes.default.svc.cluster", "kubernetes.default.svc.cluster.local" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "k8s", "OU": "System" } ] } - 如果 hosts 字段不為空則需要指定授權使用該證書的 IP 或域名清單，由于該證書後續被
etcd
kubernetes master
etcd
kubernetes master
kubernetes
kue-apiserver
service-cluster-ip-range
生成 kubernetes 證書和私鑰
$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes
$ ls kuberntes*
kubernetes.csr kubernetes-csr.json kubernetes-key.pem kubernetes.pem 或者直接在指令行上指定相關參數：
$ echo '{"CN":"kubernetes","hosts":[""],"key":{"algo":"rsa","size":2048}}' | cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes -hostname="127.0.0.1,172.20.0.112,172.20.0.113,172.20.0.114,172.20.0.115,kubernetes,kubernetes.default" - | cfssljson -bare kubernetes 建立 admin 證書
建立 admin 證書簽名請求
$ cat admin-csr.json
{ "CN": "admin", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "system:masters", "OU": "System" } ] } - 後續
kube-apiserver
RBAC
kubelet
kube-proxy
Pod
-
kube-apiserver
RBAC
RoleBindings
cluster-admin
system:masters
cluster-admin
kube-apiserver
- OU 指定該證書的 Group 為
system:masters
kubelet
kube-apiserver
system:masters
生成 admin 證書和私鑰
$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin
$ ls admin*
admin.csr admin-csr.json admin-key.pem admin.pem 建立 kube-proxy 證書
建立 kube-proxy 證書簽名請求
$ cat kube-proxy-csr.json
{ "CN": "system:kube-proxy", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "k8s", "OU": "System" } ] } - CN 指定該證書的 User 為
system:kube-proxy
-
kube-apiserver
cluster-admin
system:kube-proxy
system:node-proxier
kube-apiserver
生成 kube-proxy 用戶端證書和私鑰
$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
$ ls kube-proxy*
kube-proxy.csr kube-proxy-csr.json kube-proxy-key.pem kube-proxy.pem 校驗證書
以 kubernetes 證書為例
opsnssl
opsnssl
$ openssl x509 -noout -text -in kubernetes.pem
... Signature Algorithm: sha256WithRSAEncryption
Issuer: C=CN, ST=BeiJing, L=BeiJing, O=k8s, OU=System, CN=Kubernetes Validity Not Before: Apr 5 05:36:00 2017 GMT
Not After : Apr 5 05:36:00 2018 GMT
Subject: C=CN, ST=BeiJing, L=BeiJing, O=k8s, OU=System, CN=kubernetes
...
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Subject Key Identifier:
DD:52:04:43:10:13:A9:29:24:17:3A:0E:D7:14:DB:36:F8:6C:E0:E0
X509v3 Authority Key Identifier:
keyid:44:04:3B:60:BD:69:78:14:68:AF:A0:41:13:F6:17:07:13:63:58:CD
X509v3 Subject Alternative Name:
DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster, DNS:kubernetes.default.svc.cluster.local, IP Address:127.0.0.1, IP Address:172.20.0.112, IP Address:172.20.0.113, IP Address:172.20.0.114, IP Address:172.20.0.115, IP Address:10.254.0.1 ... - 确認
Issuer
ca-csr.json
-
Subject
kubernetes-csr.json
-
X509v3 Subject Alternative Name
kubernetes-csr.json
-
X509v3 Key Usage、Extended Key Usage
ca-config.json
kubernetes
cfssl-certinfo
cfssl-certinfo
$ cfssl-certinfo -cert kubernetes.pem
... { "subject": { "common_name": "kubernetes", "country": "CN", "organization": "k8s", "organizational_unit": "System", "locality": "BeiJing", "province": "BeiJing", "names": [ "CN", "BeiJing", "BeiJing", "k8s", "System", "kubernetes" ] }, "issuer": { "common_name": "Kubernetes", "country": "CN", "organization": "k8s", "organizational_unit": "System", "locality": "BeiJing", "province": "BeiJing", "names": [ "CN", "BeiJing", "BeiJing", "k8s", "System", "Kubernetes" ] }, "serial_number": "174360492872423263473151971632292895707129022309", "sans": [ "kubernetes", "kubernetes.default", "kubernetes.default.svc", "kubernetes.default.svc.cluster", "kubernetes.default.svc.cluster.local", "127.0.0.1", "10.64.3.7", "10.254.0.1" ], "not_before": "2017-04-05T05:36:00Z", "not_after": "2018-04-05T05:36:00Z", "sigalg": "SHA256WithRSA", ... 分發證書
将生成的證書和秘鑰檔案（字尾名為
.pem
）拷貝到所有機器的
/etc/kubernetes/ssl
目錄下備用；
$ sudo mkdir -p /etc/kubernetes/ssl
$ sudo cp *.pem /etc/kubernetes/ssl 參考
- Generate self-signed certificates
- Setting up a Certificate Authority and Creating TLS Certificates
- Client Certificates V/s Server Certificates
- 數字證書及 CA 的掃盲介紹
二、建立 kubeconfig 檔案
kubelet
kube-proxy
等 Node 機器上的程序與 Master 機器的
kube-apiserver
程序通信時需要認證和授權；
kubernetes 1.4 開始支援由
kube-apiserver
為用戶端生成 TLS 證書的
TLS Bootstrapping功能，這樣就不需要為每個用戶端生成證書了；該功能目前僅支援為
kubelet
生成證書；
建立 TLS Bootstrapping Token
Token auth file
Token可以是任意的包涵128 bit的字元串，可以使用安全的随機數發生器生成。
export BOOTSTRAP_TOKEN=$(head -c 16 /dev/urandom | od -An -t x | tr -d ' ')
cat > token.csv <<EOF ${BOOTSTRAP_TOKEN},kubelet-bootstrap,10001,"system:kubelet-bootstrap" EOF 後三行是一句，直接複制上面的腳本運作即可。
将token.csv發到所有機器（Master 和 Node）的
/etc/kubernetes/
目錄。
$cp token.csv /etc/kubernetes/ 建立 kubelet bootstrapping kubeconfig 檔案
$ cd /etc/kubernetes
$ export KUBE_APISERVER="https://172.20.0.113:6443"
$ # 設定叢集參數
$ kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=bootstrap.kubeconfig
$ # 設定用戶端認證參數
$ kubectl config set-credentials kubelet-bootstrap \
--token=${BOOTSTRAP_TOKEN} \
--kubeconfig=bootstrap.kubeconfig
$ # 設定上下文參數
$ kubectl config set-context default \
--cluster=kubernetes \
--user=kubelet-bootstrap \
--kubeconfig=bootstrap.kubeconfig
$ # 設定預設上下文
$ kubectl config use-context default --kubeconfig=bootstrap.kubeconfig -
--embed-certs
true
certificate-authority
bootstrap.kubeconfig
- 設定用戶端認證參數時沒有指定秘鑰和證書，後續由
kube-apiserver
建立 kube-proxy kubeconfig 檔案
$ export KUBE_APISERVER="https://172.20.0.113:6443"
$ # 設定叢集參數
$ kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=kube-proxy.kubeconfig
$ # 設定用戶端認證參數
$ kubectl config set-credentials kube-proxy \
--client-certificate=/etc/kubernetes/ssl/kube-proxy.pem \
--client-key=/etc/kubernetes/ssl/kube-proxy-key.pem \
--embed-certs=true \
--kubeconfig=kube-proxy.kubeconfig
$ # 設定上下文參數
$ kubectl config set-context default \
--cluster=kubernetes \
--user=kube-proxy \
--kubeconfig=kube-proxy.kubeconfig
$ # 設定預設上下文
$ kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig - 設定叢集參數和用戶端認證參數時
--embed-certs
true
certificate-authority
client-certificate
client-key
kube-proxy.kubeconfig
-
kube-proxy.pem
system:kube-proxy
kube-apiserver
cluster-admin
system:kube-proxy
system:node-proxier
kube-apiserver
分發 kubeconfig 檔案
将兩個 kubeconfig 檔案分發到所有 Node 機器的
/etc/kubernetes/
目錄
$ cp bootstrap.kubeconfig kube-proxy.kubeconfig /etc/kubernetes/ 三、建立高可用 etcd 叢集
kuberntes 系統使用 etcd 存儲所有資料，本文檔介紹部署一個三節點高可用 etcd 叢集的步驟，這三個節點複用 kubernetes master 機器，分别命名為
sz-pg-oam-docker-test-001.tendcloud.com
sz-pg-oam-docker-test-002.tendcloud.com
sz-pg-oam-docker-test-003.tendcloud.com
：
- sz-pg-oam-docker-test-001.tendcloud.com：172.20.0.113
- sz-pg-oam-docker-test-002.tendcloud.com：172.20.0.114
- sz-pg-oam-docker-test-003.tendcloud.com：172.20.0.115
TLS 認證檔案
需要為 etcd 叢集建立加密通信的 TLS 證書，這裡複用以前建立的 kubernetes 證書
$ cp ca.pem kubernetes-key.pem kubernetes.pem /etc/kubernetes/ssl - kubernetes 證書的
hosts
下載下傳二進制檔案
到
https://github.com/coreos/etcd/releases
頁面下載下傳最新版本的二進制檔案
$ https://github.com/coreos/etcd/releases/download/v3.1.5/etcd-v3.1.5-linux-amd64.tar.gz
$ tar -xvf etcd-v3.1.4-linux-amd64.tar.gz
$ sudo mv etcd-v3.1.4-linux-amd64/etcd* /root/local/bin 建立 etcd 的 systemd unit 檔案
注意替換
ETCD_NAME
INTERNAL_IP
變量的值；
$ export ETCD_NAME=sz-pg-oam-docker-test-001.tendcloud.com
$ export INTERNAL_IP=172.20.0.113
$ sudo mkdir -p /var/lib/etcd /var/lib/etcd
$ cat > etcd.service <<EOF [Unit] Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target Documentation=https://github.com/coreos [Service] Type=notify WorkingDirectory=/var/lib/etcd/ EnvironmentFile=-/etc/etcd/etcd.conf ExecStart=/root/local/bin/etcd \\ --name ${ETCD_NAME} \\ --cert-file=/etc/kubernetes/ssl/kubernetes.pem \\ --key-file=/etc/kubernetes/ssl/kubernetes-key.pem \\ --peer-cert-file=/etc/kubernetes/ssl/kubernetes.pem \\ --peer-key-file=/etc/kubernetes/ssl/kubernetes-key.pem \\ --trusted-ca-file=/etc/kubernetes/ssl/ca.pem \\ --peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \\ --initial-advertise-peer-urls https://${INTERNAL_IP}:2380 \\ --listen-peer-urls https://${INTERNAL_IP}:2380 \\ --listen-client-urls https://${INTERNAL_IP}:2379,https://127.0.0.1:2379 \\ --advertise-client-urls https://${INTERNAL_IP}:2379 \\ --initial-cluster-token etcd-cluster-0 \\ --initial-cluster sz-pg-oam-docker-test-001.tendcloud.com=https://172.20.0.113:2380,sz-pg-oam-docker-test-002.tendcloud.com=https://172.20.0.114:2380,sz-pg-oam-docker-test-003.tendcloud.com=https://172.20.0.115:2380 \\ --initial-cluster-state new \\ --data-dir=/var/lib/etcd Restart=on-failure RestartSec=5 LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF - 指定
etcd
/var/lib/etcd
/var/lib/etcd
- 為了保證通信安全，需要指定 etcd 的公私鑰(cert-file和key-file)、Peers 通信的公私鑰和 CA 證書(peer-cert-file、peer-key-file、peer-trusted-ca-file)、用戶端的CA憑證（trusted-ca-file）；
- 建立
kubernetes.pem
kubernetes-csr.json
hosts
-
--initial-cluster-state
new
--name
--initial-cluster
完整 unit 檔案見：
etcd.service啟動 etcd 服務
$ sudo mv etcd.service /etc/systemd/system/
$ sudo systemctl daemon-reload
$ sudo systemctl enable etcd
$ sudo systemctl start etcd
$ systemctl status etcd 在所有的 kubernetes master 節點重複上面的步驟，直到所有機器的 etcd 服務都已啟動。
驗證服務
在任一 kubernetes master 機器上執行如下指令：
$ etcdctl \
--ca-file=/etc/kubernetes/ssl/ca.pem \
--cert-file=/etc/kubernetes/ssl/kubernetes.pem \
--key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
cluster-health
2017-04-11 15:17:09.082250 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
2017-04-11 15:17:09.083681 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
member 9a2ec640d25672e5 is healthy: got healthy result from https://172.20.0.115:2379
member bc6f27ae3be34308 is healthy: got healthy result from https://172.20.0.114:2379
member e5c92ea26c4edba0 is healthy: got healthy result from https://172.20.0.113:2379
cluster is healthy 結果最後一行為
cluster is healthy
時表示叢集服務正常。
sz-pg-oam-docker-test-001.tendcloud.com
sz-pg-oam-docker-test-002.tendcloud.com
sz-pg-oam-docker-test-003.tendcloud.com
$ cp ca.pem kubernetes-key.pem kubernetes.pem /etc/kubernetes/ssl -
hosts
https://github.com/coreos/etcd/releases
$ https://github.com/coreos/etcd/releases/download/v3.1.5/etcd-v3.1.5-linux-amd64.tar.gz
$ tar -xvf etcd-v3.1.4-linux-amd64.tar.gz
$ sudo mv etcd-v3.1.4-linux-amd64/etcd* /root/local/bin ETCD_NAME
INTERNAL_IP
$ export ETCD_NAME=sz-pg-oam-docker-test-001.tendcloud.com
$ export INTERNAL_IP=172.20.0.113
$ sudo mkdir -p /var/lib/etcd /var/lib/etcd
$ cat > etcd.service <<EOF [Unit] Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target Documentation=https://github.com/coreos [Service] Type=notify WorkingDirectory=/var/lib/etcd/ EnvironmentFile=-/etc/etcd/etcd.conf ExecStart=/root/local/bin/etcd \\ --name ${ETCD_NAME} \\ --cert-file=/etc/kubernetes/ssl/kubernetes.pem \\ --key-file=/etc/kubernetes/ssl/kubernetes-key.pem \\ --peer-cert-file=/etc/kubernetes/ssl/kubernetes.pem \\ --peer-key-file=/etc/kubernetes/ssl/kubernetes-key.pem \\ --trusted-ca-file=/etc/kubernetes/ssl/ca.pem \\ --peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \\ --initial-advertise-peer-urls https://${INTERNAL_IP}:2380 \\ --listen-peer-urls https://${INTERNAL_IP}:2380 \\ --listen-client-urls https://${INTERNAL_IP}:2379,https://127.0.0.1:2379 \\ --advertise-client-urls https://${INTERNAL_IP}:2379 \\ --initial-cluster-token etcd-cluster-0 \\ --initial-cluster sz-pg-oam-docker-test-001.tendcloud.com=https://172.20.0.113:2380,sz-pg-oam-docker-test-002.tendcloud.com=https://172.20.0.114:2380,sz-pg-oam-docker-test-003.tendcloud.com=https://172.20.0.115:2380 \\ --initial-cluster-state new \\ --data-dir=/var/lib/etcd Restart=on-failure RestartSec=5 LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF -
etcd
/var/lib/etcd
/var/lib/etcd
-
kubernetes.pem
kubernetes-csr.json
hosts
-
--initial-cluster-state
new
--name
--initial-cluster
$ sudo mv etcd.service /etc/systemd/system/
$ sudo systemctl daemon-reload
$ sudo systemctl enable etcd
$ sudo systemctl start etcd
$ systemctl status etcd $ etcdctl \
--ca-file=/etc/kubernetes/ssl/ca.pem \
--cert-file=/etc/kubernetes/ssl/kubernetes.pem \
--key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
cluster-health
2017-04-11 15:17:09.082250 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
2017-04-11 15:17:09.083681 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
member 9a2ec640d25672e5 is healthy: got healthy result from https://172.20.0.115:2379
member bc6f27ae3be34308 is healthy: got healthy result from https://172.20.0.114:2379
member e5c92ea26c4edba0 is healthy: got healthy result from https://172.20.0.113:2379
cluster is healthy cluster is healthy
四、下載下傳和配置 kubectl 指令行工具
本文檔介紹下載下傳和配置 kubernetes 叢集指令行工具 kubelet 的步驟。
下載下傳 kubectl
$ wget https://dl.k8s.io/v1.6.0/kubernetes-client-linux-amd64.tar.gz
$ tar -xzvf kubernetes-client-linux-amd64.tar.gz
$ cp kubernetes/client/bin/kube* /usr/bin/
$ chmod a+x /usr/bin/kube* 建立 kubectl kubeconfig 檔案
$ export KUBE_APISERVER="https://172.20.0.113:6443"
$ # 設定叢集參數
$ kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER}
$ # 設定用戶端認證參數
$ kubectl config set-credentials admin \
--client-certificate=/etc/kubernetes/ssl/admin.pem \
--embed-certs=true \
--client-key=/etc/kubernetes/ssl/admin-key.pem
$ # 設定上下文參數
$ kubectl config set-context kubernetes \
--cluster=kubernetes \
--user=admin
$ # 設定預設上下文
$ kubectl config use-context kubernetes -
admin.pem
system:masters
kube-apiserver
cluster-admin
system:masters
cluster-admin
kube-apiserver
- 生成的 kubeconfig 被儲存到
~/.kube/config
五、部署高可用 kubernetes master 叢集
kubernetes master 節點包含的元件：
- kube-apiserver
- kube-scheduler
- kube-controller-manager
目前這三個元件需要部署在同一台機器上。
-
kube-scheduler
kube-controller-manager
kube-apiserver
- 同時隻能有一個
kube-scheduler
kube-controller-manager
本文檔記錄部署一個三個節點的高可用 kubernetes master 叢集步驟。（後續建立一個 load balancer 來代理通路 kube-apiserver 的請求）
TLS 證書檔案
pem和token.csv證書檔案我們在
TLS證書和秘鑰這一步中已經建立過了。我們再檢查一下。
$ ls /etc/kubernetes/ssl
admin-key.pem admin.pem ca-key.pem ca.pem kube-proxy-key.pem kube-proxy.pem kubernetes-key.pem kubernetes.pem 下載下傳最新版本的二進制檔案
有兩種下載下傳方式
方式一
從
github release 頁面下載下傳釋出版 tarball，解壓後再執行下載下傳腳本
$ wget https://github.com/kubernetes/kubernetes/releases/download/v1.6.0/kubernetes.tar.gz
$ tar -xzvf kubernetes.tar.gz
...
$ cd kubernetes
$ ./cluster/get-kube-binaries.sh
... 方式二
CHANGELOG
client
或
server
tarball 檔案
server
的 tarball
kubernetes-server-linux-amd64.tar.gz
已經包含了
client
(
kubectl
) 二進制檔案，是以不用單獨下載下傳
kubernetes-client-linux-amd64.tar.gz
$ # wget https://dl.k8s.io/v1.6.0/kubernetes-client-linux-amd64.tar.gz
$ wget https://dl.k8s.io/v1.6.0/kubernetes-server-linux-amd64.tar.gz
$ tar -xzvf kubernetes-server-linux-amd64.tar.gz
...
$ cd kubernetes
$ tar -xzvf kubernetes-src.tar.gz 将二進制檔案拷貝到指定路徑
$ cp -r server/bin/{kube-apiserver,kube-controller-manager,kube-scheduler,kubectl,kube-proxy,kubelet} /root/local/bin/ 配置和啟動 kube-apiserver
建立 kube-apiserver的service配置檔案
serivce配置檔案
/usr/lib/systemd/system/kube-apiserver.service
内容：
[Unit] Description=Kubernetes API Service Documentation=https://github.com/GoogleCloudPlatform/kubernetes After=network.target
After=etcd.service
[Service] EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/apiserver
ExecStart=/usr/bin/kube-apiserver \
$KUBE_LOGTOSTDERR \
$KUBE_LOG_LEVEL \
$KUBE_ETCD_SERVERS \
$KUBE_API_ADDRESS \
$KUBE_API_PORT \
$KUBELET_PORT \
$KUBE_ALLOW_PRIV \
$KUBE_SERVICE_ADDRESSES \
$KUBE_ADMISSION_CONTROL \
$KUBE_API_ARGS
Restart=on-failure
Type=notify
LimitNOFILE=65536 [Install] WantedBy=multi-user.target /etc/kubernetes/config
檔案的内容為：
### # kubernetes system config # # The following values are used to configure various aspects of all # kubernetes services, including # # kube-apiserver.service # kube-controller-manager.service # kube-scheduler.service # kubelet.service # kube-proxy.service # logging to stderr means we get it in the systemd journal KUBE_LOGTOSTDERR="--logtostderr=true" # journal message level, 0 is debug KUBE_LOG_LEVEL="--v=0" # Should this cluster be allowed to run privileged docker containers KUBE_ALLOW_PRIV="--allow-privileged=true" # How the controller-manager, scheduler, and proxy find the apiserver #KUBE_MASTER="--master=http://sz-pg-oam-docker-test-001.tendcloud.com:8080" KUBE_MASTER="--master=http://172.20.0.113:8080" 該配置檔案同時被kube-apiserver、kube-controller-manager、kube-scheduler、kubelet、kube-proxy使用。
apiserver配置檔案
/etc/kubernetes/apiserver
内容為：
### ## kubernetes system config ## ## The following values are used to configure the kube-apiserver ## # ## The address on the local server to listen to. #KUBE_API_ADDRESS="--insecure-bind-address=sz-pg-oam-docker-test-001.tendcloud.com" KUBE_API_ADDRESS="--advertise-address=172.20.0.113 --bind-address=172.20.0.113 --insecure-bind-address=172.20.0.113" # ## The port on the local server to listen on. #KUBE_API_PORT="--port=8080" # ## Port minions listen on #KUBELET_PORT="--kubelet-port=10250" # ## Comma separated list of nodes in the etcd cluster KUBE_ETCD_SERVERS="--etcd-servers=https://172.20.0.113:2379,172.20.0.114:2379,172.20.0.115:2379" # ## Address range to use for services KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16" # ## default admission control policies KUBE_ADMISSION_CONTROL="--admission-control=ServiceAccount,NamespaceLifecycle,NamespaceExists,LimitRanger,ResourceQuota" # ## Add your own! KUBE_API_ARGS="--authorization-mode=RBAC --runtime-config=rbac.authorization.k8s.io/v1beta1 --kubelet-https=true --experimental-bootstrap-token-auth --token-auth-file=/etc/kubernetes/token.csv --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem --tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem --client-ca-file=/etc/kubernetes/ssl/ca.pem --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem --etcd-cafile=/etc/kubernetes/ssl/ca.pem --etcd-certfile=/etc/kubernetes/ssl/kubernetes.pem --etcd-keyfile=/etc/kubernetes/ssl/kubernetes-key.pem --enable-swagger-ui=true --apiserver-count=3 --audit-log-maxage=30 --audit-log-maxbackup=3 --audit-log-maxsize=100 --audit-log-path=/var/lib/audit.log --event-ttl=1h" -
--authorization-mode=RBAC
- kube-scheduler、kube-controller-manager 一般和 kube-apiserver 部署在同一台機器上，它們使用非安全端口和 kube-apiserver通信;
- kubelet、kube-proxy、kubectl 部署在其它 Node 節點上，如果通過安全端口通路 kube-apiserver，則必須先通過 TLS 證書認證，再通過 RBAC 授權；
- kube-proxy、kubectl 通過在使用的證書裡指定相關的 User、Group 來達到通過 RBAC 授權的目的；
- 如果使用了 kubelet TLS Boostrap 機制，則不能再指定
--kubelet-certificate-authority
--kubelet-client-certificate
--kubelet-client-key
-
--admission-control
ServiceAccount
-
--bind-address
127.0.0.1
-
runtime-config
rbac.authorization.k8s.io/v1beta1
-
--service-cluster-ip-range
- 預設情況下 kubernetes 對象儲存在 etcd
/registry
--etcd-prefix
完整 unit 見
kube-apiserver.service啟動kube-apiserver
$ systemctl daemon-reload
$ systemctl enable kube-apiserver
$ systemctl start kube-apiserver
$ systemctl status kube-apiserver 配置和啟動 kube-controller-manager
建立 kube-controller-manager的serivce配置檔案
檔案路徑
/usr/lib/systemd/system/kube-controller-manager.service
Description=Kubernetes Controller Manager Documentation=https://github.com/GoogleCloudPlatform/kubernetes [Service] EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/controller-manager
ExecStart=/usr/bin/kube-controller-manager \
$KUBE_LOGTOSTDERR \
$KUBE_LOG_LEVEL \
$KUBE_MASTER \
$KUBE_CONTROLLER_MANAGER_ARGS
Restart=on-failure
LimitNOFILE=65536 [Install] WantedBy=multi-user.target 配置檔案
/etc/kubernetes/controller-manager
### # The following values are used to configure the kubernetes controller-manager # defaults from config and apiserver should be adequate # Add your own! KUBE_CONTROLLER_MANAGER_ARGS="--address=127.0.0.1 --service-cluster-ip-range=10.254.0.0/16 --cluster-name=kubernetes --cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem --cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem --service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem --root-ca-file=/etc/kubernetes/ssl/ca.pem --leader-elect=true" -
--service-cluster-ip-range
-
--cluster-signing-*
-
--root-ca-file
-
--address
127.0.0.1
參考： https://github.com/kubernetes-incubator/bootkube/issues/64$ kubectl get componentstatuses NAME STATUS MESSAGE ERROR scheduler Unhealthy Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: getsockopt: connection refused controller-manager Healthy ok etcd-2 Unhealthy Get http://172.20.0.113:2379/health: malformed HTTP response "\x15\x03\x01\x00\x02\x02" etcd-0 Healthy {"health": "true"} etcd-1 Healthy {"health": "true"}
啟動 kube-controller-manager
$ systemctl daemon-reload
$ systemctl enable kube-controller-manager
$ systemctl start kube-controller-manager 配置和啟動 kube-scheduler
建立 kube-scheduler的serivce配置檔案
/usr/lib/systemd/system/kube-scheduler.serivce
[Unit] Description=Kubernetes Scheduler Plugin Documentation=https://github.com/GoogleCloudPlatform/kubernetes [Service] EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/scheduler
ExecStart=/usr/bin/kube-scheduler \
$KUBE_LOGTOSTDERR \
$KUBE_LOG_LEVEL \
$KUBE_MASTER \
$KUBE_SCHEDULER_ARGS
Restart=on-failure
LimitNOFILE=65536 [Install] WantedBy=multi-user.target /etc/kubernetes/scheduler
### # kubernetes scheduler config # default config should be adequate # Add your own! KUBE_SCHEDULER_ARGS="--leader-elect=true --address=127.0.0.1" -
--address
127.0.0.1
啟動 kube-scheduler
$ systemctl daemon-reload
$ systemctl enable kube-scheduler
$ systemctl start kube-scheduler 驗證 master 節點功能
$ kubectl get componentstatuses
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Healthy {"health": "true"}
etcd-1 Healthy {"health": "true"}
etcd-2 Healthy {"health": "true"} 六、部署kubernetes node節點
kubernetes node 節點包含如下元件：
- Flanneld：參考我之前寫的文章 Kubernetes基于Flannel的網絡配置 ，之前沒有配置TLS，現在需要在serivce配置檔案中增加TLS配置。
- Docker1.12.5：docker的安裝很簡單，這裡也不說了。
- kubelet
- kube-proxy
下面着重講
kubelet
kube-proxy
的安裝，同時還要将之前安裝的flannel內建TLS驗證。
目錄和檔案
我們再檢查一下三個節點上，經過前幾步操作生成的配置檔案。
$ ls /etc/kubernetes/ssl
admin-key.pem admin.pem ca-key.pem ca.pem kube-proxy-key.pem kube-proxy.pem kubernetes-key.pem kubernetes.pem
$ ls /etc/kubernetes/
apiserver bootstrap.kubeconfig config controller-manager kubelet kube-proxy.kubeconfig proxy scheduler ssl token.csv 配置Flanneld
參考我之前寫的文章
service配置檔案
/usr/lib/systemd/system/flanneld.service
[Unit] Description=Flanneld overlay address etcd agent
After=network.target
After=network-online.target
Wants=network-online.target
After=etcd.service
Before=docker.service
[Service] Type=notify
EnvironmentFile=/etc/sysconfig/flanneld
EnvironmentFile=-/etc/sysconfig/docker-network
ExecStart=/usr/bin/flanneld-start $FLANNEL_OPTIONS
ExecStartPost=/usr/libexec/flannel/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker
Restart=on-failure
[Install] WantedBy=multi-user.target
RequiredBy=docker.service /etc/sysconfig/flanneld
配置檔案。
# Flanneld configuration options # etcd url location. Point this to the server where etcd runs FLANNEL_ETCD_ENDPOINTS="https://172.20.0.113:2379,https://172.20.0.114:2379,https://172.20.0.115:2379" # etcd config key. This is the configuration key that flannel queries # For address range assignment FLANNEL_ETCD_PREFIX="/kube-centos/network" # Any additional options that you want to pass FLANNEL_OPTIONS="-etcd-cafile=/etc/kubernetes/ssl/ca.pem -etcd-certfile=/etc/kubernetes/ssl/kubernetes.pem -etcd-keyfile=/etc/kubernetes/ssl/kubernetes-key.pem" 在FLANNEL_OPTIONS中增加TLS的配置。
安裝和配置 kubelet
kubelet 啟動時向 kube-apiserver 發送 TLS bootstrapping 請求，需要先将 bootstrap token 檔案中的 kubelet-bootstrap 使用者賦予 system:node-bootstrapper cluster 角色(role)， 然後 kubelet 才能有權限建立認證請求(certificate signing requests)：
$ cd /etc/kubernetes
$ kubectl create clusterrolebinding kubelet-bootstrap \
--clusterrole=system:node-bootstrapper \
--user=kubelet-bootstrap -
--user=kubelet-bootstrap
/etc/kubernetes/token.csv
/etc/kubernetes/bootstrap.kubeconfig
下載下傳最新的 kubelet 和 kube-proxy 二進制檔案
$ wget https://dl.k8s.io/v1.6.0/kubernetes-server-linux-amd64.tar.gz
$ tar -xzvf kubernetes-server-linux-amd64.tar.gz
$ cd kubernetes
$ tar -xzvf kubernetes-src.tar.gz
$ cp -r ./server/bin/{kube-proxy,kubelet} /usr/bin/ 建立 kubelet 的service配置檔案
檔案位置
/usr/lib/systemd/system/kubelet.serivce
[Unit] Description=Kubernetes Kubelet Server Documentation=https://github.com/GoogleCloudPlatform/kubernetes After=docker.service
Requires=docker.service
[Service] WorkingDirectory=/var/lib/kubelet
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/kubelet
ExecStart=/usr/bin/kubelet \
$KUBE_LOGTOSTDERR \
$KUBE_LOG_LEVEL \
$KUBELET_API_SERVER \
$KUBELET_ADDRESS \
$KUBELET_PORT \
$KUBELET_HOSTNAME \
$KUBE_ALLOW_PRIV \
$KUBELET_POD_INFRA_CONTAINER \
$KUBELET_ARGS
Restart=on-failure
[Install] WantedBy=multi-user.target kubelet的配置檔案
/etc/kubernetes/kubelet
。其中的IP位址更改為你的每台node節點的IP位址。
### ## kubernetes kubelet (minion) config # ## The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)
KUBELET_ADDRESS="--address=172.20.0.113" # ## The port for the info server to serve on #KUBELET_PORT="--port=10250" # ## You may leave this blank to use the actual hostname
KUBELET_HOSTNAME="--hostname-override=172.20.0.113" # ## location of the api-server
KUBELET_API_SERVER="--api-servers=http://172.20.0.113:8080" # ## pod infrastructure container
KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=sz-pg-oam-docker-hub-001.tendcloud.com/library/pod-infrastructure:rhel7" # ## Add your own!
KUBELET_ARGS="--cgroup-driver=systemd --cluster-dns=10.254.0.2 --experimental-bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --require-kubeconfig --cert-dir=/etc/kubernetes/ssl --cluster-domain=cluster.local. --hairpin-mode promiscuous-bridge --serialize-image-pulls=false" -
--address
127.0.0.1
127.0.0.1
- 如果設定了
--hostname-override
kube-proxy
-
--experimental-bootstrap-kubeconfig
- 管理者通過了 CSR 請求後，kubelet 自動在
--cert-dir
kubelet-client.crt
kubelet-client.key
--kubeconfig
- 建議在
--kubeconfig
kube-apiserver
--api-servers
--require-kubeconfig
kubectl get nodes
-
--cluster-dns
--cluster-domain
啟動kublet
$ systemctl daemon-reload
$ systemctl enable kubelet
$ systemctl start kubelet
$ systemctl status kubelet 通過 kublet 的 TLS 證書請求
kubelet 首次啟動時向 kube-apiserver 發送證書簽名請求，必須通過後 kubernetes 系統才會将該 Node 加入到叢集。
檢視未授權的 CSR 請求
$ kubectl get csr
NAME AGE REQUESTOR CONDITION
csr-2b308 4m kubelet-bootstrap Pending
$ kubectl get nodes
No resources found. 通過 CSR 請求
$ kubectl certificate approve csr-2b308
certificatesigningrequest "csr-2b308" approved
$ kubectl get nodes
NAME STATUS AGE VERSION
10.64.3.7 Ready 49m v1.6.1 自動生成了 kubelet kubeconfig 檔案和公私鑰
$ ls -l /etc/kubernetes/kubelet.kubeconfig
-rw------- 1 root root 2284 Apr 7 02:07 /etc/kubernetes/kubelet.kubeconfig
$ ls -l /etc/kubernetes/ssl/kubelet* -rw-r--r-- 1 root root 1046 Apr 7 02:07 /etc/kubernetes/ssl/kubelet-client.crt
-rw------- 1 root root 227 Apr 7 02:04 /etc/kubernetes/ssl/kubelet-client.key
-rw-r--r-- 1 root root 1103 Apr 7 02:07 /etc/kubernetes/ssl/kubelet.crt
-rw------- 1 root root 1675 Apr 7 02:07 /etc/kubernetes/ssl/kubelet.key 配置 kube-proxy
建立 kube-proxy 的service配置檔案
/usr/lib/systemd/system/kube-proxy.service
[Unit] Description=Kubernetes Kube-Proxy Server Documentation=https://github.com/GoogleCloudPlatform/kubernetes After=network.target
[Service] EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/proxy
ExecStart=/usr/bin/kube-proxy \
$KUBE_LOGTOSTDERR \
$KUBE_LOG_LEVEL \
$KUBE_MASTER \
$KUBE_PROXY_ARGS
Restart=on-failure
LimitNOFILE=65536 [Install] WantedBy=multi-user.target kube-proxy配置檔案
/etc/kubernetes/proxy
### # kubernetes proxy config # default config should be adequate # Add your own!
KUBE_PROXY_ARGS="--bind-address=172.20.0.113 --hostname-override=172.20.0.113 --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig --cluster-cidr=10.254.0.0/16" -
--hostname-override
- kube-proxy 根據
--cluster-cidr
--cluster-cidr
--masquerade-all
-
--kubeconfig
-
cluster-admin
system:kube-proxy
system:node-proxier
kube-apiserver
啟動 kube-proxy
$ systemctl daemon-reload
$ systemctl enable kube-proxy
$ systemctl start kube-proxy
$ systemctl status kube-proxy 驗證測試
我們建立一個niginx的service試一下叢集是否可用。
$ kubectl run nginx --replicas=2 --labels="run=load-balancer-example" --image=sz-pg-oam-docker-hub-001.tendcloud.com/library/nginx:1.9 --port=80
deployment "nginx" created
$ kubectl expose deployment nginx --type=NodePort --name=example-service
service "example-service" exposed
$ kubectl describe svc example-service
Name: example-service
Namespace: default Labels: run=load-balancer-example
Annotations: <none> Selector: run=load-balancer-example
Type: NodePort
IP: 10.254.62.207 Port: <unset> 80/TCP
NodePort: <unset> 32724/TCP
Endpoints: 172.30.60.2:80,172.30.94.2:80 Session Affinity: None Events: <none>
$ curl "10.254.62.207:80" <!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title> <style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; } </style> </head> <body> <h1>Welcome to nginx!</h1> <p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p> <p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/> Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p> <p><em>Thank you for using nginx.</em></p> </body> </html> 通路
172.20.0.113:32724
172.20.0.114:32724
或者
172.20.0.115:32724
都可以得到nginx的頁面。
七、安裝和配置 kubedns 插件
官方的yaml檔案目錄：
kubernetes/cluster/addons/dns
該插件直接使用kubernetes部署，官方的配置檔案中包含以下鏡像：
gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.1
gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.1
gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.1 我clone了上述鏡像，上傳到我的私有鏡像倉庫：
sz-pg-oam-docker-hub-001.tendcloud.com/library/k8s-dns-dnsmasq-nanny-amd64:1.14.1
sz-pg-oam-docker-hub-001.tendcloud.com/library/k8s-dns-kube-dns-amd64:1.14.1
sz-pg-oam-docker-hub-001.tendcloud.com/library/k8s-dns-sidecar-amd64:1.14.1 同時上傳了一份到時速雲備份：
index.tenxcloud.com/jimmy/k8s-dns-dnsmasq-nanny-amd64:1.14.1
index.tenxcloud.com/jimmy/k8s-dns-kube-dns-amd64:1.14.1
index.tenxcloud.com/jimmy/k8s-dns-sidecar-amd64:1.14.1 以下yaml配置檔案中使用的是私有鏡像倉庫中的鏡像。
kubedns-cm.yaml
kubedns-sa.yaml
kubedns-controller.yaml
kubedns-svc.yaml 已經修改好的 yaml 檔案見：
dns系統預定義的 RoleBinding
system:kube-dns
将 kube-system 命名空間的
kube-dns
ServiceAccount 與
system:kube-dns
Role 綁定， 該 Role 具有通路 kube-apiserver DNS 相關 API 的權限；
$ kubectl get clusterrolebindings system:kube-dns -o yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
creationTimestamp: 2017-04-11T11:20:42Z
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:kube-dns
resourceVersion: "58"
selfLink: /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindingssystem%3Akube-dns
uid: e61f4d92-1ea8-11e7-8cd7-f4e9d49f8ed0
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:kube-dns
subjects: - kind: ServiceAccount
name: kube-dns
namespace: kube-system kubedns-controller.yaml
中定義的 Pods 時使用了
kubedns-sa.yaml
檔案定義的
kube-dns
ServiceAccount，是以具有通路 kube-apiserver DNS 相關 API 的權限。
配置 kube-dns ServiceAccount
無需修改。
配置 kube-dns
kube-dns
$ diff kubedns-svc.yaml.base kubedns-svc.yaml
30c30 < clusterIP: __PILLAR__DNS__SERVER__
--- > clusterIP: 10.254.0.2 - spec.clusterIP = 10.254.0.2，即明确指定了 kube-dns Service IP，這個 IP 需要和 kubelet 的
--cluster-dns
kube-dns
kube-dns
$ diff kubedns-controller.yaml.base kubedns-controller.yaml
58c58 < image: gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.1 --- > image: sz-pg-oam-docker-hub-001.tendcloud.com/library/k8s-dns-kube-dns-amd64:v1.14.1 88c88 < - --domain=__PILLAR__DNS__DOMAIN__. --- > - --domain=cluster.local. 92c92 < __PILLAR__FEDERATIONS__DOMAIN__MAP__
--- > #__PILLAR__FEDERATIONS__DOMAIN__MAP__ 110c110 < image: gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.1 --- > image: sz-pg-oam-docker-hub-001.tendcloud.com/library/k8s-dns-dnsmasq-nanny-amd64:v1.14.1 129c129 < - --server=/__PILLAR__DNS__DOMAIN__/127.0.0.1#10053 --- > - --server=/cluster.local./127.0.0.1#10053 148c148 < image: gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.1 --- > image: sz-pg-oam-docker-hub-001.tendcloud.com/library/k8s-dns-sidecar-amd64:v1.14.1 161,162c161,162 < - --probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.__PILLAR__DNS__DOMAIN__,5,A
< - --probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.__PILLAR__DNS__DOMAIN__,5,A
--- > - --probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local.,5,A
> - --probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local.,5,A - 使用系統已經做了 RoleBinding 的
kube-dns
執行所有定義檔案
$ pwd /root/kubedns
$ ls *.yaml
kubedns-cm.yaml kubedns-controller.yaml kubedns-sa.yaml kubedns-svc.yaml
$ kubectl create -f . 檢查 kubedns 功能
建立一個 Deployment
$ cat my-nginx.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: my-nginx
spec:
replicas: 2 template:
metadata:
labels:
run: my-nginx
spec:
containers: - name: my-nginx
image: sz-pg-oam-docker-hub-001.tendcloud.com/library/nginx:1.9
ports: - containerPort: 80
$ kubectl create -f my-nginx.yaml Export 該 Deployment, 生成
my-nginx
$ kubectl expose deploy my-nginx
$ kubectl get services --all-namespaces |grep my-nginx
default my-nginx 10.254.179.239 <none> 80/TCP 42m 建立另一個 Pod，檢視
/etc/resolv.conf
是否包含
kubelet
配置的
--cluster-dns
--cluster-domain
，是否能夠将服務
my-nginx
解析到 Cluster IP
10.254.179.239
$ kubectl create -f nginx-pod.yaml
$ kubectl exec nginx -i -t -- /bin/bash
root@nginx:/# cat /etc/resolv.conf
nameserver 10.254.0.2
search default.svc.cluster.local. svc.cluster.local. cluster.local. tendcloud.com
options ndots:5
root@nginx:/# ping my-nginx
PING my-nginx.default.svc.cluster.local (10.254.179.239): 56 data bytes
76 bytes from 119.147.223.109: Destination Net Unreachable
^C--- my-nginx.default.svc.cluster.local ping statistics ---
root@nginx:/# ping kubernetes
PING kubernetes.default.svc.cluster.local (10.254.0.1): 56 data bytes
^C--- kubernetes.default.svc.cluster.local ping statistics --- 11 packets transmitted, 0 packets received, 100% packet loss
root@nginx:/# ping kube-dns.kube-system.svc.cluster.local
PING kube-dns.kube-system.svc.cluster.local (10.254.0.2): 56 data bytes
^C--- kube-dns.kube-system.svc.cluster.local ping statistics --- 6 packets transmitted, 0 packets received, 100% packet loss 從結果來看，service名稱可以正常解析。
八、配置和安裝 dashboard
官方檔案目錄：
kubernetes/cluster/addons/dashboard
我們使用的檔案
$ ls *.yaml
dashboard-controller.yaml dashboard-service.yaml dashboard-rbac.yaml 由于
kube-apiserver
啟用了
RBAC
授權，而官方源碼目錄的
dashboard-controller.yaml
沒有定義授權的 ServiceAccount，是以後續通路
kube-apiserver
的 API 時會被拒絕，web中提示：
Forbidden (403) User "system:serviceaccount:kube-system:default" cannot list jobs.batch in the namespace "default". (get jobs.batch) 增加了一個
dashboard-rbac.yaml
檔案，定義一個名為 dashboard 的 ServiceAccount，然後将它和 Cluster Role view 綁定。
配置dashboard-service
$ diff dashboard-service.yaml.orig dashboard-service.yaml
10a11 > type: NodePort - 指定端口類型為 NodePort，這樣外界可以通過位址 nodeIP:nodePort 通路 dashboard；
配置dashboard-controller
$ diff dashboard-controller.yaml.orig dashboard-controller.yaml
23c23 < image: gcr.io/google_containers/kubernetes-dashboard-amd64:v1.6.0 --- > image: sz-pg-oam-docker-hub-001.tendcloud.com/library/kubernetes-dashboard-amd64:v1.6.0 $ pwd /root/kubernetes/cluster/addons/dashboard
$ ls *.yaml
dashboard-controller.yaml dashboard-service.yaml
$ kubectl create -f .
service "kubernetes-dashboard" created
deployment "kubernetes-dashboard" created 檢查執行結果
檢視配置設定的 NodePort
$ kubectl get services kubernetes-dashboard -n kube-system
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes-dashboard 10.254.224.130 <nodes> 80:30312/TCP 25s - NodePort 30312映射到 dashboard pod 80端口；
檢查 controller
$ kubectl get deployment kubernetes-dashboard -n kube-system
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
kubernetes-dashboard 1 1 1 1 3m
$ kubectl get pods -n kube-system | grep dashboard
kubernetes-dashboard-1339745653-pmn6z 1/1 Running 0 4m 通路dashboard
有以下三種方式：
- kubernetes-dashboard 服務暴露了 NodePort，可以使用
http://NodeIP:nodePort
- 通過 kube-apiserver 通路 dashboard（https 6443端口和http 8080端口方式）；
- 通過 kubectl proxy 通路 dashboard：
通過 kubectl proxy 通路 dashboard
啟動代理
$ kubectl proxy --address='172.20.0.113' --port=8086 --accept-hosts='^*$' Starting to serve on 172.20.0.113:8086 - 需要指定
--accept-hosts
浏覽器通路 URL：
http://172.20.0.113:8086/ui
自動跳轉到：
http://172.20.0.113:8086/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard/#/workload?namespace=default
通過 kube-apiserver 通路dashboard
擷取叢集服務位址清單
$ kubectl cluster-info
Kubernetes master is running at https://172.20.0.113:6443 KubeDNS is running at https://172.20.0.113:6443/api/v1/proxy/namespaces/kube-system/services/kube-dns
kubernetes-dashboard is running at https://172.20.0.113:6443/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard https://172.20.0.113:6443/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard
（浏覽器會提示證書驗證，因為通過加密通道，以改方式通路的話，需要提前導入證書到你的計算機中）。這是我當時在這遇到的坑：
通過 kube-apiserver 通路dashboard，提示User “system:anonymous” cannot proxy services in the namespace “kube-system”. #5，已經解決。
導入證書
将生成的admin.pem證書轉換格式
openssl pkcs12 -export -in admin.pem -out admin.p12 -inkey admin-key.pem 将生成的
admin.p12
證書導入的你的電腦，導出的時候記住你設定的密碼，導入的時候還要用到。
如果你不想使用https的話，可以直接通路insecure port 8080端口:
http://172.20.0.113:8080/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard
由于缺少 Heapster 插件，目前 dashboard 不能展示 Pod、Nodes 的 CPU、記憶體等 metric 圖形。
九、配置和安裝 Heapster
heapster release 頁面下載下傳最新版本的 heapster。
$ wget https://github.com/kubernetes/heapster/archive/v1.3.0.zip
$ unzip v1.3.0.zip
$ mv v1.3.0.zip heapster-1.3.0 檔案目錄：
heapster-1.3.0/deploy/kube-config/influxdb
$ cd heapster-1.3.0/deploy/kube-config/influxdb
$ ls *.yaml
grafana-deployment.yaml grafana-service.yaml heapster-deployment.yaml heapster-service.yaml influxdb-deployment.yaml influxdb-service.yaml heapster-rbac.yaml 我們自己建立了heapster的rbac配置
heapster-rbac.yaml
配置 grafana-deployment
$ diff grafana-deployment.yaml.orig grafana-deployment.yaml
16c16 < image: gcr.io/google_containers/heapster-grafana-amd64:v4.0.2 --- > image: sz-pg-oam-docker-hub-001.tendcloud.com/library/heapster-grafana-amd64:v4.0.2 40,41c40,41 < # value: /api/v1/proxy/namespaces/kube-system/services/monitoring-grafana/ < value: /
---
> value: /api/v1/proxy/namespaces/kube-system/services/monitoring-grafana/ > #value: / - 如果後續使用 kube-apiserver 或者 kubectl proxy 通路 grafana dashboard，則必須将
GF_SERVER_ROOT_URL
/api/v1/proxy/namespaces/kube-system/services/monitoring-grafana/
http://10.64.3.7:8086/api/v1/proxy/namespaces/kube-system/services/monitoring-grafana/api/dashboards/home
配置 heapster-deployment
$ diff heapster-deployment.yaml.orig heapster-deployment.yaml
16c16 < image: gcr.io/google_containers/heapster-amd64:v1.3.0-beta.1 --- > image: sz-pg-oam-docker-hub-001.tendcloud.com/library/heapster-amd64:v1.3.0-beta.1 配置 influxdb-deployment
influxdb 官方建議使用指令行或 HTTP API 接口來查詢資料庫，從 v1.1.0 版本開始預設關閉 admin UI，将在後續版本中移除 admin UI 插件。
開啟鏡像中 admin UI的辦法如下：先導出鏡像中的 influxdb 配置檔案，開啟 admin 插件後，再将配置檔案内容寫入 ConfigMap，最後挂載到鏡像中，達到覆寫原始配置的目的：
注意：manifests 目錄已經提供了
修改後的 ConfigMap 定義檔案$ # 導出鏡像中的 influxdb 配置檔案
$ docker run --rm --entrypoint 'cat' -ti lvanneo/heapster-influxdb-amd64:v1.1.1 /etc/config.toml >config.toml.orig
$ cp config.toml.orig config.toml
$ # 修改：啟用 admin 接口
$ vim config.toml
$ diff config.toml.orig config.toml
35c35 < enabled = false --- > enabled = true
$ # 将修改後的配置寫入到 ConfigMap 對象中
$ kubectl create configmap influxdb-config --from-file=config.toml -n kube-system
configmap "influxdb-config" created
$ # 将 ConfigMap 中的配置檔案挂載到 Pod 中，達到覆寫原始配置的目的
$ diff influxdb-deployment.yaml.orig influxdb-deployment.yaml
16c16 < image: grc.io/google_containers/heapster-influxdb-amd64:v1.1.1 --- > image: sz-pg-oam-docker-hub-001.tendcloud.com/library/heapster-influxdb-amd64:v1.1.1 19a20,21 > - mountPath: /etc/ > name: influxdb-config
22a25,27 > - name: influxdb-config
> configMap: > name: influxdb-config 配置 monitoring-influxdb Service
$ diff influxdb-service.yaml.orig influxdb-service.yaml
12a13 > type: NodePort 15a17,20 > name: http
> - port: 8083 > targetPort: 8083 > name: admin - 定義端口類型為 NodePort，額外增加了 admin 端口映射，用于後續浏覽器通路 influxdb 的 admin UI 界面；
$ pwd /root/heapster-1.3.0/deploy/kube-config/influxdb
$ ls *.yaml
grafana-service.yaml heapster-rbac.yaml influxdb-cm.yaml influxdb-service.yaml
grafana-deployment.yaml heapster-deployment.yaml heapster-service.yaml influxdb-deployment.yaml
$ kubectl create -f .
deployment "monitoring-grafana" created
service "monitoring-grafana" created
deployment "heapster" created
serviceaccount "heapster" created
clusterrolebinding "heapster" created
service "heapster" created
configmap "influxdb-config" created
deployment "monitoring-influxdb" created
service "monitoring-influxdb" created 檢查 Deployment
$ kubectl get deployments -n kube-system | grep -E 'heapster|monitoring'
heapster 1 1 1 1 2m
monitoring-grafana 1 1 1 1 2m
monitoring-influxdb 1 1 1 1 2m 檢查 Pods
$ kubectl get pods -n kube-system | grep -E 'heapster|monitoring'
heapster-110704576-gpg8v 1/1 Running 0 2m
monitoring-grafana-2861879979-9z89f 1/1 Running 0 2m
monitoring-influxdb-1411048194-lzrpc 1/1 Running 0 2m 檢查 kubernets dashboard 界面，看是顯示各 Nodes、Pods 的 CPU、記憶體、負載等使用率曲線圖；
通路 grafana
- 通過 kube-apiserver 通路：擷取 monitoring-grafana 服務 URL
$ kubectl cluster-info Kubernetes master is running at https://172.20.0.113:6443 Heapster is running at https://172.20.0.113:6443/api/v1/proxy/namespaces/kube-system/services/heapster KubeDNS is running at https://172.20.0.113:6443/api/v1/proxy/namespaces/kube-system/services/kube-dns kubernetes-dashboard is running at https://172.20.0.113:6443/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard monitoring-grafana is running at https://172.20.0.113:6443/api/v1/proxy/namespaces/kube-system/services/monitoring-grafana monitoring-influxdb is running at https://172.20.0.113:6443/api/v1/proxy/namespaces/kube-system/services/monitoring-influxdb To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.http://172.20.0.113:8080/api/v1/proxy/namespaces/kube-system/services/monitoring-grafana
- 通過 kubectl proxy 通路：建立代理
$ kubectl proxy --address='172.20.0.113' --port=8086 --accept-hosts='^*$' Starting to serve on 172.20.0.113:8086http://172.20.0.113:8086/api/v1/proxy/namespaces/kube-system/services/monitoring-grafana
通路 influxdb admin UI
擷取 influxdb http 8086 映射的 NodePort
$ kubectl get svc -n kube-system|grep influxdb
monitoring-influxdb 10.254.22.46 <nodes> 8086:32299/TCP,8083:30269/TCP 9m 通過 kube-apiserver 的非安全端口通路 influxdb 的 admin UI 界面：
http://172.20.0.113:8080/api/v1/proxy/namespaces/kube-system/services/monitoring-influxdb:8083/
在頁面的 “Connection Settings” 的 Host 中輸入 node IP， Port 中輸入 8086 映射的 nodePort 如上面的 32299，點選 “Save” 即可（我的叢集中的位址是172.20.0.113:32299）：
十、配置和安裝 EFK
cluster/addons/fluentd-elasticsearch
$ ls *.yaml
es-controller.yaml es-service.yaml fluentd-es-ds.yaml kibana-controller.yaml kibana-service.yaml efk-rbac.yaml 同樣EFK服務也需要一個
efk-rbac.yaml
檔案，配置serviceaccount為
efk
配置 es-controller.yaml
$ diff es-controller.yaml.orig es-controller.yaml
24c24 < - image: gcr.io/google_containers/elasticsearch:v2.4.1-2 --- > - image: sz-pg-oam-docker-hub-001.tendcloud.com/library/elasticsearch:v2.4.1-2 配置 es-service.yaml
無需配置；
配置 fluentd-es-ds.yaml
$ diff fluentd-es-ds.yaml.orig fluentd-es-ds.yaml
26c26 < image: gcr.io/google_containers/fluentd-elasticsearch:1.22 --- > image: sz-pg-oam-docker-hub-001.tendcloud.com/library/fluentd-elasticsearch:1.22 配置 kibana-controller.yaml
$ diff kibana-controller.yaml.orig kibana-controller.yaml
22c22 < image: gcr.io/google_containers/kibana:v4.6.1-1 --- > image: sz-pg-oam-docker-hub-001.tendcloud.com/library/kibana:v4.6.1-1 給 Node 設定标簽
定義 DaemonSet
fluentd-es-v1.22
時設定了 nodeSelector
beta.kubernetes.io/fluentd-ds-ready=true
，是以需要在期望運作 fluentd 的 Node 上設定該标簽；
$ kubectl get nodes
NAME STATUS AGE VERSION
172.20.0.113 Ready 1d v1.6.0
$ kubectl label nodes 172.20.0.113 beta.kubernetes.io/fluentd-ds-ready=true
node "172.20.0.113" labeled 給其他兩台node打上同樣的标簽。
執行定義檔案
$ kubectl create -f .
serviceaccount "efk" created
clusterrolebinding "efk" created
replicationcontroller "elasticsearch-logging-v1" created
service "elasticsearch-logging" created
daemonset "fluentd-es-v1.22" created
deployment "kibana-logging" created
service "kibana-logging" created $ kubectl get deployment -n kube-system|grep kibana
kibana-logging 1 1 1 1 2m
$ kubectl get pods -n kube-system|grep -E 'elasticsearch|fluentd|kibana'
elasticsearch-logging-v1-mlstp 1/1 Running 0 1m
elasticsearch-logging-v1-nfbbf 1/1 Running 0 1m
fluentd-es-v1.22-31sm0 1/1 Running 0 1m
fluentd-es-v1.22-bpgqs 1/1 Running 0 1m
fluentd-es-v1.22-qmn7h 1/1 Running 0 1m
kibana-logging-1432287342-0gdng 1/1 Running 0 1m
$ kubectl get service -n kube-system|grep -E 'elasticsearch|kibana'
elasticsearch-logging 10.254.77.62 <none> 9200/TCP 2m
kibana-logging 10.254.8.113 <none> 5601/TCP 2m kibana Pod 第一次啟動時會用**較長時間(10-20分鐘)**來優化和 Cache 狀态頁面，可以 tailf 該 Pod 的日志觀察進度：
$ kubectl logs kibana-logging-1432287342-0gdng -n kube-system -f
ELASTICSEARCH_URL=http://elasticsearch-logging:9200
server.basePath: /api/v1/proxy/namespaces/kube-system/services/kibana-logging
{"type":"log","@timestamp":"2017-04-12T13:08:06Z","tags":["info","optimize"],"pid":7,"message":"Optimizing and caching bundles for kibana and statusPage. This may take a few minutes"} {"type":"log","@timestamp":"2017-04-12T13:18:17Z","tags":["info","optimize"],"pid":7,"message":"Optimization of bundles for kibana and statusPage complete in 610.40 seconds"} {"type":"log","@timestamp":"2017-04-12T13:18:17Z","tags":["status","plugin:[email protected]","info"],"pid":7,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2017-04-12T13:18:18Z","tags":["status","plugin:[email protected]","info"],"pid":7,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2017-04-12T13:18:19Z","tags":["status","plugin:[email protected]","info"],"pid":7,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2017-04-12T13:18:19Z","tags":["status","plugin:[email protected]","info"],"pid":7,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2017-04-12T13:18:19Z","tags":["status","plugin:[email protected]","info"],"pid":7,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2017-04-12T13:18:19Z","tags":["status","plugin:[email protected]","info"],"pid":7,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2017-04-12T13:18:19Z","tags":["status","plugin:[email protected]","info"],"pid":7,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2017-04-12T13:18:19Z","tags":["status","plugin:[email protected]","info"],"pid":7,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2017-04-12T13:18:19Z","tags":["listening","info"],"pid":7,"message":"Server running at http://0.0.0.0:5601"} {"type":"log","@timestamp":"2017-04-12T13:18:24Z","tags":["status","plugin:[email protected]","info"],"pid":7,"state":"yellow","message":"Status changed from yellow to yellow - No existing Kibana index found","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"} {"type":"log","@timestamp":"2017-04-12T13:18:29Z","tags":["status","plugin:[email protected]","info"],"pid":7,"state":"green","message":"Status changed from yellow to green - Kibana index ready","prevState":"yellow","prevMsg":"No existing Kibana index found"} 通路 kibana
-
$ kubectl cluster-info Kubernetes master is running at https://172.20.0.113:6443 Elasticsearch is running at https://172.20.0.113:6443/api/v1/proxy/namespaces/kube-system/services/elasticsearch-logging Heapster is running at https://172.20.0.113:6443/api/v1/proxy/namespaces/kube-system/services/heapster Kibana is running at https://172.20.0.113:6443/api/v1/proxy/namespaces/kube-system/services/kibana-logging KubeDNS is running at https://172.20.0.113:6443/api/v1/proxy/namespaces/kube-system/services/kube-dns kubernetes-dashboard is running at https://172.20.0.113:6443/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard monitoring-grafana is running at https://172.20.0.113:6443/api/v1/proxy/namespaces/kube-system/services/monitoring-grafana monitoring-influxdb is running at https://172.20.0.113:6443/api/v1/proxy/namespaces/kube-system/services/monitoring-influxdbhttps://172.20.0.113:6443/api/v1/proxy/namespaces/kube-system/services/kibana-logging/app/kibana
-
$ kubectl proxy --address='172.20.0.113' --port=8086 --accept-hosts='^*$' Starting to serve on 172.20.0.113:8086http://172.20.0.113:8086/api/v1/proxy/namespaces/kube-system/services/kibana-logging
在 Settings -> Indices 頁面建立一個 index（相當于 mysql 中的一個 database），選中
Index contains time-based events
，使用預設的
logstash-*
pattern，點選
Create
;
可能遇到的問題
如果你在這裡發現Create按鈕是灰色的無法點選，且Time-filed name中沒有選項，fluentd要讀取
/var/log/containers/
目錄下的log日志，這些日志是從
/var/lib/docker/containers/${CONTAINER_ID}/${CONTAINER_ID}-json.log
連結過來的，檢視你的docker配置，
—log-dirver
需要設定為json-file格式，預設的可能是journald，參考docker logging。
建立Index後，可以在
Discover
下看到 ElasticSearch logging 中彙聚的日志；
本文轉自中文社群-
Kubernetes叢集安裝文檔-v1.6版本