集群规划

网络配置

节点网络： 192.168.18.0/24
service网络： 10.96.0.0/12
pod网络： 10.244.0.0/16

etcd

部署在master节点上。

部署方法：

ansible部署

github上有人将部署方式用playbook实现。

使用kubeadm部署

基本情况

kubeadm项目链接地址
master、node：安装kuberlet， kubeadm， docker
master： kubeadm init
node： kubeadm join
apiserver，scheduler，Controller-manager，etcd在master上以Pod运行
kubeproxy以Pod方式运行在每一个node节点上。
以上pod均为静态Pod
每一个节点都需要运行flannel（也是以Pod方式运行），以提供Pod网络
kubeadm的介绍

安装步骤

master，node需要安装kubelet， kubeadm， docker
master节点上运行 kubeadm init
node节点上运行 kubeadm join 加入集群

开始部署

我的环境

[root@master yum.repos.d]# cat /etc/redhat-release 
CentOS Linux release 7.4.1708 (Core) 
[root@master yum.repos.d]# uname -a
Linux master 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

节点解析

通过hosts文件解析

192.168.18.128 master.test.com master
192.168.18.129 node01.test.com node01
192.168.18.130 node02.test.com node02

集群通过时间服务器做时钟同步，我没做。

节点互信

可以按照此

文档

配置节点互信。

选择版本

使用kubernetes v1.11.2

开始

确保iptables firewalld等未启动,且不开机自启动

配置yum仓库

使用aliyun源，

链接

docker源使用如下命令获取

cd /etc/yum.repos.d
wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

kubernetes源

[root@master yum.repos.d]# cat kubernetes.repo  [kubernetes] name=Kubernetes Repo
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
gpgcheck=1 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
enabled=1

查看源是否生效

# yum clean all # yum repolist ***** ***** Determining fastest mirrors kubernetes 243/243 repo id repo name status base/7/x86_64 CentOS-7 - Base - 163.com 9,911 docker-ce-stable/x86_64 Docker CE Stable - x86_64 16 extras/7/x86_64 CentOS-7 - Extras - 163.com 370 kubernetes Kubernetes Repo 243 updates/7/x86_64 CentOS-7 - Updates - 163.com 1,054 repolist: 11,594

安装软件

三台机器都需要安装

使用

yum install docker-ce kubelet kubeadm kubectl

安装

安装的软件包如下：

Installing : kubectl-1.11.2-0.x86_64 1/7 Installing : cri-tools-1.11.0-0.x86_64 2/7 Installing : socat-1.7.3.2-2.el7.x86_64 3/7 Installing : kubernetes-cni-0.6.0-0.x86_64 4/7 Installing : kubelet-1.11.2-0.x86_64 5/7 Installing : kubeadm-1.11.2-0.x86_64 6/7 Installing : docker-ce-18.06.0.ce-3.el7.x86_64 7/7

启动docker服务等

由于国内网络原因，kubernetes的镜像托管在google云上，无法直接下载，需要设置proxy

在

/usr/lib/systemd/system/docker.service

文件中添加如下两行

[root@master ~]# cat /usr/lib/systemd/system/docker.service |grep PROXY Environment="HTTPS_PROXY=http://www.ik8s.io:10080" Environment="NO_PROXY=127.0.0.0/8,192.168.18.0/24"

之后，启动docker

systemctl daemon-reload systemctl start docker systemctl enable docker

确认proc的这两个参数如下，均为1：

[root@master ~]# cat /proc/sys/net/bridge/bridge-nf-call-iptables  1
[root@master ~]# cat /proc/sys/net/bridge/bridge-nf-call-ip6tables  1

设置kubelet

查看kubelet安装生成了哪些文件

[root@master ~]# rpm -ql kubelet
/etc/kubernetes/manifests # 清单目录
/etc/sysconfig/kubelet # 配置文件
/etc/systemd/system/kubelet.service # unit file
/usr/bin/kubelet # 主程序

默认的配置文件

[root@master ~]# cat /etc/sysconfig/kubelet KUBELET_EXTRA_ARGS=

修改kubelet的配置文件

[root@master ~]# cat /etc/sysconfig/kubelet KUBELET_EXTRA_ARGS="--fail-swap-on=false"

此时还无法正常启动kubelet，先设置kubelet开机自启动，使用如下命令：

systemctl enable kubelet

。

kubeadm init

在master节点上执行

kubeadm init --kubernetes-version=v1.11.2 --pod-network-cidr=10.244.0.0/16 --service-cidr=10.96.0.0/12 --ignore-preflight-errors=Swap

kubeadm init的输出可见于此

此命令，下载了如下image

[root@master ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
k8s.gcr.io/kube-proxy-amd64 v1.11.2 46a3cd725628 7 days ago 97.8MB
k8s.gcr.io/kube-apiserver-amd64 v1.11.2 821507941e9c 7 days ago 187MB
k8s.gcr.io/kube-controller-manager-amd64 v1.11.2 38521457c799 7 days ago 155MB
k8s.gcr.io/kube-scheduler-amd64 v1.11.2 37a1403e6c1a 7 days ago 56.8MB
k8s.gcr.io/coredns 1.1.3 b3b94275d97c 2 months ago 45.6MB
k8s.gcr.io/etcd-amd64 3.2.18 b8df3b177be2 4 months ago 219MB
k8s.gcr.io/pause 3.1 da86e6ba6ca1 7 months ago 742kB

现在，正在运行的docker如下

[root@master ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 1c03e043e6b7 46a3cd725628 "/usr/local/bin/kube?? 3 minutes ago Up 3 minutes k8s_kube-proxy_kube-proxy-6fgjm_kube-system_f85e8660-a090-11e8-8ee7-000c29f71e04_0 5f166bd11566 k8s.gcr.io/pause:3.1 "/pause" 3 minutes ago Up 3 minutes k8s_POD_kube-proxy-6fgjm_kube-system_f85e8660-a090-11e8-8ee7-000c29f71e04_0 0f306f98cc52 b8df3b177be2 "etcd --advertise-cl?? 3 minutes ago Up 3 minutes k8s_etcd_etcd-master_kube-system_2cc1c8a24b68ab9b46bca47e153e74c6_0 8f01317b9e20 37a1403e6c1a "kube-scheduler --ad?? 3 minutes ago Up 3 minutes k8s_kube-scheduler_kube-scheduler-master_kube-system_a00c35e56ebd0bdfcd77d53674a5d2a1_0 4e6a71ab20d3 821507941e9c "kube-apiserver --au?? 3 minutes ago Up 3 minutes k8s_kube-apiserver_kube-apiserver-master_kube-system_d25d40ebb427821464356bb27a38f487_0 69e4c5dae335 38521457c799 "kube-controller-man?? 3 minutes ago Up 3 minutes k8s_kube-controller-manager_kube-controller-manager-master_kube-system_6363f7ebf727b0b95d9a9ef72516a0e5_0 da5981dc546a k8s.gcr.io/pause:3.1 "/pause" 3 minutes ago Up 3 minutes k8s_POD_kube-controller-manager-master_kube-system_6363f7ebf727b0b95d9a9ef72516a0e5_0 b7a8fdc35029 k8s.gcr.io/pause:3.1 "/pause" 3 minutes ago Up 3 minutes k8s_POD_kube-apiserver-master_kube-system_d25d40ebb427821464356bb27a38f487_0 b09efc7ff7bd k8s.gcr.io/pause:3.1 "/pause" 3 minutes ago Up 3 minutes k8s_POD_kube-scheduler-master_kube-system_a00c35e56ebd0bdfcd77d53674a5d2a1_0 ab11d6ffadab k8s.gcr.io/pause:3.1 "/pause" 3 minutes ago Up 3 minutes k8s_POD_etcd-master_kube-system_2cc1c8a24b68ab9b46bca47e153e74c6_0

node节点可以通过如下命令加入集群

kubeadm join 192.168.18.128:6443 --token n84v6t.c7d83cn4mo2z8wyr --discovery-token-ca-cert-hash sha256:b946c145416fe1995e1d4d002c149e71a897acc7b106d94cee2920cb2c85ce29

在kubeadm init的

输出

中，提示我们需要以普通用户做如下操作，我此时用root执行

[root@master ~]# mkdir -p $HOME/.kube
[root@master ~]# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

此时可以通过

kubelet get

获取各种资源信息。比如

[root@master ~]# kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok 
controller-manager Healthy ok 
etcd-0 Healthy {"health": "true"}

此时的监听状态

[root@master ~]# ss -tnl State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 128 127.0.0.1:2379 *:* 
LISTEN 0 128 127.0.0.1:10251 *:* 
LISTEN 0 128 127.0.0.1:2380 *:* 
LISTEN 0 128 127.0.0.1:10252 *:* 
LISTEN 0 128 *:22 *:* 
LISTEN 0 128 127.0.0.1:33881 *:* 
LISTEN 0 100 127.0.0.1:25 *:* 
LISTEN 0 128 192.168.18.128:10010 *:* 
LISTEN 0 128 127.0.0.1:10248 *:* 
LISTEN 0 128 127.0.0.1:10249 *:* 
LISTEN 0 128 :::6443 :::* 
LISTEN 0 128 :::10256 :::* 
LISTEN 0 128 :::22 :::* 
LISTEN 0 100 ::1:25 :::* 
LISTEN 0 128 :::10250 :::*

此时的节点状态

[root@master ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION master NotReady master 9m v1.11.2

状态为

NotReady

，需要部署 flannel，

部署flannel

在文档中，找到如下命令，部署

[root@master ~]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.extensions/kube-flannel-ds-amd64 created
daemonset.extensions/kube-flannel-ds-arm64 created
daemonset.extensions/kube-flannel-ds-arm created
daemonset.extensions/kube-flannel-ds-ppc64le created
daemonset.extensions/kube-flannel-ds-s390x created

按如下方法查看：

[root@master ~]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-78fcdf6894-cv4gp 1/1 Running 0 13m
coredns-78fcdf6894-wmd25 1/1 Running 0 13m
etcd-master 1/1 Running 0 49s
kube-apiserver-master 1/1 Running 0 49s
kube-controller-manager-master 1/1 Running 0 48s
kube-flannel-ds-amd64-r42wr 1/1 Running 0 2m
kube-proxy-6fgjm 1/1 Running 0 13m
kube-scheduler-master 1/1 Running 0 48s
[root@master ~]# docker images |grep flannel
quay.io/coreos/flannel v0.10.0-amd64 f0fad859c909 6 months ago 44.6MB
[root@master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready master 14m v1.11.2

此时master节点状态变为

Ready

在node节点上执行 `kubeadm join` 命令

使用master节点执行

kubeadm init

命令的输出，在node上执行，使其加入集群。

[root@node01 ~]# kubeadm join 192.168.18.128:6443 --token n84v6t.c7d83cn4mo2z8wyr --discovery-token-ca-cert-hash sha256:b946c145416fe1995e1d4d002c149e71a897acc7b106d94cee2920cb2c85ce29 --ignore-preflight-errors=Swap

[preflight] running pre-flight checks
 [WARNING RequiredIPVSKernelModulesAvailable]: the IPVS proxier will not be used, because the following required kernel modules are not loaded: [ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh] or no builtin kernel ipvs support: map[ip_vs_sh:{} nf_conntrack_ipv4:{} ip_vs:{} ip_vs_rr:{} ip_vs_wrr:{}]
you can solve this problem with following methods:
 1. Run 'modprobe -- ' to load missing kernel modules;
2. Provide the missing builtin kernel ipvs support

 [WARNING Swap]: running with swap on is not supported. Please disable swap
I0815 22:02:30.751069 15460 kernel_validator.go:81] Validating kernel version
I0815 22:02:30.751145 15460 kernel_validator.go:96] Validating kernel config
 [WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 18.06.0-ce. Max validated version: 17.03
[discovery] Trying to connect to API Server "192.168.18.128:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://192.168.18.128:6443"
[discovery] Requesting info from "https://192.168.18.128:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "192.168.18.128:6443"
[discovery] Successfully established connection with API Server "192.168.18.128:6443"
[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.11" ConfigMap in the kube-system namespace
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[preflight] Activating the kubelet service
[tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap...
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "node01" as an annotation

This node has joined the cluster:
* Certificate signing request was sent to master and a response
 was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the master to see this node join the cluster.

在两个节点上，执行完毕上述命令后，在master上查看

[root@master ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION master Ready master 23m v1.11.2 node01 Ready <none> 2m v1.11.2 node02 Ready <none> 1m v1.11.2

部署成功。

部署完成后的观察

检查现在正在运行的pod

[root@master ~]# kubectl get pods -n kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
coredns-78fcdf6894-cv4gp  1/1 Running  0 28m  10.244.0.3 master  <none>
coredns-78fcdf6894-wmd25    1/1 Running  0 28m  10.244.0.2 master  <none>
etcd-master     1/1 Running 0 15m  192.168.18.128 master <none>
kube-apiserver-master     1/1 Running 0 15m  192.168.18.128 master  <none>
kube-controller-manager-master 1/1 Running 0 15m 192.168.18.128 master  <none>
kube-flannel-ds-amd64-48rvq  1/1 Running 3 6m  192.168.18.130 node02 <none>
kube-flannel-ds-amd64-7dw42 1/1 Running 3  7m  192.168.18.129 node01 <none>
kube-flannel-ds-amd64-r42wr 1/1  Running 0  16m  192.168.18.128  master <none>
kube-proxy-6fgjm     1/1   Running 0  28m  192.168.18.128 master <none>
kube-proxy-6mngv     1/1   Running 0  7m  192.168.18.129  node01 <none>
kube-proxy-9sh2n    1/1  Running 0  6m  192.168.18.130 node02 <none>
kube-scheduler-master 1/1 Running 0   15m  192.168.18.128 master <none>

可以发现：

kube-apiserver， kube-scheduler， kube-controller，etcd-master运行在master上，
kube-flannel在三个节点上均有运行
kube-proxy 在三个节点均有运行

本文转自SegmentFault-

使用kubeadm部署kubernetes集群

使用kubeadm部署kubernetes集群集群规划部署方法：开始部署部署完成后的观察