文章目錄

前言
一、高可用叢集
- 1.1 高可用叢集技術細節
二、部署高可用叢集
- 2.1 準備環境
- 2.2 所有master節點部署keepalived
- - 2.2.1 安裝相關包和keepalived
  - 2.2.2 配置master節點
  - 2.2.3 啟動和檢查
- 2.3 部署haproxy
- 2.4 所有節點安裝Docker/kubeadm/kubelet
- - 2.4.1 安裝Docker
  - 2.4.2 添加kubernetes軟體源
  - 2.4.3 安裝kubeadm，kubelet和kubectl
- 2.5 部署Kubernetes Master
- 2.5.1 建立kubeadm配置檔案
- 2.6 安裝叢集網絡
- 2.7 master2節點加入叢集
- - 2.7.1 複制密鑰及相關檔案
  - 2.7.2 master2加入叢集
  - 2.8 加入Kubernetes Node

前言

之前我們搭建的叢集，隻有一個master節點，當master節點當機的時候，通過node将無法繼續通路，而master主要是管理作用，是以整個叢集将無法提供服務

一、高可用叢集

下面我們就需要搭建一個多master節點的高可用叢集，不會存在單點故障問題

但是在node 和 master節點之間，需要存在一個 LoadBalancer元件，作用如下：

負載
檢查master節點的狀态

16、Kubernetes搭建高可用叢集前言一、高可用叢集二、部署高可用叢集

對外有一個統一的VIP：虛拟ip來對外進行通路

1.1 高可用叢集技術細節

16、Kubernetes搭建高可用叢集前言一、高可用叢集二、部署高可用叢集

keepalived：配置虛拟ip，檢查節點的狀态
haproxy：負載均衡服務【類似于nginx】
apiserver：
controller：
manager：
scheduler

二、部署高可用叢集

2.1 準備環境

角色	IP
master1	192.168.88.10
master2	192.168.88.11
node1	192.168.88.12
VIP（虛拟ip）	192.168.88.100

需要在這三個節點上進行操作

# 關閉防火牆
[[email protected] ~]$ systemctl stop firewalld && systemctl disable firewalld

# 關閉selinux
# 永久關閉
[[email protected] ~]$ sed -i 's/enforcing/disabled/' /etc/selinux/config  && setenforce 0  
# 臨時關閉


# 關閉swap
# 臨時
[[email protected] ~]$ swapoff -a 
# 永久關閉
[[email protected] ~]$ sed -ri 's/.*swap.*/#&/' /etc/fstab

# 根據規劃設定主機名【master1節點上操作】
[[email protected] ~]$ hostnamectl set-hostname master1

# 根據規劃設定主機名【master2節點上操作】
[[email protected] ~]$ hostnamectl set-hostname master2

# 根據規劃設定主機名【node1節點操作】
[[email protected] ~]$ hostnamectl set-hostname node1

# 在master添加hosts
[[email protected] ~]$ cat >> /etc/hosts << EOF
192.168.88.100  k8smaster
192.168.88.10 master01.k8s.io master1
192.168.88.11 master02.k8s.io master2
192.168.88.12 node01.k8s.io node1
EOF


# 将橋接的IPv4流量傳遞到iptables的鍊【3個節點上都執行】
[[email protected] ~]$ cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF

# 生效
[[email protected] ~]$ sysctl --system  

# 時間同步
[[email protected] ~]$ yum install ntpdate -y
[[email protected] ~]$ ntpdate time.windows.com

2.2 所有master節點部署keepalived

2.2.1 安裝相關包和keepalived

[[email protected] ~]$ yum install -y conntrack-tools libseccomp libtool-ltdl

[[email protected] ~]$ yum install -y keepalived

2.2.2 配置master節點

[[email protected] ~]$ cat > /etc/keepalived/keepalived.conf <<EOF 
! Configuration File for keepalived

global_defs {
   router_id k8s
}

vrrp_script check_haproxy {
    script "killall -0 haproxy"
    interval 3
    weight -2
    fall 10
    rise 2
}

vrrp_instance VI_1 {
    state MASTER 
    interface ens33 
    virtual_router_id 51
    priority 250
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass ceb1b3ec013d66163d6ab
    }
    virtual_ipaddress {
        192.168.88.100   #vip
    }
    track_script {
        check_haproxy
    }

}
EOF

master2節點配置

[[email protected] ~]$ cat > /etc/keepalived/keepalived.conf <<EOF 
! Configuration File for keepalived

global_defs {
   router_id k8s
}

vrrp_script check_haproxy {
    script "killall -0 haproxy"
    interval 3
    weight -2
    fall 10
    rise 2
}

vrrp_instance VI_1 {
    state BACKUP 
    interface ens33 
    virtual_router_id 51
    priority 200
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass ceb1b3ec013d66163d6ab
    }
    virtual_ipaddress {
        192.168.88.100
    }
    track_script {
        check_haproxy
    }

}
EOF

2.2.3 啟動和檢查

在兩台master節點都執行

# 啟動keepalived，設定開機啟動
[[email protected] ~]$ systemctl start keepalived.service && systemctl enable keepalived.service
# 檢視啟動狀态
[[email protected] ~]$ systemctl status keepalived.service

啟動後檢視master1的網卡資訊

[[email protected] ~]$ ip a s ens33

2.3 部署haproxy

安裝

[[email protected] ~]$ yum install -y haproxy

配置

兩台master節點的配置均相同，配置中聲明了後端代理的兩個master節點伺服器，指定了haproxy運作的端口為16443等，是以16443端口為叢集的入口

[[email protected] ~]$ cat > /etc/haproxy/haproxy.cfg << EOF
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    # to have these messages end up in /var/log/haproxy.log you will
    # need to:
    # 1) configure syslog to accept network log events.  This is done
    #    by adding the '-r' option to the SYSLOGD_OPTIONS in
    #    /etc/sysconfig/syslog
    # 2) configure local2 events to go to the /var/log/haproxy.log
    #   file. A line like the following can be added to
    #   /etc/sysconfig/syslog
    #
    #    local2.*                       /var/log/haproxy.log
    #
    log         127.0.0.1 local2
    
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon 
       
    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------  
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000
#---------------------------------------------------------------------
# kubernetes apiserver frontend which proxys to the backends
#--------------------------------------------------------------------- 
frontend kubernetes-apiserver
    mode                 tcp
    bind                 *:16443
    option               tcplog
    default_backend      kubernetes-apiserver    
#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend kubernetes-apiserver
    mode        tcp
    balance     roundrobin
    server      master01.k8s.io   192.168.44.155:6443 check
    server      master02.k8s.io   192.168.44.156:6443 check
#---------------------------------------------------------------------
# collection haproxy statistics message
#---------------------------------------------------------------------
listen stats
    bind                 *:1080
    stats auth           admin:awesomePassword
    stats refresh        5s
    stats realm          HAProxy\ Statistics
    stats uri            /admin?stats
EOF

兩台master都啟動

# 設定開機啟動
[[email protected] ~]$ systemctl enable haproxy && systemctl start haproxy
# 檢視啟動狀态
[[email protected] ~]$ systemctl status haproxy

檢查端口

[[email protected] ~]$ netstat -lntup|grep haproxy

2.4 所有節點安裝Docker/kubeadm/kubelet

Kubernetes預設CRI（容器運作時）為Docker，是以先安裝Docker。

2.4.1 安裝Docker

#配置一下Docker的阿裡yum源
[[email protected] ~]$ wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo

#yum方式安裝docker
[[email protected] ~]$ yum -y install docker-ce-18.06.1.ce-3.el7

# 啟動docker
[[email protected] ~]$ systemctl enable docker && systemctl start docker
[[email protected] ~]$ docker --version
Docker version 18.06.1-ce, build e68fc7a

配置docker的鏡像源

[[email protected] ~]$ cat >> /etc/docker/daemon.json << EOF
{
  "registry-mirrors": ["https://b9pmyelo.mirror.aliyuncs.com"]
}
EOF

然後重新開機docker

[[email protected] ~]$ systemctl restart docker

2.4.2 添加kubernetes軟體源

配置一下yum的k8s軟體源

[[email protected] ~]$ cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

2.4.3 安裝kubeadm，kubelet和kubectl

[[email protected] ~]$ yum install -y kubelet-1.18.0 kubeadm-1.18.0 kubectl-1.18.0
[[email protected] ~]$ systemctl enable kubelet && systemctl start kubelet

2.5 部署Kubernetes Master

2.5.1 建立kubeadm配置檔案

在具有vip的master上進行初始化操作，這裡為master1

# 建立檔案夾
[[email protected] ~]$ mkdir /usr/local/kubernetes/manifests -p

# 到manifests目錄
[[email protected] ~]$ cd /usr/local/kubernetes/manifests/

# 建立yaml檔案
[[email protected] ~]$ cat > kubeadm-config.yaml << EOF
apiServer:
  certSANs:
    - master1
    - master2
    - master.k8s.io
    - 192.168.88.10
    - 192.168.88.11
    - 192.168.88.12
    - 127.0.0.1
  extraArgs:
    authorization-mode: Node,RBAC
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta1
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: "master.k8s.io:16443"
controllerManager: {}
dns: 
  type: CoreDNS
etcd:
  local:    
    dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.16.3
networking: 
  dnsDomain: cluster.local  
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.1.0.0/16
scheduler: {}
EOF

在master1節點執行

[[email protected] ~]$ kubeadm init --config kubeadm-config.yaml

按照提示配置環境變量，使用kubectl工具

# 執行下方指令
[[email protected] ~]$ mkdir -p $HOME/.kube
[[email protected] ~]$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[[email protected] ~]$ sudo chown $(id -u):$(id -g) $HOME/.kube/config

# 檢視節點
[[email protected] ~]$ kubectl get nodes

# 檢視pod
[[email protected] ~]$ kubectl get pods -n kube-system

按照提示儲存以下内容，一會要使用：

#注意，每個人的都不一樣
kubeadm join master.k8s.io:16443 --token jv5z7n.3y1zi95p952y9p65 \
    --discovery-token-ca-cert-hash sha256:403bca185c2f3a4791685013499e7ce58f9848e2213e27194b75a2e3293d8812 \
    --control-plane

–control-plane ：隻有在添加master節點的時候才有

檢視叢集狀态

# 檢視叢集狀态
[[email protected] ~]$ kubectl get cs

# 檢視pod
[[email protected] ~]$ kubectl get pods -n kube-system

2.6 安裝叢集網絡

擷取到flannel的yaml，在master1上執行

# 建立檔案夾
[[email protected] ~]$ mkdir flannel && cd flannel
# 下載下傳yaml檔案
[[email protected] ~]$ wget https://www.chenleilei.net/soft/k8s/kube-flannel.yaml 
#直接使用位址這個即可

#替換倉庫位址，quay.io國内通路不到，需要修改為quay-mirror.qiniu.com
[[email protected] ~]$ sed -i 's#quay.io#quay-mirror.qiniu.com#g' kube-flannel.yaml   


 
[[email protected] ~]$ kubectl apply -f kube-flannel.yaml

檢查

[[email protected] ~]$ kubectl get pods -n kube-system

2.7 master2節點加入叢集

2.7.1 複制密鑰及相關檔案

從master1複制密鑰及相關檔案到master2

[[email protected] ~]$ ssh [email protected].168.88.11 mkdir -p /etc/kubernetes/pki/etcd

[[email protected] ~]$ scp /etc/kubernetes/admin.conf [email protected].168.88.11 :/etc/kubernetes
   
[[email protected] ~]$ scp /etc/kubernetes/pki/{ca.*,sa.*,front-proxy-ca.*} [email protected].168.88.11 :/etc/kubernetes/pki
   
[[email protected] ~]$ scp /etc/kubernetes/pki/etcd/ca.* [email protected].168.88.11 :/etc/kubernetes/pki/etcd

2.7.2 master2加入叢集

執行在master1上init後輸出的join指令,需要帶上參數

--control-plane

表示把master控制節點加入叢集

[[email protected] ~]$ kubeadm join master.k8s.io:16443 --token ckf7bs.30576l0okocepg8b     --discovery-token-ca-cert-hash sha256:19afac8b11182f61073e254fb57b9f19ab4d798b70501036fc69ebef46094aba --control-plane

檢查狀态

[[email protected] ~]$ kubectl get node

[[email protected] ~]$ kubectl get pods --all-namespaces

2.8 加入Kubernetes Node

在node1上執行

向叢集添加新節點，執行在kubeadm init輸出的kubeadm join指令：

[[email protected] ~]$ kubeadm join master.k8s.io:16443 --token ckf7bs.30576l0okocepg8b     --discovery-token-ca-cert-hash sha256:19afac8b11182f61073e254fb57b9f19ab4d798b70501036fc69ebef46094aba

叢集網絡重新安裝，因為添加了新的node節點

檢查狀态

[[email protected] ~]$ kubectl get node
[[email protected] ~]$ kubectl get pods --all-namespaces

測試kubernetes叢集

在Kubernetes叢集中建立一個pod，驗證是否正常運作

# 建立nginx deployment
[[email protected] ~]$ kubectl create deployment nginx --image=nginx

# 暴露端口
[[email protected] ~]$ kubectl expose deployment nginx --port=80 --type=NodePort

# 檢視狀态
[[email protected] ~]$ kubectl get pod,svc

通過任何一個節點，都能夠通路我們的nginx頁面，通路位址：http://NodeIP:Port

16、Kubernetes搭建高可用叢集前言一、高可用叢集二、部署高可用叢集

文章目錄

前言

一、高可用叢集

1.1 高可用叢集技術細節

二、部署高可用叢集

2.1 準備環境

2.2 所有master節點部署keepalived

2.2.1 安裝相關包和keepalived

2.2.2 配置master節點

2.2.3 啟動和檢查

2.3 部署haproxy

2.4 所有節點安裝Docker/kubeadm/kubelet

2.4.1 安裝Docker

2.4.2 添加kubernetes軟體源

2.4.3 安裝kubeadm，kubelet和kubectl

2.5 部署Kubernetes Master

2.5.1 建立kubeadm配置檔案

2.6 安裝叢集網絡

2.7 master2節點加入叢集

2.7.1 複制密鑰及相關檔案

2.7.2 master2加入叢集

2.8 加入Kubernetes Node

繼續閱讀

pod的深入了解

深入了解Kubernetes資源限制：記憶體深入了解Kubernetes資源限制：記憶體

安裝harborInstallation and Configuration Guide

【k8s學習系列】第2篇，規模和更新部署k8s學習系列前言規模和更新部署總結

Kubernetes - Kubernetes 元件

k8s部署es叢集和kibana

kubernetes學習筆記--挂載GlusterFS存儲卷

Kubernetes - Xshell連接配接虛拟機 & 搭建Kubernetes基礎叢集

Kubernetes學習--資源管理方式

kubernetes-雲原生技術進階第18講：Kubernetes 排程和資源管理第18講：Kubernetes 排程和資源管理一、Kubernetes 排程過程二、Kubernetes 基礎排程力三、Kubernetes 進階排程能力

通過serviceAccount的secret通路kubernetes API Server前提設定環境變量通過curl通路restAPI額外部分

cephadm離線搭建v17.2.0 Quincy版本Ceph叢集叢集規劃準備工作

使用jvm監控工具(jconsole、jvisualvm)通過jmx遠端連接配接kubernetes上的java應用

Error: docker-ce conflicts with 2:docker-1.13.1-53.git774336d.el7.centos.x86_64

golang建構Dockerfile，并打包成鏡像，運作在docker和k8s上

使用kubeadm+calico部署kubernetes v1.25.3