k8s中利用rook安裝ceph叢集

k8s中利用rook 1.6安裝ceph叢集

環境準備

系統版本(使用sealos建構k8s叢集)

k8s: 1.19.6
rook: 1.6
ubuntu: 1.20

安裝條件

您已經安裝了 Kubernetes 叢集，且叢集版本不低于 v1.17.0，安裝方法請參考安裝 Kubernetes 叢集；
- Kubernetes 叢集有至少 3 個工作節點，且每個工作節點都有一塊初系統盤以外的未格式化的裸盤（工作節點是虛拟機時，未格式化的裸盤可以是虛拟磁盤），用于建立 3 個 Ceph OSD；
- 也可以隻有 1 個工作節點，挂載了一塊未格式化的裸盤；
- 在節點機器上執行 lsblk -f 指令可以檢視磁盤是否需被格式化，輸出結果如下：

NAME                  FSTYPE      LABEL UUID                                   MOUNTPOINT
vda
└─vda1                LVM2_member       >eSO50t-GkUV-YKTH-WsGq-hNJY-eKNf-3i07IB
 ├─ubuntu--vg-root   ext4              c2366f76-6e21-4f10-a8f3-6776212e2fe4   /
 └─ubuntu--vg-swap_1 swap              9492a3dc-ad75-47cd-9596-678e8cf17ff9   [SWAP]
vdb

注：如果以前安裝過ceph，那麼現在重新安裝，可能需要将挂載的硬碟重新格式化挂載，請參考這篇文章：Rook Ceph OSD異常，格式化osd硬碟重新挂載

安裝過程

安裝rook，rbd和cephfs的差別請看官方文檔：ceph-csi-drivers

# 下載下傳源碼
$ git clone --single-branch --branch v1.6.0 https://github.com/rook/rook.git
$ cd rook/cluster/examples/kubernetes/ceph
$ kubectl create -f crds.yaml -f common.yaml -f operator.yaml

# 安裝叢集
$ kubectl create -f cluster.yaml

# 安裝toolbox工具
$ kubectl create -f toolbox.yaml

# 安裝dashboard
$ kubectl create -f dashboard-external-https.yaml

# 建立cephfs類型的storageclass
$ k apply -f ./csi/cephfs/storageclass.yaml

驗證安裝

# 等一會兒，會發現所有的pod都啟動了
$ watch kubectl get pods -n rook-ceph
NAME                                                 READY   STATUS      RESTARTS   AGE
csi-cephfsplugin-9phgd                               3/3     Running     0          19m
csi-cephfsplugin-b5zm7                               3/3     Running     0          19m
csi-cephfsplugin-mvx7b                               3/3     Running     0          19m
csi-cephfsplugin-ngkpt                               3/3     Running     0          19m
csi-cephfsplugin-provisioner-db45f85f5-77658         6/6     Running     0          19m
csi-cephfsplugin-provisioner-db45f85f5-88xm9         6/6     Running     0          19m
csi-cephfsplugin-xvrfz                               3/3     Running     0          19m
csi-rbdplugin-59f94                                  3/3     Running     0          19m
csi-rbdplugin-76g7n                                  3/3     Running     0          19m
csi-rbdplugin-p4twb                                  3/3     Running     0          19m
csi-rbdplugin-pjsw9                                  3/3     Running     0          19m
csi-rbdplugin-provisioner-d85cbdb48-cm8zb            6/6     Running     0          19m
csi-rbdplugin-provisioner-d85cbdb48-xg7ph            6/6     Running     0          19m
csi-rbdplugin-tj2vr                                  3/3     Running     0          19m
rook-ceph-crashcollector-gpu-1-5b68d5cd59-mtn7s      1/1     Running     0          19m
rook-ceph-crashcollector-gpu-2-868f498db-p4cg5       1/1     Running     0          16m
rook-ceph-crashcollector-gpu-3-6959b695d5-ns6hf      1/1     Running     0          16m
rook-ceph-crashcollector-worker01-6446f7c66d-4dgvr   1/1     Running     0          18m
rook-ceph-crashcollector-worker02-7c5dbc645-q4xsh    1/1     Running     0          19m
rook-ceph-mgr-a-56dc6bd5dd-ss6d5                     1/1     Running     0          18m
rook-ceph-mon-a-7cf96d4f9f-9l5kq                     1/1     Running     0          19m
rook-ceph-mon-b-7c4d4c48c6-g9z2w                     1/1     Running     0          19m
rook-ceph-mon-c-7dc65846d-hmn6b                      1/1     Running     0          19m
rook-ceph-operator-54cf7487d4-9zhn6                  1/1     Running     0          20m
rook-ceph-osd-0-765fbb9f79-66v97                     1/1     Running     0          18m
rook-ceph-osd-1-d566bfc77-w5xpd                      1/1     Running     1          16m
rook-ceph-osd-2-76754d4875-lzqrz                     1/1     Running     0          16m
rook-ceph-osd-prepare-gpu-1-c6zgm                    0/1     Completed   0          15m
rook-ceph-osd-prepare-gpu-2-r8m9r                    0/1     Completed   0          15m
rook-ceph-osd-prepare-gpu-3-l8skf                    0/1     Completed   0          15m
rook-ceph-osd-prepare-worker01-nq4g2                 0/1     Completed   0          15m
rook-ceph-osd-prepare-worker02-d4qrw                 0/1     Completed   0          15m

# 檢視 ceph 狀态
$ kubectl -n rook-ceph exec -it rook-ceph-tools-76c7d559b6-8w7bk -- sh -c 'ceph status'
  cluster:
    id:     5db57586-6d6f-4529-a956-b41242046ff2
    health: HEALTH_WARN
            clock skew detected on mon.b, mon.c
            mon c is low on available space
 
  services:
    mon: 3 daemons, quorum a,b,c (age 30m)
    mgr: a(active, since 26m)
    osd: 3 osds: 3 up (since 27m), 3 in (since 27m)
 
  data:
    pools:   1 pools, 1 pgs
    objects: 0 objects, 0 B
    usage:   3.0 GiB used, 87 GiB / 90 GiB avail
    pgs:     1 active+clean

# 檢視dashboard
$ kubectl -n rook-ceph get service
NAME                                     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
csi-cephfsplugin-metrics                 ClusterIP   10.106.161.37    <none>        8080/TCP,8081/TCP   3h13m
csi-rbdplugin-metrics                    ClusterIP   10.106.22.108    <none>        8080/TCP,8081/TCP   3h13m
rook-ceph-mgr                            ClusterIP   10.99.57.141     <none>        9283/TCP            3h12m
rook-ceph-mgr-dashboard                  ClusterIP   10.109.130.98    <none>        8443/TCP            3h12m
rook-ceph-mgr-dashboard-external-http    NodePort    10.98.243.88     <none>        7000:30574/TCP      9m49s
rook-ceph-mgr-dashboard-external-https   NodePort    10.96.251.99     <none>        8443:32066/TCP      5s
rook-ceph-mon-a                          ClusterIP   10.100.24.39     <none>        6789/TCP,3300/TCP   3h13m
rook-ceph-mon-b                          ClusterIP   10.107.108.211   <none>        6789/TCP,3300/TCP   3h13m
rook-ceph-mon-c                          ClusterIP   10.96.149.72     <none>        6789/TCP,3300/TCP   3h12m
NAME                                     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
csi-cephfsplugin-metrics                 ClusterIP   10.106.161.37    <none>        8080/TCP,8081/TCP   3h13m
csi-rbdplugin-metrics                    ClusterIP   10.106.22.108    <none>        8080/TCP,8081/TCP   3h13m
rook-ceph-mgr                            ClusterIP   10.99.57.141     <none>        9283/TCP            3h12m
rook-ceph-mgr-dashboard                  ClusterIP   10.109.130.98    <none>        8443/TCP            3h12m
rook-ceph-mgr-dashboard-external-http    NodePort    10.98.243.88     <none>        7000:30574/TCP      9m49s
rook-ceph-mgr-dashboard-external-https   NodePort    10.96.251.99     <none>        8443:32066/TCP      5s
rook-ceph-mon-a                          ClusterIP   10.100.24.39     <none>        6789/TCP,3300/TCP   3h13m
rook-ceph-mon-b                          ClusterIP   10.107.108.211   <none>        6789/TCP,3300/TCP   3h13m
rook-ceph-mon-c                          ClusterIP   10.96.149.72     <none>        6789/TCP,3300/TCP   3h12m

# 擷取dashboard的密碼,使用者名是admin
$ kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath="{['data']['password']}" | base64 --decode && echo

FAQ

crashcollector pod啟動報錯： Unable to attach or mount volumes: unmounted volumes=[rook-ceph-crash-collector-keyring], unattached volumes=[default-token-vttr8 rook-ceph-crash-collector-keyring rook-config-override rook-ceph-log rook-ceph-crash]: timed out waiting for the condition A：按照如下步驟操作：

$ kubectl delete -f cluster.yaml
$ kubectl delete -f operator.yaml -f common.yaml -f crds.yaml 

# 叢集中的每台機器上面執行指令
$ rm -rf /var/lib/rook /var/lib/kubelet/plugins_registry/* /var/lib/kubelet/plugins/

# 重新部署cluster
$ kubectl apply -f cluster.yaml

如果還是不行，就走開篇的重裝ceph的步驟吧

執行ceph status指令的時候報warn： clock skew detected on mon.b, mon.c A: 這個問題是因為mon.b,mon.c服務所在node的時間差距超過了限制，參考clockdiff-檢測兩台linux主機的時間差

# 首先确定mon.b, mon.c服務所在的節點
$ k get pod rook-ceph-mon-b-7c4d4c48c6-g9z2w -n rook-ceph -o wide
NAME                               READY   STATUS    RESTARTS   AGE   IP               NODE       NOMINATED NODE   READINESS GATES
rook-ceph-mon-b-7c4d4c48c6-g9z2w   1/1     Running   0          98m   100.103.88.144   worker02   <none>           <none>

# 檢視worker02的節點時間和master機器的時間差
$ apt install iputils-clockdiff
$ clockdiff 10.20.17.193
.
host=10.20.17.193 rtt=750(187)ms/0ms delta=0ms/0ms Thu Apr 22 20:44:20 2021

# 去每台機器上面檢查ntp服務是否啟動
$ systemctl status ntp                                                           
● ntp.service - Network Time Service
   Loaded: loaded (/lib/systemd/system/ntp.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2021-04-22 20:26:46 CST; 19min ago
     Docs: man:ntpd(8)
 Main PID: 32140 (ntpd)
    Tasks: 2 (limit: 4915)
   CGroup: /system.slice/ntp.service
           └─32140 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 111:115

Apr 22 20:37:03 master ntpd[32140]: 192.168.222.1 local addr 10.20.17.193 -> <null>
Apr 22 20:37:05 master ntpd[32140]: 192.168.222.1 local addr 10.20.17.193 -> <null>
Apr 22 20:37:07 master ntpd[32140]: 192.168.222.1 local addr 10.20.17.193 -> <null>
Apr 22 20:37:07 master ntpd[32140]: 192.168.222.1 local addr 10.20.17.193 -> <null>
Apr 22 20:39:22 master ntpd[32140]: 192.168.222.1 local addr 10.20.17.193 -> <null>
Apr 22 20:39:43 master ntpd[32140]: 192.168.222.1 local addr 10.20.17.193 -> <null>
Apr 22 20:39:54 master ntpd[32140]: 192.168.222.1 local addr 10.20.17.193 -> <null>
Apr 22 20:40:14 master ntpd[32140]: 192.168.222.1 local addr 10.20.17.193 -> <null>
Apr 22 20:40:24 master ntpd[32140]: 192.168.222.1 local addr 10.20.17.193 -> <null>
Apr 22 20:40:25 master ntpd[32140]: 192.168.222.1 local addr 10.20.17.193 -> <null>

# 啟動ntp服務
$ systemctl status ntp

# 開機自啟動ntp服務 
$ systemctl enable ntp

執行ceph status指令的時候報warn： mon c is low on available space A: 這個問題是因為mon.c服務所在主控端節點的磁盤空間不足，注意不是挂載的ceph磁盤不足。

k8s中利用rook安裝ceph叢集

k8s中利用rook 1.6安裝ceph叢集

安裝過程

FAQ

繼續閱讀

【k8s學習系列】第2篇，規模和更新部署k8s學習系列前言規模和更新部署總結

資料庫設計（5）_實體結構設計

Kubernetes - Kubernetes 元件

Ceph介紹（一）：基本原理

k8s部署es叢集和kibana

kubernetes學習筆記--挂載GlusterFS存儲卷

Kubernetes - Xshell連接配接虛拟機 & 搭建Kubernetes基礎叢集

Kubernetes學習--資源管理方式

kubernetes-雲原生技術進階第18講：Kubernetes 排程和資源管理第18講：Kubernetes 排程和資源管理一、Kubernetes 排程過程二、Kubernetes 基礎排程力三、Kubernetes 進階排程能力

分布式存儲伺服器架設不二之選拓普龍S265-12體驗解析

通過serviceAccount的secret通路kubernetes API Server前提設定環境變量通過curl通路restAPI額外部分

cephadm離線搭建v17.2.0 Quincy版本Ceph叢集叢集規劃準備工作

使用jvm監控工具(jconsole、jvisualvm)通過jmx遠端連接配接kubernetes上的java應用

Error: docker-ce conflicts with 2:docker-1.13.1-53.git774336d.el7.centos.x86_64

golang建構Dockerfile，并打包成鏡像，運作在docker和k8s上

使用kubeadm+calico部署kubernetes v1.25.3