Kubernetes 安裝 Rook Ceph 存儲

資料：

官網：https://rook.io 

官網教學：https://rook.io/docs/rook/v1.6/ceph-quickstart.html

1、kubernetes叢集安裝

            kubernetes叢集安裝

2、檢查所有磁盤

lsblk -f

NAME               FSTYPE      LABEL UUID                                   MOUNTPOINT
sda                                                                         
├─sda1             vfat              4903-0788                              /boot/efi
├─sda2             xfs               a16acff6-f085-46a7-b5f5-3cce6ffcc9be   /boot
└─sda3             LVM2_member       vvS0sK-oaBA-lu6w-q34K-628p-MfVN-xZ0090 
  ├─centos-root    xfs               09334c34-6c55-41a0-9190-194f3177cc04   /
  ├─centos-swap    swap              3120a952-eb9c-442c-bcd5-9010cec98c87   
  └─centos-home    xfs               2bdb4e09-8929-447b-a9d1-6d45e70751eb   /home
sdc                                                                         
sdd                
           

3、Git 下載下傳yaml檔案      

git clone --single-branch --branch v1.6.8 https://github.com/rook/rook.git
           

4、私有倉庫處理

        非私有倉庫的跳過這一步

        編輯 rook/cluster/examples/kubernetes/ceph/operator.yaml

vi rook/cluster/examples/kubernetes/ceph/operator.yaml

#編輯鏡像到私有鏡像 去除#符号
ROOK_CSI_CEPH_IMAGE: "xxxx.xx/cephcsi/cephcsi:v3.3.1"
ROOK_CSI_REGISTRAR_IMAGE: "xxxx.xx/sig-storage/csi-node-driver-registrar:v2.2.0"
ROOK_CSI_RESIZER_IMAGE: "xxxx.xx/sig-storage/csi-resizer:v1.2.0"
ROOK_CSI_PROVISIONER_IMAGE: "xxxx.xx/sig-storage/csi-provisioner:v2.2.2"
ROOK_CSI_SNAPSHOTTER_IMAGE: "xxxx.xx/sig-storage/csi-snapshotter:v4.1.1"
ROOK_CSI_ATTACHER_IMAGE: "xxxx.xx/sig-storage/csi-attacher:v3.2.1"
CSI_VOLUME_REPLICATION_IMAGE: "xxxx.xx/csiaddons/volumereplication-operator:v0.1.0"

image: xxxx.xx/rook/ceph:v1.6.8
           

         同時從公網拉取鏡像到私有鏡像伺服器

5、部署rook-ceph公共資源

kubectl create -f crds.yaml -f common.yaml -f operator.yaml

kubectl -n rook-ceph get pod
           

6、修改cluster.yaml

vi cluster.yaml
           

cephVersion:
    image: xxxx.xx/ceph/ceph:v15.2.13  #修改私有鏡像

  storage: # cluster level storage configuration and selection
    useAllNodes: false  #修改成false,手工指定節點
    useAllDevices: false  #修改成false,手工指定磁盤
    #指定伺服器和磁盤
    nodes:
     - name: "k8s-node-01"
       devices:
       - name: "sdc"
       - name: "sdd"
     - name: "k8s-node-02"
       devices:
       - name: "sdc"
       - name: "sdd"
     - name: "k8s-node-03"
       devices:
       - name: "sdc"
       - name: "sdd"
     - name: "k8s-node-04"
       devices:
       - name: "sdc"
       - name: "sdd"
           

7、初始化rook-ceph叢集

kubectl apply -f cluster.yaml
           

8、使用toolbox檢視叢集狀态

  8.1修改toolbox.yaml改私有鏡像

vi toolbox.yaml
           

spec:
  template:
    spec:
      containers:
        - name: rook-ceph-tools
          image: xxxxx.xx/rook/ceph:v1.6.8   #修改成私有鏡像
           

8.2 部署toolbox

kubectl apply -f toolbox.yaml
           

8.3 運作toolbox

kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- bash
           

8.4查詢ceph狀态

ceph status
           

cluster:
    id:     01cb2eb5-862c-410a-9db2-6712d5e096f1
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum a,b,c (age 49m)
    mgr: a(active, since 46m), standbys: b
    osd: 8 osds: 8 up (since 46m), 8 in (since 46m)
 
  data:
    pools:   1 pools, 1 pgs
    objects: 0 objects, 0 B
    usage:   8.0 GiB used, 17 TiB / 17 TiB avail
    pgs:     1 active+clean
           

9、建立storageclass

kubectl apply -f filesystem.yaml
kubectl apply -f csi/cephfs/storageclass.yaml
kubectl apply -f csi/rbd/storageclass.yaml