天天看點

ceph對接k8s storage class

簡介

對接ceph的rbd和cephfs到k8s中提供持久化存儲

環境

主機名 IP role 作業系統
ceph-01 172.16.31.11 mon osd CentOS7.8
ceph-02 172.16.31.12 Osd CentOS7.8
ceph-03 172.16.31.13 osd CentOS7.8

這個是官網的圖

[外鍊圖檔轉存失敗,源站可能有防盜鍊機制,建議将圖檔儲存下來直接上傳(img-v0yioV1B-1610092913362)(https://docs.ceph.com/en/mimic/_images/7c79c29c18fe63f3e72dc8af9524487f9068e450166d6cc79775bd37c8191317.png)]

步驟

安裝ceph

主機名設定

## ceph-01
hostnamectl set-hostname ceph-01
## ceph-02
hostnamectl set-hostname ceph-01
## ceph-03
hostnamectl set-hostname ceph-01
           

添加主機映射

cat << EOF >> /etc/hosts
172.16.31.11 ceph-01
172.16.31.12 ceph-02
172.16.31.13 ceph-03
EOF
           

關閉防火牆

systemctl stop firewalld && systemctl stop firewalld
setenforce 0 && sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config 
iptables -F && iptables -X && iptables -Z
           

時間同步

yum install -y ntpdate
ntpdate
           

ssh無密鑰通路

## ceph-01節點執行
ssh-keygen 

ssh-copy-id ceph-01

ssh-copy-id ceph-02

ssh-copy-id ceph-03
           

準備repo

yum install epel-release -y
cat << EOF > /etc/yum.repos.d/ceph-deploy.repo
[ceph-noarch]
name=Ceph noarch packages
baseurl=https://download.ceph.com/rpm-luminous/el7/noarch
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://download.ceph.com/keys/release.asc
EOF
           

國内使用者可以用阿裡的倉庫

wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
cat << EOF > /etc/yum.repos.d/ceph-deploy.repo
[ceph-noarch]
name=Ceph noarch packages
baseurl=https://mirrors.aliyun.com/ceph/rpm-luminous/el7/noarch/
enabled=1
gpgcheck=0
EOF
           

安裝ceph-deploy軟體包

ceph-1節點

yum install ceph-deploy yum-plugin-priorities python2-pip bash-completion -y
           

其他節點安裝

yum install yum-plugin-priorities python2-pip bash-completion -y
           

建立一個ceph目錄

ceph-01節點

mkdir ceph-cluster
cd ceph-cluster
           

初始化ceph叢集

ceph-deploy new ceph-01
           

(可選)修改網絡接口

如果有兩個網卡,可以将管理和存儲網分離

public_network = 172.16.0.0/16
cluster_network = 192.168.31.0/24
           

安裝ceph軟體包

ceph-deploy install ceph-01 ceph-02 ceph-03
           

國内加速可以指定阿裡雲鏡像位址,先在所有節點添加這個倉庫

cat << EOF > /etc/yum.repos.d/ceph-luminous.repo
[ceph]
name=Ceph packages for x86_64
baseurl=http://mirrors.aliyun.com/ceph/rpm-luminous/el7/x86_64
enabled=1
gpgcheck=0
EOF
           

然後執行

ceph-deploy install ceph-01 ceph-02 ceph-03 --no-adjust-repos
           

建立mon

ceph-deploy mon create-initial
           

執行完後會建立*.keyring 密鑰環

複制配置和秘鑰到對應的節點上

ceph-deploy admin ceph-01 ceph-02 ceph-03 
           

部署mgr

ceph-deploy mgr create ceph-01
           

mgr是ceph-12.x版本(luminous)新增的元件

部署osd

ceph-deploy osd create --data /dev/sdb ceph-01
ceph-deploy osd create --data /dev/sdb ceph-02
ceph-deploy osd create --data /dev/sdb ceph-03
           

檢查叢集狀态

ceph health
ceph -s
           

測試

建立pool

ceph osd pool create test 8 8
echo `date` > date.txt
rados put test-object-1 date.txt --pool=test
           
上傳到存儲池中
echo `date` > date.txt
rados put test-object-1 date.txt --pool=test
           
檢視存儲池和對象映射
rados -p test ls
ceph osd map test test-object-1
           
删除
rados rm test-object-1 --pool=test
ceph osd pool rm test test --yes-i-really-really-mean-it
           

這裡删不掉的話,需要添加這個配置

然後重新開機mon

ceph-deploy --overwrite-conf admin ceph-01 ceph-02 ceph-03
systemctl restart [email protected]
## 再執行删除
ceph osd pool rm test test --yes-i-really-really-mean-it
           

ceph rbd對接kubernetes

參考github連接配接:https://kubernetes.io/zh/docs/concepts/storage/storage-classes/#ceph-rbd

建立pool

ceph osd pool create kube-pool 64 64
           

導入admin keyring

擷取admin keyring

ceph auth get-key client.admin
           

将key換成上一步輸出的結果

kubectl create secret generic ceph-secret  -n kube-system \
  --type="kubernetes.io/rbd" \
  --from-literal=key='AQDYuPZfdjykCxAAXApI8weHFiZdEPcoc8EaRA=='
           

建立 user secret

ceph auth add client.kube mon 'allow r' osd 'allow rwx pool=kube-pool'
ceph auth get-key client.kube
kubectl create secret generic ceph-secret-user -n kube-system  --from-literal=key='AQAH2vZfe8wWIhAA0w81hjSAoqmjayS5SmWuVQ=='  --type=kubernetes.io/rbd
           

建立StorageClass

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ceph-rbd
provisioner: kubernetes.io/rbd
parameters:
  monitors: 172.16.31.11:6789
  adminId: admin
  adminSecretName: ceph-secret
  adminSecretNamespace: kube-system
  pool: kube-pool
  userId: kube
  userSecretName: ceph-secret-user
  userSecretNamespace: kube-system
  fsType: ext4
  imageFormat: "2"
  imageFeatures: "layering"
           

worker節點安裝ceph-common

cat << EOF > /etc/yum.repos.d/ceph-luminous.repo
[ceph]
name=Ceph packages for x86_64
baseurl=http://mirrors.aliyun.com/ceph/rpm-luminous/el7/x86_64
enabled=1
gpgcheck=0
EOF
yum install -y ceph-common
           

建立PVC

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rbd-1
  namespace: default
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: ceph-rbd
           

建立deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: test-rbd
  name: test-rbd
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test-rbd
  template:
    metadata:
      labels:
        app: test-rbd
    spec:
      containers:
      - image: zerchin/network
        imagePullPolicy: IfNotPresent
        name: test-rbd
        volumeMounts:
        - mountPath: /data
          name: rbd
      volumes:
      - name: rbd
        persistentVolumeClaim:
          claimName: rbd-1
           

常見問題

問題1:rbd未加載報錯
MountVolume.WaitForAttach failed for volume "pvc-8d8a8ed9-bcdb-4de8-a725-9121fcb89c84" : rbd: map failed exit status 2, rbd output: libkmod: ERROR ../libkmod/libkmod.c:586 kmod_search_moddep: could not open moddep file '/lib/modules/4.4.247-1.el7.elrepo.x86_64/modules.dep.bin' modinfo: ERROR: Module alias rbd not found. modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.4.247-1.el7.elrepo.x86_64/modules.dep.bin' modprobe: FATAL: Module rbd not found in directory /lib/modules/4.4.247-1.el7.elrepo.x86_64 rbd: failed to load rbd kernel module (1) rbd: sysfs write failed In some cases useful info is found in syslog - try "dmesg | tail". rbd: map failed: (2) No such file or directory
           

原因

主要就是沒有加載rbd子產品,需要到所有的worker節點上加載rbd子產品

解決

modprobe rbd
           

參考:https://forums.cnrancher.com/q_445.html

問題2:挂載失敗
MountVolume.WaitForAttach failed for volume "pvc-aa0d2e46-3df3-4c70-a318-ad95d4d0810a" : rbd: map failed exit status 110, rbd output: rbd: sysfs write failed In some cases useful info is found in syslog - try "dmesg | tail". rbd: map failed: (110) Connection timed out
           

解決

ceph osd crush tunables hammer
           

參考:https://github.com/rancher/rancher/issues/13198#issuecomment-391920740

問題3:ceph HEALTH_WARN

解決

ceph health detail
ceph osd pool application enable kube-pool rbd
           

部署cephfs檔案系統

k8s預設沒有cephfs的provisioner,是以需要手動部署一個provisioner去對接cephfs

參考github連結:https://github.com/kubernetes-retired/external-storage/tree/master/ceph/cephfs

部署mds(中繼資料服務)

ceph-deploy mds create ceph-01
           

建立兩個存儲池,用來存放實際的資料以及中繼資料

ceph osd pool create cephfs_data 64
ceph osd pool create cephfs_metadata 64
           

建立cephfs檔案系統

ceph fs new cephfs cephfs_metadata cephfs_data
           

檢視mds狀态

ceph mds stat
ceph -s
           

部署provisioner

這裡有兩種方式部署provisioner,其中一種是直接

docker run

的方式部署,另一種是通過

deployment

的方式部署到k8s中

docker run方式部署ceph-provisioner
docker run -tid -v /root/.kube:/kube -v /var/run/kubernetes:/var/run/kubernetes --privileged --net=host --name ceph-provisioner quay.io/external_storage/cephfs-provisioner   /usr/local/bin/cephfs-provisioner   -master=https://172.16.0.99:6443   -kubeconfig=/kube/config -id=cephfs-provisioner-1 -disable-ceph-namespace-isolation
           
deployment方式部署到k8s中

rbac相關yaml

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cephfs-provisioner
  namespace: cephfs
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
  - apiGroups: [""]
    resources: ["services"]
    resourceNames: ["kube-dns","coredns"]
    verbs: ["list", "get"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cephfs-provisioner
subjects:
  - kind: ServiceAccount
    name: cephfs-provisioner
    namespace: cephfs
roleRef:
  kind: ClusterRole
  name: cephfs-provisioner
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: cephfs-provisioner
  namespace: cephfs
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["create", "get", "delete"]
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: cephfs-provisioner
  namespace: cephfs
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: cephfs-provisioner
subjects:
- kind: ServiceAccount
  name: cephfs-provisioner
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: cephfs-provisioner
  namespace: cephfs
           

ceph-provisioner-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cephfs-provisioner
  namespace: cephfs
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cephfs-provisioner
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: cephfs-provisioner
    spec:
      containers:
      - name: cephfs-provisioner
        image: "quay.io/external_storage/cephfs-provisioner:latest"
        env:
        - name: PROVISIONER_NAME
          value: ceph.com/cephfs
        - name: PROVISIONER_SECRET_NAMESPACE
          value: cephfs
        command:
        - "/usr/local/bin/cephfs-provisioner"
        args:
        - "-id=cephfs-provisioner-1"
        - "-disable-ceph-namespace-isolation"
      serviceAccount: cephfs-provisioner
           

儲存上述兩個檔案,并執行

kubectl apply

kubectl apply -f cephfs-provisioner-rbac.yaml
kubectl apply -f cephfs-provisioner-deployment.yaml
           

導入秘鑰

ceph auth get-key client.admin > /tmp/secret
kubectl create ns cephfs
kubectl create secret generic ceph-secret-admin --from-file=/tmp/secret --namespace=cephfs
           

建立Storage class

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: cephfs
provisioner: ceph.com/cephfs
parameters:
    monitors: 172.16.31.11:6789
    adminId: admin
    adminSecretName: ceph-secret-admin
    adminSecretNamespace: "cephfs"
    claimRoot: /pvc-volumes
           

建立pvc

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: cephfs
spec:
  storageClassName: cephfs
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
           

建立deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: test-cephfs
  name: test-cephfs
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test-cephfs
  template:
    metadata:
      labels:
        app: test-cephfs
    spec:
      containers:
      - image: zerchin/network
        imagePullPolicy: IfNotPresent
        name: test-rbd
        volumeMounts:
        - mountPath: /data
          name: cephfs
      volumes:
      - name: cephfs
        persistentVolumeClaim:
          claimName: cephfs
           

常見問題

問題:無法挂載cephfs
MountVolume.SetUp failed for volume "pvc-e4373999-8380-4211-99c5-5d096f234b35" : CephFS: mount failed: mount failed: exit status 5 Mounting command: mount Mounting arguments: -t ceph -o <masked>,<masked> 172.16.29.5:6789:/pvc-volumes/kubernetes/kubernetes-dynamic-pvc-b3e72054-4dc7-11eb-abdd-f21be6c36b31 /var/lib/kubelet/pods/5986dc99-b707-4ea9-b6b2-ae7ffd457c99/volumes/kubernetes.io~cephfs/pvc-e4373999-8380-4211-99c5-5d096f234b35 Output: modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.4.247-1.el7.elrepo.x86_64/modules.dep.bin' modprobe: FATAL: Module ceph not found in directory /lib/modules/4.4.247-1.el7.elrepo.x86_64 failed to load ceph kernel module (1) mount error 5 = Input/output error
           

此時手動在背景使用mount.ceph挂載對應的目錄也挂載不上

原因

在 cephfs_provisoner.py 的實作中,預設添加了對 cephfs namespace 的支援,是以在對 volume 授權時會添加對 namespace 相關的權限設定。因為,我們使用的 ceph 版本 luminous 沒有對 namespace 進行支援,是以,在使用時産生了建立的 volume 挂載到 pod 内後沒有讀寫權限"input/output error"的問題。 此時,你在 cephfs 端檢視卷的讀寫權限時,你可以看到目錄讀寫權限都是問号。于是我們修改了這部分邏輯,去掉了 namespace 相關的部分。

解決

設定ceph-provisioner啟動時添加該參數

-disable-ceph-namespace-isolation

參考:https://www.infoq.cn/article/jqhjzvvl11escvfydruc

擴充

高可用

添加多個mon

ceph-deploy mon add ceph-02 ceph-03
           

當ceph叢集有多個mon時,ceph會同步mon并形成仲裁,檢查仲裁狀态指令如下:

ceph quorum_status --format json-pretty
           

添加mgr

ceph-deploy mgr create ceph-02 ceph-03
           

檢視叢集狀态

ceph -s
           

節點清理ceph

ceph-deploy purge [ceph-node]
ceph-deploy purgedata [ceph-node]
ceph-deploy forgetkeys
rm ceph.*
           

繼續閱讀