如何在K8S中部署redis叢集并使用ceph作為持久化存儲

1、環境準備

請安裝完成K8S環境，此示例中我們有一個master：10.41.10.61，3個node：10.41.10.71-73，1個registry鏡像倉庫：10.41.10.81，1個ceph分布式存儲叢集：10.41.10.81-83

請在K8S所有節點上配置/etc/docker/daemon.json，添加：“insecure-registries”: [“10.41.10.81:5000”]

2、在鏡像倉庫服務上下載下傳Percona_xtradb_cluster鏡像

docker pull redis
docker tag docker.io/redis 10.41.10.81:5000/redis
docker push 10.41.10.81:5000/redis #送出到鏡像倉庫
           

3、在master上部署2個服務

cat <<eof >redis-master-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: redis-master-svc
spec:
  selector:
    node: redis-master
  ports:
  - name: redis
    port: 6379
  type: NodePort
  externalIPs:
  - 10.41.10.60
eof
cat <<eof >redis-slave-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: redis-slave-svc
spec:
  selector:
    node: redis-slave
  ports:
  - name: redis
    port: 6379
eof
kubectl apply -f redis-master-service.yaml #生效服務  
kubectl apply -f redis-slave-service.yaml
kubectl get svc #檢視服務，這一步需要得到redis-master-svc的内部叢集位址，後面添加slave節點時需要它作為master的IP位址
           

4、添加兩個節點的持久化存儲

cat <<eof >redis-master-pv-pvc.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: redis-master-pv
spec:
  capacity:
    storage: 20Gi
  accessModes:
    - ReadWriteMany
  cephfs:
    monitors:
      - 10.41.10.81:6789,10.41.10.82:6789,10.41.10.83:6789
    path: /redis/master
    user: admin
    readOnly: false
    secretRef:
      name: ceph-secret
  persistentVolumeReclaimPolicy: Recycle
---
apiVersion: v1
metadata:
  name: redis-master-pvc
spec:
  accessModes:
    - ReadWriteMany
  volumeName: redis-master-pv
  resources:
    requests:
      storage: 20Gi
eof
cat <<eof >redis-slave-pv-pvc.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: redis-slave-pv
spec:
  capacity:
    storage: 20Gi
  accessModes:
    - ReadWriteMany
  cephfs:
    monitors:
      - 10.41.10.81:6789,10.41.10.82:6789,10.41.10.83:6789
    path: /redis/slave
    user: admin
    readOnly: false
    secretRef:
      name: ceph-secret
  persistentVolumeReclaimPolicy: Recycle
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: redis-slave-pvc
spec:
  accessModes:
    - ReadWriteMany
  volumeName: redis-slave-pv
  resources:
    requests:
      storage: 20Gi
eof
kubectl apply -f redis-master-pv-pvc.yaml #生效存儲
kubectl apply -f redis-slave-pv-pvc.yaml
kubectl get pv #檢視存儲
           

5、在存儲中添加redis配置檔案

我的下載下傳裡找名稱：“k8s部署redis所需要的配置檔案”

上傳以上兩個檔案分别到存儲目錄下的master與slave檔案下，并分别修改檔案名為redis.conf

這個dump.rdb隻有已經進行資料快照才會産生，這裡不用管它。

修改slave檔案夾下面的redis.conf，主要關注以下幾項

requirepass #這裡配置redis的密碼
masterauth #這裡配置slave的複制密碼
slaveof 10.111.80.141 6379 #這裡的主節點就是要用上面我們檢視到的k8s的redis-master-svc叢集位址
           

修改master檔案夾下面的redis.conf，注意slaveof這一項需要注釋掉，其它同上。

6、K8S中添加部署

cat <<eof >redis-master-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis-master
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis-master
  template:
    metadata:
      labels:
        app: redis-master
        node: redis-master
        unit: redis-cluster
    spec:
      containers:
      - name: redis-master
        image: 10.41.10.81:5000/redis
        volumeMounts:
        - mountPath: "/data"
          name: data
        ports:
        - containerPort: 6379
        command: ["redis-server"]
        args: ["/data/redis.conf"]
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: redis-master-pvc
eof
cat <<eof >redis-slave-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis-slave
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis-slave
  template:
    metadata:
      labels:
        app: redis-slave
        node: redis-slave
        unit: redis-cluster
    spec:
      containers:
      - name: redis-slave
        image: 10.41.10.81:5000/redis
        command: ["redis-server"]
        args: ["/data/redis.conf"]
        volumeMounts:
        - mountPath: "/data"
          name: data
        ports:
        - containerPort: 6379
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: redis-slave-pvc
eof
kubectl apply -f redis-master-deployment.yaml #部署主節點
kubectl apply -f redis-slave-deployment.yaml #部署次節點
kubectl get pods -o wide #檢視部署情況
           

7、驗證

分别進入pod，檢視叢集狀态

kubectl exec -it redis-master-75fdfdb8f8-bqwmx /bin/bash
redis-cli
auth [email protected] #輸入密碼
info replication #檢視叢集
           

進入slave節點

kubectl exec -it redis-slave-7b764f878b-tfrdp /bin/bash

redis-cli

auth [email protected] #輸入密碼

info replication #檢視叢集

大功告成！