阿裡雲Kubernetes CSI實踐 - TopologyAware雲盤卷

Topology Aware

對于雲盤這種存儲卷類型，隻能在相同zone的ecs上進行挂載，在多可用區的叢集環境中，就需要對pv、pvc和pod進行一緻性zone排程；

1. 傳統使用方式：

通過PV、PVC所在zoneid資訊，把pod排程到相應zone；

即在PV中定義相應的zoneid資訊，當一個pod使用這個pv的時候，scheduler會根據pv的zoneid把pod排程到相應可用區；

具體配置：

apiVersion: v1
kind: PersistentVolume
metadata:
  labels:
    failure-domain.beta.kubernetes.io/region: cn-beijing
    failure-domain.beta.kubernetes.io/zone: cn-beijing-e
  name: d-2ze6wbcvihjeaw5bae0c
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 20Gi
  csi:
    driver: diskplugin.csi.alibabacloud.com
    volumeHandle: d-bp1fqy1enb2rtymx5g5y
  persistentVolumeReclaimPolicy: Delete
  storageClassName: alicloud-disk-ssd           

通過在PV中添加下面label來完成排程：

labels:
    failure-domain.beta.kubernetes.io/region: cn-beijing
    failure-domain.beta.kubernetes.io/zone: cn-beijing-e           

2. 通過Topology Aware方式

上述方式需要先确定雲盤的可用區，然後與pod進行關聯排程。

而Topology Aware方式是雲盤根據pod所在的zone進行建立。

使用下面storageclass，WaitForFirstConsumer配置會使pv controller在pod啟動的時候去建立pv，并通過pod所在的zone資訊來确定pv所在的zone。

kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
  name: alicloud-disk-aware
provisioner: diskplugin.csi.alibabacloud.com
volumeBindingMode: WaitForFirstConsumer
parameters:
  type: cloud_ssd           

環境準備

叢集建立、依賴配置、CSI插件部署等請參考：

CSI部署詳解

建立StorageClass

WaitForFirstConsumer：表示隻有pod消費這個pvc的時候才驅動Provisioner進行pv建立；

kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
  name: alicloud-disk-aware
provisioner: diskplugin.csi.alibabacloud.com
volumeBindingMode: WaitForFirstConsumer
parameters:
  type: cloud_ssd           

建立動态PVC PV

通過下面模闆建立動态卷PV:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: disk-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 25Gi
  storageClassName: alicloud-disk-aware           

# kubectl get pvc
NAME       STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS          AGE
disk-pvc   Pending                                      alicloud-disk-aware   23m           

建立應用

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-disk
  labels:
    app: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80
        volumeMounts:
          - name: disk-pvc
            mountPath: "/data"
      volumes:
        - name: disk-pvc
          persistentVolumeClaim:
            claimName: disk-pvc           

驗證挂載、高可用

檢視pvc已經綁定完成
# kubectl get pvc
NAME       STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS          AGE
disk-pvc   Bound    pvc-bf7732a6-a7cf-11e9-8dec-00163e0a6ecc   25Gi       RWO            alicloud-disk-aware   18s
# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM              STORAGECLASS          REASON   AGE
pvc-bf7732a6-a7cf-11e9-8dec-00163e0a6ecc   25Gi       RWO            Delete           Bound    default/disk-pvc   alicloud-disk-aware            7s

檢視pvc詳情，volume.kubernetes.io/selected-node即為pv的排程資訊：
# kubectl describe pvc disk-pvc
Name:          disk-pvc
Namespace:     default
StorageClass:  alicloud-disk-aware
Status:        Bound
Volume:        pvc-bf7732a6-a7cf-11e9-8dec-00163e0a6ecc
Labels:        <none>
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: diskplugin.csi.alibabacloud.com
               volume.kubernetes.io/selected-node: cn-hangzhou.192.168.6.149
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      25Gi
Access Modes:  RWO
VolumeMode:    Filesystem

檢視pod，驗證雲盤挂載成功，建立測試檔案；
# kubectl get pod
NAME                          READY   STATUS    RESTARTS   AGE
nginx-disk-6d5659d745-2ts26   1/1     Running   0          2m1s
# kubectl exec nginx-disk-6d5659d745-2ts26 ls /data
lost+found
# kubectl exec nginx-disk-6d5659d745-2ts26 touch /data/test
# kubectl exec nginx-disk-6d5659d745-2ts26 ls /data
lost+found
test

删除Pod，檢視重建Pod是否資料穩定；
# kubectl delete pod nginx-disk-6d5659d745-2ts26
pod "nginx-disk-6d5659d745-2ts26" deleted

# kubectl get pod
NAME                          READY   STATUS    RESTARTS   AGE
nginx-disk-6d5659d745-m5z9p   1/1     Running   0          11s
# kubectl exec nginx-disk-6d5659d745-m5z9p ls /data
lost+found
test