Kubernetes 環境中使用 NAS 動态存儲卷

原文作者：阚俊寶 原文連結 更多雲原生技術資訊可關注 阿裡巴巴雲原生技術圈

介紹

K8S在應用中經常會用到nas共享存儲卷，通常的使用方法是通過一個pv、pvc進行挂載，但當需要pv、pvc數量很大的時候，手動建立顯得非常繁瑣，這時動态資料卷的功能可以滿足您的需求。

以下場景：

當為不同的使用者提供不同的nas子目錄進行挂載的時候，可以考慮使用nas動态存儲卷；

當使用SatefulSet部署應用，每個pod配置使用不同nas資料卷的時候；

本方案中實作的動态nas卷，是在某個nas檔案系統下通過建立子目錄、并把子目錄映射為一個動态pv提供給應用；

部署詳解

1. 部署Controller

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: alicloud-nas-controller
  namespace: kube-system
spec:
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: alicloud-nas-controller
    spec:
      tolerations:
      - operator: "Exists"
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 1
            preference:
              matchExpressions:
              - key: node-role.kubernetes.io/master
                operator: Exists
      priorityClassName: system-node-critical
      serviceAccount: admin
      hostNetwork: true
      containers:
        - name: nfs-provisioner
          image: registry.cn-hangzhou.aliyuncs.com/acs/alicloud-nas-controller:v1.14.3.8-58bf821-aliyun
          env:
          - name: PROVISIONER_NAME
            value: alicloud/nas
          securityContext:
            privileged: true
          volumeMounts:
          - mountPath: /var/log
            name: log
      volumes:
      - hostPath:
          path: /var/log
        name: log           

2. StorageClass配置：

Nas Controller支援兩種挂載驅動，flexvolume、nfs方式；

flexvolume：表示使用阿裡雲提供的flexvolume(alicloud/nas)作為挂載驅動，叢集中需要部署flexvolume；

nfs：表示用時Kubernetes提供的标準挂載驅動；

Flexvolume方式：

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: alicloud-nas
mountOptions:
- nolock,tcp,noresvport
- vers=3
parameters:
  server: "23a9649583-iaq37.cn-shenzhen.nas.aliyuncs.com:/nasroot1/,23a9649583-ajv79.cn-shenzhen.nas.aliyuncs.com:/nasroot2/"
  driver: flexvolume
provisioner: alicloud/nas
reclaimPolicy: Delete           

mountOptions：

表示生成的pv option 配置，挂載nas卷時使用這個option挂載；

server：

表示生成目标pv所使用nas挂載點清單；格式為 nfsurl1:/path1,nfsurl2:/path2；

當配置多個server的時候，通過此storageclass建立的pv會輪詢使用上述server作為配置參數；

driver：

支援flexvolume、nfs，預設為nfs；

reclaimPolicy：

pv的回收政策，建議配置為Retain；當配置為Delete時，删除pv後nas檔案系統中的對應目錄會預設修改名字（path-name ==> archived-path-name）;

如果需要删除檔案系統中對應的存儲目錄，可在storageclass中配置archiveOnDelete為false；

NFS 方式：

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: alicloud-nas
mountOptions:
- noresvport
- vers=4.0
parameters:
  server: "23a9649583-ajv79.cn-shenzhen.nas.aliyuncs.com:/nasroot2/"
  driver: nfs
provisioner: alicloud/nas
reclaimPolicy: Delete           

應用示例

1. 建立Flexvolume驅動資料卷

建立存儲類：# kubectl create -f -

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: alicloud-nas
mountOptions:
- nolock,tcp,noresvport
- vers=3
parameters:
  server: "23a9649583-iaq37.cn-shenzhen.nas.aliyuncs.com:/nasroot1/,23a9649583-ajv79.cn-shenzhen.nas.aliyuncs.com:/nasroot2/"
  driver: flexvolume
provisioner: alicloud/nas
reclaimPolicy: Delete           

建立statefulset應用，配置動态存儲卷：

apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  ports:
  - port: 80
    name: web
  clusterIP: None
  selector:
    app: nginx
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: web
spec:
  serviceName: "nginx"
  replicas: 5
  volumeClaimTemplates:
  - metadata:
      name: html
    spec:
      accessModes:
        - ReadWriteOnce
      storageClassName: alicloud-nas
      resources:
        requests:
          storage: 2Gi
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:alpine
        volumeMounts:
        - mountPath: "/data"
          name: html           

檢查生成的pvc、pv；

# kubectl get pvc | grep web
html-web-0      Bound         default-html-web-0-pvc-58450866-b9e1-11e9-91a9-00163e10b7cf   2Gi        RWO            alicloud-nas        96s
html-web-1      Bound         default-html-web-1-pvc-6b108a5b-b9e0-11e9-91a9-00163e10b7cf   2Gi        RWO            alicloud-nas        8m14s
html-web-2      Bound         default-html-web-2-pvc-661ebd73-b9e1-11e9-91a9-00163e10b7cf   2Gi        RWO            alicloud-nas        73s
html-web-3      Bound         default-html-web-3-pvc-69569748-b9e1-11e9-91a9-00163e10b7cf   2Gi        RWO            alicloud-nas        68s
html-web-4      Bound         default-html-web-4-pvc-6c52a277-b9e1-11e9-91a9-00163e10b7cf   2Gi        RWO            alicloud-nas        63s
下面看到使用了flexvolume驅動建立的pv：
# kubectl get pv default-html-web-0-pvc-58450866-b9e1-11e9-91a9-00163e10b7cf -oyaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: alicloud/nas
  creationTimestamp: "2019-08-08T13:35:20Z"
  finalizers:
  - kubernetes.io/pv-protection
  labels:
    createdby.aliyun.com: alicloud-nas-controller
    version.controller.aliyun.com: v1.14.3-1173ac8
  name: default-html-web-0-pvc-58450866-b9e1-11e9-91a9-00163e10b7cf
  resourceVersion: "2241971"
  selfLink: /api/v1/persistentvolumes/default-html-web-0-pvc-58450866-b9e1-11e9-91a9-00163e10b7cf
  uid: 5da7280c-b9e1-11e9-91a9-00163e10b7cf
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 2Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: html-web-0
    namespace: default
    resourceVersion: "2241962"
    uid: 58450866-b9e1-11e9-91a9-00163e10b7cf
  flexVolume:
    driver: alicloud/nas
    options:
      options: nolock,tcp,noresvport,vers=3
      path: /nasroot2/default-html-web-0-pvc-58450866-b9e1-11e9-91a9-00163e10b7cf
      server: 23a9649583-ajv79.cn-shenzhen.nas.aliyuncs.com
      vers: "3"
  persistentVolumeReclaimPolicy: Delete
  storageClassName: alicloud-nas
  volumeMode: Filesystem
status:
  phase: Bound           

檢查5個pv是否輪詢使用server中的不同挂載點：

# for item in `kubectl get pv | grep web | awk '{print $1}'`; do  kubectl get pv $item -oyaml | grep server; done
      server: 23a9649583-ajv79.cn-shenzhen.nas.aliyuncs.com
      server: 23a9649583-ajv79.cn-shenzhen.nas.aliyuncs.com
      server: 23a9649583-iaq37.cn-shenzhen.nas.aliyuncs.com
      server: 23a9649583-ajv79.cn-shenzhen.nas.aliyuncs.com
      server: 23a9649583-iaq37.cn-shenzhen.nas.aliyuncs.com
      
# for item in `kubectl get pv | grep web | awk '{print $1}'`; do  kubectl get pv $item -oyaml | grep path; done
      path: /nasroot2/default-html-web-0-pvc-58450866-b9e1-11e9-91a9-00163e10b7cf
      path: /nasroot2/default-html-web-1-pvc-6b108a5b-b9e0-11e9-91a9-00163e10b7cf
      path: /nasroot1/default-html-web-2-pvc-661ebd73-b9e1-11e9-91a9-00163e10b7cf
      path: /nasroot2/default-html-web-3-pvc-69569748-b9e1-11e9-91a9-00163e10b7cf
      path: /nasroot1/default-html-web-4-pvc-6c52a277-b9e1-11e9-91a9-00163e10b7cf           

nas檔案系統中生成不同的子目錄：

# ls nasroot1
default-html-web-2-pvc-661ebd73-b9e1-11e9-91a9-00163e10b7cf  default-html-web-4-pvc-6c52a277-b9e1-11e9-91a9-00163e10b7cf
# ls nasroot2
default-html-web-0-pvc-58450866-b9e1-11e9-91a9-00163e10b7cf  default-html-web-3-pvc-69569748-b9e1-11e9-91a9-00163e10b7cf
default-html-web-1-pvc-6b108a5b-b9e0-11e9-91a9-00163e10b7cf           

删除應用pv、pvc，nas檔案系統中目錄名字改為archived-__

# ls nasroot1
archived-default-html-web-2-pvc-661ebd73-b9e1-11e9-91a9-00163e10b7cf.2019-08-08-13:38:37
archived-default-html-web-4-pvc-6c52a277-b9e1-11e9-91a9-00163e10b7cf.2019-08-08-13:38:37
# ls nasroot2
archived-default-html-web-0-pvc-58450866-b9e1-11e9-91a9-00163e10b7cf.2019-08-08-13:38:37
archived-default-html-web-1-pvc-6b108a5b-b9e0-11e9-91a9-00163e10b7cf.2019-08-08-13:38:37
archived-default-html-web-3-pvc-69569748-b9e1-11e9-91a9-00163e10b7cf.2019-08-08-13:38:37           

2. 建立NFS驅動資料卷

archiveOnDelete: 配置為false，表示pv回收政策為Delete時，删除pv會把檔案系統中的目錄一起删除；

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: alicloud-nas
mountOptions:
- noresvport
- vers=4.0
parameters:
  server: "23a9649583-ajv79.cn-shenzhen.nas.aliyuncs.com:/nasroot2/"
  driver: nfs
  archiveOnDelete: "false"
provisioner: alicloud/nas
reclaimPolicy: Delete           

建立statefulset，檢查pv、pvc：

# kubectl get pvc | grep web
html-web-0      Bound         default-html-web-0-pvc-fed53a89-b9e1-11e9-91a9-00163e10b7cf   2Gi        RWO            alicloud-nas        2m4s
html-web-1      Bound         default-html-web-1-pvc-015441f5-b9e2-11e9-91a9-00163e10b7cf   2Gi        RWO            alicloud-nas        2m
html-web-2      Bound         default-html-web-2-pvc-3edcb47b-b9e2-11e9-91a9-00163e10b7cf   2Gi        RWO            alicloud-nas        17s
html-web-3      Bound         default-html-web-3-pvc-420f9bd8-b9e2-11e9-91a9-00163e10b7cf   2Gi        RWO            alicloud-nas        11s
html-web-4      Bound         default-html-web-4-pvc-4843579f-b9e2-11e9-91a9-00163e10b7cf   2Gi        RWO            alicloud-nas        1s
下面資訊顯示，pv使用的驅動為NFS：
# kubectl get pv default-html-web-0-pvc-fed53a89-b9e1-11e9-91a9-00163e10b7cf -oyaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: alicloud/nas
  creationTimestamp: "2019-08-08T13:39:50Z"
  finalizers:
  - kubernetes.io/pv-protection
  labels:
    createdby.aliyun.com: alicloud-nas-controller
    version.controller.aliyun.com: v1.14.3-1173ac8
  name: default-html-web-0-pvc-fed53a89-b9e1-11e9-91a9-00163e10b7cf
  resourceVersion: "2243033"
  selfLink: /api/v1/persistentvolumes/default-html-web-0-pvc-fed53a89-b9e1-11e9-91a9-00163e10b7cf
  uid: fef7e4d1-b9e1-11e9-91a9-00163e10b7cf
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 2Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: html-web-0
    namespace: default
    resourceVersion: "2243017"
    uid: fed53a89-b9e1-11e9-91a9-00163e10b7cf
  mountOptions:
  - noresvport
  - vers=4.0
  nfs:
    path: /nasroot2/default-html-web-0-pvc-fed53a89-b9e1-11e9-91a9-00163e10b7cf
    server: 23a9649583-ajv79.cn-shenzhen.nas.aliyuncs.com
  persistentVolumeReclaimPolicy: Delete
  storageClassName: alicloud-nas
  volumeMode: Filesystem
status:
  phase: Bound           

檢查5個pv的挂載點：

# for item in `kubectl get pv | grep web | awk '{print $1}'`; do  kubectl get pv $item -oyaml | grep server; done
    server: 23a9649583-ajv79.cn-shenzhen.nas.aliyuncs.com
    server: 23a9649583-ajv79.cn-shenzhen.nas.aliyuncs.com
    server: 23a9649583-ajv79.cn-shenzhen.nas.aliyuncs.com
    server: 23a9649583-ajv79.cn-shenzhen.nas.aliyuncs.com
    server: 23a9649583-ajv79.cn-shenzhen.nas.aliyuncs.com
# for item in `kubectl get pv | grep web | awk '{print $1}'`; do  kubectl get pv $item -oyaml | grep path; done
    path: /nasroot2/default-html-web-0-pvc-fed53a89-b9e1-11e9-91a9-00163e10b7cf
    path: /nasroot2/default-html-web-1-pvc-015441f5-b9e2-11e9-91a9-00163e10b7cf
    path: /nasroot2/default-html-web-2-pvc-3edcb47b-b9e2-11e9-91a9-00163e10b7cf
    path: /nasroot2/default-html-web-3-pvc-420f9bd8-b9e2-11e9-91a9-00163e10b7cf
    path: /nasroot2/default-html-web-4-pvc-4843579f-b9e2-11e9-91a9-00163e10b7cf           

# ls nasroot2/
default-html-web-0-pvc-fed53a89-b9e1-11e9-91a9-00163e10b7cf  default-html-web-3-pvc-420f9bd8-b9e2-11e9-91a9-00163e10b7cf
default-html-web-1-pvc-015441f5-b9e2-11e9-91a9-00163e10b7cf  default-html-web-4-pvc-4843579f-b9e2-11e9-91a9-00163e10b7cf
default-html-web-2-pvc-3edcb47b-b9e2-11e9-91a9-00163e10b7cf           

删除應用pv、pvc，nas檔案系統中目錄同時被删除；（StorageClass中配置了archiveOnDelete: "false"）

“ 阿裡巴巴雲原生 關注微服務、Serverless、容器、Service Mesh 等技術領域、聚焦雲原生流行技術趨勢、雲原生大規模的落地實踐，做最懂雲原生開發者的技術圈。”