在ACK 叢集中使用本地盤建立LVM資料卷

阿裡雲容器服務團隊提供的CSI存儲插件支援在容器場景使用多種存儲服務，包括雲盤、NAS、OSS、LVM、記憶體卷等；本文介紹如何通過CSI插件在阿裡雲本地盤上自動化建立lvm資料卷。

介紹：

LVM卷屬于本地存儲的一種類型，CSI插件提供對多種本地資料卷統一管理的方式：localplugin.csi.alibabacloud.com；lvm卷為local plugin管理的一種存儲類型。插件包含兩部分：ControllerServer 和 NodeServer；

ControllerServer：實作lvm卷的建立、删除功能；

NodeServer：實作lvm卷的挂載、解除安裝、格式化、lvm建立、清除等能力；

目前支援的lvm功能包括：

LVM卷的全生命周期管理；

LVM卷的格式化、挂載、删除等基本功能；

自動化建立VG；

實作lvm卷自動化擴容功能；

依賴：

建立Kubernetes叢集，推薦使用阿裡雲ACK服務；

添加有本地盤的ecs節點；

叢集版本在1.14或者以上；

部署：

部署CSI NodeServer: (csi-local-plugin)

apiVersion: storage.k8s.io/v1beta1
kind: CSIDriver
metadata:
  name: localplugin.csi.alibabacloud.com
spec:
  attachRequired: false
  podInfoOnMount: true
---
kind: DaemonSet
apiVersion: apps/v1
metadata:
  name: csi-local-plugin
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: csi-local-plugin
  template:
    metadata:
      labels:
        app: csi-local-plugin
    spec:
      tolerations:
        - operator: Exists
      serviceAccount: admin
      priorityClassName: system-node-critical
      hostNetwork: true
      hostPID: true
      containers:
        - name: driver-registrar
          image: registry.cn-hangzhou.aliyuncs.com/acs/csi-node-driver-registrar:v1.1.0
          imagePullPolicy: Always
          args:
            - "--v=5"
            - "--csi-address=/csi/csi.sock"
            - "--kubelet-registration-path=/var/lib/kubelet/csi-plugins/localplugin.csi.alibabacloud.com/csi.sock"
          env:
            - name: KUBE_NODE_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: spec.nodeName
          volumeMounts:
            - name: plugin-dir
              mountPath: /csi
            - name: registration-dir
              mountPath: /registration

        - name: csi-localplugin
          securityContext:
            privileged: true
            capabilities:
              add: ["SYS_ADMIN"]
            allowPrivilegeEscalation: true
          image: registry.cn-hangzhou.aliyuncs.com/plugins/csi-plugin:v1.14-9fa71837c
          imagePullPolicy: "Always"
          args :
            - "--endpoint=$(CSI_ENDPOINT)"
            - "--v=5"
            - "--nodeid=$(KUBE_NODE_NAME)"
            - "--driver=localplugin.csi.alibabacloud.com"
          env:
            - name: KUBE_NODE_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: spec.nodeName
            - name: CSI_ENDPOINT
              value: unix://var/lib/kubelet/csi-plugins/localplugin.csi.alibabacloud.com/csi.sock
          volumeMounts:
            - name: pods-mount-dir
              mountPath: /var/lib/kubelet
              mountPropagation: "Bidirectional"
            - mountPath: /dev
              mountPropagation: "HostToContainer"
              name: host-dev
            - mountPath: /var/log/
              name: host-log
      volumes:
        - name: plugin-dir
          hostPath:
            path: /var/lib/kubelet/csi-plugins/localplugin.csi.alibabacloud.com
            type: DirectoryOrCreate
        - name: registration-dir
          hostPath:
            path: /var/lib/kubelet/plugins_registry
            type: DirectoryOrCreate
        - name: pods-mount-dir
          hostPath:
            path: /var/lib/kubelet
            type: Directory
        - name: host-dev
          hostPath:
            path: /dev
        - name: host-log
          hostPath:
            path: /var/log/
  updateStrategy:
    rollingUpdate:
      maxUnavailable: 10%
    type: RollingUpdate

部署ControllerServer: (csi-local-provisioner)

kind: Deployment
apiVersion: apps/v1
metadata:
  name: csi-local-provisioner
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: csi-local-provisioner
  replicas: 2
  template:
    metadata:
      labels:
        app: csi-local-provisioner
    spec:
      tolerations:
      - operator: "Exists"
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 1
            preference:
              matchExpressions:
              - key: node-role.kubernetes.io/master
                operator: Exists
      priorityClassName: system-node-critical
      serviceAccount: admin
      hostNetwork: true
      containers:
        - name: external-local-provisioner
          image: registry.cn-hangzhou.aliyuncs.com/acs/csi-provisioner:v1-5f99079e0-ack
          args:
            - "--csi-address=$(ADDRESS)"
            - "--feature-gates=Topology=True"
            - "--volume-name-prefix=local"
            - "--strict-topology=true"
            - "--timeout=150s"
            - "--extra-create-metadata=true"
            - "--enable-leader-election=true"
            - "--leader-election-type=leases"
            - "--retry-interval-start=500ms"
            - "--v=5"
          env:
            - name: ADDRESS
              value: /socketDir/csi.sock
          imagePullPolicy: "Always"
          volumeMounts:
            - name: socket-dir
              mountPath: /socketDir
      volumes:
        - name: socket-dir
          hostPath:
            path: /var/lib/kubelet/csi-plugins/localplugin.csi.alibabacloud.com
            type: DirectoryOrCreate

插件建立完成：

]# kubectl get pod -nkube-system | grep local
csi-local-plugin-54zvk                             2/2     Running   0          48m
csi-local-plugin-dxxq9                             2/2     Running   0          48m
csi-local-plugin-f5gr4                             2/2     Running   0          48m
csi-local-plugin-fq88g                             2/2     Running   0          48m
csi-local-plugin-tn5vh                             2/2     Running   0          48m
csi-local-provisioner-759699c8d4-6srrq             1/1     Running   3          3h42m
csi-local-provisioner-759699c8d4-fxlb5             1/1     Running   3          3h42m

使用：

1. 叢集中添加包含本地盤的節點：

# kubectl get node
cn-shanghai.192.168.2.70   Ready    <none>   86s     v1.16.6-aliyun.1

# ls /dev/vd*
/dev/vda  /dev/vda1  /dev/vdb  /dev/vdc

執行vgdisplay，顯示為空；
# vgdisplay

2. 建立StorageClass：

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
    name: csi-local
provisioner: localplugin.csi.alibabacloud.com
parameters:
    volumeType: LVM
    vgName: volumegroup1
    fsType: ext4
    pvType: "localdisk"
    LvmType: "striping"
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

provisioner：代表使用的驅動類型，即本地存儲驅動；

volumeType：定義使用哪種本地存儲類型，這裡選擇為LVM類型；

fsType：定義lvm卷的格式化檔案系統類型；

pvType：表示建立vg使用的底層資料盤類型，localdisk标志阿裡雲本地盤類型；

LvmTypeTag：表示lvm卷類型，支援linear(線性配置設定)、striping(條帶化方案)；

volumeBindingMode：表示使用延遲綁定模式，即隻有pod使用pvc的時候，才出發自動建立pv；

allowVolumeExpansion：配置是否運作自動擴容功能；

3. 建立PVC：

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: lvm-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
  storageClassName: csi-local

# kubectl create -f pvc.yaml
persistentvolumeclaim/lvm-pvc created

由于使用延遲綁定，pvc會等待pod消費他的時候，才觸發建立pv，否則一直pending；
# kubectl get pvc
NAME      STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
lvm-pvc   Pending                                      csi-local      11s

4. 建立Pod消費pvc：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment-lvm
  labels:
    app: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        volumeMounts:
          - name: lvm-pvc
            mountPath: "/data"
      volumes:
        - name: lvm-pvc
          persistentVolumeClaim:
            claimName: lvm-pvc

建立pod如下：

# kubectl create -f deploy.yaml
deployment.apps/deployment-lvm created

# kubectl get pvc
NAME      STATUS   VOLUME                                       CAPACITY   ACCESS MODES   STORAGECLASS   AGE
lvm-pvc   Bound    local-d451fbc2-3e13-4105-9e58-8ffaf5837fbf   2Gi        RWO            csi-local      47s

# kubectl get pod
NAME                             READY   STATUS    RESTARTS   AGE
deployment-lvm-9f798687c-4h2bs   1/1     Running   0          56s

LVM 資料卷資訊如下：
# kubectl get pv local-d451fbc2-3e13-4105-9e58-8ffaf5837fbf -oyaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: localplugin.csi.alibabacloud.com
  name: local-d451fbc2-3e13-4105-9e58-8ffaf5837fbf
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 2Gi
  csi:
    driver: localplugin.csi.alibabacloud.com
    fsType: ext4
    volumeAttributes:
      LvmTypeTag: striping
      csi.storage.k8s.io/pv/name: local-d451fbc2-3e13-4105-9e58-8ffaf5837fbf
      csi.storage.k8s.io/pvc/name: lvm-pvc
      csi.storage.k8s.io/pvc/namespace: default
      fsType: ext4
      pvType: localdisk
      storage.kubernetes.io/csiProvisionerIdentity: 1584985469710-8081-localplugin.csi.alibabacloud.com
      vgName: volumegroup1
      volume.beta.kubernetes.io/storage-provisioner: localplugin.csi.alibabacloud.com
      volume.kubernetes.io/selected-node: cn-shanghai.192.168.2.70
      volumeType: LVM
    volumeHandle: local-d451fbc2-3e13-4105-9e58-8ffaf5837fbf
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - cn-shanghai.192.168.2.70
  persistentVolumeReclaimPolicy: Delete
  storageClassName: csi-local

5. 登陸節點驗證lvm卷：

vg資訊如下：

# vgdisplay
  --- Volume group ---
  VG Name               volumegroup1
  System ID
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  2
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               1
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               207.99 GiB
  PE Size               4.00 MiB
  Total PE              53246
  Alloc PE / Size       512 / 2.00 GiB
  Free  PE / Size       52734 / 205.99 GiB
  VG UUID               04A5WZ-6dtd-LVPe-EvUh-n9uG-4ndy-fvRfaf

PV資訊如下：

# pvdisplay
  --- Physical volume ---
  PV Name               /dev/vdb
  VG Name               volumegroup1
  PV Size               104.00 GiB / not usable 4.00 MiB
  Allocatable           yes
  PE Size               4.00 MiB
  Total PE              26623
  Free PE               26111
  Allocated PE          512
  PV UUID               nBDn6O-7lf7-333d-PjvR-mezx-js6Y-pOOd57

  --- Physical volume ---
  PV Name               /dev/vdc
  VG Name               volumegroup1
  PV Size               104.00 GiB / not usable 4.00 MiB
  Allocatable           yes
  PE Size               4.00 MiB
  Total PE              26623
  Free PE               26623
  Allocated PE          0
  PV UUID               9ELK0x-Gw6e-VAUq-DcP2-kYgR-Mxva-pAUB48

LVM資訊如下：

# lvdisplay
  --- Logical volume ---
  LV Path                /dev/volumegroup1/local-d451fbc2-3e13-4105-9e58-8ffaf5837fbf
  LV Name                local-d451fbc2-3e13-4105-9e58-8ffaf5837fbf
  VG Name                volumegroup1
  LV UUID                ZHb6N9-XN7t-DxER-OWi0-rUXi-XcbL-YoY1aK
  LV Write Access        read/write
  LV Creation host, time iZuf667ifeuz5zz5j9oteeZ, 2020-03-24 01:44:31 +0800
  LV Status              available
  # open                 1
  LV Size                2.00 GiB
  Current LE             512
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           252:0

6. 删除Pod、PVC：

# kubectl delete -f deploy.yaml
deployment.apps "deployment-lvm" deleted

# kubectl delete -f pvc.yaml
persistentvolumeclaim "lvm-pvc" deleted


到節點插件lvm卷：
# vgdisplay
  --- Volume group ---
  VG Name               volumegroup1
  System ID
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  3
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                0
  Open LV               0
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               207.99 GiB
  PE Size               4.00 MiB
  Total PE              53246
  Alloc PE / Size       0 / 0
  Free  PE / Size       53246 / 207.99 GiB
  VG UUID               04A5WZ-6dtd-LVPe-EvUh-n9uG-4ndy-fvRfaf

LVM資料卷已經删除：
# lvdisplay
#

在ACK 叢集中使用本地盤建立LVM資料卷

介紹：

依賴：

部署：

使用：

1. 叢集中添加包含本地盤的節點：

2. 建立StorageClass：

3. 建立PVC：

4. 建立Pod消費pvc：

5. 登陸節點驗證lvm卷：

6. 删除Pod、PVC：

繼續閱讀

Perl與網絡監控

（Nginx）03_Nginx原理與優化一、Nginx原理二、master-workers機制三、面試題：

（Nginx）02_Nginx配置執行個體一、Nginx的應用：反向代理實作一：二、反向代理執行個體二：三、Nginx的配置執行個體三–負載均衡：四、Nginx的配置執行個體四–動靜分離：Nginx在實際開發中的應用：

nginx配置示例

轉詳解C#資料庫存取圖檔三大方式

BMP檔案結構及圖像每行位元組計算方法

磁盤結構及在Linux中的命名

niginx的location的location詳細規則和優先級關系以及測試案例2.1比對文法2.2優先級判斷2.3練習3.1處理邏輯如何執行3.2功能測試3.3流程解析

Puppet自動化Nginx+Mongrel負載均衡配置

Ubuntu修改nginx将多個域名分别配置到伺服器不同端口

修改or隐藏nginx版本号

Nginx安全改動：隐藏Nginx版本号背景解決serve_tokens指令參考

版本号隐藏

Nginx服務優化（版本隐藏、版本号僞裝、逾時管理、程序管理、日志分割）Nginx服務優化未完待續

nginx location中斜線的位置的重要性