如何在 K8S 中手動擴容雲盤資料卷？

原文作者：阚俊寶

原文連結：

https://developer.aliyun.com/article/718731?spm=a2c6h.12873581.0.0.6f7c115e50nLX7&groupCode=cloudnative 更多雲原生技術資訊可關注阿裡巴巴雲原生技術圈

介紹：

雲盤資料卷擴容包括以下幾個部分：

雲盤實體空間擴容，需要在雲盤控制台操作；

檔案系統擴容，需要挂載雲盤到一個實體節點手動操作；

PV、PVC Size更新，需要更新StorageClass、PVC；

注意：擴容雲盤前需要為雲盤打快照，以預防更新失敗導緻資料丢失；

雲盤目前無法做到線上擴容，需要重新開機應用才可以實作，可以通過兩種方法實作雲盤擴容。下面以叢集中部署Zookeeper為例介紹兩種擴容方式，Zookeeper叢集如下：

# kubectl get pod
NAME                            READY   STATUS    RESTARTS   AGE
zookeeper-default-zookeeper-0   1/1     Running   0          2m55s
zookeeper-default-zookeeper-1   1/1     Running   0          2m14s
zookeeper-default-zookeeper-2   1/1     Running   0          94s
# kubectl get pvc| grep zoo
datadir-zookeeper-default-zookeeper-0   Bound    d-8vb5teafaoa80ia7affg   20Gi       RWO            alicloud-disk-efficiency   3m12s
datadir-zookeeper-default-zookeeper-1   Bound    d-8vb60faf6epslbctnzka   20Gi       RWO            alicloud-disk-efficiency   2m31s
datadir-zookeeper-default-zookeeper-2   Bound    d-8vbidmq57w4df6k84zem   20Gi       RWO            alicloud-disk-efficiency   111s
# kubectl get pv| grep zoo
d-8vb5teafaoa80ia7affg   20Gi       RWO            Delete           Bound      default/datadir-zookeeper-default-zookeeper-0   alicloud-disk-efficiency            3m17s
d-8vb60faf6epslbctnzka   20Gi       RWO            Delete           Bound      default/datadir-zookeeper-default-zookeeper-1   alicloud-disk-efficiency            2m33s
d-8vbidmq57w4df6k84zem   20Gi       RWO            Delete           Bound      default/datadir-zookeeper-default-zookeeper-2   alicloud-disk-efficiency            107s

暫停應用方式：

如果應用可以暫停服務，可以先停掉應用（删除），手動對每個依賴資料盤進行擴容，然後再啟動應用。缺點是：應用會暫停一定時間；

上面應用使用了3個20Gi的雲盤，分别挂載在3個pod上。目标為把三個雲盤擴容到30Gi，主要步驟：

删除應用負載；

雲盤控制台線上擴容雲盤；

挂載雲盤到某節點進行檔案系統擴容；

更新PV、PVC的Size參數；

重新開機應用；

1. 删除應用：

删除zookeeper statefulset對象；

# kubectl delete sts zookeeper-default-zookeeper
# kubectl get pvc | grep zoo
datadir-zookeeper-default-zookeeper-0   Bound    d-8vb6ie0kwtyynpf4gu4l   20Gi       RWO            alicloud-disk-efficiency   22m
datadir-zookeeper-default-zookeeper-1   Bound    d-8vbhscszlr47rbot0boc   20Gi       RWO            alicloud-disk-efficiency   21m
datadir-zookeeper-default-zookeeper-2   Bound    d-8vb444t0f8xnicj9c2ov   20Gi       RWO            alicloud-disk-efficiency   21m
# kubectl get pv | grep zoo
d-8vb444t0f8xnicj9c2ov   20Gi       RWO            Delete           Bound      default/datadir-zookeeper-default-zookeeper-2   alicloud-disk-efficiency            21m
d-8vb6ie0kwtyynpf4gu4l   20Gi       RWO            Delete           Bound      default/datadir-zookeeper-default-zookeeper-0   alicloud-disk-efficiency            22m
d-8vbhscszlr47rbot0boc   20Gi       RWO            Delete           Bound      default/datadir-zookeeper-default-zookeeper-1   alicloud-disk-efficiency            21m

2.3. 雲盤擴容：

根據雲盤文檔，分别對3塊雲盤進行擴容：

https://help.aliyun.com/document_detail/113316.html https://help.aliyun.com/document_detail/25452.html

注意：擴容雲盤的同時也要完成對檔案系統的擴容。

4. 擴容PV、PVC

編輯pvc、pv所使用的StorageClass，配置allowVolumeExpansion: true标簽；

# kubectl edit sc alicloud-disk-efficiency
# kubectl get sc alicloud-disk-efficiency -o yaml
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  creationTimestamp: "2019-09-05T12:30:27Z"
  name: alicloud-disk-efficiency
  resourceVersion: "1675896"
  selfLink: /apis/storage.k8s.io/v1/storageclasses/alicloud-disk-efficiency
  uid: f1071bcc-cfd8-11e9-81cd-00163e0804c2
parameters:
  type: cloud_efficiency
provisioner: alicloud/disk
reclaimPolicy: Delete
volumeBindingMode: Immediate

更新PVC的Size為30Gi：

# kubectl patch pvc datadir-zookeeper-default-zookeeper-0 -p '{"spec":{"resources":{"requests":{"storage":"30Gi"}}}}'
persistentvolumeclaim/datadir-zookeeper-default-zookeeper-0 patched
# kubectl patch pvc datadir-zookeeper-default-zookeeper-1 -p '{"spec":{"resources":{"requests":{"storage":"30Gi"}}}}'
persistentvolumeclaim/datadir-zookeeper-default-zookeeper-1 patched
# kubectl patch pvc datadir-zookeeper-default-zookeeper-2 -p '{"spec":{"resources":{"requests":{"storage":"30Gi"}}}}'
persistentvolumeclaim/datadir-zookeeper-default-zookeeper-2 patched
# kubectl get pvc | grep zoo
datadir-zookeeper-default-zookeeper-0   Bound    d-8vb6ie0kwtyynpf4gu4l   20Gi       RWO            alicloud-disk-efficiency   51m
datadir-zookeeper-default-zookeeper-1   Bound    d-8vbhscszlr47rbot0boc   20Gi       RWO            alicloud-disk-efficiency   50m
datadir-zookeeper-default-zookeeper-2   Bound    d-8vb444t0f8xnicj9c2ov   20Gi       RWO            alicloud-disk-efficiency   49m
# kubectl get pv | grep zoo
d-8vb444t0f8xnicj9c2ov   30Gi       RWO            Delete           Bound      default/datadir-zookeeper-default-zookeeper-2   alicloud-disk-efficiency            49m
d-8vb6ie0kwtyynpf4gu4l   30Gi       RWO            Delete           Bound      default/datadir-zookeeper-default-zookeeper-0   alicloud-disk-efficiency            51m
d-8vbhscszlr47rbot0boc   30Gi       RWO            Delete           Bound      default/datadir-zookeeper-default-zookeeper-1   alicloud-disk-efficiency            50m

可見pv的大小已經更新為30Gi，pvc的大小會在Pod啟動後自動更新；

5. 重新建立應用：

# kubectl get pod
NAME                            READY   STATUS    RESTARTS   AGE
zookeeper-default-zookeeper-0   1/1     Running   0          94s
zookeeper-default-zookeeper-1   1/1     Running   0          64s
zookeeper-default-zookeeper-2   1/1     Running   0          39s
# kubectl exec -ti zookeeper-default-zookeeper-0 sh
# df -h |grep zoo
/dev/vdb         30G   45M   30G   1% /var/lib/zookeeper
# kubectl get pvc | grep zoo
datadir-zookeeper-default-zookeeper-0   Bound    d-8vb6ie0kwtyynpf4gu4l   30Gi       RWO            alicloud-disk-efficiency   56m
datadir-zookeeper-default-zookeeper-1   Bound    d-8vbhscszlr47rbot0boc   30Gi       RWO            alicloud-disk-efficiency   56m
datadir-zookeeper-default-zookeeper-2   Bound    d-8vb444t0f8xnicj9c2ov   30Gi       RWO            alicloud-disk-efficiency   55m
# kubectl get pv | grep zoo
d-8vb444t0f8xnicj9c2ov   30Gi       RWO            Delete           Bound      default/datadir-zookeeper-default-zookeeper-2   alicloud-disk-efficiency            55m
d-8vb6ie0kwtyynpf4gu4l   30Gi       RWO            Delete           Bound      default/datadir-zookeeper-default-zookeeper-0   alicloud-disk-efficiency            56m
d-8vbhscszlr47rbot0boc   30Gi       RWO            Delete           Bound      default/datadir-zookeeper-default-zookeeper-1   alicloud-disk-efficiency            56m

上面輸出日志可以看出，檔案系統已經擴容到30G，pv pvc的size也擴容到30G；

逐個Pod更新方式：

由多個Pod運作的服務，可以通過一個一個pod更新的方式實作業務不中斷，而隻是Running pod數量暫時減少。

還是以zookeeper為例，使用3個20Gi的雲盤，分别挂載在3個pod上。目标為把三個雲盤擴容到40Gi：

主要步驟：

修改某個Pod所對應pv的Zoneid label，然後删除pod；

登陸雲盤控制台線上擴容雲盤（擴容前最好對雲盤打快照，容災）；

更新PV、PVC 的Size參數，以及Pv的ZoneId Lable；

對其他Pod重複1-4步驟；

1. 修改PV的ZoneId

# kubectl get pod zookeeper-default-zookeeper-0 -oyaml | grep pers -C 1
  - name: datadir
    persistentVolumeClaim:
      claimName: datadir-zookeeper-default-zookeeper-0
# kubectl get pvc datadir-zookeeper-default-zookeeper-0
NAME                                    STATUS   VOLUME                   CAPACITY   ACCESS MODES   STORAGECLASS               AGE
datadir-zookeeper-default-zookeeper-0   Bound    d-8vbhscszlr47tgn0eheb   20Gi       RWO            alicloud-disk-efficiency   21m

上述日志得出zookeeper-default-zookeeper-0 使用的pvc為：datadir-zookeeper-default-zookeeper-0，對應的PV為d-8vbhscszlr47tgn0eheb；

更新PV Label，

如果pv中已經有如下label，則把zone對應的值改為”原來value -none“，即配置一個不存在的zone，讓pod無法排程成功；

如果pv中沒有如下label，則添加這些lables到pv；

#kubectl edit pv d-8vbhscszlr47tgn0eheb
  labels:
    failure-domain.beta.kubernetes.io/region: cn-zhangjiakou
    failure-domain.beta.kubernetes.io/zone: cn-zhangjiakou-a-none

删除Pod：zookeeper-default-zookeeper-0

# kubectl delete pod zookeeper-default-zookeeper-0

這時删除的pod一直處于Pending狀态：

# kubectl get pod
NAME                            READY   STATUS    RESTARTS   AGE
zookeeper-default-zookeeper-0   0/1     Pending   0          9s
zookeeper-default-zookeeper-1   1/1     Running   0          24m
zookeeper-default-zookeeper-2   1/1     Running   0          24m

根據雲盤文檔，分别對3塊雲盤進行庫容：

# kubectl edit sc alicloud-disk-efficiency
# kubectl get sc alicloud-disk-efficiency -o yaml
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  creationTimestamp: "2019-09-05T12:30:27Z"
  name: alicloud-disk-efficiency
  resourceVersion: "1675896"
  selfLink: /apis/storage.k8s.io/v1/storageclasses/alicloud-disk-efficiency
  uid: f1071bcc-cfd8-11e9-81cd-00163e0804c2
parameters:
  type: cloud_efficiency
provisioner: alicloud/disk
reclaimPolicy: Delete
volumeBindingMode: Immediate

更新PVC的Size為40Gi：

# kubectl patch pvc datadir-zookeeper-default-zookeeper-0 -p '{"spec":{"resources":{"requests":{"storage":"40Gi"}}}}'
persistentvolumeclaim/datadir-zookeeper-default-zookeeper-0 patched
# kubectl get pvc | grep datadir-zookeeper-default-zookeeper-0
datadir-zookeeper-default-zookeeper-0   Bound    d-8vbhscszlr47tgn0eheb   20Gi       RWO            alicloud-disk-efficiency   29m
# kubectl get pv | grep datadir-zookeeper-default-zookeeper-0
d-8vbhscszlr47tgn0eheb   40Gi       RWO            Delete           Bound      default/datadir-zookeeper-default-zookeeper-0   alicloud-disk-efficiency            29m

可見pv的大小已經更新為40Gi，pvc的大小會在Pod啟動後自動更新；

恢複PV的Lable，把zoneId的值恢複之前的value，删除相應labels（若之前沒有這些labels）：

labels:
    failure-domain.beta.kubernetes.io/region: cn-zhangjiakou
    failure-domain.beta.kubernetes.io/zone: cn-zhangjiakou-a

檢查容器挂載檔案系統大小，pv、pvc大小：

# kubectl exec -ti zookeeper-default-zookeeper-0 sh
# df -h | grep zoo
/dev/vdb         40G   48M   40G   1% /var/lib/zookeeper

# kubectl get pvc | grep datadir-zookeeper-default-zookeeper-0
datadir-zookeeper-default-zookeeper-0   Bound    d-8vbhscszlr47tgn0eheb   40Gi       RWO            alicloud-disk-efficiency   33m
# kubectl get pv | grep datadir-zookeeper-default-zookeeper-0
d-8vbhscszlr47tgn0eheb   40Gi       RWO            Delete           Bound      default/datadir-zookeeper-default-zookeeper-0   alicloud-disk-efficiency            33m

可見檔案系統、pv、pvc都已經實作了擴容；

5. 對其他pod所對應的pvc、pv、雲盤進行上述擴容

“ 阿裡巴巴雲原生關注微服務、Serverless、容器、Service Mesh 等技術領域、聚焦雲原生流行技術趨勢、雲原生大規模的落地實踐，做最懂雲原生開發者的技術圈。”

如何在 K8S 中手動擴容雲盤資料卷？

介紹：

暫停應用方式：

1. 删除應用：

2.3. 雲盤擴容：

4. 擴容PV、PVC

5. 重新建立應用：

逐個Pod更新方式：

1. 修改PV的ZoneId

5. 對其他pod所對應的pvc、pv、雲盤進行上述擴容

繼續閱讀

VBA調用cmd指令行下執行的指令

在目前位置打開指令行視窗的技巧

unit 1 - redhat Enterprise 8.0 Linux 指令行使用技巧

Windows指令行中使用SSH連接配接Linux

Linux下指令行中的複制和粘貼

1.Linux指令行使用技巧

spec檔案詳解

微服務-性能壓測\緩存redis和分布式鎖redisson和SpringCache

HK-2000資料采集儀資料庫操作說明

終端環境之tmux

查找檔案中的字元串

拒絕使用者登入:/bin/false和/usr/sbin/nologin

Shell程式設計——sort排序、uniq忽略重複、tr替換壓縮删除、cut指定删除字段、正規表達式元字元sort 指令uniq 指令tr 指令cut 指令正規表達式

Linxu常用指令技巧彙總

《Linux指令行與Shell腳本程式設計大全第2版.布盧姆》pdf

ACS基本配置-權限等級管理