本文記錄使用Velero Restic快速完成雲原生應用及PV資料從GKE到至ACK的遷移的實踐過程。 此過程也同樣适用于自建Kubernetes叢集内的應用及PV資料遷移至ACK。
實踐步驟概覽
（1）建立GKE叢集（或自建Kubernetes叢集）
（2）在GKE叢集上部署示例應用Jenkins Application并執行一個建構任務
（3）
建立ACK叢集（4）在ACK叢集中部署Minio Service用于應用遷移時資料中轉服務
（5）在GKE上部署Velero并備份整個Jenkins Application（或其他任意應用）
（6）把Jenkins Application使用的容器鏡像同步到阿裡雲容器鏡像倉庫
（7）在ACK上建立Jenkins Application所使用的StorageClass
（8）在ACK上部署Velero并恢複整個Jenkins Application
（9）在ACK上替換Jenkins Application所使用的容器鏡像
（10）驗證Jenkins Application能否正常提供服務及其建構任務是否存在
環境物料清單
（1）GKE叢集（或自建Kubernetes叢集）
（2）ACK叢集
建立GKE叢集并部署示例應用Jenkins Application
(1) 在MarketPlace中找到Jenkins應用并配置部署到GKE叢集的jenkins命名空間下：
(2)應用正在完成部署：
(3)檢視Ingress并通路jenkins服務：
(4)Jenkins應用的初始化配置：
(5)Jenkin應用上建立一個名為gke-to-ack的任務并執行建構：
這個步驟相當于往PV存儲卷中寫資料。
（6）Jenkins應用的k8s資源清單如下
$ kubectl -n jenkins get deploy
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
jenkins-jenkins-deployment 1 1 1 1 177m
$ kubectl -n jenkins get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
jenkins-jenkins-agents-connector ClusterIP 10.39.241.75 <none> 50000/TCP 177m
jenkins-jenkins-ui NodePort 10.39.253.141 <none> 8080:31759/TCP 177m
$ kubectl -n jenkins get ing
NAME HOSTS ADDRESS PORTS AGE
jenkins-jenkins-ui * xx.xx.xx.xx 80, 443 177m
$ kubectl -n jenkins get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
jenkins-jenkins-pvc Bound pvc-e64eacc4-a482-11e9-bfa9-42010a8000da 8Gi RWO standard 177m
$ kubectl -n jenkins get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-e64eacc4-a482-11e9-bfa9-42010a8000da 8Gi RWO Delete Bound jenkins/jenkins-jenkins-pvc standard 177m 建立ACK叢集并部署Minio對象存儲應用
Minio對象存儲服務用于存儲PV持久化資料， 我們即将支援阿裡雲OSS對象存儲服務。
(1) 自定義替換minio-deploy.yaml中的 MINIO_ACCESS_KEY 和 MINIO_SECRET_KEY值并部署：
kubectl apply -f minio-deploy.yaml ---
apiVersion: v1
kind: Namespace
metadata:
name: minio
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
namespace: minio
name: minio
labels:
component: minio
spec:
strategy:
type: Recreate
template:
metadata:
labels:
component: minio
spec:
volumes:
- name: storage
emptyDir: {}
- name: config
emptyDir: {}
containers:
- name: minio
image: minio/minio:latest
imagePullPolicy: IfNotPresent
args:
- server
- /storage
- --config-dir=/config
env:
- name: MINIO_ACCESS_KEY
value: "<your MINIO_ACCESS_KEY>"
- name: MINIO_SECRET_KEY
value: "your MINIO_SECRET_KEY"
ports:
- containerPort: 9000
volumeMounts:
- name: storage
mountPath: "/storage"
- name: config
mountPath: "/config"
---
apiVersion: v1
kind: Service
metadata:
namespace: minio
name: minio
labels:
component: minio
spec:
# ClusterIP is recommended for production environments.
# Change to NodePort if needed per documentation,
# but only if you run Minio in a test/trial environment, for example with Minikube.
type: LoadBalancer
ports:
- port: 9000
targetPort: 9000
protocol: TCP
selector:
component: minio (2) 檢視minio LoadBalancer資訊并登陸應用：
$ kubectl -n minio get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
minio LoadBalancer 172.21.2.216 xxx.xxx.xxx.xx 9000:30912/TCP 2d4h (3) 替換minio-job.yaml中的minio_server_url minino_access_key minio_secret_key并運作job在minio中建立名為velero的bucket：
kubectl apply -f minio-job.yaml apiVersion: batch/v1
kind: Job
metadata:
namespace: minio
name: minio-setup
labels:
component: minio
spec:
template:
metadata:
name: minio-setup
spec:
restartPolicy: OnFailure
volumes:
- name: config
emptyDir: {}
containers:
- name: mc
image: minio/mc:latest
imagePullPolicy: IfNotPresent
command:
- /bin/sh
- -c
- "mc --config-dir=/config config host add velero <your minio_server_url> <your minino_access_key> <your minio_secret_key> && mc --config-dir=/config mb -p velero/velero"
volumeMounts:
- name: config
mountPath: "/config" (4) 檢視bucket是否建立成功：
在GKE和ACK上部署velero
(1) 安裝velero client：
請從
official release下載下傳最新版本的velero用戶端：
(2) 安裝velero server：
建立credentials-velero檔案并設定aws_access_key_id 與 aws_secret_access_key的值
[default]
aws_access_key_id = <your minio_access_key>
aws_secret_access_key = <your minio_secret_key> ACK叢集中請替換minio_server_url的值并指定image參數部署velero server
velero install --provider aws --image registry.cn-hangzhou.aliyuncs.com/acs/velero:latest --bucket velero --secret-file ./credentials-velero --use-volume-snapshots=false --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=<your minio_server_url> --use-restic --wait GKE中請替換minio_server_url的值并部署velero server
velero install --provider aws --bucket velero --secret-file ./credentials-velero --use-volume-snapshots=false --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=<your minio_server_url> --use-restic --wait 在GKE叢集中備份Jenkins Application
(1)在備份帶volume資訊的pod之前，我們要通過給pod加annotation來告訴velero哪些pod需要包含volume資料， 檢視jenkins應用的Deployment資源：
$ kubectl -n jenkins get deploy jenkins-jenkins-deployment -oyaml (2)為pod加annotation:
$ kubectl -n jenkins get po
NAME READY STATUS RESTARTS AGE
jenkins-deployer-zh5p4 0/1 Completed 0 109m
jenkins-jenkins-deployment-7df86c64d4-tqqlr 1/1 Running 0 109m
$ kubectl -n jenkins annotate pod/jenkins-jenkins-deployment-7df86c64d4-tqqlr backup.velero.io/backup-volumes=jenkins-jenkins-pvc
pod/jenkins-jenkins-deployment-7df86c64d4-tqqlr annotated (3)建立備份
$ velero backup create gcloud-jenkins-backup-restic --include-namespaces jenkins --wait
Backup request "gcloud-jenkins-backup-restic" submitted successfully.
Waiting for backup to complete. You may safely press ctrl-c to stop waiting - your backup will continue in the background.
.......................
Backup completed with status: Completed. You may check for more information using the commands `velero backup describe gcloud-jenkins-backup-restic` and `velero backup logs gcloud-jenkins-backup-restic`. (4) 檢視備份
$ ./velero backup get
NAME STATUS CREATED EXPIRES STORAGE LOCATION SELECTOR
gcloud-jenkins-backup-restic Completed 2019-07-12 18:48:48 +0800 +08 29d default <none> （5）登入Minio Server檢視
同步（或批量）遷移容器鏡像
把Jenkins應用使用的容器鏡像導入到阿裡雲容器鏡像倉庫，導入到的位址為：
registry.cn-hangzhou.aliyuncs.com/haoshuwei/jenkins:2.150.3 批量遷移可參考：
https://github.com/AliyunContainerService/sync-repo.git在ACK叢集中建立Jenkins應用所使用的StorageClass standard
我們知道不同的雲廠商，後端的存儲基礎設施是不一樣的， 我們需要先建立一個相同名稱的StorageClass來屏蔽對底層存儲基礎設施差異的感覺。
$ kubectl apply -f storageclass.yaml kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
name: standard
provisioner: alicloud/disk
parameters:
type: cloud
reclaimPolicy: Delete 在ACK叢集中恢複Jenkins Application
檢視備份資訊：
$ velero backup get
NAME STATUS CREATED EXPIRES STORAGE LOCATION SELECTOR
gcloud-jenkins-backup-restic Completed 2019-07-12 18:48:48 +0800 CST 29d default <none> 恢複應用：
$ velero restore create --from-backup gcloud-jenkins-backup-restic 此處需要編輯deployment修改image為registry.cn-hangzhou.aliyuncs.com/haoshuwei/jenkins:2.150.3
檢視restore的狀态：
$ velero restore describe gcloud-jenkins-backup-restic-20190712190536
$ velero restore logs gcloud-jenkins-backup-restic-20190712190536 restore完成後的狀态為：
$ velero restore get
NAME BACKUP STATUS WARNINGS ERRORS CREATED SELECTOR
gcloud-jenkins-backup-restic-20190712190536 gcloud-jenkins-backup-restic Completed 0 0 2019-07-12 19:05:36 +0800 CST <none> 檢視ACK叢集上jenkins應用的ingress并通路服務進行驗證
$ kubectl -n jenkins get ing
NAME HOSTS ADDRESS PORTS AGE
jenkins-jenkins-ui * xx.xx.xx.xx 80, 443 56m 至此，一個帶pv存儲的jenkins應用被完整地從GKE遷移到ACK上。
參考連結：
https://velero.io/docs/v1.0.0/get-started/ https://velero.io/docs/v1.0.0/restic/#troubleshooting https://github.com/heptio/velero https://github.com/AliyunContainerService/velero-plugin