Openshift4 Pipeline 調優

通過調整 pipeline 循序漸進的提高過程效率，并适應多場景。

與之前在openshift使用jenkins 做pipeline 時候的思路是一緻的，通過配置私服位址和緩存編譯檔案到持久化存儲，以提高編譯效率。

調整前後，pipeline運作用時分别為 14min -> 9min -> 4min

demo 場景介紹

使用到的pipeline 和自定義 task 存在github

https://github.com/cai11745/ocp4-userguide/tree/master/attachment/pipeline

pipeline #1： s2i-build-and-deploy.yaml

openshift自帶demo，pipeline一共三步，使用到了三個 clustertask： git-clone，s2i-java-8，openshift-client

pipeline #2： s2i-build-and-deploy-mvn-mirror.yaml

在上一個基礎上增加了MAVEN_MIRROR_URL參數，可以使用從内網拉取編譯所需檔案，如 nexus

pipeline #3： mvn-cache-and-deploy.yaml

pipeline 一共4步，第一步拉取代碼。

第二步編譯，使用自定義task mvn-repo-task.yaml，并到了新的workspace: maven-cache 并做持久化，用于緩存編譯所需依賴。

第三步通過Dockerfile build image and push，使用 clustertask： buildah，Dockerfile 預設路徑是git 根目錄下，推送到openshift内置倉庫，使用的 serviceaccount 叫 pipeline，每個project下面已經有了，具備push image 權限，若是外部倉庫，需要另做權限處理。

第四步通過 deployment/service/route yaml 釋出應用，yaml 檔案放在 git repo 的 k8s目錄，通過 environments/dev/kustomization.yaml 指定yaml路徑，每次執行 pipeline 會更新 deployment 的鏡像。使用到了自定義task deploy-app-task.yaml

pipeline #4: mvn-cache-build-cache-and-deploy

這一步的初衷是想把鏡像檔案持久化，不需要每次拉取基礎鏡像。不過沒有達到預期效果，要5分鐘，比pipeline#3 還多了一分鐘。

待再調整，調整 storage driver 或者使用 node 節點鏡像檔案資料。

使用的java應用源碼

https://github.com/cai11745/spring-petclinic.git

在官方基礎增加了 Dockerfile，k8s目錄（deployment/service/route yaml），environments目錄。這些内容在pipeline #3才會用到。

整個demo過程需要使用到私有git庫（線上的也行，可能拉取會慢一些），私有nexus（必備）

并準備幾個網絡存儲用于緩存資料，nfs或其他都可。（必備）

pipeline #1 s2i-build-and-deploy

所有 yaml 以github 最新為準。

1-s2i-build-and-deploy.yaml

1-pipelinerun-s2i-build-and-deploy.yaml

[[email protected] pipeline]# oc create -f 1-s2i-build-and-deploy.yaml 
[[email protected] pipeline]# oc create -f 1-pipelinerun-s2i-build-and-deploy.yaml

pipelinerun-xx.yaml 内置了預設參數，可以通過oc create -f 1-pipelinerun-xx.yaml 快速生成pipelinerun，不需要通過頁面去 start pipeline

pipelinerun-xx.yaml 需要用 oc create -f 指令建立，不能用 apply -f ，generateName: s2i-build-and-deploy- 最後會加一串随機數作為新的pipelinerun 的名稱

這個pipeline 比較簡單，通過拉取源代碼，然後s2i 制作鏡像，最後 oc new-app --docker-image 指令釋出應用，這個pipelinerun 反複執行應該會報錯，應用最後建立應用時候會同名，需要修改下 pipelinerun，IMAGE_NAME 參數裡 image-registry.openshift-image-registry.svc:5000/demo/s2i-java:v1 ， s2i-java 是應用名稱，重複執行需要不一樣。

在 pipeline #3 開始将不存在這個問題，主要是調整下 deploy 這一步的邏輯。

Openshift4 Pipeline 調優

[[email protected] pipeline]# cat 1-s2i-build-and-deploy.yaml 
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: s2i-build-and-deploy
spec:
  params:
    - name: IMAGE_NAME
      type: string
      default: image-registry.openshift-image-registry.svc:5000/PROJECT/APP_NAME:TAG
      description: The application image to build. Need to replace the PROJECT,APP_NAME,TAG.
    - name: GIT_REPO
      type: string
      description: The application git repository
    - name: GIT_REVISION
      type: string
      default: master
      description: The application git revision
  workspaces:
    - name: workspace
  tasks:
    - name: fetch-repository
      taskRef:
        name: git-clone
        kind: ClusterTask
      workspaces:
        - name: output
          workspace: workspace
      params:
        - name: url
          value: $(params.GIT_REPO)
        - name: revision
          value: $(params.GIT_REVISION)
        - name: subdirectory
          value: ""
        - name: deleteExisting
          value: "true"

    - name: build
      taskRef:
        name: s2i-java-8
        kind: ClusterTask
      runAfter:
        - fetch-repository
      workspaces:
        - name: source
          workspace: workspace
      params:
        - name: IMAGE
          value: $(params.IMAGE_NAME)
        - name: TLSVERIFY
          value: "false"

    - name: deploy
      taskRef:
        name: openshift-client
        kind: ClusterTask
      runAfter:
        - build
      params:
        - name: ARGS
          value:
            - "new-app"
            - "--docker-image"
            - "$(params.IMAGE_NAME)"

[[email protected] pipeline]# cat 1-pipelinerun-s2i-build-and-deploy.yaml 
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  generateName: s2i-build-and-deploy-
spec:
  params:
  - name: IMAGE_NAME
    value: image-registry.openshift-image-registry.svc:5000/demo/s2i-java:v1
  - name: GIT_REPO
    value: http://gogs-demo-cicd.apps.ocp4.example.com/gogs/spring-petclinic.git
  - name: GIT_REVISION
    value: master
  pipelineRef:
    name: s2i-build-and-deploy
  serviceAccountName: pipeline
  workspaces:
  - name: workspace
    persistentVolumeClaim:
      claimName: petclinic-dev-workspace

pipeline #2 s2i-build-and-deploy-mvn-mirror

2-s2i-build-and-deploy-mvn-mirror.yaml

2-pipelinerun-s2i-build-and-deploy-mvn-mirror.yaml

與 pipeline#1 的差別就是在build 這一步，把 MAVEN_MIRROR_URL 參數添加到了pipeline中，這個參數 s2i-java-8 clustertask 原本就是支援的。每次從私服拉取編譯所需jar，并且私服會緩存檔案，不需要每次從網際網路擷取。

Openshift4 Pipeline 調優

[[email protected] pipeline]# cat 2-s2i-build-and-deploy-mvn-mirror.yaml 
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: s2i-build-and-deploy-mvn-mirror
spec:
...

    - name: MAVEN_MIRROR_URL
      type: string
      default: ""
      description: The base URL of a mirror used for retrieving artifacts. e.g. http://nexus_addr/repository/maven-public
  workspaces:
    - name: workspace
  tasks:
    - name: fetch-repository
...
    - name: build
      taskRef:
        name: s2i-java-8
        kind: ClusterTask
      runAfter:
        - fetch-repository
      workspaces:
        - name: source
          workspace: workspace
      params:
        - name: IMAGE
          value: $(params.IMAGE_NAME)
        - name: TLSVERIFY
          value: "false"
        - name: PATH_CONTEXT
          value: spring-petclinic
        - name: MAVEN_MIRROR_URL
          value: $(params.MAVEN_MIRROR_URL)

    - name: deploy
...

[[email protected] pipeline]# cat 2-pipelinerun-s2i-build-and-deploy-mvn-mirror.yaml 
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  generateName: s2i-build-and-deploy-mvn-mirror-
spec:
  params:
  - name: IMAGE_NAME
    value: image-registry.openshift-image-registry.svc:5000/demo/java-mvn-cache:v1
  - name: GIT_REPO
    value: http://gogs-demo-cicd.apps.ocp4.example.com/gogs/spring-petclinic.git
  - name: GIT_REVISION
    value: master
  - name: MAVEN_MIRROR_URL
    value: http://nexus-demo-cicd.apps.ocp4.example.com/repository/maven-public/
  pipelineRef:
    name: s2i-build-and-deploy-mvn-mirror
  workspaces:
  - name: workspace
    persistentVolumeClaim:
      claimName: petclinic-dev-workspace

pipeline #3 mvn-cache-and-deploy

mvn-repo-task.yaml

deploy-app-task.yaml

3-mvn-cache-and-deploy.yaml

3-pipelinerun-mvn-cache-and-deploy.yaml

Openshift4 Pipeline 調優

使用到了兩個自定義task。

mvn-repo-task.yaml 相較 clustertask maven增加了一個workspace，并調整了maven參數 args，增加了 Dmaven.repo.local=$(workspaces.mvn-repo.path)/.m2 使用者将編譯檔案緩存到workspace

[[email protected] pipeline]# cat mvn-repo-task.yaml 
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  annotations:
    tekton.dev/pipelines.minVersion: 0.12.1
    tekton.dev/tags: build-tool
  name: maven-repo
  labels:
    app.kubernetes.io/version: '0.1'
    operator.tekton.dev/provider-type: community
spec:
  description: This Task can be used to run a Maven build, and '/root/.m2' repository data will be stored to the Workspace.
  params:
    - default: registry.example.com:5000/cloud-builders/mvn:3.5.0-jdk-8
      description: Maven base image
      name: MAVEN_IMAGE
      type: string
    - default:
        - package
      description: maven goals to run
      name: GOALS
      type: array
    - default: ''
      description: The Maven repository mirror url
      name: MAVEN_MIRROR_URL
      type: string
...

  workspaces:
    - description: The workspace consisting of maven project.
      name: source
    - description: The workspace consisting of the custom maven settings provided by the user.
      name: maven-settings
    - description: The workspace to store local maven  repositories.
      name: mvn-repo
  steps:
    - image: 'registry.example.com:5000/ubi8/ubi-minimal:8.2'
      name: mvn-settings
      resources: {}
      script: >
 
 ...

    - args:
        - -Dmaven.repo.local=$(workspaces.mvn-repo.path)/.m2
        - '-s'
        - $(workspaces.maven-settings.path)/settings.xml
        - $(params.GOALS)
      command:
        - /usr/bin/mvn
      image: $(params.MAVEN_IMAGE)
      name: mvn-goals
      resources: {}
      workingDir: $(workspaces.source.path)

deploy-app-task.yaml

先生成一個 kustomization.yaml，替換 k8s 目錄下的 deployment.yaml route.yaml service.yaml 相關參數，如鏡像，并将yaml導入openshift

[[email protected] pipeline]# cat deploy-app-task.yaml 
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: deploy-app
spec:
  params:
    - name: DEPLOYMENT_NAME
      description: The name of deployment
      type: string
    - name: CURRENT_IMAGE
      description: The current image repo/image:tag in the manifests for the deployment
      type: string
    - name: NEW_IMAGE_NAME
      description: The new image repo/image to be deployed
      type: string
    - name: NEW_IMAGE_TAG
      description: The new image tag to be deployed
      type: string
      default: ""
    - name: NEW_IMAGE_DIGEST
      description: The digest of the new image to get deployed
      type: string
      default: ""
    - name: NAMESPACE
      description: The namespace for the deployment
      type: string
    - name: KUSTOMIZE_OVERLAY_DIR
      description: The subdirectory in configs git repo for the kustomize overlay to be applied
  workspaces:
  - description: The workspace consisting of maven project.
    name: source
  steps:
    - name: apply-manifests
      workingDir: $(workspaces.source.path)
      image: image-registry.openshift-image-registry.svc:5000/openshift/cli
      script: |

        find $(workspaces.source.path)

        cat >> $(workspaces.source.path)/$(params.KUSTOMIZE_OVERLAY_DIR)/kustomization.yaml <<- EOF

        images:
        - name: quay.io/siamaksade/spring-petclinic:latest
          newName: $(params.NEW_IMAGE_NAME)
          newTag: $(params.NEW_IMAGE_TAG)
          digest: $(params.NEW_IMAGE_DIGEST)
        EOF

        [[ "x$(params.NEW_IMAGE_DIGEST)" == "x" ]] && sed -i "/digest/d" $(workspaces.source.path)/$(params.KUSTOMIZE_OVERLAY_DIR)/kustomization.yaml
        [[ "x$(params.NEW_IMAGE_TAG)" == "x" ]] && sed -i "/newTag/d" $(workspaces.source.path)/$(params.KUSTOMIZE_OVERLAY_DIR)/kustomization.yaml

        echo "########################"  
        echo "## kustomization.yaml ##"
        echo "########################"

        cat $(workspaces.source.path)/$(params.KUSTOMIZE_OVERLAY_DIR)/kustomization.yaml

        echo "######## DRY RUN #######"
        oc apply -k $(params.KUSTOMIZE_OVERLAY_DIR) --dry-run=client -o yaml -n $(params.NAMESPACE)
        echo "########################"

        oc apply -k $(params.KUSTOMIZE_OVERLAY_DIR) -n $(params.NAMESPACE)
        oc rollout status deploy/$(params.DEPLOYMENT_NAME) -n $(params.NAMESPACE)

導入 task, pipeline，并執行 pipelinerun

第一次run的時候，緩存jar 到網絡存儲，可能最終時長會比 pipeline#2 還久，再次run 就會體會到使用本地緩存的效率。

[[email protected] pipeline]# oc create -f mvn-repo-task.yaml 
[[email protected] pipeline]# oc create -f deploy-app-task.yaml 
[[email protected] pipeline]# oc create -f 3-mvn-cache-and-deploy.yaml 
[[email protected] pipeline]# oc create -f 3-pipelinerun-mvn-cache-and-deploy.yaml

[[email protected] pipeline]# cat 3-mvn-cache-and-deploy.yaml 
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: mvn-cache-and-deploy
spec:
  params:
    - name: NAMESPACE
      description: The namespace for the deployment
      type: string
    - name: APP_IMAGE
      type: string
      default: image-registry.openshift-image-registry.svc:5000/demo/spring-petclinic
      description: The application image to build
    - name: APP_IMAGE_TAG
      type: string
      default: v1
      description: The application image tag to build
    - name: GIT_REPO
      type: string
      description: The application git repository
      default: http://gogs-demo-cicd.apps.ocp4.example.com/gogs/spring-petclinic.git
    - name: pathToContext
      description: The path to the build context, used by Kaniko - within the workspace
      default: ./
    - name: GIT_REVISION
      type: string
      default: master
      description: The application git revision
    - name: MAVEN_MIRROR_URL
      type: string
      default: http://nexus-demo-cicd.apps.ocp4.example.com/repository/maven-public/
      description: The base URL of a mirror used for retrieving artifacts. e.g. http://nexus_addr/repository/maven-public
  workspaces:
    - name: source-cache
    - name: maven-settings
    - name: maven-cache
  tasks:
    - name: fetch-repository
      taskRef:
        name: git-clone
        kind: ClusterTask
      workspaces:
        - name: output
          workspace: source-cache
      params:
        - name: url
          value: $(params.GIT_REPO)
        - name: revision
          value: $(params.GIT_REVISION)
        - name: deleteExisting
          value: "true"
    - name: mvn-package
      taskRef:
        name: maven-repo
      runAfter:
      - fetch-repository
      workspaces:
      - name: source
        workspace: source-cache
      - name: mvn-repo
        workspace: maven-cache
      - name: maven-settings
        workspace: maven-settings
      params:
      - name: MAVEN_MIRROR_URL
        value: $(params.MAVEN_MIRROR_URL)
      - name: MAVEN_IMAGE
        value: registry.cn-hangzhou.aliyuncs.com/laocai/mvn:3.5.0-jdk-8
      - name: GOALS
        value:
        - package
        - -DskipTests=true

    - name: build-image
      taskRef:
        name: buildah
        kind: ClusterTask
      runAfter:
        - mvn-package
      workspaces:
        - name: source
          workspace: source-cache
      params:
        - name: IMAGE
          value: $(params.APP_IMAGE):$(params.APP_IMAGE_TAG)
        - name: BUILDER_IMAGE
          value: registry.example.com:5000/rhel8/buildah:v1

    - name: deploy-dev
      taskRef:
        name: deploy-app
      runAfter:
        - build-image
      params:
      - name: DEPLOYMENT_NAME
        value: spring-petclinic
      - name: CURRENT_IMAGE
        value: "quay.io/siamaksade/spring-petclinic:latest"
      - name: NEW_IMAGE_NAME
        value: $(params.APP_IMAGE)
      - name: NEW_IMAGE_TAG
        value: $(params.APP_IMAGE_TAG)
      - name: NAMESPACE
        value: $(params.NAMESPACE)
      - name: KUSTOMIZE_OVERLAY_DIR
        value: environments/dev
      workspaces:
      - name: source
        workspace: source-cache

pipelinerun

[[email protected] pipeline]# cat 3-pipelinerun-mvn-cache-and-deploy.yaml 
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  generateName: mvn-cache-and-deploy-
spec:
  params:
  - name: NAMESPACE
    value: demo
  pipelineRef:
    name: mvn-cache-and-deploy
  workspaces:
  - name: source-cache
    persistentVolumeClaim:
      claimName: petclinic-dev-workspace
  - name: maven-settings
    emptyDir: {}
  - name: maven-cache
    persistentVolumeClaim:
      claimName: m2-cache

pipeline #4 mvn-cache-build-cache-and-deploy

buildah-cache-task.yaml

4-mvn-cache-build-cache-and-deploy.yaml

4-pipelinerun-mvn-cache-build-cache-and-deploy.yaml

這是一個沒有達到預期效果的pipeline，因為比上一個pipeline 用時反而更長了。

與上一個pipeline 的差別是第三步 build-image，我根據 clustertask buildah 修改了一個新的task，檔案是buildah-cache-task.yaml

在 volumes varlibcontainers 這邊使用了pvc，原來是 emptyDir

[[email protected] pipeline]# cat buildah-cache-task.yaml 
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: buildah-cache
...
spec:
  params:
  - default: vfs
    description: Set buildah storage driver
    name: STORAGE_DRIVER
    type: string

...

  steps:
  - image: $(params.BUILDER_IMAGE)
    name: build
    resources: {}
    script: |
      buildah --storage-driver=$(params.STORAGE_DRIVER) bud \
        $(params.BUILD_EXTRA_ARGS) --format=$(params.FORMAT) \
        --tls-verify=$(params.TLSVERIFY) --no-cache \
        -f $(params.DOCKERFILE) -t $(params.IMAGE) $(params.CONTEXT)
    volumeMounts:
    - mountPath: /var/lib/containers
      name: varlibcontainers
    workingDir: $(workspaces.source.path)
  - image: $(params.BUILDER_IMAGE)
    name: push
    resources: {}
    script: |
      buildah --storage-driver=$(params.STORAGE_DRIVER) push \
        $(params.PUSH_EXTRA_ARGS) --tls-verify=$(params.TLSVERIFY) \
        --digestfile $(workspaces.source.path)/image-digest $(params.IMAGE) \
        docker://$(params.IMAGE)
    volumeMounts:
    - mountPath: /var/lib/containers
      name: varlibcontainers
    workingDir: $(workspaces.source.path)
...

  volumes:
  - name: varlibcontainers
    persistentVolumeClaim:
      claimName: var-lib-containers
  workspaces:
  - name: source

Openshift4 Pipeline 調優

一些可能的原因和後續改進方向：

調整 storage-driver，預設是 vfs 方式，一個鏡像占用了 1.5G，制作鏡像時候占用2.1G。而 java:8 基礎鏡像530M, jar 48M

目前試過 overlay，overlay2，在儲存鏡像檔案時候報錯。

+ buildah --storage-driver=overlay bud --format=oci --tls-verify=true --no-cache -f ./Dockerfile -t image-registry.openshift-image-registry.svc:5000/demo-cicd/spring-petclinic:v1 .
level=error msg="'overlay' is not supported over nfs at \"/var/lib/containers/storage/overlay\""
kernel does not support overlay fs: 'overlay' is not supported over nfs at "/var/lib/containers/storage/overlay": backing file system is unsupported for this graph driver
level=error msg="exit status 125"

查了一下vfs 的介紹，docker裡運作docker，是一個典型的使用場景

VFS存儲驅動程式不是聯合檔案系統；相反，每個圖層都是磁盤上的目錄，并且沒有寫時複制支援。要建立一個新圖層，先前的圖層會進行“深層複制”。與其他存儲驅動程式相比，這會導緻磁盤性能下降和磁盤使用空間更多。但是，它強大，穩定，适用于各種環境。它也可以用作在測試環境中驗證其他存儲後端的機制。

但并不建議在實際或者生産環境使用，但是對于需要進行簡單驗證的場景，或者需要測試Docker引擎的其他部件的場景，是很有價值的。對于在Docker中運作Docker的場景也很有用。

build 一步，buildah 原來有個 no-cache 參數，我去掉之後，并沒有效果。
使用 node 節點鏡像資料檔案，還沒想好怎麼做。這樣的話，最好将執行 pipeline 的project 固定在幾個node上。

一些注意事項

以上每個pipeline 共用了一個存儲，且都是直接使用的根目錄。且每個pipeline 初始化時候都會清空目錄，是以共用存儲情況下，要避免同時執行。

還有一個辦法，就是使用子目錄，比如 git-clone 這個task，通過配置 subdirectory 參數可以使用 workspace 下的子目錄，就避免了多個流水線共用一個根目錄的情況。
使用到的鏡像，用于執行任務的在 clustertask 或者pipeline 中可以指定，init container 需要在 csv 中修改

[[email protected] ~]# oc -n openshift-operators get clusterserviceversions
NAME                                DISPLAY                       VERSION   REPLACES                            PHASE
redhat-openshift-pipelines.v1.4.0   Red Hat OpenShift Pipelines   1.4.0     redhat-openshift-pipelines.v1.3.1   Succeeded

# 修改 csv 中對應環境變量，然後檢視 openshift-pipelines-operator pod 是否更新變量  
[[email protected] ~]# oc -n openshift-operators edit clusterserviceversions redhat-openshift-pipelines.v1.4.0

參照這篇文章最後

https://github.com/cai11745/ocp4-userguide/blob/master/DevOps/openshift-pipeline-Tekton-install.md

pipelinerun 預設使用的 serviceaccount 叫 pipeline，如果把鏡像推到到其他project并釋出，需要oc adm policy 授權。

FAQ

pipeline #4 使用nfs 存儲緩存鏡像檔案報錯

因為容器運作使用者為 root，nfs server端存儲權限配置為 /nfs/storageclass *(rw,sync,all_squash)

即所有使用者使用nfs存儲時候，使用者都會被限制為 nfsnobody，導緻容器使用時候 chown 失敗

解決方法：把nfs server 權限改為 /nfs/storageclass *(rw,sync,no_root_squash)

即root 使用不限制，其他使用者使用還是會被限制為 nfsnobody

重新建立pvc即可

STEP-BUILD
+ buildah --storage-driver=vfs bud --format=oci --tls-verify=true --no-cache -f ./Dockerfile -t image-registry.openshift-image-registry.svc:5000/demo/spring-petclinic:v1 .
chown /var/lib/containers/storage/vfs: operation not permitted
level=error msg="exit status 125"

sh-4.4# id
uid=0(root) gid=0(root) groups=0(root),1000670000

[[email protected] storageclass]# ls  demo-cicd-var-lib-containers-pvc-16b94c24-cace-45a8-be7a-285439fea406/storage/ -l
total 0
drwx------ 2 nfsnobody nfsnobody 6 May  7 23:18 mounts
-rw-r--r-- 1 nfsnobody nfsnobody 0 May  7 23:18 storage.lock
drwx------ 2 nfsnobody nfsnobody 6 May  7 23:18 tmp
-rw-r--r-- 1 nfsnobody nfsnobody 0 May  7 23:18 userns.lock
drwx------ 2 nfsnobody nfsnobody 6 May  7 23:18 vfs

參考連結

https://ibm-developer.gitbook.io/cloudpakforapplications-appmod/ci-cd/tekton-tutorial-openshift

Openshift4 Pipeline 調優