二進制部署K8s叢集第27節之helm踩坑部署harbor

一、建立PVC

需要先部署NFS

https://blog.51cto.com/yht1990/2630775《storageClass動态挂載對接NFS存儲》

kubectl create ns harbor
cat > harbor-pvc.yaml <<'eof'
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: harbor-pvc
  namespace: harbor
spec:
  accessModes:     
    - ReadWriteOnce
  storageClassName: storage-nfs 
  resources:
    requests:
      storage: 20Gi
eof
kubectl apply -f harbor-pvc.yaml           

二、拉取harbor的chart到本地

[root@k8s-master harbor]# helm repo add harbor https://helm.goharbor.io
[root@k8s-master harbor]# helm repo update
[root@k8s-master harbor]# helm search repo harbor
NAME            CHART VERSION   APP VERSION     DESCRIPTION                                       
harbor/harbor   1.5.1           2.1.1           An open source trusted cloud native registry th...
[root@k8s-master harbor]# helm repo ls
NAME    URL                                      
stable  http://mirror.azure.cn/kubernetes/charts/
harbor  https://helm.goharbor.io  
[root@k8s-master harbor]# helm pull harbor/harbor --version 1.5.1           

三、Chart參數設定

生産環境size必須調大

[root@k8s-master harbor]# tar xf harbor-1.5.1.tgz
[root@k8s-master harbor]# cd harbor
[root@k8s-master harbor]#  cp values.yaml values.yaml.bak
[root@k8s-master harbor]#  vim values.yaml 
...
 36       core: harbor.od.com
...
101 externalURL: https://harbor.od.com  # 設定通路域名
...
108 externalURL: https://harbor.od.com
185 193   persistentVolumeClaim:
194     registry:
195       # Use the existing PVC which must be created manually before bound,
196       # and specify the "subPath" if the PVC is shared with other components
197       existingClaim: "harbor-pvc"
198       # Specify the "storageClass" used to provision the volume. Or the default
199       # StorageClass will be used(the default).
200       # Set it to "-" to disable dynamic provisioning
201       storageClass: ""
202       subPath: "registry"
203       accessMode: ReadWriteOnce
204       size: 5Gi
205     chartmuseum:
206       existingClaim: "harbor-pvc"
207       storageClass: ""
208       subPath: "chartmuseum"
209       accessMode: ReadWriteOnce
210       size: 5Gi
211     jobservice:
212       existingClaim: "harbor-pvc"
213       storageClass: ""
214       subPath: "jobservice"
215       accessMode: ReadWriteOnce
216       size: 1Gi
217     # If external database is used, the following settings for database will
218     # be ignored
219     database:
220       existingClaim: "harbor-pvc"
221       storageClass: ""
222       subPath: "database"
223       accessMode: ReadWriteOnce
224       size: 1Gi
225     # If external Redis is used, the following settings for Redis will
226     # be ignored
227     redis:
228       existingClaim: "harbor-pvc"
229       storageClass: ""
230       subPath: "redis"
231       accessMode: ReadWriteOnce
232       size: 1Gi
233     trivy:
234       existingClaim: "harbor-pvc"
235       storageClass: ""
236       subPath: "trivy"
237       accessMode: ReadWriteOnce
238       size: 5Gi
 ...
539 clair:
540   enabled: false
 ...
569 trivy:
570   # enabled the flag to enable Trivy scanner
571   enabled: false
  ...
626 notary:
627   enabled: false
 ...           

四、踩坑一

redis持久化資料目錄權限導緻無法登入

redis資料目錄，/var/lib/redis，需要設定redis的使用者及使用者組權限

/root/harbor/templates/redis/statefulset.yaml
      initContainers:
      - name: "change-permission-of-directory"
        securityContext:
          runAsUser: 0
        image: {{ .Values.database.internal.image.repository }}:{{ .Values.database.internal.image.tag }}
        imagePullPolicy: {{ .Values.imagePullPolicy }}
        command: ["/bin/sh"]
        args: ["-c", "chown -R 999:999 /var/lib/redis"]
        volumeMounts:
        - name: data
          mountPath: /var/lib/redis
          subPath: {{ $redis.subPath }}           

五、踩坑二

registry元件的鏡像存儲目錄權限導緻鏡像推送失敗

registry的鏡像存儲目錄，需要設定registry使用者的使用者及使用者組，不然鏡像推送失敗

/root/harbor/templates/registry/registry-dpl.yaml
      initContainers:
      - name: "change-permission-of-directory"
        securityContext:
          runAsUser: 0
        image: {{ .Values.database.internal.image.repository }}:{{ .Values.database.internal.image.tag }}
        imagePullPolicy: {{ .Values.imagePullPolicy }}
        command: ["/bin/sh"]
        args: ["-c", "chown -R 10000:10000 {{ .Values.persistence.imageChartStorage.filesystem.rootdirectory }}"]
        volumeMounts:
        - name: registry-data
          mountPath: {{ .Values.persistence.imageChartStorage.filesystem.rootdirectory }}
          subPath: {{ .Values.persistence.persistentVolumeClaim.registry.subPath }}           

六、踩坑三

chartmuseum存儲目錄權限，導緻chart推送失敗

/root/harbor/templates/chartmuseum/chartmuseum-dpl.yaml
      initContainers:
      - name: "change-permission-of-directory"
        securityContext:
          runAsUser: 0
        image: {{ .Values.database.internal.image.repository }}:{{ .Values.database.internal.image.tag }}
        imagePullPolicy: {{ .Values.imagePullPolicy }}
        command: ["/bin/sh"]
        args: ["-c", "chown -R 10000:10000 /chart_storage"]
        volumeMounts:
        - name: chartmuseum-data
          mountPath: /chart_storage
          subPath: {{ .Values.persistence.persistentVolumeClaim.chartmuseum.subPath }}           

七、安裝harbor

cd 
helm install harbor ./harbor -n harbor
helm -n harbor ls
kubectl -n harbor get po           

八、配置通路推送

8.1、域名配置

8.2、配置docker daemon

cat /etc/docker/daemon.json
  "insecure-registries": [  
 "harbor.od.com"   
  ],       
systemctl restart docker           

8.3 推送chart

docker login harbor.od.com
helm plugin install https://github.com/chartmuseum/helm-push
helm plugin ls
kubectl get secret harbor-harbor-ingress -n harbor -o jsonpath="{.data.ca\.crt}" | base64 -d >harbor.ca.crt
cp harbor.ca.crt /etc/pki/ca-trust/source/anchors
update-ca-trust enable; update-ca-trust extract
helm repo add myharbor https://harbor.od.com/chartrepo/library --ca-file=harbor.ca.crt
helm repo ls
helm push harbor myharbor --ca-file=harbor.ca.crt -u admin -p Harbor12345