天天看點
傳回

helm入門helm入門

helm入門

簡介

Helm是一個由CNCF孵化和管理的項目，用于對需要在Kubernetes 上部署的複雜應用進行定義、安裝和更新。Helm以Chart的方式對應用軟體進行描述，可以友善地建立、版本化、共享和釋出複雜的應用軟體。

helm3架構

helm入門helm入門

helm3安裝 

#github位址
#https://github.com/helm/helm
#本文示例使用的是v3.7.0版本
wget https://get.helm.sh/helm-v3.7.0-linux-amd64.tar.gz
#解壓->helm放入PATH一個路徑下

helm中三大概念

  • Chart：一個Helm包，其中包含運作一個應用所需要的工具和資源定義，還可能包含Kubernetes叢集中的服務定義，類似于Homebrew 中的formula、APT中的dpkg或者Yum中的RPM檔案。
  • Release：在Kubernetes叢集上運作的一個Chart執行個體。在同一個 叢集上，一個Chart可以被安裝多次。例如有一個MySQL Chart，如果想在伺服器上運作兩個MySQL資料庫，就可以基于這個Chart安裝兩次。 每次安裝都會生成新的Release，會有獨立的Release名稱。
  • Repository：用于存放和共享Chart倉庫。 簡單來說，Helm整個系統的主要任務就是，在倉庫中查找需要的 Chart，然後将Chart以Release的形式安裝到Kubernetes叢集中。

Helm Chart的使用

下面将使用一個例子展示helm chart的使用。

建立 

$ helm create nginx

該指令會建立一個nginx檔案目錄，tree檢視目錄結構

$ tree
.
├── charts  #包含chart依賴的其他chart
├── Chart.yaml #包含了chart資訊的YAML檔案
├── templates #模闆目錄， 當和values 結合時，可生成有效的Kubernetes manifest檔案
│   ├── deployment.yaml
│   ├── _helpers.tpl
│   ├── hpa.yaml
│   ├── ingress.yaml
│   ├── NOTES.txt
│   ├── serviceaccount.yaml
│   ├── service.yaml
│   └── tests #測試
│       └── test-connection.yaml
└── values.yaml #chart 預設的配置值

Chart.yaml

$ cat Chart.yaml 
apiVersion: v2 #在heml3中apiVersion必須是v2
name: nginx #chart名字
description: A Helm chart for Kubernetes #chart描述
type: application #chart類型 application（預設）、library
version: 0.1.0 #chart的版本
appVersion: "1.16.0" #應用的版本

values.yaml

$ cat values.yaml 
# Default values for nginx.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

replicaCount: 1

image:
  repository: nginx
  pullPolicy: IfNotPresent
  # Overrides the image tag whose default is the chart appVersion.
  tag: ""

imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""

serviceAccount:
  # Specifies whether a service account should be created
  create: true
  # Annotations to add to the service account
  annotations: {}
  # The name of the service account to use.
  # If not set and create is true, a name is generated using the fullname template
  name: ""

podAnnotations: {}

podSecurityContext: {}
  # fsGroup: 2000

securityContext: {}
  # capabilities:
  #   drop:
  #   - ALL
  # readOnlyRootFilesystem: true
  # runAsNonRoot: true
  # runAsUser: 1000

service:
  type: ClusterIP
  port: 80

ingress:
  enabled: false
  className: ""
  annotations: {}
    # kubernetes.io/ingress.class: nginx
    # kubernetes.io/tls-acme: "true"
  hosts:
    - host: chart-example.local
      paths:
        - path: /
          pathType: ImplementationSpecific
  tls: []
  #  - secretName: chart-example-tls
  #    hosts:
  #      - chart-example.local

resources: {}
  # We usually recommend not to specify default resources and to leave this as a conscious
  # choice for the user. This also increases chances charts run on environments with little
  # resources, such as Minikube. If you do want to specify resources, uncomment the following
  # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
  # limits:
  #   cpu: 100m
  #   memory: 128Mi
  # requests:
  #   cpu: 100m
  #   memory: 128Mi

autoscaling:
  enabled: false
  minReplicas: 1
  maxReplicas: 100
  targetCPUUtilizationPercentage: 80
  # targetMemoryUtilizationPercentage: 80

nodeSelector: {}

tolerations: []

affinity: {}

templates目錄下存放了應用編排檔案。

  • （_）開頭的檔案用來存儲局部和輔助對象，供其他chart模闆使用。模闆指令都是嵌入在

    {{

     和

    }}

     之間的。 
    cat _helpers.tpl 
{{/*
Expand the name of the chart.
*/}}
{{- define "nginx.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
{{- end }}

{{/*
Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
If release name contains chart name it will be used as a full name.
*/}}
{{- define "nginx.fullname" -}}
{{- if .Values.fullnameOverride }}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- $name := default .Chart.Name .Values.nameOverride }}
{{- if contains $name .Release.Name }}
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
{{- end }}
{{- end }}
{{- end }}

{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "nginx.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
{{- end }}

{{/*
Common labels
*/}}
{{- define "nginx.labels" -}}
helm.sh/chart: {{ include "nginx.chart" . }}
{{ include "nginx.selectorLabels" . }}
{{- if .Chart.AppVersion }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
{{- end }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end }}

{{/*
Selector labels
*/}}
{{- define "nginx.selectorLabels" -}}
app.kubernetes.io/name: {{ include "nginx.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}

{{/*
Create the name of the service account to use
*/}}
{{- define "nginx.serviceAccountName" -}}
{{- if .Values.serviceAccount.create }}
{{- default (include "nginx.fullname" .) .Values.serviceAccount.name }}
{{- else }}
{{- default "default" .Values.serviceAccount.name }}
{{- end }}
{{- end }}
  • yaml格式的編排檔案中将某些字段設定為“模闆指令”，這些“模闆指令”會在Helm部署應用時進行參數注入和模闆的動态渲染。 
    $ cat deployment.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "nginx.fullname" . }}
  labels:
    {{- include "nginx.labels" . | nindent 4 }}
spec:
  {{- if not .Values.autoscaling.enabled }}
  replicas: {{ .Values.replicaCount }}
  {{- end }}
  selector:
    matchLabels:
      {{- include "nginx.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      {{- with .Values.podAnnotations }}
      annotations:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      labels:
        {{- include "nginx.selectorLabels" . | nindent 8 }}
    spec:
      {{- with .Values.imagePullSecrets }}
      imagePullSecrets:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      serviceAccountName: {{ include "nginx.serviceAccountName" . }}
      securityContext:
        {{- toYaml .Values.podSecurityContext | nindent 8 }}
      containers:
        - name: {{ .Chart.Name }}
          securityContext:
            {{- toYaml .Values.securityContext | nindent 12 }}
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          ports:
            - name: http
              containerPort: 80
              protocol: TCP
          livenessProbe:
            httpGet:
              path: /
              port: http
          readinessProbe:
            httpGet:
              path: /
              port: http
          resources:
            {{- toYaml .Values.resources | nindent 12 }}
      {{- with .Values.nodeSelector }}
      nodeSelector:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      {{- with .Values.affinity }}
      affinity:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      {{- with .Values.tolerations }}
      tolerations:
        {{- toYaml . | nindent 8 }}
      {{- end }}
    可是使用

    helm template [NAME] [CHART] [flags]

     指令在本地渲染Helm Chart并列印最終的應用模闆内容，values.yaml和（_）開頭的檔案中的值會被注入到yaml中，參數

    --set

     則會覆寫value.yaml檔案中的值。 
    $ helm template nginx  ./nginx --set image.repository=docker.io/library/nginx,service.type=NodePort
...
...
---
# Source: nginx/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    helm.sh/chart: nginx-0.1.0
    app.kubernetes.io/name: nginx
    app.kubernetes.io/instance: nginx
    app.kubernetes.io/version: "1.16.0"
    app.kubernetes.io/managed-by: Helm
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: nginx
      app.kubernetes.io/instance: nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/name: nginx
        app.kubernetes.io/instance: nginx
    spec:
      serviceAccountName: nginx
      securityContext:
        {}
      containers:
        - name: nginx
          securityContext:
            {}
          image: "docker.io/library/nginx:1.16.0"
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 80
              protocol: TCP
          livenessProbe:
            httpGet:
              path: /
              port: http
          readinessProbe:
            httpGet:
              path: /
              port: http
          resources:
            {}
---
...
...
  • NOTES.txt在

    helm install

     或

    helm upgrade

     指令的最後，Helm會列印出對使用者有用的資訊，該檔案是純文字，但會像模闆一樣處理， 所有正常的模闆函數和對象都是可用的。 
    $ cat NOTES.txt 
1. Get the application URL by running these commands:
{{- if .Values.ingress.enabled }}
{{- range $host := .Values.ingress.hosts }}
  {{- range .paths }}
  http{{ if $.Values.ingress.tls }}s{{ end }}://{{ $host.host }}{{ .path }}
  {{- end }}
{{- end }}
{{- else if contains "NodePort" .Values.service.type }}
  export NODE_PORT=$(kubectl get --namespace {{ .Release.Namespace }} -o jsonpath="{.spec.ports[0].nodePort}" services {{ include "nginx.fullname" . }})
  export NODE_IP=$(kubectl get nodes --namespace {{ .Release.Namespace }} -o jsonpath="{.items[0].status.addresses[0].address}")
  echo http://$NODE_IP:$NODE_PORT
{{- else if contains "LoadBalancer" .Values.service.type }}
     NOTE: It may take a few minutes for the LoadBalancer IP to be available.
           You can watch the status of by running 'kubectl get --namespace {{ .Release.Namespace }} svc -w {{ include "nginx.fullname" . }}'
  export SERVICE_IP=$(kubectl get svc --namespace {{ .Release.Namespace }} {{ include "nginx.fullname" . }} --template "{{"{{ range (index .status.loadBalancer.ingress 0) }}{{.}}{{ end }}"}}")
  echo http://$SERVICE_IP:{{ .Values.service.port }}
{{- else if contains "ClusterIP" .Values.service.type }}
  export POD_NAME=$(kubectl get pods --namespace {{ .Release.Namespace }} -l "app.kubernetes.io/name={{ include "nginx.name" . }},app.kubernetes.io/instance={{ .Release.Name }}" -o jsonpath="{.items[0].metadata.name}")
  export CONTAINER_PORT=$(kubectl get pod --namespace {{ .Release.Namespace }} $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
  echo "Visit http://127.0.0.1:8080 to use your application"
  kubectl --namespace {{ .Release.Namespace }} port-forward $POD_NAME 8080:$CONTAINER_PORT
{{- end }}

部署

helm install [NAME] [CHART] [flags]

指令安裝部署，同樣values.yaml和（_）開頭的檔案中的值會被注入到yaml中，參數

--set

則會覆寫value.yaml檔案中的值。

$ helm -n default install nginx ./nginx --set image.repository=docker.io/library/nginx,service.type=NodePort
NAME: nginx
LAST DEPLOYED: Sat Oct  9 16:26:50 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
  export NODE_PORT=$(kubectl get --namespace default -o jsonpath="{.spec.ports[0].nodePort}" services nginx)
  export NODE_IP=$(kubectl get nodes --namespace default -o jsonpath="{.items[0].status.addresses[0].address}")
  echo http://$NODE_IP:$NODE_PORT
  • 檢視部署的release 
    $ helm -n default ls
NAME 	NAMESPACE	REVISION	UPDATED                                	STATUS  	CHART      	APP VERSION
nginx	default  	1       	2021-10-09 16:26:50.820014356 +0800 CST	deployed	nginx-0.1.0	1.16.0

  • release版本資訊存儲

    預設存儲在release相應namespace下的secret（可配置configmap、sql），每一個REVISION都會生成一個secret。

    $ kubectl -n default get secret -l "owner=helm"
NAME                          TYPE                 DATA   AGE
sh.helm.release.v1.nginx.v1   helm.sh/release.v1   1      1m
    檢視存儲内容 
    kubectl -n default get secret sh.helm.release.v1.nginx.v1 -ojsonpath={.data.release} |base64 -d |base64 -d |gzip -d
  • 測試release是否符合預期（需等待所有部署的資源對象建立成功），原理：通過在叢集中部署templates/tests/下的yaml，根據容器退出狀态判斷（exit 0：通過）是否測試通過。測試的yaml必須含有注解：helm.sh/hook: test。 
    $ helm test nginx
NAME: nginx
LAST DEPLOYED: Sat Oct  9 16:26:50 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE:     nginx-test-connection
Last Started:   Sat Oct  9 16:47:02 2021
Last Completed: Sat Oct  9 16:47:20 2021
Phase:          Succeeded
NOTES:
1. Get the application URL by running these commands:
  export NODE_PORT=$(kubectl get --namespace default -o jsonpath="{.spec.ports[0].nodePort}" services nginx)
  export NODE_IP=$(kubectl get nodes --namespace default -o jsonpath="{.items[0].status.addresses[0].address}")
  echo http://$NODE_IP:$NODE_PORT

更新、復原、解除安裝

  • helm upgrade [RELEASE] [CHART] [flags]

    指令進行更新。

    更新之前檢視nginx release目前版本：REVISION=1

    helm -n default ls
NAME 	NAMESPACE	REVISION	UPDATED                                	STATUS  	CHART      	APP VERSION
nginx	default  	1       	2021-10-09 16:26:50.820014356 +0800 CST	deployed	nginx-0.1.0	1.16.0
    使用更新指令，将deployment中的鏡像tag更新到1.17.0，可使用

    --dry-run

     試運作測試是否可以更新成功，并列印更新清單。 
    $ helm -n default upgrade nginx ./nginx/ --set image.repository=docker.io/library/nginx,service.type=NodePort,image.tag=1.17.0
Release "nginx" has been upgraded. Happy Helming!
NAME: nginx
LAST DEPLOYED: Sun Oct 10 13:41:17 2021
NAMESPACE: default
STATUS: deployed
REVISION: 2
NOTES:
1. Get the application URL by running these commands:
  export NODE_PORT=$(kubectl get --namespace default -o jsonpath="{.spec.ports[0].nodePort}" services nginx)
  export NODE_IP=$(kubectl get nodes --namespace default -o jsonpath="{.items[0].status.addresses[0].address}")
  echo http://$NODE_IP:$NODE_PORT
    再次檢視nginx release目前版本：REVISION=2 
    $ helm -n default ls
NAME 	NAMESPACE	REVISION	UPDATED                                	STATUS  	CHART      	APP VERSION
nginx	default  	2       	2021-10-10 13:41:17.486033411 +0800 CST	deployed	nginx-0.1.0	1.16.0

  • helm rollback <RELEASE> [REVISION] [flags]

    指令進行復原。

    復原之前可使用

    helm history RELEASE_NAME [flags]

     檢視nginx release曆史版本。 
    $ helm -n default history nginx
REVISION	UPDATED                 	STATUS    	CHART      	APP VERSION	DESCRIPTION     
1       	Sat Oct  9 16:26:50 2021	superseded	nginx-0.1.0	1.16.0     	Install complete
2       	Sun Oct 10 13:41:17 2021	deployed  	nginx-0.1.0	1.16.0     	Upgrade complete
    将nginx release復原到REVISION=1，復原完成後可檢視deployment image tag：1.16.0。 
    $ helm -n default rollback nginx 1
Rollback was a success! Happy Helming!

  • helm uninstall RELEASE_NAME [...] [flags]

     指令解除安裝release，解除安裝完成後release、release secret、k8s manifest都被删除。 
    $ helm -n default uninstall nginx
release "nginx" uninstalled

倉庫使用

Harbor

  • 使用helm安裝harbor 
    $ helm repo add harbor https://helm.goharbor.io
"harbor" has been added to your repositories
#檢視本地repo清單
$ helm repo ls
NAME  	URL                     
harbor	https://helm.goharbor.io
#檢視harbor倉庫helm chart版本
$ helm search repo harbor
NAME         	CHART VERSION	APP VERSION	DESCRIPTION                                       
harbor/harbor	1.7.3        	2.3.3      	An open source trusted cloud native registry th...
#部署harbor release
#persistence.enabled=false 關閉持久化存儲
#expose.type=nodePort service類型設定為nodeport友善暴露服務
#expose.tls.enabled=false 關閉tls
#--create-namespace 沒有harbor命名空間，則會建立
#externalURL中的ip為本機網卡ip
$ helm -n harbor install harbor harbor/harbor --set persistence.enabled=false,expose.type=nodePort,expose.tls.enabled=false,externalURL=http://10.23.18.211:30002 --create-namespace
NAME: harbor
LAST DEPLOYED: Mon Oct 11 18:38:47 2021
NAMESPACE: harbor
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Please wait for several minutes for Harbor deployment to complete.
Then you should be able to visit the Harbor portal at http://10.23.18.211:30002
For more details, please visit https://github.com/goharbor/harbor
#等待所有pod running
$ kubectl -n harbor get pod
NAME                                    READY   STATUS    RESTARTS   AGE
harbor-chartmuseum-6c9f9c84f8-ztpgp     1/1     Running   0          3m20s
harbor-core-c6dc8c895-vn5kx             1/1     Running   0          3m19s
harbor-database-0                       1/1     Running   0          3m20s
harbor-jobservice-6c4d647bcb-nd64t      1/1     Running   0          3m20s
harbor-nginx-7bc597c58b-67x6x           1/1     Running   0          3m19s
harbor-notary-server-5f598b9555-4qjpm   1/1     Running   0          3m20s
harbor-notary-signer-65b84d688d-g92tv   1/1     Running   0          3m19s
harbor-portal-945d945f-ks6tx            1/1     Running   0          3m20s
harbor-redis-0                          1/1     Running   0          3m20s
harbor-registry-6d976587dc-dbrnt        2/2     Running   0          3m20s
harbor-trivy-0                          1/1     Running   0          3m20s
    浏覽器通路http://10.23.18.211:30002即可通路harbor頁面，admin/Harbor12345為預設使用者密碼。

  • 推送nginx chart到倉庫

    登入harbor頁面，項目–>建立項目，建立一個名為helm的helm chart倉庫。

    添加helm倉庫。

    $ helm repo add my-harbor http://10.23.18.211:30002/chartrepo/helm --username=admin --password=Harbor12345
"my-harbor" has been added to your repositories
#檢視目前repo
$ helm repo ls
NAME     	URL                                     
harbor   	https://helm.goharbor.io                
my-harbor	http://10.23.18.211:30002/chartrepo/helm
    由于原生helm沒有推送指令，是以需要安裝一個推送插件。 
    $ helm plugin install https://github.com/chartmuseum/helm-push.git
Downloading and installing helm-push v0.10.0 ...
https://github.com/chartmuseum/helm-push/releases/download/v0.10.0/helm-push_0.10.0_linux_amd64.tar.gz
Installed plugin: cm-push
#檢視已安裝的插件
$ helm plugin ls
NAME   	VERSION	DESCRIPTION                      
cm-push	0.10.0 	Push chart package to ChartMuseum
    （備選方案）如果伺服器不能通路外網。可以從能夠通路外網的機器clone插件代碼和插件壓縮包，然後放到此伺服器上。 
    #在能夠通路外網的機器上
$ git clone https://github.com/chartmuseum/helm-push.git
$curl https://github.com/chartmuseum/helm-push/releases/download/v0.10.0/helm-push_0.10.0_linux_amd64.tar.gz
#将插件代碼上傳到伺服器/root/.cache/helm/plugins目錄下，建立release目錄
$ mkdiir /root/.cache/helm/plugins/helm-push/release
#将插件壓縮包上傳到伺服器/root/.cache/helm/plugins/helm-push/releases下
$ cd /root/.cache/helm/plugins
#vim 編輯helm-push/scripts/install_plugin.sh，根據需要注釋掉 `# Download with curl if possible.`下邊代碼，以注釋掉curl為例：
$ vim helm-push/scripts/install_plugin.sh
$ cat helm-push/scripts/install_plugin.sh
...
...
# Download with curl if possible.
if [ -x "$(which curl 2>/dev/null)" ]; then
  #  curl -sSL "${url}" -o "releases/v${version}.tar.gz"
     echo "jump curl"
else
    wget -q "${url}" -O "releases/v${version}.tar.gz"
fi
...
...
#安裝推送插件
$ helm plugin install ./helm-push/
Downloading and installing helm-push v0.10.0 ...
https://github.com/chartmuseum/helm-push/releases/download/v0.10.0/helm-push_0.10.0_linux_amd64.tar.gz
jump curl
Installed plugin: cm-push
[[email protected] plugins]# helm plugin ls
NAME   	VERSION	DESCRIPTION                      
cm-push	0.10.0 	Push chart package to ChartMuseum
    推送nginx chart。 
    #cd到nginx目錄，推送
$ helm cm-push ./nginx my-harbor
Pushing nginx-0.1.0.tgz to my-harbor...
Done.
#更新本地緩存的my-harbor倉庫資訊，可以确認nginx chart上傳到harbor上，也可以在harbor頁面上确認
$ helm repo  update my-harbor 
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "my-harbor" chart repository
Update Complete. ⎈Happy Helming!⎈
$ helm search repo my-harbor
NAME           	CHART VERSION	APP VERSION	DESCRIPTION                
my-harbor/nginx	0.1.0        	1.16.0     	A Helm chart for Kubernetes
  • 拉取、部署倉庫中的chart

    helm pull [chart URL | repo/chartname] [...] [flags]

     拉取倉庫中chart，pull隻會拉取到本地目前目錄下的.tgz壓縮包，可使用

    tar xzvf xxx.tgz

     解壓。 
    #更新本地helm repo 緩存
$ helm repo update
#pull拉取nginx chart
$ helm pull my-harbor/nginx

    helm install [NAME] [CHART] [flags]

     部署倉庫中的chart，和上邊部署篇幅類似，把使用本地檔案改成使用倉庫中的chart即可。 
    $ helm -n test install nginx my-harbor/nginx --set image.repository=docker.io/library/nginx,service.type=NodePort --create-namespace
NAME: nginx
LAST DEPLOYED: Mon Oct 11 19:47:36 2021
NAMESPACE: test
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
  export NODE_PORT=$(kubectl get --namespace test -o jsonpath="{.spec.ports[0].nodePort}" services nginx)
  export NODE_IP=$(kubectl get nodes --namespace test -o jsonpath="{.items[0].status.addresses[0].address}")
  echo http://$NODE_IP:$NODE_PORT
helm golang 運維 PaaS CNCF devops
上一篇: 解決windows平台下vscode編譯arduino esp8266控制台輸出亂碼
下一篇: Enable In-Memory Database Configuration

繼續閱讀