天天看點
傳回

Jenkins釋出項目到K8S叢集CI/CD常用服務部署Jenkins添加Kubenetes叢集制作jenkins-slave鏡像使用jenkins部署項目到K8S配置檔案示例

CI/CD常用服務部署

Jenkins && Github

docker-compose檔案

version: "3"

services:
  jenkins:
    container_name: jenkins
    image: jenkins:latest
    restart: always
    user: root
    ports:
      - 8080:8080
      - 50000:50000
    volumes:
      - /data/jenkins/jenkins_home:/var/jenkins_home
      - /etc/localtime:/etc/localtime
    deploy:
      resources:
        limits:
          cpus: '2'
          memory: 2G

  gitlab:
    container_name: gitlab
    image: gitlab/gitlab-ce:latest
    restart: always
    environment:
      TZ: 'Asia/Shanghai'
      GITLAB_OMNIBUS_CONFIG: |
        external_url 'http://192.168.2.10'
    ports:
      - 33022:22/tcp
      - 80:80
      - 443:443
    volumes:
      - /data/gitlab/config:/etc/gitlab
      - /data/gitlab/data:/var/opt/gitlab
      - /data/gitlab/logs:/var/log/gitlab
      - /etc/localtime:/etc/localtime
    deploy:
      resources:
        limits:
          cpus: '2'
          memory: 2G

Harbor鏡像倉庫

前置條件: 已安裝docker-compose

下載下傳位址

https://github.com/goharbor/harbor/releases

tar -zxf harbor-offline-installer-v2.1.0.tgz
mv harbor /usr/local/
cp harbor.yml.tmpl  harbor.yml

## 編輯配置
vim harbor.yml
hostname: 192.168.2.10
# http related config
http:
  # port for http, default is 80. If https enabled, this port will redirect to https port
  port: 85
# https related config
#https:
  # https port for harbor, default is 443
  #  port: 443
  # The path of cert and key files for nginx
  # certificate: /your/certificate/path
  # private_key: /your/private/key/path
# 修改data目錄
data_volume: /data/harbor/data
# 修改log目錄
	location: /data/harbor/log/harbor
    
           
## 安裝
./install.sh

通路: ip:85

預設賬戶密碼：admin/Harbor12345

配置項目,使用者

docker登入私有倉庫

docker login --username=wangshui898 http://192.168.2.10:85

啟動關閉

進入目錄下

docker-compose up -d 啟動
docker-compose stop 停止
docker-compose restart 重新啟動

上傳鏡像

配置daemon.json,把鏡像倉庫位址設定為可信位址

vi /etc/docker/daemon.json
{
"registry-mirrors": ["https://gsm39obv.mirror.aliyuncs.com"],
"insecure-registries": ["192.168.2.10:85"]
}

重新開機docker生效

systemctl restart docker

Jenkins添加Kubenetes叢集

生成所需K8S叢集證書

admin帳号請求證書json檔案

cat >> admin-csr.json <<EOF
{
  "CN": "admin",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "hangzhou",
      "L": "XS",
      "O": "system:masters",
      "OU": "System"
    }
  ]
}
EOF

生成admin證書

  • 前置條件: 已有k8s證書生成工具,有叢集ca相關配置和公鑰私鑰(說白了就是部署叢集時候用的那套工具以及ca配置和證書) 
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin
ls
admin.csr  admin-csr.json  admin-key.pem  admin.pem

生成pfx格式證書

生成pfx格式證書,并下載下傳到電腦

openssl pkcs12 -export -out ./jenkins-admin.pfx -inkey ./admin-key.pem -in ./admin.pem -passout pass:pfx_passwd
ls jenkins-admin.pfx 
jenkins-admin.pfx
注意: -passout 參數後的pass:pfx_passwd ,pfx_passwd是fpx檔案格式的密碼,上傳jenkins時候需要

證書上傳到jenkins

Jenkins釋出項目到K8S叢集CI/CD常用服務部署Jenkins添加Kubenetes叢集制作jenkins-slave鏡像使用jenkins部署項目到K8S配置檔案示例

在憑據管理中選擇Certificate類型–>Upload PKCS#12 certificate證書,并根據提示上傳之前的jenkins-admin.pfx

注意填寫密碼,點選Change Password,填入剛剛設定的pfx_passwd

配置jenkins雲節點,添加kubenetes叢集

插件: kubenetes

方法一:

系統管理-> 系統配置 -> cloud;點選跳轉連接配接a separate configuration page,會跳轉到k8s叢集配置頁面

方法二:

系統管理-> 節點管理 -> Configure Clouds

選擇 Add a new cloud -> kubenetes,點選Kubernetes Cloud details…進行配置

  1. 叢集名稱随意填寫
  2. Kubernetes 位址填寫apiserver對外通路位址
  3. Kubernetes 服務證書 key複制ca.pem公鑰内容
  4. 憑據選擇上傳的pfx證書
Jenkins釋出項目到K8S叢集CI/CD常用服務部署Jenkins添加Kubenetes叢集制作jenkins-slave鏡像使用jenkins部署項目到K8S配置檔案示例

點選連接配接測試,如出現success(老版本)或者Connected to Kubernetes v1.18.10,表示連接配接k8s叢集成功

如果jenkins部署在k8s叢集内部,則Kubernetes位址隻需要填寫k8s叢集名稱即可,可通過kubectl get svc擷取叢集名稱

制作jenkins-slave鏡像

擷取agent.jar

也叫slave.jar,實際jenkins中的remoting

官網位址

https://www.jenkins.io/zh/projects/remoting/

jar包下載下傳位址

https://repo.jenkins-ci.org/public/org/jenkins-ci/main/remoting/

擷取jenkins-slave

git位址

https://github.com/jenkinsci/docker-inbound-agent/blob/master/jenkins-agent

編輯Dockerfile

這裡隻加了jdk環境和maven環境,可以根據實際情況添加 
ls -1
apache-maven-3.6.3-bin.tar.gz
Dockerfile
jdk-8u251-linux-x64.tar.gz
jenkins-agent
remoting-3.40.1.jar
remoting-4.6.jar
settings.xml
kubectl
           
FROM centos:7

RUN yum install git crul wget openssl openssl-devel\
	&& mkdir -p /usr/share/jenkins

ADD  jdk-8u251-linux-x64.tar.gz /usr/local/
ADD  apache-maven-3.6.3-bin.tar.gz /usr/local/
COPY settings.xml /usr/local/apache-maven-3.6.3/conf/settings.xml
COPY jenkins-agent /usr/local/bin/jenkins-slave
COPY remoting-3.40.1.jar /usr/share/jenkins/agent.jar
COPY kubectl /sbin/kubectl

RUN chmod 755 /usr/share/jenkins/agent.jar \
        && chmod +x /usr/local/bin/jenkins-slave

USER root

WORKDIR /home/jenkins

ENV M3_HOME=/usr/local/apache-maven-3.6.3
ENV JAVA_HOME=/usr/local/jdk1.8.0_251
ENV PATH=${PATH}:${JAVA_HOME}/bin:${M3_HOME}/bin

ENTRYPOINT ["jenkins-slave"]
#ENTRYPOINT ["/bin/ping"]
#CMD ["www.baidu.com"]
remoting.jar具體重命名和所放目錄,要參照jenkins-slave腳本的執行指令參數來修改

使用jenkins部署項目到K8S

生成kube-admin.kubeconfig檔案

插件: Kubernetes Continuous Deploy

生成admin使用者的kubeconfig檔案

ls admin*
admin.csr  admin-csr.json  admin-key.pem  admin.pem
ls -1 /opt/k8s/ssl/ca* 
ca-key.pem	ca.pem

# 設定叢集參數
kubectl config set-cluster kubernetes \
--certificate-authority=/opt/k8s/ssl/ca.pem \
--embed-certs=true \
--server=https://192.168.2.100:6443 \
--kubeconfig=kube-admin.kubeconfig

# 設定用戶端認證參數-設定證書資訊
kubectl config set-credentials admin \
--client-certificate=/data/TLS/k8s/admin.pem \
--client-key=/data/TLS/k8s/admin-key.pem \
--embed-certs=true \
--kubeconfig=kube-admin.kubeconfig

# 設定上下文參數-綁定賬号和管理的叢集
kubectl config set-context kubernetes \
--cluster=kubernetes \
--user=admin \
--kubeconfig=kube-admin.kubeconfig

# 設定預設上下文
kubectl config use-context kubernetes \
--kubeconfig=kube-admin.kubeconfig
–embed-certs=true表示把證書内容寫入到檔案中

将kube-admin.kubeconfig儲存到jenkins

安裝好Kubernetes Continuous Deploy插件後,憑據管理中選擇Kubernetes configuration (kubeconfig)–>Enter directily,将kube-admin.kubeconfig内容拷貝到輸入框,儲存.

jenkins釋出到K8S邏輯說明

jenkins生成動态jenkins-slave節點,裡面有kubectl指令,通過憑據儲存的kube-admin.kubeconfig證書檔案,在節點内儲存到/root/.kube/config,動态節點即可操作叢集,通過項目代碼内的deployment.yaml釋出到叢集

配置檔案示例

deployment.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: NS_NAME 
---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: APP_NAME
  namespace: NS_NAME
  labels:
    app: APP_NAME
spec:
  replicas: RS_NUM
  selector:
    matchLabels:
      app: APP_NAME
  template:
    metadata:
      labels:
        app: APP_NAME
    spec:
      imagePullSecrets:
      - name: SECRET_NAME
      containers:
      - image: IMAGE_NAME
        name: APP_NAME        
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: APP_NAME
  name: APP_NAME-service
  namespace: NS_NAME
spec:
  ports:
  - port: 8091
    protocol: TCP
    targetPort: 8080
  selector:
    app: APP_NAME

Jenkinsfile

// 插件: pipeline / git / Git Parameter / Kubernetes / Kubernetes Continuous Deploy

// 項目資訊
def project_name = "apps"
def app_name = "book"
// 鏡像資訊
def harbor_url = "192.168.2.10:85"
def harbor_auth = "715b0f10-d7ae-4a8a-bcc7-d5a188f5ba23"
def image_tag = "${BUILD_NUMBER}"
def image_name = "${harbor_url}/${project_name}/${app_name}:${image_tag}"
// Git資訊
def git_url = "http://192.168.2.10/ipanel/book.git"
def git_auth = "715b0f10-d7ae-4a8a-bcc7-d5a188f5ba23"
// k8s叢集資訊
def k8s_auth = ""
def k8s_kubeconfig_auth = "a37784ad-f7f2-49df-96f5-f1dc9b5ebd6b"
def Secret_Name = "jenkins-slave-secret"
pipeline {
    agent {
        kubernetes {
            cloud 'kubernetes'
            defaultContainer 'jnlp'
            yaml """
                apiVersion: v1
                kind: Pod
                spec:
                  containers:
                  - name: jnlp
                    image: ${harbor_url}/library/jenkins-slave:v1
                    imagePullPolicy: Always
                    volumeMounts:
                    - mountPath: /etc/resolv.conf
                      name: resolv
                      readOnly: false
                    - mountPath: /var/run/docker.sock
                      name: docker-sock
                      readOnly: false
                    - mountPath: /data/maven/repos
                      name: maven-date
                      readOnly: false
                    - mountPath: /usr/bin/docker
                      name: docker-bin
                      readOnly: false
                    - mountPath: /home/jenkins/agent
                      name: agent-home
                      readOnly: false
                  nodeSelector:
                    kubernetes.io/os: linux
                  restartPolicy: Never
                  
                  volumes:
                  - hostPath:
                      path: /usr/bin/docker
                    name: docker-bin
                  - hostPath:
                      path: /etc/resolv.conf
                    name: resolv
                  - hostPath:
                      path: /var/run/docker.sock
                    name: docker-sock
                  - emptyDir:
                    name: agent-home
                  - name: maven-date
                    nfs:
                      path: /data/NFS/maven
                      readOnly: false
                      server: 192.168.2.10
                """
            
        }
    }
    parameters {
    gitParameter name: 'Branch', 
                 type: 'PT_BRANCH',
                 branchFilter: '.*',
                 defaultValue: 'master',
                 selectedValue: 'NONE',
                 sortMode: 'NONE',
				 description: 'Select your branch or tag.'
    choice(name: 'ReplicasNum', choices: ['1', '3', '5'], description: '請選擇副本數')
	choice(name: 'NameSpace', choices: ['default', 'dev', 'apps'], description: '請選擇命名空間')
    }
    stages {
        stage('源碼拉取') {
            steps {
                container('jnlp') {
                    checkout([$class: 'GitSCM', 
                    branches: [[name: "${Branch}"]], 
                    doGenerateSubmoduleConfigurations: false, 
                    extensions: [], submoduleCfg: [], 
                    userRemoteConfigs: [[credentialsId: "${git_auth}", url: "${git_url}"]]])
                    sh 'ls'
                }
            }
        }
        stage('代碼建構') {
            steps {
                container('jnlp') {
                    sh "mvn clean package -Dmaven.test.skip=true"
                }
            }
        }
        stage('鏡像建構') {
            steps {
                container('jnlp') {
                    withCredentials([usernamePassword(credentialsId: "${harbor_auth}", passwordVariable: 'passwd', usernameVariable: 'user')]) {
                        sh """
						echo "Docker Login"
                        docker login -u "$user" -p "$passwd" "${harbor_url}"
                        echo "Add Dockfile"
                        echo '
                        	FROM ${harbor_url}/devops/centos-jdk1.8:v1
    						COPY  target/*.jar /home/app/book.jar
    						USER root
    						WORKDIR /home/app
    						EXPOSE 8080
    						ENTRYPOINT ["java","-jar","book.jar"]                                
                            ' > Dockerfile
                        echo "Docker Build"
                        docker build -t ${image_name} .
                        echo "Docker push"
                        docker push ${image_name}
                        echo "Docker rm images"
                        docker rmi ${image_name}
                        """
                   }
                }
            }
        }
        stage('釋出到K8S') {
            steps {
                container('jnlp') {
                withCredentials([kubeconfigContent(credentialsId: "${k8s_kubeconfig_auth}", variable: 'KUBECONFIG_CONTENT')]) {
					withCredentials([usernamePassword(credentialsId: "${harbor_auth}", passwordVariable: 'passwd', usernameVariable: 'user')]) {
						sh """
                        mkdir /root/.kube
                        echo "${KUBECONFIG_CONTENT}" > /root/.kube/config
                        sed -i 's#IMAGE_NAME#${image_name}#g' deployment.yaml
                        sed -i 's#APP_NAME#${app_name}#g' deployment.yaml
                        sed -i 's#NS_NAME#${params.NameSpace}#g' deployment.yaml
                        sed -i 's#RS_NUM#${params.ReplicasNum}#g' deployment.yaml
						sed -i 's#SECRET_NAME#${Secret_Name}#g' deployment.yaml
                        cat deployment.yaml
						kubectl create secret docker-registry "${Secret_Name}" --docker-server="${harbor_url}" --docker-username="$user" --docker-password="$passwd" -n "${params.NameSpace}" |true
                        kubectl apply -f deployment.yaml --record
						kubectl get node
                        kubectl get deploy -o wide -n "${params.NameSpace}"
                        kubectl get pod -o wide -n "${params.NameSpace}"
						kubectl rollout history deploy/${app_name} -n "${params.NameSpace}"
						"""
						}                    
                    }
                }
            }
        }
        
    }
}
如果使用NFS挂載方式,node節點需要安裝nfs-utils
devops CI/CD jenkins kubernetes k8s
上一篇: k8s 部署 nacos 1.4.1 叢集
下一篇: k8s部署 nacos 叢集

繼續閱讀