公司的CI/CD是使用Jenkins，開發、測試、預發的CI和CD都是在一起的，而生産環境的CI/CD我們是分開的
CI任務結束之後，開發可以選擇釋出哪個release版本。
可以看一下整體預覽情況：
每個Job的Pipeline狀态：
自定義釋出機器、同時有釋出及復原功能：
我們都是基于maven的Java應用，進行編譯打包其實比較簡單，這裡的CI較為簡單，我這裡隻簡單說明及其需要注意的點
maven編譯打包時，可以去掉單元測試
若Jenkins的機器配置比較高，可以可以開啟maven的并發編譯（3.3版本以上，預設支援并發）
适當調整maven的JVM
下面着重介紹CD部分的配置
以： 我們的msg-server服務為例
添加參數化建構
a.添加PROJECT(字元參數):辨別：應用名稱
b.添加DEPLOY_TYPE(Active Choices Parameter ):辨別：釋出還是復原
c.添加Version（ Active Choices Reactive Parameter的）：辨別釋出的版本
d.添加 HOTS（ Active Choices Reactive Parameter），表示釋出的主機清單
e.添加 DEPLOYED_HOSTS（Active Choices Reactive Parameter ），辨別：已經釋出過的主機清單
2.基礎共用腳本部分
cat get_versions.sh
#!/bin/bash
#prod
packageDir="/data/jenkins/repos/master/"
#test
#packageDir="/data/jenkins/repos/test"
#dev
#packageDir="/data/jenkins/repos/dev"
project=$1
deploy_type=$2
if [[ "${deploy_type}" == "Rollback" ]] ; then
lasted_version=`ls -lt ${packageDir}/${project}|grep ${project}|awk '{print $9}'|awk -F '.' '{print $1}'`
echo ${lasted_version}|awk -F " " '{for(i=1;i<=NF;i++) a[i,NR]=$i}END{for(i=1;i<=NF;i++) {for(j=1;j<=NR;j++) printf a[i,j] " ";print ""}}'
else
fi
cat get_deploy_lists.sh
PACKAGE_DIR=/data/jenkins/repos/release_package
version=$3
db_url='ops-db.xxxxxx.com'
db_user='xxxxx'
db_pwd="xxxxxxxx"
db_port=3306
format_version=`echo ${version}|awk -F '##' '{print $1}'`
#查詢已經釋出或復原過的主機清單
query_sql="select host_ip from deploy_history where status=1 and deploy_type='${deploy_type}' and version='${format_version}' and hostname like '%${project}%' "
result=`mysql -u${db_user} -p${db_pwd} -h${db_url} -P${db_port} -B jenkins -e "${query_sql}" |awk 'NR>1'`
echo ${result}|awk -F " " '{for(i=1;i<=NF;i++) a[i,NR]=$i}END{for(i=1;i<=NF;i++) {for(j=1;j<=NR;j++) printf a[i,j] " ";print ""}}'
cat get_undeploy_lists.sh
db_url='ops-db.xxxxxxx.com'
#查詢待釋出的主機
#if [ "${deploy_type}" == "Deploy" ] ; then
#query_sql="select host_ip from hosts where host_ip not in (select host_ip from deploy_history where status=1 and deploy_type='${deploy_type}' and version='${format_version}' and hostname like '%${project}%' ) "
#result=`mysql -u${db_user} -p${db_pwd} -h${db_url} -P${db_port} -B jenkins -e "${query_sql}" |awk 'NR>1'`
#echo ${result}|awk -F " " '{for(i=1;i<=NF;i++) a[i,NR]=$i}END{for(i=1;i<=NF;i++) {for(j=1;j<=NR;j++) printf a[i,j] " ";print ""}}'
#fi
#if [ "${deploy_type}" == "Rollback" ] ; then
#if rollback ,
# query_sql="select host_ip from hosts where host_ip not in (select host_ip from deploy_history where status=1 and deploy_type='Rollback' and version='${format_version}' and hostname like '%${project}%' ) "
# result=`mysql -u${db_user} -p${db_pwd} -h${db_url} -P${db_port} -B jenkins -e "${query_sql}" |awk 'NR>1'`
# echo ${result}|awk -F " " '{for(i=1;i<=NF;i++) a[i,NR]=$i}END{for(i=1;i<=NF;i++) {for(j=1;j<=NR;j++) printf a[i,j] " ";print ""}}'
#fi
query_sql="select host_ip from hosts where host_ip not in (select host_ip from deploy_history where status=1 and deploy_type='${deploy_type}' and version='${format_version}' and hostname like '%${project}%' ) and hostname like '%${project}%' "
result=`mysql -u${db_user} -p${db_pwd} -h${db_url} -P${db_port} -B jenkins -e "${query_sql}" |awk 'NR>1'`
echo ${result}|awk -F " " '{for(i=1;i<=NF;i++) a[i,NR]=$i}END{for(i=1;i<=NF;i++) {for(j=1;j<=NR;j++) printf a[i,j] " ";print ""}}'
cat deploy_success_update.sh
host_ip=$2
deploy_type=$3
version=$4
db_url='ops-db.xxxxxxxx.com'
db_pwd="xxxxxx"
insert_sql="insert into deploy_history(hostname,host_ip,deploy_type,status,version) values(\"${project}\",\"${host_ip}\",\"${deploy_type}\",1,\"${version}\" )"
echo ${insert_sql}
count_result=`mysql -u${db_user} -p${db_pwd} -h${db_url} -P${db_port} -B jenkins -e "${insert_sql}" `
3.建立資料庫及定義表結構
/*
Navicat MySQL Data Transfer
Source Server : ops-db
Source Server Version : 50728
Source Host : ops-db. xxxxx.com:3306
Source Database : jenkins
Target Server Type : MYSQL
Target Server Version : 50728
File Encoding : 65001
Date: 2020-08-13 14:25:52
*/
Create Database: CREATE DATABASE `jenkins` /*!40100 DEFAULT CHARACTER SET utf8mb4 */
SET FOREIGN_KEY_CHECKS=0;
-- ----------------------------
-- Table structure for deploy_history
CREATE TABLE `deploy_history` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`hostname` varchar(255) NOT NULL COMMENT '釋出的主機名',
`host_ip` varchar(255) NOT NULL COMMENT '釋出主機IP位址',
`deploy_type` varchar(255) NOT NULL,
`status` tinyint(1) NOT NULL COMMENT '0 釋出失敗; 1釋出成功',
`version` varchar(255) NOT NULL,
`ctm_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
KEY `udx_deploy_type_version` (`deploy_type`,`version`) USING BTREE,
KEY `idx_host_ip` (`host_ip`) USING BTREE
) ENGINE=InnoDB AUTO_INCREMENT=128 DEFAULT CHARSET=utf8mb4;
-- Table structure for hosts
CREATE TABLE `hosts` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`hostname` varchar(255) NOT NULL,
`host_ip` varchar(255) NOT NULL,
`comment` varchar(255) DEFAULT NULL,
) ENGINE=InnoDB AUTO_INCREMENT=19 DEFAULT CHARSET=utf8mb4;
-- Table structure for hosts-test
CREATE TABLE `hosts-test` (
#可以檢視以往的曆史釋出記錄：
4.Pipelin流水線代碼
#Groovy代碼如下：
/**
* Description:Master環境JavaSpringBoot的Pipeline腳本
* Author: ledi
* Date:2020-03-01
*/
pipeline {
agent any
environment {
packageDir="/data/jenkins/repos/master"
backupDir="/data/release_package"
deployDir="/data/www"
ansible_Dir="/etc/ansible/roles/masterv2"
port="20881"
}
stages {
stage("推包至遠端"){
steps{
script{
if ( "${DEPLOY_TYPE}" == 'Deploy' ){
sh '''
HOSTS=`echo ${HOSTS}|sed s/[[:space:]]//g`
source /etc/profile &> /dev/null
#/bin/mv ${WORKSPACE}/${PROJECT}/target/${PROJECT}.jar ${packageDir}/${PROJECT}_${BUILD_NUMBER}.jar
old_version=`echo "${VERSION}"`
new_version=`echo "${VERSION}"|awk -F '##' '{print $1}'`
ansible-playbook -i ${ansible_Dir}/hosts ${ansible_Dir}/copy_jar.yaml -e "backupDir=${backupDir} packageDir=${packageDir} old_version=${old_version} new_version=${new_version} project=${PROJECT}" --limit ${HOSTS}
'''
}else{
echo "執行復原操作,無需推包至遠端"
}
} // end script
} // end steps
} //end stage
stage("停應用") {
steps {
script {
ansible-playbook -i ${ansible_Dir}/hosts ${ansible_Dir}/stop_app.yaml -e "project=${PROJECT} " --limit ${HOSTS}
'''
}
}
}
stage("啟應用"){
new_version=`echo "${VERSION}"|awk -F '##' '{print $1}'`
ansible-playbook -i ${ansible_Dir}/hosts ${ansible_Dir}/start_app.yaml -e "project=${PROJECT} new_version=${new_version} " --limit ${HOSTS}
stage("健康檢查"){
hosts_ip=`echo ${HOSTS} |sed "s/,/ /g" `
function check_health(){
for (( i=1;i<="$#";i++ )); do
ansible ${!i} -u www -i ${ansible_Dir}/hosts -m shell -a "tail -1000 /data/logs/${PROJECT}/nohup.out"
for (( j=1;j<=60;j++ ))
do
echo "ip is ${!i}"
if [[ `(echo "status -l ";sleep 1;exit)|telnet ${!i} ${port} |grep "server"| grep -o "OK"` == "OK" ]]; then
echo " ^_^^_^ IP ${!i} ${port} 端口檢查成功 ^_^^_^"
sh /data/jenkins/scripts/deploy_success_update.sh ${PROJECT} ${!i} ${DEPLOY_TYPE} ${new_version}
break 1;
else
echo "==== IP ${!i} ${port} 端口異常,繼續探測 ==== "
sleep 3
fi
done
if [[ `(echo "status -l ";sleep 1;exit)|telnet ${!i} ${port} |grep "server"| grep -o "OK"` != "OK" ]] ; then
echo "==== IP ${!i} ${port} 端口異常,啟動失敗,請檢查應用 === "
exit 1
fi
done
}
check_health ${hosts_ip}
'''
}// end script
} // end stage heal
} //end stages
說明：
這裡也可以将此Groovy代碼和應用放在一起，放入GitLab中，這樣更容易維護及管理。
5.Ansible 公用部分
cat copy_jar.yaml
---
- hosts: all
gather_facts: False
remote_user: root
vars:
backupDir: {backupDir}
packageDir: {packageDir}
project: {PROJECT}
old_version: {old_version}
new_version: {new_version}
tasks:
- name: Copy Jar to Remoute Machine
copy: src={{ packageDir }}/{{project }}/{{ old_version }}.jar dest={{ backupDir }}/{{ new_version }}.jar owner=www group=www mode=0664 force=yes
cat stop_app.yaml
- name: Stop Java App
shell: sh /home/www/scripts/stop_springboot.sh {{ project }} owner=www group=www
cat start_app.yaml
remote_user: www
- name: Start Java App
shell: /bin/bash /home/www/scripts/start_springboot.sh {{ project }} {{ new_version }} owner=www group=www
最後，開發們可以自定義釋出主機，假設一個微服務有20台機器，可以先選擇5台進行釋出，通過後，再選擇5台或以上，然後慢慢類似滾動釋出。
存在不足的點：
若一個微服務有20台機器，若采用每5台釋出，則開發需要手工進行4次操作，改進的地方：若第一次5台成功後，則自動執行後面的操作
在使用Ansible推包到阿裡雲ECS時，很慢；這個是我們的網絡機房問題，因為打包機器在公司内部，和阿裡雲ECS通過虛拟隧道過去的，是以比較慢，改進的地方：
一是将打包機器也遷移到雲端；二是通過專線将公司網絡與阿裡雲互通，形成實體線路；不過上面兩個都是需要￥的哈,運維同學不單單考慮系統可用性、穩定性，也要為公司節約一定成本哈。