Docker 部署 Prometheus

說明：

監控端安裝：

Prometheus Server(普羅米修斯監控主伺服器 )

Node Exporter (收集Host硬體和作業系統資訊)

cAdvisor (負責收集Host上運作的容器資訊)

Grafana (展示普羅米修斯監控界面）

被監控安裝：

Node Exporter (收集Host硬體和作業系統資訊)

cAdvisor (負責收集Host上運作的容器資訊)

1.安裝Node Exporter

所有伺服器安裝
Node Exporter 收集系統資訊，用于監控CPU、記憶體、磁盤使用率、磁盤讀寫等系統資訊
–net=host，這樣 Prometheus Server 可以直接與 Node Exporter 通信

docker run -d -p 9100:9100 \
-v "/proc:/host/proc" \
-v "/sys:/host/sys" \
-v "/:/rootfs" \
-v "/etc/localtime:/etc/localtime" \
--net=host \
prom/node-exporter \
--path.procfs /host/proc \
--path.sysfs /host/sys \
--collector.filesystem.ignored-mount-points "^/(sys|proc|dev|host|etc)($|/)"

[[email protected] ~]# docker ps|grep exporter
ee30add8d207   prom/node-exporter                                  "/bin/node_exporter …"   About a minute ago   Up About a minute                 condescending_shirley

2.安裝cAdvisor

所有伺服器安裝
cAdvisor 收集docker資訊，用于展示docker的cpu、記憶體、上傳下載下傳等資訊
–net=host，這樣 Prometheus Server 可以直接與 cAdvisor 通信

docker run -d \
-v "/etc/localtime:/etc/localtime" \
--volume=/:/rootfs:ro \
--volume=/var/run:/var/run:rw \
--volume=/sys:/sys:ro \
--volume=/var/lib/docker/:/var/lib/docker:ro \
--volume=/dev/disk/:/dev/disk:ro \
--publish=18104:8080 \
--detach=true \
--name=cadvisor \
--privileged=true \
google/cadvisor:latest

[[email protected] ~]# docker ps|grep cadvisor
cf6af6118055        google/cadvisor:latest                            "/usr/bin/cadvisor -…"   38 seconds ago       Up 37 seconds       0.0.0.0:18104->8080/tcp   cadvisor

可以進入容器檢視：
[[email protected] ~]# sudo docker exec -it 容器id /bin/sh

3.安裝 Prometheus Server

監控端安裝

1）編輯配置檔案

首先在本地建立 prometheus.yml 這是普羅米修斯的配置檔案
将下方内容寫入到檔案中
将監聽的位址改為自己本機位址

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    #監聽的位址
    - targets: ['localhost:9090','172.23.0.241:8088','172.23.0.241:9090']

2）啟動容器

1> prometheus.yml配置檔案

prometheus.yml内需配置外網ip，内網ip除了本機，在grafana識别不到！

# my global confi
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    #監聽的位址（此處為伺服器内網ip）
    - targets: ['10.27.158.33:9090','10.27.158.33:9100','10.27.158.33:18104']
    - targets: ['10.29.46.54:9100','10.29.46.54:18104']
    - targets: ['10.27.163.172:9100','10.27.163.172:18104']

#  - job_name: 'GitLab'
#    metrics_path: '/-/metrics'
#    static_configs:
#    - targets: ['172.23.0.241:10101']

  - job_name: 'jenkins'
    metrics_path: '/prometheus/'
    scheme: http
    bearer_token: bearer_token
    static_configs:
    - targets: ['172.23.0.242:8080']

  - job_name: "Nginx"
    metrics_path: '/status/format/prometheus'
    static_configs:
    - targets: ['172.23.0.242:8088']

2>啟動指令

–net=host，這樣 Prometheus Server 可以直接與 Exporter 和 Grafana 通

docker run -d -p 9090:9090 \
-v /root/Prometheus/prometheus.yml:/etc/prometheus/prometheus.yml \
-v "/etc/localtime:/etc/localtime" \
--name prometheus \
--net=host \
prom/prometheus:latest

# 當Prometheus容器啟動成功後通路
# PS：伺服器需開啟eth0的外網端口，才可用浏覽器通路 9090 0.0.0.0
106.15.0.11:9090

4.建立運作Grafana

監控伺服器安裝
用于圖像化顯示

docker run -d -i -p 3000:3000 \
-v "/etc/localtime:/etc/localtime" \
-e "GF_SERVER_ROOT_URL=http://grafana.server.name" \
-e "GF_SECURITY_ADMIN_PASSWORD=admin8888" \
--net=host \
grafana/grafana

# PS：伺服器需開啟eth0的外網端口，才可用浏覽器通路：3000 0.0.0.0
Grafana啟動後，在浏覽器中打開 172.23.0.241:3000 登入界面，登入：
	使用者名：admin
	密碼：admin8888

1）添加普羅米修斯伺服器

Docker 部署 PrometheusDocker 部署 Prometheus配置Prometheus監控NginxDocker 搭建 Consul叢集（未完）

然後為添加好的資料源做圖形顯示

Docker 部署 PrometheusDocker 部署 Prometheus配置Prometheus監控NginxDocker 搭建 Consul叢集（未完）

5.添加監控模闆

自己手工建立dashboard有點困難，可以借助開元的力量通路 [監控模闆位址]https://grafana.com/grafana/dashboards将會看到很多用于監控 Docker 的 Dashboard。監控模闆位址(多種監控模闆根據自己需求下載下傳不同的模闆)

監控模闆位址
有些dashboard可以下載下傳後直接導入，而有些需要修改後再導入，需要看dashboard的overview
最後效果

Docker 部署 PrometheusDocker 部署 Prometheus配置Prometheus監控NginxDocker 搭建 Consul叢集（未完）

這時候可以自己選擇編譯對應的模闆，在prometheus上取值

傳到grafana上。就可以了。挺好用的！

6.鍵值查詢

通過名額 io_namespace_http_requests_total 我們可以：

查詢應用的請求總量
	sum(io_namespace_http_requests_total)
查詢每秒Http請求量
	sum(rate(io_wise2c_gateway_requests_total[5m]))
查詢目前應用請求量Top N的URI
	topk(10, sum(io_namespace_http_requests_total) by (path))

配置Prometheus監控Nginx

1、需給Nginx安裝兩個子產品，才可用Prometheus來監控：nginx-module-vts、geoip

2、思路：原來無論是編譯、還是yum裝的nginx，都需要下載下傳同版本的tar包，基于原來安裝選項的基礎上，增加以上兩個子產品選項，進行編譯安裝，來替換原來的nginx，最終将原nginx目錄的配置檔案如nginx.conf檔案、conf.d目錄再移動到編譯安裝後的nignx目錄内，最後啟動nginx即可。

這裡官方源安裝：

1）配置官方源

[[email protected] ~]# vim /etc/yum.repos.d/nginx.repo
[nginx-stable]
name=nginx stable repo
baseurl=http://nginx.org/packages/centos/7/$basearch/
gpgcheck=1
enabled=1
gpgkey=https://nginx.org/keys/nginx_signing.key
module_hotfixes=true

2）安裝依賴

yum install -y gcc gcc-c++ autoconf pcre pcre-devel make automake wget httpd-tools vim tree

3）安裝nginx

4）配置nginx

[[email protected] ~]# vim /etc/nginx/nginx.conf
user www;

5）啟動服務

1.方法一：直接啟動如果有報錯==》重大錯誤，80端口有占用==》檢視占用端口的服務HTTPD，停掉，在重新開機nginx
[[email protected] ~]# systemctl start nginx
2.方法二：
[[email protected] ~]# nginx

1.檢視目前Nginx安裝選項

[[email protected] nginx-1.12.2]# nginx -V
[[email protected] nginx-1.12.2]# ./configure --prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/lib/nginx/tmp/client_body --http-proxy-temp-path=/var/lib/nginx/tmp/proxy --http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi --http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi --http-scgi-temp-path=/var/lib/nginx/tmp/scgi --pid-path=/run/nginx.pid --lock-path=/run/lock/subsys/nginx --user=nginx --group=nginx --with-compat --with-debug --with-file-aio --with-google_perftools_module --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_degradation_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_mp4_module --with-http_perl_module=dynamic --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-http_xslt_module=dynamic --with-mail=dynamic --with-mail_ssl_module --with-pcre --with-pcre-jit --with-stream=dynamic --with-stream_ssl_module --with-stream_ssl_preread_module --with-threads --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic' --with-ld-opt='-Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Wl,-E'

2.準備子產品

# 下載下傳、解壓新包
[[email protected] packages]# wget http://nginx.org/download/nginx-1.16.1.tar.gz
[[email protected] packages]# tar xf nginx-1.16.1.tar.gz

#克隆下載下傳 nginx-module-vts 子產品
[[email protected] packages]# git clone https://github.com/vozlt/nginx-module-vts

# .安裝GeoIP子產品
[[email protected] packages]# yum -y install epel-release geoip-devel

3.停止Nginx服務

# 停止nginx服務
[[email protected] packages]# nginx -s stop

# 備份原nginx啟動檔案
[[email protected] packages]# which nginx
/usr/sbin/nginx
[[email protected] packages]# mv /usr/sbin/nginx /usr/sbin/nginx.bak

# 備份原nignx目錄
[[email protected]8s-n1 packages]# mv /etc/nginx nginx-1.12.2.bak

4.編譯安裝

1> 安裝所需依賴

編譯安裝時可能會出現 `make: *** 沒有規則可以建立“default”需要的目标“build”。 停止`的報錯，是因為缺少依賴導緻

# 管他三七21，裝一波兒在進行編譯，否則裝完依賴還得重新./configure ~
yum install -y gcc gcc++ bash-completion vim lrzsz wget expect net-tools nc nmap tree dos2unix htop iftop iotop unzip telnet sl psmisc nethogs glances bc pcre-devel zlib zlib-devel openssl openssl-devel libxml2 libxml2-dev libxslt-devel gd gd-devel perl-devel perl-ExtUtils-Embed GeoIP GeoIP-devel GeoIP-data pcre-devel

2> 編譯安裝

進入剛剛解壓的nginx目錄，編譯安裝
基于原來安裝參數，尾部追加連個參數
- –add-module=/root/packages/nginx-module-vts
- –with-http_geoip_module

# 編譯安裝
# -j 多核編譯（配置低的不建議使用此參數，會卡住~）
[[email protected] nginx-1.12.2]# make -j && make install

5.配置Nginx

[[email protected] packages]# cp -r nginx-1.12.2.bak/conf.d/ /etc/nginx/
[[email protected] packages]# cp -r nginx-1.12.2.bak/nginx.conf /etc/nginx/
[[email protected] packages]# rm -f /etc/nginx/conf.d/default.conf

配置Nginx配置檔案
- http層
- server層

···
http {	
	···
    include /etc/nginx/conf.d/*.conf;

	##################### 1.http層：添加三行配置 ##################### 
    vhost_traffic_status_zone;
    vhost_traffic_status_filter_by_host on;
    geoip_country /usr/share/GeoIP/GeoIP.dat;

	##################### 2.server層：指定server層端口号，建議8088端口，不沖突直接複制粘貼即可#####################
    server {
        listen       8088;
        server_name  localhost;
        # 以下vhost配置寫在此location内
        location /status {
        vhost_traffic_status on;	# 流量狀态，預設即為on，可不寫此行
        vhost_traffic_status_display;
        vhost_traffic_status_display_format html;
        vhost_traffic_status_filter_by_set_key $uri uri::$server_name;     #每個uri通路量
        vhost_traffic_status_filter_by_set_key $geoip_country_code country::$server_name;     #不同國家/區域請求量
        vhost_traffic_status_filter_by_set_key $status $server_name;     #http code統計
        vhost_traffic_status_filter_by_set_key $upstream_addr upstream::backend;     #後端>轉發統計
        vhost_traffic_status_filter_by_set_key $remote_port client::ports::$server_name;     #請求端口統計
        vhost_traffic_status_filter_by_set_key $remote_addr client::addr::$server_name;     #請求IP統計

        location ~ ^/storage/(.+)/.*$ {
            set $volume $1;
            vhost_traffic_status_filter_by_set_key $volume storage::$server_name;     #請求路徑統計
        }
        }
    }
   	##################### server層：可建立一個server，或在原有的不打緊的配置上修改也可以#####################
}

6.啟動Nginx

[[email protected] packages]# nginx
[[email protected] packages]# netstat -lntp|grep nginx
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      62214/nginx: master 
tcp        0      0 0.0.0.0:8088            0.0.0.0:*               LISTEN      62214/nginx: master 

浏覽器通路：
	172.23.0.243:80			# nginx 預設官方頁面
	172.23.0.243:8088/status # nignx 監控項頁面

Docker 部署 PrometheusDocker 部署 Prometheus配置Prometheus監控NginxDocker 搭建 Consul叢集（未完）

7.使用Prometheus監控Nginx

prometheus服務端配置prometheus.yml，并重新開機prometheus容器
metrics_path：定義接口字尾類型，預設為/metrics
即我們輸入ip+端口後，浏覽器會自動追加/metrics字尾

[[email protected] ~]# vim prometheus.yml
···
scrape_configs:
  - job_name: "Nginx"
    metrics_path: '/status/format/prometheus'
    static_configs:
    - targets: ['172.23.0.243:8088']
···
[[email protected] ~]# docker restart prometheus

# 此時進入prometheus管理頁面，則能查詢nginx的監控項

8.各個監控項的含義

Nginx-module-vts提供了多種監控項，了解監控項含義，有助于幫助自己生成需要的圖表

# HELP nginx_vts_info Nginx info
# TYPE nginx_vts_info gauge
nginx_vts_info{hostname="hbhly_21_205",version="1.16.1"} 1
# HELP nginx_vts_start_time_seconds Nginx start time
# TYPE nginx_vts_start_time_seconds gauge
nginx_vts_start_time_seconds 1584268136.439
# HELP nginx_vts_main_connections Nginx connections
# TYPE nginx_vts_main_connections gauge

# 區分狀态的nginx連接配接數
nginx_vts_main_connections{status="accepted"} 9271
nginx_vts_main_connections{status="active"} 7
nginx_vts_main_connections{status="handled"} 9271
nginx_vts_main_connections{status="reading"} 0
nginx_vts_main_connections{status="requests"} 438850
nginx_vts_main_connections{status="waiting"} 6
nginx_vts_main_connections{status="writing"} 1
# HELP nginx_vts_main_shm_usage_bytes Shared memory [ngx_http_vhost_traffic_status] info
# TYPE nginx_vts_main_shm_usage_bytes gauge

# 記憶體使用量
nginx_vts_main_shm_usage_bytes{shared="max_size"} 1048575
nginx_vts_main_shm_usage_bytes{shared="used_size"} 24689
nginx_vts_main_shm_usage_bytes{shared="used_node"} 7
# HELP nginx_vts_server_bytes_total The request/response bytes
# TYPE nginx_vts_server_bytes_total counter
# HELP nginx_vts_server_requests_total The requests counter
# TYPE nginx_vts_server_requests_total counter
# HELP nginx_vts_server_request_seconds_total The request processing time in seconds
# TYPE nginx_vts_server_request_seconds_total counter
# HELP nginx_vts_server_request_seconds The average of request processing times in seconds
# TYPE nginx_vts_server_request_seconds gauge
# HELP nginx_vts_server_request_duration_seconds The histogram of request processing time
# TYPE nginx_vts_server_request_duration_seconds histogram
# HELP nginx_vts_server_cache_total The requests cache counter
# TYPE nginx_vts_server_cache_total counter

# 分Host的進出流量
nginx_vts_server_bytes_total{host="10.160.21.205",direction="in"} 22921464
nginx_vts_server_bytes_total{host="10.160.21.205",direction="out"} 1098196005

# 分狀态碼的請求數量統計 1** 2** 3** 4** 5**
nginx_vts_server_requests_total{host="10.160.21.205",code="1xx"} 0
nginx_vts_server_requests_total{host="10.160.21.205",code="2xx"} 86809
nginx_vts_server_requests_total{host="10.160.21.205",code="3xx"} 0
nginx_vts_server_requests_total{host="10.160.21.205",code="4xx"} 2
nginx_vts_server_requests_total{host="10.160.21.205",code="5xx"} 0
nginx_vts_server_requests_total{host="10.160.21.205",code="total"} 86811

# 響應時間
nginx_vts_server_request_seconds_total{host="10.160.21.205"} 0.000
nginx_vts_server_request_seconds{host="10.160.21.205"} 0.000

# 分狀态的緩存的統計
nginx_vts_server_cache_total{host="10.160.21.205",status="miss"} 0
nginx_vts_server_cache_total{host="10.160.21.205",status="bypass"} 0
nginx_vts_server_cache_total{host="10.160.21.205",status="expired"} 0
nginx_vts_server_cache_total{host="10.160.21.205",status="stale"} 0
nginx_vts_server_cache_total{host="10.160.21.205",status="updating"} 0
nginx_vts_server_cache_total{host="10.160.21.205",status="revalidated"} 0
nginx_vts_server_cache_total{host="10.160.21.205",status="hit"} 0
nginx_vts_server_cache_total{host="10.160.21.205",status="scarce"} 0
nginx_vts_server_bytes_total{host="devapi.feedback.test",direction="in"} 3044526
nginx_vts_server_bytes_total{host="devapi.feedback.test",direction="out"} 41257028

# 分狀态的連接配接數的統計
nginx_vts_server_requests_total{host="devapi.feedback.test",code="1xx"} 0
nginx_vts_server_requests_total{host="devapi.feedback.test",code="2xx"} 3983
nginx_vts_server_requests_total{host="devapi.feedback.test",code="3xx"} 0
nginx_vts_server_requests_total{host="devapi.feedback.test",code="4xx"} 24
nginx_vts_server_requests_total{host="devapi.feedback.test",code="5xx"} 11
nginx_vts_server_requests_total{host="devapi.feedback.test",code="total"} 4018
nginx_vts_server_request_seconds_total{host="devapi.feedback.test"} 327.173
nginx_vts_server_request_seconds{host="devapi.feedback.test"} 0.000

# nginx緩存電腦，精确到狀态和type
nginx_vts_server_cache_total{host="devapi.feedback.test",status="miss"} 0
nginx_vts_server_cache_total{host="devapi.feedback.test",status="bypass"} 0
nginx_vts_server_cache_total{host="devapi.feedback.test",status="expired"} 0
nginx_vts_server_cache_total{host="devapi.feedback.test",status="stale"} 0
nginx_vts_server_cache_total{host="devapi.feedback.test",status="updating"} 0
nginx_vts_server_cache_total{host="devapi.feedback.test",status="revalidated"} 0
nginx_vts_server_cache_total{host="devapi.feedback.test",status="hit"} 0
nginx_vts_server_cache_total{host="devapi.feedback.test",status="scarce"} 0
nginx_vts_server_bytes_total{host="testapi.feedback.test",direction="in"} 55553573
nginx_vts_server_bytes_total{host="testapi.feedback.test",direction="out"} 9667561188
nginx_vts_server_requests_total{host="testapi.feedback.test",code="1xx"} 0
nginx_vts_server_requests_total{host="testapi.feedback.test",code="2xx"} 347949
nginx_vts_server_requests_total{host="testapi.feedback.test",code="3xx"} 31
nginx_vts_server_requests_total{host="testapi.feedback.test",code="4xx"} 7
nginx_vts_server_requests_total{host="testapi.feedback.test",code="5xx"} 33
nginx_vts_server_requests_total{host="testapi.feedback.test",code="total"} 348020
nginx_vts_server_request_seconds_total{host="testapi.feedback.test"} 2185.177
nginx_vts_server_request_seconds{host="testapi.feedback.test"} 0.001
nginx_vts_server_cache_total{host="testapi.feedback.test",status="miss"} 0
nginx_vts_server_cache_total{host="testapi.feedback.test",status="bypass"} 0
nginx_vts_server_cache_total{host="testapi.feedback.test",status="expired"} 0
nginx_vts_server_cache_total{host="testapi.feedback.test",status="stale"} 0
nginx_vts_server_cache_total{host="testapi.feedback.test",status="updating"} 0
nginx_vts_server_cache_total{host="testapi.feedback.test",status="revalidated"} 0
nginx_vts_server_cache_total{host="testapi.feedback.test",status="hit"} 0
nginx_vts_server_cache_total{host="testapi.feedback.test",status="scarce"} 0
nginx_vts_server_bytes_total{host="*",direction="in"} 81519563
nginx_vts_server_bytes_total{host="*",direction="out"} 10807014221

# 分host請求數量統計
nginx_vts_server_requests_total{host="*",code="1xx"} 0
nginx_vts_server_requests_total{host="*",code="2xx"} 438741
nginx_vts_server_requests_total{host="*",code="3xx"} 31
nginx_vts_server_requests_total{host="*",code="4xx"} 33
nginx_vts_server_requests_total{host="*",code="5xx"} 44
nginx_vts_server_requests_total{host="*",code="total"} 438849
nginx_vts_server_request_seconds_total{host="*"} 2512.350
nginx_vts_server_request_seconds{host="*"} 0.007

# 分host緩存統計
nginx_vts_server_cache_total{host="*",status="miss"} 0
nginx_vts_server_cache_total{host="*",status="bypass"} 0
nginx_vts_server_cache_total{host="*",status="expired"} 0
nginx_vts_server_cache_total{host="*",status="stale"} 0
nginx_vts_server_cache_total{host="*",status="updating"} 0
nginx_vts_server_cache_total{host="*",status="revalidated"} 0
nginx_vts_server_cache_total{host="*",status="hit"} 0
nginx_vts_server_cache_total{host="*",status="scarce"} 0
# HELP nginx_vts_upstream_bytes_total The request/response bytes
# TYPE nginx_vts_upstream_bytes_total counter
# HELP nginx_vts_upstream_requests_total The upstream requests counter
# TYPE nginx_vts_upstream_requests_total counter
# HELP nginx_vts_upstream_request_seconds_total The request Processing time including upstream in seconds
# TYPE nginx_vts_upstream_request_seconds_total counter
# HELP nginx_vts_upstream_request_seconds The average of request processing times including upstream in seconds
# TYPE nginx_vts_upstream_request_seconds gauge
# HELP nginx_vts_upstream_response_seconds_total The only upstream response processing time in seconds
# TYPE nginx_vts_upstream_response_seconds_total counter
# HELP nginx_vts_upstream_response_seconds The average of only upstream response processing times in seconds
# TYPE nginx_vts_upstream_response_seconds gauge
# HELP nginx_vts_upstream_request_duration_seconds The histogram of request processing time including upstream
# TYPE nginx_vts_upstream_request_duration_seconds histogram
# HELP nginx_vts_upstream_response_duration_seconds The histogram of only upstream response processing time
# TYPE nginx_vts_upstream_response_duration_seconds histogram

# 分upstream流量統計
nginx_vts_upstream_bytes_total{upstream="::nogroups",backend="10.144.227.162:80",direction="in"} 12296
nginx_vts_upstream_bytes_total{upstream="::nogroups",backend="10.144.227.162:80",direction="out"} 13582924
nginx_vts_upstream_requests_total{upstream="::nogroups",backend="10.144.227.162:80",code="1xx"} 0
nginx_vts_upstream_requests_total{upstream="::nogroups",backend="10.144.227.162:80",code="2xx"} 25
nginx_vts_upstream_requests_total{upstream="::nogroups",backend="10.144.227.162:80",code="3xx"} 0
nginx_vts_upstream_requests_total{upstream="::nogroups",backend="10.144.227.162:80",code="4xx"} 0
nginx_vts_upstream_requests_total{upstream="::nogroups",backend="10.144.227.162:80",code="5xx"} 0
nginx_vts_upstream_requests_total{upstream="::nogroups",backend="10.144.227.162:80",code="total"} 25
nginx_vts_upstream_request_seconds_total{upstream="::nogroups",backend="10.144.227.162:80"} 1.483
nginx_vts_upstream_request_seconds{upstream="::nogroups",backend="10.144.227.162:80"} 0.000
nginx_vts_upstream_response_seconds_total{upstream="::nogroups",backend="10.144.227.162:80"} 1.484
nginx_vts_upstream_response_seconds{upstream="::nogroups",backend="10.144.227.162:80"} 0.000
nginx_vts_upstream_bytes_total{upstream="::nogroups",backend="10.152.218.149:80",direction="in"} 12471
nginx_vts_upstream_bytes_total{upstream="::nogroups",backend="10.152.218.149:80",direction="out"} 11790508
nginx_vts_upstream_requests_total{upstream="::nogroups",backend="10.152.218.149:80",code="1xx"} 0
nginx_vts_upstream_requests_total{upstream="::nogroups",backend="10.152.218.149:80",code="2xx"} 24
nginx_vts_upstream_requests_total{upstream="::nogroups",backend="10.152.218.149:80",code="3xx"} 0
nginx_vts_upstream_requests_total{upstream="::nogroups",backend="10.152.218.149:80",code="4xx"} 0
nginx_vts_upstream_requests_total{upstream="::nogroups",backend="10.152.218.149:80",code="5xx"} 0
nginx_vts_upstream_requests_total{upstream="::nogroups",backend="10.152.218.149:80",code="total"} 24
nginx_vts_upstream_request_seconds_total{upstream="::nogroups",backend="10.152.218.149:80"} 1.169
nginx_vts_upstream_request_seconds{upstream="::nogroups",backend="10.152.218.149:80"} 0.000
nginx_vts_upstream_response_seconds_total{upstream="::nogroups",backend="10.152.218.149:80"} 1.168
nginx_vts_upstream_response_seconds{upstream="::nogroups",backend="10.152.218.149:80"} 0.000
nginx_vts_upstream_bytes_total{upstream="::nogroups",backend="10.160.21.205:8081",direction="in"} 3036924
nginx_vts_upstream_bytes_total{upstream="::nogroups",backend="10.160.21.205:8081",direction="out"} 33355357
nginx_vts_upstream_requests_total{upstream="::nogroups",backend="10.160.21.205:8081",code="1xx"} 0
nginx_vts_upstream_requests_total{upstream="::nogroups",backend="10.160.21.205:8081",code="2xx"} 3971
nginx_vts_upstream_requests_total{upstream="::nogroups",backend="10.160.21.205:8081",code="3xx"} 0
nginx_vts_upstream_requests_total{upstream="::nogroups",backend="10.160.21.205:8081",code="4xx"} 24
nginx_vts_upstream_requests_total{upstream="::nogroups",backend="10.160.21.205:8081",code="5xx"} 11
nginx_vts_upstream_requests_total{upstream="::nogroups",backend="10.160.21.205:8081",code="total"} 4006
nginx_vts_upstream_request_seconds_total{upstream="::nogroups",backend="10.160.21.205:8081"} 326.427
nginx_vts_upstream_request_seconds{upstream="::nogroups",backend="10.160.21.205:8081"} 0.000
nginx_vts_upstream_response_seconds_total{upstream="::nogroups",backend="10.160.21.205:8081"} 300.722
nginx_vts_upstream_response_seconds{upstream="::nogroups",backend="10.160.21.205:8081"} 0.000
nginx_vts_upstream_bytes_total{upstream="::nogroups",backend="10.160.21.205:8082",direction="in"} 55536408
nginx_vts_upstream_bytes_total{upstream="::nogroups",backend="10.160.21.205:8082",direction="out"} 9650089427
nginx_vts_upstream_requests_total{upstream="::nogroups",backend="10.160.21.205:8082",code="1xx"} 0
nginx_vts_upstream_requests_total{upstream="::nogroups",backend="10.160.21.205:8082",code="2xx"} 347912
nginx_vts_upstream_requests_total{upstream="::nogroups",backend="10.160.21.205:8082",code="3xx"} 31
nginx_vts_upstream_requests_total{upstream="::nogroups",backend="10.160.21.205:8082",code="4xx"} 7
nginx_vts_upstream_requests_total{upstream="::nogroups",backend="10.160.21.205:8082",code="5xx"} 33
nginx_vts_upstream_requests_total{upstream="::nogroups",backend="10.160.21.205:8082",code="total"} 347983
nginx_vts_upstream_request_seconds_total{upstream="::nogroups",backend="10.160.21.205:8082"} 2183.271
nginx_vts_upstream_request_seconds{upstream="::nogroups",backend="10.160.21.205:8082"} 0.001
nginx_vts_upstream_response_seconds_total{upstream="::nogroups",backend="10.160.21.205:8082"} 2180.893
nginx_vts_upstream_response_seconds{upstream="::nogroups",backend="10.160.21.205:8082"} 0.001

9.Prometheus UI中Target表達式查詢

1）CAdvisor中擷取的典型監控名額

名額名稱	類型	含義
container_cpu_load_average_10s	gauge	過去10秒内容器CPU的平均負載
container_cpu_usage_seconds_total	counter	容器在每個CPU核心上的累積占用時間 (機關：秒)
container_cpu_system_seconds_total	counter	System CPU累積占用時間（機關：秒）
container_cpu_user_seconds_total	counter	User CPU累積占用時間（機關：秒）
container_fs_usge_bytes	gauge	容器中檔案系統的使用量(機關：位元組)
container_network_receive_bytes_total	counter	容器網絡累計接受資料總量（機關: 位元組）
container_network_transmit_bytes_total	counter	容器網絡累計傳輸資料總量（機關: 位元組）

2）容器相關

# 容器的CPU使用率
sum(irate(container_cpu_usage_seconds_total{image!=""}[1m])) without (cpu)

# 容器記憶體使用量（機關: 位元組）
container_memory_usage_bytes{image!=""}

# 容器網絡接收量速率（機關: 位元組/秒）
sum(rate(container_network_receive_bytes_total{image!=""}[1m])) without (interface)

# 容器網絡傳輸量速率
sum(rate(container_network_transmit_bytes_total{image!=""}[1m])) without (interface)

# 容器檔案系統讀取速率
sum(rate(container_fs_reads_bytes_total{image!=""}[1m])) without (device)

# 容器檔案系統寫入速率（機關: 位元組/秒）
sum(rate(container_fs_writes_bytes_total{image!=""}[1m])) without (device)

3）http相關

# HTTP請求總數
prometheus_http_requests_total

# HTTP請求持續時間秒桶
prometheus_http_request_duration_seconds_bucket

# HTTP請求持續時間秒數計數
prometheus_http_request_duration_seconds_count

# HTTP請求持續時間秒數之和
prometheus_http_request_duration_seconds_sum

# HTTP響應大小位元組
prometheus_http_response_size_bytes_bucket

# HTTP響應大小位元組計數計數
prometheus_http_response_size_bytes_count

# HTTP響應大小位元組的總和
prometheus_http_response_size_bytes_sum

4）Nginx相關

# Nginxvts過濾位元組總數
nginx_vts_filter_bytes_total

# Nginx VTS過濾器緩存總數
nginx_vts_filter_cache_total

# Nginx VTS過濾請求秒數
nginx_vts_filter_request_seconds

# Nginx VTS過濾器請求總秒數
nginx_vts_filter_request_seconds_total

# Nginx VTS過濾器請求總數
nginx_vts_filter_requests_total

# nginx資訊
nginx_vts_info

# Nginx VTS主連接配接
nginx_vts_main_connections

# Nginx VTS主SHM使用位元組
nginx_vts_main_shm_usage_bytes

# Nginx VTS伺服器位元組總數
nginx_vts_server_bytes_total

# Nginx VTS伺服器緩存總數
nginx_vts_server_cache_total

# Nginx_vts伺服器請求秒
nginx_vts_server_request_seconds

# Nginx_vts伺服器請求總秒數
nginx_vts_server_request_seconds_total

# Nginx_vts服務總請求數
nginx_vts_server_requests_total

# Nginx VTS開始時間秒數
nginx_vts_start_time_seconds

10.安裝blackbox_exporter

blackbox收集服務狀态資訊，如判斷服務http請求是否傳回200繼而報警
blackbox_exporter是Prometheus 官方提供的 exporter 之一，可以提供 http、dns、tcp、icmp 的監控資料采集

功能：
HTTP 測試
    定義 Request Header 資訊
    判斷 Http status / Http Respones Header / Http Body 内容
    
TCP 測試
    業務元件端口狀态監聽
    應用層協定定義與監聽
    
ICMP 測試
	主機探活機制
	
POST 測試
	接口聯通性
	
SSL 證書過期時間

# 下載下傳、解壓
[[email protected] Prometheus]# wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.14.0/blackbox_exporter-0.14.0.linux-amd64.tar.gz
[[email protected] Prometheus]# tar -xvf blackbox_exporter-0.14.0.linux-amd64.tar.gz
[[email protected] Prometheus]# mv blackbox_exporter-0.14.0.linux-amd64 /usr/local/blackbox_exporter

# 檢視安裝是否成功
[[email protected] Prometheus]# /usr/local/blackbox_exporter/blackbox_exporter --version
blackbox_exporter, version 0.14.0 (branch: HEAD, revision: bba7ef76193948a333a5868a1ab38b864f7d968a)
  build user:       [email protected]
  build date:       20190315-13:32:31
  go version:       go1.11.5

# 加入systemd管理
[[email protected] Prometheus]# cat /usr//lib/systemd/system/blackbox_exporter.service
[Unit]
Description=blackbox_exporter
 
[Service]
User=root
Type=simple
ExecStart=/usr/local/blackbox_exporter/blackbox_exporter --config.file=/usr/local/blackbox_exporter/blackbox.yml
Restart=on-failure
[[email protected] Prometheus]# 

# 啟動
[[email protected] Prometheus]# systemctl daemon-reload
[[email protected] Prometheus]# systemctl enable --now blackbox_exporter

11.Docker部署nginx-module-vts子產品

由于yum安裝的nginx，預設是沒有nginx-module-vts子產品的，需要下載下傳對應的nginx源碼，進行重新編譯才行。

Docker 搭建 Consul叢集（未完）

1.啟動第一個consul服務：consul1

docker run --name consul1 -d -p 8500:8500 -p 8300:8300 -p 8301:8301 -p 8302:8302 -p 8600:8600 --restart=always consul:latest agent -server -bootstrap-expect 2 -ui -bind=0.0.0.0 -client=0.0.0.0

# 擷取 consul server1 的 ip 位址
docker inspect --format '{{ .NetworkSettings.IPAddress }}' consul1
172.17.0.2

# PS:
    8500 http 端口，用于 http 接口和 web ui
    8300 server rpc 端口，同一資料中心 consul server 之間通過該端口通信
    8301 serf lan 端口，同一資料中心 consul client 通過該端口通信
    8302 serf wan 端口，不同資料中心 consul server 通過該端口通信
    8600 dns 端口，用于服務發現
    -bbostrap-expect 2: 叢集至少兩台伺服器，才能選舉叢集leader
    -ui：運作 web 控制台
    -bind： 監聽網口，0.0.0.0 表示所有網口，如果不指定預設為127.0.0.1，則無法和容器通信
    -client ： 限制某些網口可以通路

2.啟動第二個consul服務：consul2，并加入consul1（使用join指令）

docker run -d --name consul2 -d -p 8501:8500 consul agent -server -ui -bind=0.0.0.0 -client=0.0.0.0 -join 172.17.0.2

docker run -d -p 8501:8500 --restart=always -v /XiLife/consul/data/server3:/consul/data -v /XiLife/consul/conf/server2:/consul/config -e CONSUL_BIND_INTERFACE='eth0' --privileged=true --name=consu2 consul agent -server -ui -node=consul2 -client='0.0.0.0' -datacenter=xdp_dc -data-dir /consul/data -config-dir /consul/config -join=172.17.0.2

3.啟動第三個consul服務：consul3，并加入consul1

docker run --name consul3 -d -p 8502:8500 consul agent -server -ui -bind=0.0.0.0 -client=0.0.0.0 -join 172.17.0.2

4.檢視運作的容器（consul叢集狀态）

[[email protected] consul]# docker exec -it consul1 consul members
Node          Address          Status  Type    Build   Protocol  DC   Segment
013a4a7e74d2  172.17.0.4:8301  alive   server  1.10.0  2         dc1  <all>
3c118fa83d47  172.17.0.3:8301  alive   server  1.10.0  2         dc1  <all>
4b5123c97c2b  172.17.0.5:8301  alive   server  1.10.0  2         dc1  <all>
a7d272ad157a  172.17.0.2:8301  alive   server  1.10.0  2         dc1  <all>

5.服務注冊與剔除

接下來，我們要注冊服務到 Consul 中，可以通過其提供的 API 标準接口來添加
那麼先注冊一個測試服務，該測試資料為本機 node-exporter 服務資訊，服務位址及端口為 node-exporter 預設提供名額資料的位址，執行如下指令

# 注冊241的 node-exporter 服務資訊
curl -X PUT -d '{"id": "node-exporter","name": "node-exporter-172.23.0.241","address": "172.23.0.241","port": 9100,"tags": ["prometheus"],"checks": [{"http": "http://172.23.0.241:9100/metrics", "interval": "5s"}]}'  http://172.23.0.241:8500/v1/agent/service/register

# 注冊242的 node-exporter 服務資訊
将上面所有IP位址改為242的即可，端口不變

如果要登出掉某個服務，可以通過如下 API 指令操作，例如登出上邊添加的 node-exporter 服務

curl -X PUT http://172.23.0.241:8500/v1/agent/service/deregister/node-exporter

附：更新Centos6核心

rpm -Uvh https://hkg.mirror.rackspace.com/elrepo/kernel/el6/x86_64/RPMS/elrepo-release-6-12.el6.elrepo.noarch.rpm

yum源報錯解決：找不到鏡像源
cd /etc/yum.repos.d
mv CentOS-Base.repo CentOS-Base.repo.backup
wget http://mirrors.163.com/.help/CentOS6-Base-163.repo
mv CentOS6-Base-163.repo CentOS-Base.repo
yum clean all
wget -O /etc/yum.repos.d/CentOS-Base.repo http://file.kangle.odata.cc/repo/Centos-6.repo
wget -O /etc/yum.repos.d/epel.repo http://file.kangle.odata.cc/repo/epel-6.repo
yum makecache

Docker 部署 PrometheusDocker 部署 Prometheus配置Prometheus監控NginxDocker 搭建 Consul叢集（未完）

Docker 部署 Prometheus

說明：

1.安裝Node Exporter

2.安裝cAdvisor

3.安裝 Prometheus Server

1）編輯配置檔案

2）啟動容器

1> prometheus.yml配置檔案

2>啟動指令

4.建立運作Grafana

1）添加普羅米修斯伺服器

5.添加監控模闆

6.鍵值查詢

配置Prometheus監控Nginx

1.檢視目前Nginx安裝選項

2.準備子產品

3.停止Nginx服務

4.編譯安裝

1> 安裝所需依賴

2> 編譯安裝

5.配置Nginx

6.啟動Nginx

7.使用Prometheus監控Nginx

8.各個監控項的含義

9.Prometheus UI中Target表達式查詢

1）CAdvisor中擷取的典型監控名額

2）容器相關

3）http相關

4）Nginx相關

10.安裝blackbox_exporter

11.Docker部署nginx-module-vts子產品

Docker 搭建 Consul叢集（未完）

1.啟動第一個consul服務：consul1

2.啟動第二個consul服務：consul2， 并加入consul1（使用join指令）

3.啟動第三個consul服務：consul3，并加入consul1

4.檢視運作的容器（consul叢集狀态）

5.服務注冊與剔除

附：更新Centos6核心

繼續閱讀

2.啟動第二個consul服務：consul2，并加入consul1（使用join指令）