Prometheus 使用 blackbox 監控 http tcp

Prometheus http tcp 黑盒監控 blackbox_exporter

Blackbox_exporter

blackbox_exporter 是 Prometheus 拿來對 http/https、tcp、icmp、dns、進行的黑盒監控工具

什麼是黑盒監控？以下介紹是抄 zhangguanzhang 的 ​​prometheus的黑盒監控​​

正常的各種exporter都是和需要監控的機器一起安裝的,如果需要監控一些tcp端口和七層應用層的狀态呢,這個時候就需要黑盒監控了,不需要安裝在目标機器上即可從外部去監控。

安裝

二進制安裝

wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.16.0/blackbox_exporter-0.16.0.linux-amd64.tar.gz
tar zxvf blackbox_exporter-0.16.0.linux-amd64.tar.gz
cd blackbox_exporter-0.16.0.linux-amd64
./blackbox_exporter <flags>      

blackbox_exporter 有用的參數大概是如下幾個

# ./blackbox_exporter --help
usage: blackbox_exporter [<flags>]

Flags:
  -h, --help                     Show context-sensitive help (also try --help-long and --help-man).
      --config.file="blackbox.yml"  
                                 Blackbox exporter configuration file.
      --web.listen-address=":9115"  
                                 The address to listen on for HTTP requests.

      --log.level=info           Only log messages with the given severity or above. One of: [debug, info, warn, error]      

啟動

# 預設端口為9115
nohup ./blackbox_exporter --config.file="blackbox.yml" &      

docker安裝

由于 Prometheus 所有元件預設時區都使用的 UTC ，是以推薦用 Docker 去運作它

# 如果你不需要開 debug，請去掉最後的 --log.level=debug
docker run --rm -d -p 9115:9115 --name blackbox_exporter -v /usr/share/zoneinfo/Asia/Shanghai:/etc/localtime:ro -v /data/prometheus/blackbox_exporter/blackbox.yml:/config/blackbox.yml prom/blackbox-exporter:master --config.file=/config/blackbox.yml --log.level=debug      

blackbox.yml

# 官方預設的配置檔案
modules:
  http_2xx:
    prober: http
  http_post_2xx:
    prober: http
    http:
      method: POST
  tcp_connect:
    prober: tcp
  pop3s_banner:
    prober: tcp
    tcp:
      query_response:
      - expect: "^+OK"
      tls: true
      tls_config:
        insecure_skip_verify: false
  ssh_banner:
    prober: tcp
    tcp:
      query_response:
      - expect: "^SSH-2.0-"
  irc_banner:
    prober: tcp
    tcp:
      query_response:
      - send: "NICK prober"
      - send: "USER prober prober prober :prober"
      - expect: "PING :([^ ]+)"
        send: "PONG ${1}"
      - expect: "^:[^ ]+ 001"
  icmp:
    prober: icmp      

配置

prometheus.yml

HTTP 配置

scrape_configs:
  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]  # 子產品對應 blackbox.yml 
    static_configs:
      - targets:
        - http://baidu.com    # http
        - https://baidu.com   # https
        - http://xx.com:8080 # 8080端口的域名
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9115  # blackbox安裝在哪台機器      

TCP 配置

- job_name: blackbox_tcp
    metrics_path: /probe
    params:
      module: [tcp_connect]
    static_configs:
      - targets:
        - 192.168.1.2:280
        - 192.168.1.2:7013

    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 192.168.1.99:9115 # Blackbox exporter.      

重新開機 Prometheus ，打開 targets ，即可看到。

想檢視有哪些名額，則點選 Endpoint 下面的 URL 即可：

HTTP 比較值得關注的有如下幾項：

# 傳回的 http 狀态碼
probe_http_status_code 200
# https 證書過期時間，預設使用 unixtime
probe_ssl_earliest_cert_expiry 1.637745419e+09
# 如果探測成功則為 1，反之為 0
probe_success 1      

告警規則

# 以下兩條二選一
groups:
  - name: http
    rules:
    - alert: xxx域名解析失敗
      expr: probe_success{instance="https://xx.com"} == 0
      for: 1m
      labels:
        severity: "error"
      annotations:
        summary: "xxx域名解析失敗"
    - alert: xxx域名解析失敗
      expr: probe_http_status_code{instance="https://xx.com"} != 200
      for: 5m
      labels:
        severity: "error"
      annotations:
        summary: "xxx域名解析失敗"      

自定義子產品

有時可能對于某些 URL 需要帶參數，如 header、body 之類的，就需要自定義一個子產品，​​官方例子​​。

編輯 blackbox.yml

http_2xx_wxjj:
    prober: http
    timeout: 5s
    http:
      method: GET
      headers:
        Cookie: JSESSIONID=C123455dfdf
        sid: 41c912344555-24rkjkffd
        appid: 1221kj2h1k3hjk13hk
      body: '{}'      

編輯 Prometheus.yml

- job_name: 'blackbox_wxjl'
    metrics_path: /probe
    params:
      module: [http_2xx_wxjj]  # Look for a HTTP 200 response.
    static_configs:
      - targets:
        - http://192.168.201.173:808/byxxxxx/41234456661f-4357c9?head=APP_GeList&user=%E9%BB%84%E5%AE%15
   # Target to probe with http.

    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 172.18.11.154:9115  # The blackbox exporter's real hostname:port.      

開啟 debug

當你覺得自己設定沒錯，http 狀态碼卻傳回不正确，想要調試一下，就需要打開debug。

• 啟動時指定 --log.level=debug
• targets 後面帶上 &debug=true，即 ​​http://172.18.11.154:9115/probe?module=http_2xx_wxjj&target=http://192.168.201.173:808/byxxxxx/41234456661f-4357c9?head=APP_GeList&user=黃��&debug=true​​

targets 開啟 debug 會比正常連結輸出更多資訊

Module configuration:
prober: http
timeout: 5s
http:
    ip_protocol_fallback: true
    method: GET
    headers:
        Cookie: JSESSIONID=C123455dfdf
        appid: 41c912344555-24rkjkffd
        sid: 1221kj2h1k3hjk13hk
    body: '{}'
tcp:
    ip_protocol_fallback: true
icmp:
    ip_protocol_fallback: true
dns:
    ip_protocol_fallback: true      

FAQ

需要安裝多少個 blackbox_exporter ？