Prometheus Consul Blackbox | export 監控實作

雲和安全管理服務專家新钛雲服 郭鵬超原創

前言：

• blackbox_exporter

是Prometheus 官方提供的 exporter 之一,主要提供http、dns、tcp、icmp 的監控資料采集。

• Consul

主要提供，服務發現，健康檢查，等功能，本次內建主要使用到服務發現功能。

本文主要實作，基于consul_sd_config & consul 的 prometheus 服務發現，實作網路裝置ping監控，站點可用行監控，以及證書相關資訊監控。

安裝環境：

• k8s

• consul

• Prometheus

• blackbox_exporter

1: Consul 安裝

1.1：使用helm 安裝 consul

Bash
# 添加 consul helm 源
helm repo add hashicorp https://helm.releases.hashicorp.com
# 安裝consul 
helm -n consul  install  \
--set storageClass=alicloud-disk-efficiency  \
consul hashicorp/consul \
--version=0.32.1
           

1.2：檢視服務安裝狀态

Bash
[root@xxxxxxxx consul_install]# kubectl -n consul get pods
NAME                     READY   STATUS    RESTARTS   AGE
consul-consul-9lxfc      1/1     Running   0          6d1h
consul-consul-ntqcf      1/1     Running   0          6d1h
consul-consul-q7c6f      1/1     Running   0          6d1h
consul-consul-server-0   1/1     Running   0          6d1h
consul-consul-server-1   1/1     Running   0          6d1h
consul-consul-server-2   1/1     Running   0          6d1h
           

1.3：nginx-ingress consul

• consul_ingress.yml

Bash
# consul.xxxxxx.cn  ----->  替換為正确域名

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: consul-ingress
  namespace: consul
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: consul.xxxxxx.cn
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: consul-consul-ui
            port: 
              number: 80
           

• 執行部署

Bash
kubectl apply -f consul_ingress.yml
           

1.4：通路測試

2: Blackbox_export

2.1：blackbox 安裝

• blackbox-exporter-config.yaml

Bash
apiVersion: v1
kind: ConfigMap
metadata:
  name: blackbox-exporter
  labels:
    app: blackbox-exporter
data:
  blackbox.yml: |-
    modules:
      ## ----------- DNS 檢測配置 -----------
      dns_tcp:  
        prober: dns
        dns:
          transport_protocol: "tcp"
          preferred_ip_protocol: "ip4"
          query_name: "kubernetes.default.svc.cluster.local" # 用于檢測域名可用的網址
          query_type: "A" 
      ## ----------- TCP 檢測子產品配置 -----------
      tcp_connect:
        prober: tcp
        timeout: 5s
      ## ----------- ICMP 檢測配置 -----------
      ping:
        prober: icmp
        timeout: 5s
        icmp:
          preferred_ip_protocol: "ip4"
      ## ----------- HTTP GET 2xx 檢測子產品配置 -----------
      http_get_2xx:  
        prober: http
        timeout: 10s
        http:
          method: GET
          preferred_ip_protocol: "ip4"
          valid_http_versions: ["HTTP/1.1","HTTP/2"]
          valid_status_codes: [200]           # 驗證的HTTP狀态碼,預設為2xx
          no_follow_redirects: false          # 是否不跟随重定向
      ## ----------- HTTP GET 3xx 檢測子產品配置 -----------
      http_get_3xx:  
        prober: http
        timeout: 10s
        http:
          method: GET
          preferred_ip_protocol: "ip4"
          valid_http_versions: ["HTTP/1.1","HTTP/2"]
          valid_status_codes: [301,302,304,305,306,307]  # 驗證的HTTP狀态碼,預設為2xx
          no_follow_redirects: false                     # 是否不跟随重定向
      ## ----------- HTTP POST 監測子產品 -----------
      http_post_2xx: 
        prober: http
        timeout: 10s
        http:
          method: POST
          preferred_ip_protocol: "ip4"
          valid_http_versions: ["HTTP/1.1", "HTTP/2"]
          #headers:                             # HTTP頭設定
          #  Content-Type: application/json
          #body: '{}'                           # 請求體設定
           

• blackbox-exporter-deploy.yaml

Bash
apiVersion: v1
kind: Service
metadata:
  name: blackbox-exporter
  labels:
    k8s-app: blackbox-exporter
spec:
  type: ClusterIP
  ports:
  - name: http
    port: 9115
    targetPort: 9115
  selector:
    k8s-app: blackbox-exporter
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: blackbox-exporter
  labels:
    k8s-app: blackbox-exporter
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: blackbox-exporter
  template:
    metadata:
      labels:
        k8s-app: blackbox-exporter
    spec:
      containers:
      - name: blackbox-exporter
        image: prom/blackbox-exporter:v0.19.0
        args:
        - --config.file=/etc/blackbox_exporter/blackbox.yml
        - --web.listen-address=:9115
        - --log.level=info
        ports:
        - name: http
          containerPort: 9115
        resources:
          limits:
            cpu: 3
            memory: 6000Mi
          requests:
            cpu: 100m
            memory: 50Mi
        livenessProbe:
          tcpSocket:
            port: 9115
          initialDelaySeconds: 5
          timeoutSeconds: 5
          periodSeconds: 10
          successThreshold: 1
          failureThreshold: 3
        readinessProbe:
          tcpSocket:
            port: 9115
          initialDelaySeconds: 5
          timeoutSeconds: 5
          periodSeconds: 10
          successThreshold: 1
          failureThreshold: 3
        volumeMounts:
        - name: config
          mountPath: /etc/blackbox_exporter
      volumes:
      - name: config
        configMap:
          name: blackbox-exporter
          defaultMode: 420
           

• 執行安裝

Bash
kubectl apply -f blackbox-exporter-deploy.yaml
kubectl apply -f blackbox-exporter-config.yaml
           

2.2：nginx ingress blackbox-exporter • blackbox_ingress.yml

Bash
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: blackbox-ingress
  namespace: monitoring
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: blackbox-devops.lululemon.cn
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: blackbox-exporter
            port:
              number: 9115
           

• 執行安裝

Bash
kubectl apply -f blackbox_ingress.yml
           

3: rometheus 添加 服務動态發現

Bash
#####  http_get_2xx 資料擷取
 - job_name: http_get_2xx
      params:
        module:
        - http_get_2xx
      scrape_interval: 2s
      scrape_timeout: 2s
      metrics_path: /probe
      consul_sd_configs:
      # consul 服務位址
      - server: consul-consul-server.consul.svc.cluster.local:8500
        tag_separator: ','
        services:
        - http_get_2xx
      relabel_configs:
        - source_labels: ['__meta_consul_service_address']
          target_label: __param_target
        - source_labels: ['__meta_consul_service_address']
          target_label: instance
        - target_label: __address__
          ## blackbox-export  位址
          replacement: blackbox-exporter.monitoring.svc.cluster.local:9115
 ####### icmp 配置 
     - job_name: blackbox_icmp
      params:
        module:
        - ping
      scrape_interval: 2s
      scrape_timeout: 2s
      metrics_path: /probe
      consul_sd_configs:
      # consul 服務位址
      - server: consul-consul-server.consul.svc.cluster.local:8500
        tag_separator: ','
        services:
        - ping
      relabel_configs:
        - source_labels: ['__meta_consul_service_address']
          target_label: __param_target
        - source_labels: ['__meta_consul_service_address']
          target_label: instance
        - target_label: __address__
          ## blackbox-export  位址
          replacement: blackbox-exporter.monitoring.svc.cluster.local:9115
           

4：添加 icmp 監控

4.1：添加監控位址到consul

• icmp_list

Bash
192.168.1.1
192.168.1.2
           

• add_consul_service_icmp.sh

Bash
#!/usr/bin/env bash

ip_addr=$1


if test "$ip_addr";then
        curl -X PUT -d '{
            "id": "icmp_'${ip_addr}'",
            "name": "ping",
            "address": "'${ip_addr}'",
            "port": 443,
            "Meta": {
              "env": "prod",
              "team": "network",
              "project": "network",
              "owner": "Mike"
            },
            "tags": ["node"],
            "checks": [{"http": "http://blackbox-exporter.monitoring.svc.cluster.local:9115/","interval": "15s"}]}' \
            http://consul-consul-server:8500/v1/agent/service/register
else
        echo "請輸入參數"
fi
           

• 添加service ping

Bash
for i in `cat icmp_list`;do bash add_consul_service_icmp.sh  $i;done
           

4.2：檢視consul 服務

4.3：删除ping 監控位址腳本

Bash
#!/usr/bin/env bash
ip_addr=$1

curl -X PUT http://consul-consul-server:8500/v1/agent/service/deregister/icmp_${ip_addr}
           

5: 添加http_get_2xx

5.1：添加監控域名

• domain_name_list

Bash
wwww.baidu.com
wwww.1111.com
wwww.2222.com
           

• add_consul_service_http_get_2xx.sh

Bash
#!/usr/bin/env bash

service_name=$1


if test "$service_name";then
        curl -X PUT -d '{
            "id": "http_get_2xx_'${service_name}'",
            "name": "http_get_2xx",
            "address": "https://'${service_name}'",
            "port": 443,
            "Meta": {
              "env": "prod",
              "team": "web",
              "project": "web",
              "owner": "Devops"
            },
            "tags": ["node"],
            "checks": [{"http": "http://blackbox-exporter.monitoring.svc.cluster.local:9115/","interval": "15s"}]}' \
            http://consul-consul-server:8500/v1/agent/service/register
else
        echo "請輸入參數"
fi
           

• 添加 service http_get_2xx

Bash
for i  in `cat domain_name_list`;do bash  add_consul_service_http_get_2xx.sh  $i;done
           

5.2：檢視consul 服務

5.3：删除域名監控腳本

• del_consul_service_http_get_2xx.sh

Bash
#!/usr/bin/env bash
ip_addr=$1

curl -X PUT http://consul-consul-server:8500/v1/agent/service/deregister/http_get_2xx_${ip_addr}
           

6：檢視prometheus 監控

總結：

使用上述方案，黑盒監控與自建cmdb 平台很容易進行內建，使其監控自動化，不需要過多的人工幹預，可以省去大量的人工成本，grafana 的配置這裡就不進行過多介紹，自行通過谷歌完成。