Spring boot項目服務監控-docker-Prometheus-Grafana-AlertManager-webhook-釘釘

目錄

• • 一　Docker Prometheus Grafana 監控 Spring boot項目
  • • 1.1 Springboot 內建 micrometer
    • 1.2 deploy prometheus
    • 1.3 deploy grafana
  • 二　Docker Prometheus AlertManager webhook 報警
  • • 2.1 配置釘釘機器人
    • 2.2 deploy alertManager
    • 2.3 deploy 釘釘插件
    • 2.4 驗證

一　Docker Prometheus Grafana 監控 Spring boot項目

1.1 Springboot 內建 micrometer

• 1.1.1 pom.xml

<dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <dependency>
            <groupId>io.micrometer</groupId>
            <artifactId>micrometer-registry-prometheus</artifactId>
        </dependency>
           

• 1.1.2 application.properties

# Actuator prometheus
management.endpoints.web.exposure.include=*
# 将應用名稱添加到計量器的 tag 中去,以便 Prometheus 根據應用名區分不同服務
management.metrics.tags.application=${spring.application.name}
           

• 1.1.3 jvm監控配置

package com.car.life.service.wallet.config

import io.micrometer.core.instrument.MeterRegistry
import org.springframework.beans.factory.annotation.Value
import org.springframework.boot.actuate.autoconfigure.metrics.MeterRegistryCustomizer
import org.springframework.context.annotation.Bean
import org.springframework.context.annotation.Configuration


/**
 * @program: car-life-project
 *
 * @description: MircoMeterConfig
 *
 * @author: loulvlin
 *
 * @create: 2021-02-22 14:01
 */
@Configuration
class MircoMeterConfig {

    @Bean
    fun configurer(
            @Value("\${spring.application.name}") applicationName: String): MeterRegistryCustomizer<MeterRegistry> {
        return MeterRegistryCustomizer { registry: MeterRegistry -> registry.config().commonTags("application", applicationName) }
    }
}
           

• 1.1.4 接口/actuator/prometheus權限放行

if(request.requestURI=="/actuator/prometheus" || request.requestURI=="/status"  || request.requestURI=="/favicon.ico"){
            return super.preHandle(request, response, handler)
        }
           

• 1.1.5 啟動項目
• 1.1.6 通路接口檢視監控名額資料

http://192.168.40.234:11000/actuator/prometheus
http://localhost:11000/actuator/prometheus
           

1.2 deploy prometheus

• 1.2.1 create folder

mkdir -p /home/prometheus/
mkdir -p /home/prometheus/rules
           

• 1.2.3 get prometheus.yml

cd /home/prometheus/

docker run --name prometheus -d -p 127.0.0.1:9090:9090 prom/prometheus

docker cp /etc/prometheus/prometheus.yml .

docker rm -f prometheus
           

• 1.2.3 modify prometheus.yml

添加一個job_name 為wallet-service的job

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:9090']

  - job_name: 'wallet-service'
    scrape_interval: 15s
    scrape_timeout: 10s
    metrics_path: '/actuator/prometheus'
    static_configs:
      - targets: ['192.168.40.234:11000']
        "labels": {
            "instance": "wallet-service",
            "service": "car-life-project"
        }
           

• 1.2.4 download image and start container

如果要修改時區，需要自己建構image，參數配置不起作用

docker pull prom/prometheus

docker run --name prometheus -d --restart always \
    --hostname prometheus \
    -p 9090:9090 \
    -v /home/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml \
    -v /home/prometheus/rules:/home/prometheus/rules \
    -u root prom/prometheus
           

• 1.2.5 通路prometheus

http://192.168.0.35:9090/targets #監控目标清單頁
#查詢控制台 可以查詢一個名額，例如 http_server_requests_seconds_sum
http://192.168.0.35:9090/graph 
           

1.3 deploy grafana

• 1.3.1 deploy

docker pull grafana/grafana

docker run -d --restart always \
    -p 3000:3000 \
    --name=grafana \
    grafana/grafana
    
# 啟動後，通路：http://192.168.0.35:3000，預設使用者名密碼 admin/admin   新密碼修改成123456
           

• 1.3.2 配置prometheus資料源

1. 配置中添加 Prometheus 資料源［url:http://192.168.0.35:9090/,HTTP Method
:GET］最後Save&Test
           

• 1.3.2 配置dashboard

1. +-->import  輸入4701  或   12271[推薦]
2. 儲存dashboard
3. 頁面背景色--->頁面左下角-->admin-->Preference
           

• 1.3.３ 監控圖

  

二　Docker Prometheus AlertManager webhook 報警

2.1 配置釘釘機器人

注意：如果機器人配置了多個關鍵字，推送到釘釘中的消息内容必須包含至少一個關鍵詞，才能推送成功

• 2.1.1 建群＂服務監控＂并添加＂DEV服務監控＂機器人
• 為機器人配置關鍵字，可以配置多個關鍵字，比如＂服務監控＂

2.2 deploy alertManager

• 2.2.1 get alertmanager.yml

mkdir -p /home/alertManager && cd /home/alertManager

docker run -d --restart always --name alertmanager -p 9093:9093 prom/alertmanager

docker cp alertmanager:/etc/alertmanager/alertmanager.yml .

docker rm -f  alertmanager
           

• 2.2.2 modify alertmanager.yml

192.168.0.35:8060為後面prometheus-webhook-dingtalk部署的ip和端口

global:
  resolve_timeout: 5m
route:
  receiver: webhook
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 5m
  group_by: [alertname]
  routes:
  - receiver: webhook
    group_wait: 10s
    match:
      team: mobile-dev
receivers:
- name: webhook
  webhook_configs:
  - url: http://192.168.0.35:8060/dingtalk/webhook1/send
    send_resolved: true
           

• 2.2.3 編寫wallet-service告警規則

cd /home/prometheus/rules

vim wallet-service.yml
           

groups:
- name: wallet-service
  rules:
  - alert: wallet-service
    expr: up{job="wallet-service"} == 0
    for: 15s
    labels:
      severity: critical
      team: mobile-dev
    annotations:
      summary: "服務監控,恭喜您，您的wallet-service服務已經挂掉啦"
      description: "{{$labels.instance}} of job {{$labels.job}} 已經停止."
           

• 2.2.4 prometheus配置告警

cd /home/prometheus

vim prometheus.yml
           

192.168.0.35:9093為alertmanager部署的ip和端口

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets: ['192.168.0.35:9093']
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
- /home/prometheus/rules/*.yml
  # - "first_rules.yml"
  # - "second_rules.yml"

           

删除prometheus容器并重新開機[由于前面已經做過rules映射配置，故重新開機容器即可]

docker run --name prometheus -d --restart always \
    --hostname prometheus \
    -p 9090:9090 \
    -v /home/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml \
    -v /home/prometheus/rules:/home/prometheus/rules \
    -u root prom/prometheus
           

• 2.2.5 啟動alertmanager容器

docker pull prom/alertmanager

docker run -d \
  -p 9093:9093 \
  --name alertmanager \
  --restart=always \
  -v /etc/localtime:/etc/localtime \
  -v /home/alertManager/alertmanager.yml:/etc/alertmanager/alertmanager.yml \
  prom/alertmanager
           

2.3 deploy 釘釘插件

• 2.3.1 擷取token

https://oapi.dingtalk.com/robot/send?access_token=XXXXXX
           

• 2.3.2 啟動插件

docker pull timonwong/prometheus-webhook-dingtalk

docker run -d --restart always -p 8060:8060 --name webhook timonwong/prometheus-webhook-dingtalk --ding.profile="webhook1=https://oapi.dingtalk.com/robot/send?access_token=XXXXXX"
           

2.4 驗證

• 2.4.1 檢視告警規則

http://192.168.0.35:9090/rules
           

• 2.4.2 關閉服務

http://192.168.0.35:9090/targets

http://192.168.0.35:9090/alerts
           

targets

alerts

釘釘消息：

• 2.4.2 啟動服務

http://192.168.0.35:9090/targets

http://192.168.0.35:9090/alerts
           

targets:

alerts:

釘釘消息：