Prometheus監控學習筆記之使用JMX Exporter監控微服務JVM

Prometheus JMX Exporter 微服務 JVM

0x00 概述

本文實作微服務JVM監控的方法為，使用volume HostPath挂載的JMX Exporter的方式在容器内以in-process的方式實作對微服務的JMV監控。

主要坑點為HostPath挂載JMX Exporter javaagent導緻微服務無法啟動。

0x02 部署

2.1 下載下傳JMX Exporter javaagent包

$ mkdir -p /tmp/jmx_exporter
$ wget -O /tmp/jmx_exporter/jmx_prometheus_javaagent-0.17.0.jar https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.17.0/jmx_prometheus_javaagent-0.17.0.jar

2.2 配置YAML檔案

此處鏡像采用Deployment+Tomcat的鏡像作為樣例，自測的時候注意及時更換鏡像；

'''
1. 采用HostPath的方式，将下載下傳的JMX Exporter的jar包挂載到伺服器容内啟動；
2. JMX Exporter的配置檔案以configmap的形式挂載到服務容器内；
'''

Tomcat服務的Deployment檔案：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tomcat-with-jvmExporter
  namespace: default
  labels:
    app: tomcat-with-jvmExporter
spec:
  replicas: 1
  selector:
    matchLabels:
      app: tomcat-with-jvmExporter
  template:
    metadata:
      labels:
        app: tomcat-with-jvmExporter
    spec:
      nodeSelector:
        kubernetes.io/hostname: docker49
      restartPolicy: Always
      containers:
      - name: tomcat-with-jvmExporter
        image: tomcat:9.0.54-jdk8-openjdk
        imagePullPolicy: Always
        env:
        # JVM env
        - name: JAVA_OPTS
          value: "-Xms500m -Xmx500m -javaagent:/jmx_prometheus_javaagent-0.17.0.jar=5555:/jmx/prometheus-jmx-config.yaml"
          #value: "-Xms500m -Xmx500m"
        volumeMounts:
        - mountPath: /jmx_prometheus_javaagent-0.17.0.jar
          name: jmx-prometheus-javaagent
        - mountPath: /jmx
          name: prometheus-jmx-config
        resources:
          limits:
            cpu: 2000m
            memory: "4096Mi"
          requests:
            cpu: 2000m
            memory: "4096Mi"
      volumes:
      - name: jmx-prometheus-javaagent
        hostPath:
          path: /tmp/jmx_prometheus_javaagent-0.17.0.jar
      - configMap:
          name: prometheus-jmx-config
        name: prometheus-jmx-config

'''
1. 通過hostpath挂載的方式，對于pod來說，不一定會排程到這個節點伺服器上；

2. 如果pod沒排程到有這個jar包的伺服器上，在啟動pod的時候，會發現pod挂載了jar包和配置檔案，但是無法執行這個jar包；

3. 是以需要在yaml中通過nodeSelector指定pod排程到固定節點上，這樣hostPath才能取到檔案，pod才會成功挂載這個檔案并正常運作

4. 上面我們将jar包下載下傳到了docker49這台機器上，是以指定将tomcat的pod排程到docker49，注意紅字部分

'''

未配置nodeSelector的話，會出現如下報錯：

'''
Error opening zip file or JAR manifest missing : /jmx_prometheus_javaagent-0.17.0.jar
Error occurred during initialization of VM
agent library failed to init: instrument
'''

直接搜尋以上報錯日志，StackOverFlow上大部分都會說是權限問題，但是無論你改成755還是777都一樣，配置nodeSelector，将pod排程到JMX Exporter jar包的機器上即可。

Tomcat服務的Service檔案：

apiVersion: v1
kind: Service
metadata:
  name: tomcat-with-jvmExporter-svc
  namespace: default
  labels:
    app: tomcat-with-jvmExporter-svc
  annotations:
    prometheus.io/port: "5555"
    prometheus.io/jvm: "true"
spec:
  type: NodePort
  ports:
  - port: 8080
    name: tomcat-with-jvmExporter-svc
    targetPort: 30899
    nodePort: 30899
  - name: jmx-metrics
    port: 5555
    protocol: TCP
  selector:
    app: tomcat-with-jvmExporter

Tomcat服務的configmap檔案（此處為友善測試采用最小化配置檔案，官方配置檔案樣例）：

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-jmx-config
  namespace: default
data:
  prometheus-jmx-config.yaml: |
#    lowercaseOutputLabelNames: true
#    lowercaseOutputName: true
    rules:
    - pattern: ".*"
#        - pattern: 'Catalina<type=GlobalRequestProcessor, name=\"(\w+-\w+)-(\d+)\"><>(\w+):'
#          name: tomcat_$3_total
#          labels:
#            port: "$2"
#            protocol: "$1"
#          help: Tomcat global $3
#          type: COUNTER
#        - pattern: 'Catalina<j2eeType=Servlet, WebModule=//([-a-zA-Z0-9+&@#/%?=~_|!:.,;]*[-a-zA-Z0-9+&@#/%=~_|]), name=([-a-zA-Z0-9+/$%~_-|!.]*), J2EEApplication=none, J2EEServer=none><>(requestCount|maxTime|processingTime|errorCount):'
#          name: tomcat_servlet_$3_total
#          labels:
#            module: "$1"
#            servlet: "$2"
#          help: Tomcat servlet $3 total
#          type: COUNTER
#        - pattern: 'Catalina<type=ThreadPool, name="(\w+-\w+)-(\d+)"><>(currentThreadCount|currentThreadsBusy|keepAliveCount|pollerThreadCount|connectionCount):'
#          name: tomcat_threadpool_$3
#          labels:
#            port: "$2"
#            protocol: "$1"
#          help: Tomcat threadpool $3
#          type: GAUGE
#        - pattern: 'Catalina<type=Manager, host=([-a-zA-Z0-9+&@#/%?=~_|!:.,;]*[-a-zA-Z0-9+&@#/%=~_|]), context=([-a-zA-Z0-9+/$%~_-|!.]*)><>(processingTime|sessionCounter|rejectedSessions|expiredSessions):'
#          name: tomcat_session_$3_total
#          labels:
#            context: "$2"
#            host: "$1"
#          help: Tomcat session $3 total
#          type: COUNTER

2.3 All in One yaml檔案

tomcat-with-jvmExporter-AllinOne.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tomcat-wtih-jvmExporter
  namespace: default
  labels:
    app: tomcat-with-jvmExporter
spec:
  replicas: 1
  selector:
    matchLabels:
      app: tomcat-with-jvmExporter
  template:
    metadata:
      labels:
        app: tomcat-with-jvmExporter
    spec:
      nodeSelector:
        kubernetes.io/hostname: docker49
      restartPolicy: Always
      containers:
      - name: tomcat-with-jvmExporter
        image: tomcat:9.0.54-jdk8-openjdk
        imagePullPolicy: Always
        env:
        # JVM env
        - name: JAVA_OPTS
          value: "-Xms500m -Xmx500m -javaagent:/jmx_prometheus_javaagent-0.17.0.jar=5555:/jmx/prometheus-jmx-config.yaml"
          #value: "-Xms500m -Xmx500m"
        volumeMounts:
        - mountPath: /jmx_prometheus_javaagent-0.17.0.jar
          name: jmx-prometheus-javaagent
        - mountPath: /jmx
          name: prometheus-jmx-config
        resources:
          limits:
            cpu: 2000m
            memory: "4096Mi"
          requests:
            cpu: 2000m
            memory: "4096Mi"
      volumes:
      - name: jmx-prometheus-javaagent
        hostPath:
          path: /tmp/jmx_prometheus_javaagent-0.17.0.jar
      - configMap:
          name: prometheus-jmx-config
        name: prometheus-jmx-config
---
apiVersion: v1
kind: Service
metadata:
  name: tomcat-with-jvmExporter-svc
  namespace: default
  labels:
    app: tomcat-with-jvmExporter-svc
  annotations:
    prometheus.io/port: "5555"
    prometheus.io/jvm: "true"
spec:
  type: NodePort
  ports:
  - port: 8080
    name: tomcat-with-jvmExporter-svc
    targetPort: 30899
    nodePort: 30899
  - name: jmx-metrics
    port: 5555
    protocol: TCP
  selector:
    app: tomcat-with-jvmExporter
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-jmx-config
  namespace: default
data:
  prometheus-jmx-config.yaml: |
    #lowercaseOutputLabelNames: true
    #lowercaseOutputName: true
    rules:
    - pattern: ".*"
#        - pattern: 'Catalina<type=GlobalRequestProcessor, name=\"(\w+-\w+)-(\d+)\"><>(\w+):'
#          name: tomcat_$3_total
#          labels:
#            port: "$2"
#            protocol: "$1"
#          help: Tomcat global $3
#          type: COUNTER
#        - pattern: 'Catalina<j2eeType=Servlet, WebModule=//([-a-zA-Z0-9+&@#/%?=~_|!:.,;]*[-a-zA-Z0-9+&@#/%=~_|]), name=([-a-zA-Z0-9+/$%~_-|!.]*), J2EEApplication=none, J2EEServer=none><>(requestCount|maxTime|processingTime|errorCount):'
#          name: tomcat_servlet_$3_total
#          labels:
#            module: "$1"
#            servlet: "$2"
#          help: Tomcat servlet $3 total
#          type: COUNTER
#        - pattern: 'Catalina<type=ThreadPool, name="(\w+-\w+)-(\d+)"><>(currentThreadCount|currentThreadsBusy|keepAliveCount|pollerThreadCount|connectionCount):'
#          name: tomcat_threadpool_$3
#          labels:
#            port: "$2"
#            protocol: "$1"
#          help: Tomcat threadpool $3
#          type: GAUGE
#        - pattern: 'Catalina<type=Manager, host=([-a-zA-Z0-9+&@#/%?=~_|!:.,;]*[-a-zA-Z0-9+&@#/%=~_|]), context=([-a-zA-Z0-9+/$%~_-|!.]*)><>(processingTime|sessionCounter|rejectedSessions|expiredSessions):'
#          name: tomcat_session_$3_total
#          labels:
#            context: "$2"
#            host: "$1"
#          help: Tomcat session $3 total
#          type: COUNTER

2.4 啟動并檢查服務

啟動服務：

kubectl apply -f tomcat-with-jvmExporter-AllinOne.yaml

檢查服務：

kubectl get svc -n default

檢查JVM Exporter是否已擷取JVM監控資料，此處docker49 ip：192.168.6.6，JVM Exporter對外nodeExporter為30566

curl 192.168.6.6:30566/metrics

如果傳回以下資料，說明已成功擷取監控資料：

jvm_threads_daemon 36.0
# HELP jvm_threads_peak Peak thread count of a JVM
# TYPE jvm_threads_peak gauge
jvm_threads_peak 80.0
# HELP jvm_threads_started_total Started thread count of a JVM
# TYPE jvm_threads_started_total counter
jvm_threads_started_total 155322.0
# HELP jvm_threads_deadlocked Cycles of JVM-threads that are in deadlock waiting to acquire object monitors or ownable synchronizers
# TYPE jvm_threads_deadlocked gauge
jvm_threads_deadlocked 0.0
# HELP jvm_threads_deadlocked_monitor Cycles of JVM-threads that are in deadlock waiting to acquire object monitors
# TYPE jvm_threads_deadlocked_monitor gauge
jvm_threads_deadlocked_monitor 0.0
# HELP jvm_threads_state Current count of threads by state
# TYPE jvm_threads_state gauge
jvm_threads_state{state="NEW",} 0.0
jvm_threads_state{state="BLOCKED",} 0.0
jvm_threads_state{state="TIMED_WAITING",} 22.0
jvm_threads_state{state="TERMINATED",} 0.0
jvm_threads_state{state="RUNNABLE",} 12.0
jvm_threads_state{state="WAITING",} 41.0
# HELP jvm_classes_currently_loaded The number of classes that are currently loaded in the JVM
# TYPE jvm_classes_currently_loaded gauge
jvm_classes_currently_loaded 18332.0
# HELP jvm_classes_loaded_total The total number of classes that have been loaded since the JVM has started execution
# TYPE jvm_classes_loaded_total counter
jvm_classes_loaded_total 18478.0
# HELP jvm_classes_unloaded_total The total number of classes that have been unloaded since the JVM has started execution
# TYPE jvm_classes_unloaded_total counter
jvm_classes_unloaded_total 146.0
# HELP jmx_config_reload_failure_created Number of times configuration have failed to be reloaded.
# TYPE jmx_config_reload_failure_created gauge
jmx_config_reload_failure_created 1.655778604558E9
# HELP jmx_config_reload_success_created Number of times configuration have successfully been reloaded.
# TYPE jmx_config_reload_success_created gauge
jmx_config_reload_success_created 1.655778604539E9
# HELP jvm_memory_pool_allocated_bytes_created Total bytes allocated in a given JVM memory pool. Only updated after GC, not continuously.
# TYPE jvm_memory_pool_allocated_bytes_created gauge
jvm_memory_pool_allocated_bytes_created{pool="Code Cache",} 1.655778606954E9
jvm_memory_pool_allocated_bytes_created{pool="PS Eden Space",} 1.655778606951E9
jvm_memory_pool_allocated_bytes_created{pool="PS Old Gen",} 1.655778606954E9
jvm_memory_pool_allocated_bytes_created{pool="PS Survivor Space",} 1.655778606954E9
jvm_memory_pool_allocated_bytes_created{pool="Compressed Class Space",} 1.655778606954E9
jvm_memory_pool_allocated_bytes_created{pool="Metaspace",} 1.655778606954E9

0x03 配置Prometheus擷取監控個資料

修改prometheus.yml配置檔案，新增JVM Exporter資料

- job_name: "jvm-ucm-config"
      honor_labels: true
      static_configs:
        - targets: ["192.168.6.6:30566"]

重新加載Prometheus配置檔案

kill -HUP $pid

0x04 Grafana導入JVM監控面闆

推薦面闆id：11278

需要修改面闆變量，根據以上配置，變量設定為2，job+instance

0x05 參考

prometheus/jmx_exporter官方Github

Prometheus監控學習筆記之使用JMX Exporter監控微服務JVM

0x00 概述

0x02 部署

2.1 下載下傳JMX Exporter javaagent包

2.2 配置YAML檔案

2.3 All in One yaml檔案

2.4 啟動并檢查服務

0x03 配置Prometheus擷取監控個資料

0x04 Grafana導入JVM監控面闆

0x05 參考

繼續閱讀

關于Gradle配置的小結

Java小案例——随機數猜測随機數猜測

nginx location中斜線的位置的重要性

27 Best Free Eclipse Plug-ins for Java Developer to be ProductiveCode Quality PluginsText Editor PluginsDependency ManagementVersion Control Integration PluginsFramework Development Continuous Integration Related PluginsOther Utility Plugins

Java String.format方法的簡單使用

neo4j之cypher使用文檔

GitHub連夜封殺！這份阿裡 10W 字内部 Java 字面試手冊到底有多強？

spark/scala關于【資源檔案】加載方法概述外部檔案加載方案測試資源檔案打包入jar包中小結

mybatis_入門程式Mybatis入門

AOP程式設計_Android優雅權限架構(1)概念基礎，2021金三銀四前言正文大綱正文

Effective Java 8:通用程式設計

OOM三種類型

工廠模式-三種類型

【遞歸】高效率求2的n次幂

win10本地scala和spark安裝安裝scala安裝spark

scala (3) Function 和 Method