Prometheus 基于pod服務發現監控tomcat

建構鏡像

預設tomcat鏡像不提供普羅米修斯相關的支援，是以需要自己制作鏡像。 

在tomcat釋出目錄下需要有一個metrics.war，這樣接口位址才能暴露tomcat的一些名額，以及采集名額的tomcat_exporter_client-0.0.12.jar

[root@master tomcat]# cat Dockerfile 
FROM tomcat
ADD metrics.war /usr/local/tomcat/webapps/
ADD simpleclient-0.8.0.jar  /usr/local/tomcat/lib/
ADD simpleclient_common-0.8.0.jar /usr/local/tomcat/lib/
ADD simpleclient_hotspot-0.8.0.jar /usr/local/tomcat/lib/
ADD simpleclient_servlet-0.8.0.jar /usr/local/tomcat/lib/
ADD tomcat_exporter_client-0.0.12.jar /usr/local/tomcat/lib/


[root@master tomcat]# docker build -t 'tomcat_prometheus:v1' .
Sending build context to Docker daemon    128kB
Step 1/7 : FROM tomcat
latest: Pulling from library/tomcat
5e0b432e8ba9: Pull complete 
a84cfd68b5ce: Pull complete 
e8b8f2315954: Pull complete 
Digest: sha256:429d5c140820d5eba38042823501d5f4e0f03b282133513e29c22d6358daa7c7
Status: Downloaded newer image for tomcat:latest
 ---> 24207ccc9cce
Step 2/7 : ADD metrics.war /usr/local/tomcat/webapps/
 ---> 65f67a4763ea
Step 3/7 : ADD simpleclient-0.8.0.jar  /usr/local/tomcat/lib/
 ---> 72d7414ff158
Step 4/7 : ADD simpleclient_common-0.8.0.jar /usr/local/tomcat/lib/
 ---> 8e58e81f047f
Step 5/7 : ADD simpleclient_hotspot-0.8.0.jar /usr/local/tomcat/lib/
 ---> 2324ee685ac8
Step 6/7 : ADD simpleclient_servlet-0.8.0.jar /usr/local/tomcat/lib/
 ---> deb785fb6740
Step 7/7 : ADD tomcat_exporter_client-0.0.12.jar /usr/local/tomcat/lib/
 ---> 92539e485933
Successfully built 92539e485933
Successfully tagged tomcat_prometheus:v1      

放在公有倉庫上面了。 

docker pull xianchao/tomcat_prometheus:v1 #在k8s的node節點拉取鏡像拉取鏡像

基于上面的鏡像建立一個tomcat執行個體 

pod裡面這個字段也就是說我們建立的pod可以被Prometheus抓取到。

annotations:
        prometheus.io/scrape: 'true'      

[root@master tomcat]# cat deploy.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tomcat-deployment
  namespace: monitor
spec:
  selector: 
    matchLabels: 
     app: tomcat
  replicas: 1 # tells deployment to run 2 pods matching the template
  template: # create pods using pod definition in this template
    metadata:
      labels:
        app: tomcat
      annotations:
        prometheus.io/scrape: 'true'
    spec:
      containers:
      - name: tomcat
        image: xianchao/tomcat_prometheus:v1
        ports:
        - containerPort: 8080
        securityContext: 
          privileged: true      

[root@master prometheus]# kubectl get pod -n monitor -o wide
NAME                                 READY   STATUS    RESTARTS   AGE   IP              NODE     NOMINATED NODE   READINESS GATES
grafana-757fcd5f7c-n6sj7             1/1     Running   19         26d   10.233.96.236   node2    <none>           <none>
node-exporter-26849                  1/1     Running   21         36d   192.168.100.5   master   <none>           <none>
node-exporter-jbbcq                  1/1     Running   23         36d   192.168.100.6   node1    <none>           <none>
node-exporter-mh78s                  1/1     Running   21         36d   192.168.100.7   node2    <none>           <none>
prometheus-server-5775f99578-6pxsj   1/1     Running   0          89s   10.233.90.131   node1    <none>           <none>
tomcat-deployment-699d65b574-6zzcn   1/1     Running   0          10m   10.233.96.242   node2    <none>           <none>      

我們可以看到 kubernetes-pods下面出現了很多執行個體，這是因為這裡我們使用的是 pod 類型的服務發現，那麼我們應該怎樣來過濾出這個pod來呢？

​

​keep​

​，就是隻把符合我們要求的給保留下來，哪些才是符合我們要求的呢？ 

我們可以把滑鼠放置在任意一個 target 上，可以檢視到​

​Before relabeling​

​裡面所有的中繼資料 

- job_name: 'kubernetes-pods-tomcat'
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - source_labels: [__meta_kubernetes_namespace,__meta_kubernetes_pod_label_app]
        action: keep
        regex: monitor;tomcat      

[root@master prometheus]# curl http://10.233.96.242:8080/metrics/
# HELP jvm_classes_loaded The number of classes that are currently loaded in the JVM
# TYPE jvm_classes_loaded gauge
jvm_classes_loaded 2760.0
# HELP jvm_classes_loaded_total The total number of classes that have been loaded since the JVM has started execution
# TYPE jvm_classes_loaded_total counter
jvm_classes_loaded_total 2760.0
# HELP jvm_classes_unloaded_total The total number of classes that have been unloaded since the JVM has started execution
# TYPE jvm_classes_unloaded_total counter
jvm_classes_unloaded_total 0.0
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 7.53
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.639707011452E9      

cat tomcat-service.yaml
kind: Service  #service 類型
apiVersion: v1
metadata:
#  annotations:
#    prometheus.io/scrape: 'true'
  name: tomcat-service
spec:
  selector:
    app: tomcat
  ports:
  - nodePort: 31360
    port: 80
    protocol: TCP
    targetPort: 8080
  type: NodePort