prometheus 監控二進制安裝的K8s元件

由于我的k8s是采用二進制安裝的，且prometheus-operator(prometheus-stack)沒有自帶k8s本身元件的job：kube-apiserver、kube-controller-manager、kubelet、kube-schedular、kube-proxy的，也許有，但是我學藝不精，可能沒有找到，在這裡我自己配置了對這些元件的監控

監控api-server

1、編輯配置檔案添加job

root@master prometheus]# kubectl -n middleware edit configmaps prom-prometheus-server
    - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      job_name: kubernetes-apiservers
      kubernetes_sd_configs:
      - role: endpoints
      relabel_configs:
      - action: keep
        regex: default;kubernetes;https
        source_labels:
        - __meta_kubernetes_namespace
        - __meta_kubernetes_service_name
        - __meta_kubernetes_endpoint_port_name
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true      

2、将自帶的報警配置裡面job改成自定義的job

      - alert: KubeAPIDown
        annotations:
          description: KubeAPI has disappeared from Prometheus target discovery.
          runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeapidown
          summary: Target disappeared from Prometheus target discovery.
        expr: absent(up{job="kubernetes-apiservers"} == 1)
        for: 1m
        labels:
          severity: critical      

監控kube-controller-manager

1、修改kube-controller-manager的配置檔案，添加預設監聽位址為0.0.0.0

[root@master prometheus]# cat /opt/kubernetes/cfg/kube-controller-manager 
KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=true \
--v=4 \
--master=127.0.0.1:8080 \
--leader-elect=true \
--address=0.0.0.0 \
--service-cluster-ip-range=10.0.0.0/24 \
--cluster-name=kubernetes \
--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \
--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem  \
--root-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem"      

2、添加job

    - job_name: kube-controller-manager
      static_configs:
      - targets:
        - 10.1.6.99:10252      

3、将自帶的alert裡面job改成自定義的job名稱

      - alert: KubeControllerManagerDown
        annotations:
          description: KubeControllerManager has disappeared from Prometheus target discovery.
          runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubecontrollermanagerdown
          summary: Target disappeared from Prometheus target discovery.
        expr: absent(up{job="kube-controller-manager"} == 1)
        for: 15m
        labels:
          severity: critical      

監控kube-scheduler

1、添加job

    - job_name: kube-scheduler
      static_configs:
      - targets:
        - 10.1.6.99:10251      

2、修改預設alert裡面的job名稱

      - alert: KubeSchedulerDown
        annotations:
          description: KubeScheduler has disappeared from Prometheus target discovery.
          runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeschedulerdown
          summary: Target disappeared from Prometheus target discovery.
        expr: absent(up{job="kube-scheduler"} == 1)
        for: 15m
        labels:
          severity: critical      

監控kubelet

    - job_name: kubelet
      kubernetes_sd_configs:
      - role: node
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true  # Required with Minikube.      

2、修改alert裡面job名稱為自定義job名稱

      - alert: KubeletDown
        annotations:
          description: Kubelet has disappeared from Prometheus target discovery.
          runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletdown
          summary: Target disappeared from Prometheus target discovery.
        expr: absent(up{job="kubelet"}  == 1)
        for: 15m
        labels:
          severity: critical      

監控Kube-proxy

1、修改kube-proxy配置檔案添加監聽位址0.0.0.0

KUBE_PROXY_OPTS="--logtostderr=true \
--v=4 \
--hostname-override=10.1.6.101 \
--cluster-cidr=10.0.0.0/24 \
--metrics-bind-address=0.0.0.0 \
--kubeconfig=/opt/kubernetes/cfg/kube-proxy.kubeconfig"      

    - job_name: kube-proxy
      static_configs:
      - targets:
        - 10.1.6.99:10249      

    - name: kubernetes-system-proxy
      rules:
      - alert: Kube-proxy Down
        annotations:
          description: KubeScheduler has disappeared from Prometheus target discovery.
          runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeproxydown
          summary: Target disappeared from Prometheus target discovery.
        expr: absent(up{job="kube-proxy"} == 1)
        for: 15m
        labels:
          severity: critical