實戰Elastic Stack分析K8S應用日志--部署filebeat

制作filebeat底包鏡像，接入dubbo服務消費者采集日志，并将日志輸出到kafka。

配置 filebeat

[[email protected] ~]cat filebeat-7.14.1-linux-x86_64.tar.gz.sha512

bbc18f796172ab195632b54fea8c340d38f70cf27a98c00eccf701da3a6e022ff8fe9ce0937fa6da9c71ae9b3fba42e6f723ff802739bc660bdab410bac4d15b  filebeat-7.14.1-linux-x86_64.tar.gz
           

[[email protected] ~]# mkdir /data/dockerfile/filebeat

[[email protected] ~]# cd /data/dockerfile/filebeat/

[[email protected] filebeat]# cat docker-entrypoint.sh

#!/bin/bash

ENV=${ENV:-"test"}
PROJ_NAME=${PROJ_NAME:-"no-define"}
MULTILINE=${MULTILINE:-"^\d{2}"}

cat > /etc/filebeat.yaml << EOF
filebeat.inputs:
- type: log
  fields_under_root: true
  fields:
    topic: logm-${PROJ_NAME}
  paths:
    - /logm/*.log
    - /logm/*/*.log
    - /logm/*/*/*.log
    - /logm/*/*/*/*.log
    - /logm/*/*/*/*/*.log
  scan_frequency: 120s
  max_bytes: 10485760
  multiline.pattern: '$MULTILINE'
  multiline.negate: true
  multiline.match: after
  multiline.max_lines: 100
- type: log
  fields_under_root: true
  fields:
    topic: logu-${PROJ_NAME}
  paths:
    - /logu/*.log
    - /logu/*/*.log
    - /logu/*/*/*.log
    - /logu/*/*/*/*.log
    - /logu/*/*/*/*/*.log
    - /logu/*/*/*/*/*/*.log
output.kafka:
  hosts: ["10.4.7.12:9092"]
  topic: k8s-fb-$ENV-%{[topic]}
  version: 2.0.0
  required_acks: 0
  max_message_bytes: 10485760
EOF

set -xe

# If user don't provide any command
# Run filebeat
if [[ "$1" == "" ]]; then
     exec filebeat  -c /etc/filebeat.yaml 
else
    # Else allow the user to run arbitrarily commands like bash
    exec "$@"
fi
           

/etc/filebeat.yaml解釋：

指定收集的日志檔案

配置日志輸出至 kafka

準備dockerfile

[[email protected] filebeat]# vi Dockerfile

FROM debian:jessie

ENV FILEBEAT_VERSION=7.14.1 \
    FILEBEAT_SHA1=bbc18f796172ab195632b54fea8c340d38f70cf27a98c00eccf701da3a6e022ff8fe9ce0937fa6da9c71ae9b3fba42e6f723ff802739bc660bdab410bac4d15b

RUN set -x && \
  apt-get update && \
  apt-get install -y wget && \
  wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-${FILEBEAT_VERSION}-linux-x86_64.tar.gz -O /opt/filebeat.tar.gz && \
  cd /opt && \
  echo "${FILEBEAT_SHA1}  filebeat.tar.gz" | sha512sum -c - && \
  tar xzvf filebeat.tar.gz && \
  cd filebeat-* && \
  cp filebeat /bin && \
  cd /opt && \
  rm -rf filebeat* && \
  apt-get purge -y wget && \
  apt-get autoremove -y && \
  apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

COPY docker-entrypoint.sh /
ENTRYPOINT ["/docker-entrypoint.sh"]
           

[[email protected] filebeat]# docker build . -t harbor.od.com/infra/filebeat:v7.14.1

[[email protected] filebeat]# docker push harbor.od.com/infra/filebeat:v7.14.1

準備資源清單

[[email protected] filebeat]# vi /data/k8s-yaml/test/dubbo-demo-consumer/fbdp.yaml

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: dubbo-demo-consumer
  namespace: test
  labels: 
    name: dubbo-demo-consumer
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: dubbo-demo-consumer
  template:
    metadata:
      labels: 
        app: dubbo-demo-consumer
        name: dubbo-demo-consumer
    spec:
      containers:
      - name: dubbo-demo-consumer
        image: harbor.od.com/app/dubbo-demo-web:tomcat_210831_2200
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
          protocol: TCP
        env:
        - name: C_OPTS
          value: -Denv=fat -Dapollo.meta=http://apollo-configservice:8080
        volumeMounts:
        - mountPath: /opt/tomcat/logs
          name: logm
      - name: filebeat
        image: harbor.od.com/infra/filebeat:v7.14.1
        imagePullPolicy: IfNotPresent
        env:
        - name: ENV
          value: test
        - name: PROJ_NAME
          value: dubbo-demo-web
        volumeMounts:
        - mountPath: /logm
          name: logm
      volumes:
      - emptyDir: {}
        name: logm
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext: 
        runAsUser: 0
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600
           

fbdp.yaml檔案解釋： spec-containers下有兩個容器，邊車模式（sidecar）運作在同一pod中

應用資源清單

kubectl apply -f http://k8s-yaml.od.com/test/dubbo-demo-consumer/fbdp.yaml

kubectl get pods -n test

檢視filebeat日志

[[email protected] ~]# docker ps -a | grep filebeat

25e8b0643685   71a4a9362bbe                        "/docker-entrypoint.…"   7 minutes ago    Up 7 minutes                                                    k8s_filebeat_dubbo-demo-consumer-dddc598b-bf6bx_test_35326fe2-59a1-45cc-b6f4-089d65d2ed63_0
           

此時重新整理demo-test網頁，可以看到通路日志，如下

[[email protected] ~]# docker exec -it 25e8b0643685 bash

[email protected]:/# ls
bin  boot  data  dev  docker-entrypoint.sh  etc  home  lib  lib64  logm  logs  media  mnt  opt	proc  root  run  sbin  srv  sys  tmp  usr  var
[email protected]:/# cd logm/
[email protected]:/logm# ls
catalina.2021-09-02.log  localhost.2021-09-02.log  localhost_access_log.2021-09-02.txt	stdout.log
[email protected]:/logm# tailf stdout.log 
2021-09-02 22:27:19 HelloAction接收到請求:devops
2021-09-02 22:27:19 HelloService傳回到結果:2021-09-02 22:27:19 <h1>這是Dubbo 消費者端(Tomcat服務) 版本更新2.0</h1>hello devops
2021-09-02 22:27:19 HelloAction接收到請求:devops
2021-09-02 22:27:19 HelloService傳回到結果:2021-09-02 22:27:19 <h1>這是Dubbo 消費者端(Tomcat服務) 版本更新2.0</h1>hello devops
2021-09-02 22:27:19 HelloAction接收到請求:devops
           

檢視Kafka中Topic

驗證filebeat輸出到kafka

[[email protected] kafka]# bin/kafka-topics.sh --list --zookeeper 10.4.7.11:2181

k8s-fb-test-logm-dubbo-demo-web