ELK 7.5.0（八）ELK+Filebeat采集Json日志

一、實驗環境

主機名        IP
es        192.168.14.210
kibana    192.168.14.210
logstash  192.168.14.211
Filebeat  192.168.14.213
nginx     192.168.14.213
           

二、安裝部署（内容比較多，已經分開寫）

1、ELK（elasticsearch+logstash+kibana）

具體檢視第二、三、四章節

2、Filebeat安裝

具體檢視第六章節

三、Nginx使用json格式日志

json的好處：

用戶端原生日志需要Logstash做正則比對，比較麻煩，也會消耗系統資源；

用戶端設定Json格式的日志不需要Logstash正則分析，就能直接分段采集，減少伺服器壓力。

1、nginx配置檔案添加Json日志格式

[[email protected] ~]# vi /usr/local/nginx/conf/nginx.conf
    log_format json '{"@timestamp":"$time_iso8601",'
                    '"clientip":"$remote_addr",'
                    '"status":$status,'
                    '"bodysize":$body_bytes_sent,'
                    '"referer":"$http_referer",'
                    '"ua":"$http_user_agent",'
                    '"handletime":$request_time,'
                    '"url":"$uri"}';

    access_log  logs/access.json.log  json;
           

2、啟動或重載配置檔案

[[email protected] ~]# nginx -t
nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful
[[email protected] ~]# nginx -s reload
           

3、浏覽器通路nginx，然後本機檢視nginx日志輸出是否正常

四、Filebeat采集Json日志格式

1、修改配置檔案

[[email protected] ~]# vi  /usr/local/filebeat-7.5.0/filebeat.yml
filebeat.inputs:
- type: log
  tail_files: true
  backoff: "1s"
  paths:
      - /usr/local/nginx/logs/access.json.log

output:
  logstash:
    hosts: ["192.168.14.211:5044"]
           

2、啟動或者重新開機Filebeat服務

[[email protected] ~]# pkill filebeat
[[email protected] ~]# nohup filebeat -e -c /usr/local/filebeat-7.5.0/filebeat.yml > /tmp/filebeat.log 2>&1 &
           

五、Logstash解析Json日志

1、修改配置檔案，根據需求删除不必要字段

[[email protected] ~]# vi /usr/local/logstash-7.5.0/config/logstash.conf 
input {
  beats {
    host => '0.0.0.0'
    port => 5044
  }
}
filter {
  json {
    source => "message"
     remove_field => ["message","@version","path","input","log","agent","ecs","tags"]    #不需要記錄，移除的字段
  }
}
output {
  elasticsearch {
    hosts => ["http://192.168.14.210:9200"]
    user => "elastic"
    password => "elkpwd"
    index => "logstash-%{+YYYY.MM.dd}"
  }
}
           

2、啟動或者重新開機服務

[[email protected] ~]# kill -9 14598(logstash程序ID)
[[email protected] ~]# nohup logstash -f /usr/local/logstash-7.5.0/config/logstash.conf > /tmp/logstash.log 2> /tmp/logstash.log & 
           

3、在kibana檢視json日志格式

至此，json日志格式采集并展示完成。tomcat其他服務類似此操作。