一.介紹
輕量型資料采集器Beats是一個免費且開放的平台，集合了多種單一用途資料采集器。它們從成百上千或成千上萬台機器和系統向 Logstash 或 Elasticsearch 發送資料。
如果采集資料不需要任何處理，那麼可以直接發送到Elasticsearch中。
如果采集的資料需要處理，那麼可以發送到Logstash中，處理完成後再發送到Elasticsearch。最後通過Kibana對資料進行一系列的可視化展示。
二. Filebeat
2.1介紹
Filebeat是一款輕量型日志采集器，用于監控、收集伺服器日志檔案
2.2架構
首先Filebeat指定一些日志檔案為資料輸入源，之後使用Harvester（收割機）源源不斷的讀取日志，最後通過Spooler（卷軸）将日志資料傳送到對應的目的地。
2.3安裝
- 使用rz工具将Filebeat壓縮檔案上傳到Linux虛拟機
-
解壓：
tar -zxvf filebeat-7.12.1-linux-x86_64.tar.gz -C /opt/
2.4入門案例
接下來我們使用filebeat讀取一個普通的日志檔案
-
建立一個文本檔案
vim /usr/local/mylog.log # 為該檔案随便添加一句話
Filebeat入門案例 - 在filebeat中建立配置檔案，配置文本檔案的讀取參數
# 進入filebeat檔案夾下
cd /opt/filebeat-7.12.1-linux-x86_64/
# 建立配置檔案
vim mylogconfig.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /opt/log/mylog.log
output.console:
pretty: true
enable: true - 基于配置檔案啟動filebeat
./filebeat -e -c mylogconfig.yml 參數說明：
-e：标準輸出，輸出到控制台
-c：指定配置檔案
4. 向文本檔案追加資料，測試filebeat是否能為增量資料生成日志資料
# 打開另一個會話視窗，進入文本檔案的目錄下
cd /opp/
# 向文本檔案中追加内容，再次檢視filebeat的控制台
echo '科比是我的偶像' >> mylog.log 2.5自定義字段
Filebeat讀取日志檔案後會生成json格式的日志，我們還可以為生成的日志添加一些自定義字段：
# 修改配置檔案：
filebeat.inputs:
- type: log
enabled: true
paths:
- /opt/log/mylog.log
tags: ["mylog","test"]#添加自定義标簽，便于後續處理
fields:
from: mylog
fields_under_root: true #true為添加到根節點，false為添加到子節點中
output.console:
pretty: true
enable: true
# 重新開機
filebeat ./filebeat -e -c mylogconfig.yml
# 向文本檔案追加資料
echo '科比是我的偶像' >> mylog.log 我們可以看到生成的日志資料多了兩個字段
2.6收集Nginx日志
- 安裝Nginx
tar -zxvf nginx-1.21.1.tar.gz -C /usr/local/
# 安裝依賴包
yum -y install gcc gcc-c++ pcre pcre-devel zlib zlib-devel openssl openssl-devel
# 安裝Nginx
./configure
make & make install
# 啟動Nginx
/usr/local/nginx/sbin/nginx
./nginx Nginx的日志檔案在/usr/local/nginx/logs中，正常日志存在access.log中，異常日志存在error.log
中。
- 讀取Nginx日志的配置檔案
# 在filebeat中建立配置檔案
cd /usr/local/filebeat-7.12.1-linux-x86_64/
vim nginxlogconfig.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /usr/local/nginx/logs/*.log
tags: ["nginx"]
output.console:
pretty: true
enable: true
# 啟動filebeat，如果filebeat還在啟動，關閉已啟動的filebeat
./filebeat -e -c nginxlogconfig.yml 2.72.7 Filebeat模闆
- 配置Nginx讀取模闆：
# 檢視Filebeat的日志處理模闆
./filebeat modules list
[root@node0 filebeat]# ./filebeat modules list
Enabled:
nginx
Disabled:
activemq
apache
auditd
aws
azure
barracuda
bluecoat
cef
checkpoint
cisco
coredns
crowdstrike
cyberark
cylance
elasticsearch
envoyproxy
f5
fortinet
gcp
google_workspace
googlecloud
gsuite
haproxy
ibmmq
icinga
iis
imperva
infoblox
iptables
juniper
kafka
kibana
logstash
microsoft
misp
mongodb
mssql
mysql
mysqlenterprise
nats
netflow
netscout
o365
okta
oracle
osquery
panw
pensando
postgresql
proofpoint
rabbitmq
radware
redis
santa
snort
snyk
sonicwall
sophos
squid
suricata
system
threatintel
tomcat
traefik
zeek
zoom
zscaler
# 啟用模闆
./filebeat modules enable nginx
cd modules.d/
vim nginx.yml
# 配置日志處理模闆
- module: nginx
# Access logs
access:
enabled: true
var.paths: ["/usr/local/nginx/logs/access.log"]
# Error logs
error:
enabled: true
var.paths: ["/usr/local/nginx/logs/error.log"] - 修改配置檔案：
vim nginxlogconfig.yml
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: true
output.console:
pretty: true
enable: true - 啟動filebeat，如果filebeat還在啟動，關閉已啟動的filebeat
./filebeat -e -c nginxlogconfig.yml {
"@timestamp": "2021-12-03T04:57:55.131Z",
"@metadata": {
"beat": "filebeat",
"type": "_doc",
"version": "7.12.1",
"pipeline": "filebeat-7.12.1-nginx-access-pipeline"
},
"event": {
"dataset": "nginx.access",
"module": "nginx",
"timezone": "+08:00"
},
"fileset": {
"name": "access"
},
"ecs": {
"version": "1.8.0"
},
"host": {
"name": "node0"
},
"log": {
"offset": 1746,
"file": {
"path": "/usr/local/nginx/logs/access.log"
}
},
"message": "192.168.134.1 - - [03/Dec/2021:12:57:51 +0800] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36\"",
"input": {
"type": "log"
},
"service": {
"type": "nginx"
},
"agent": {
"id": "1043e956-ee86-47e0-8edd-084cab906fd9",
"name": "node0",
"type": "filebeat",
"version": "7.12.1",
"hostname": "node0",
"ephemeral_id": "dda18f29-2c37-49e9-b1b0-755f61cf47ea"
}
} 2.8 将資料輸出到ES中
- 啟動Elasticsearch
- 啟動Kibana，連接配接Elasticsearch
- 修改Filebeat配置檔案：
vim nginxlogconfig.yml
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: true
output.elasticsearch:
hosts: ["node0:9200"] - 啟動filebeat，如果filebeat還在啟動，關閉已啟動的filebeat
./filebeat -e -c nginxlogconfig.yml - 進入Kibana檢視資料
GET /filebeat-7.12.1/_search
{
"query": {
"match_all": {}
}
} {
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 2,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "filebeat-7.12.1",
"_type" : "_doc",
"_id" : "K4XUfn0BLb2Jb20VAvp9",
"_score" : 1.0,
"_source" : {
"agent" : {
"hostname" : "node0",
"name" : "node0",
"id" : "1043e956-ee86-47e0-8edd-084cab906fd9",
"ephemeral_id" : "934e20db-a77a-40ca-af7b-6c9556d7b1af",
"type" : "filebeat",
"version" : "7.12.1"
},
"nginx" : {
"access" : {
"remote_ip_list" : [
"192.168.134.1"
]
}
},
"log" : {
"file" : {
"path" : "/usr/local/nginx/logs/access.log"
},
"offset" : 2319
},
"source" : {
"address" : "192.168.134.1",
"ip" : "192.168.134.1"
},
"fileset" : {
"name" : "access"
},
"url" : {
"original" : "/"
},
"input" : {
"type" : "log"
},
"@timestamp" : "2021-12-03T05:46:00.000Z",
"ecs" : {
"version" : "1.8.0"
},
"related" : {
"ip" : [
"192.168.134.1"
]
},
"service" : {
"type" : "nginx"
},
"host" : {
"name" : "node0"
},
"http" : {
"request" : {
"method" : "GET"
},
"response" : {
"status_code" : 304,
"body" : {
"bytes" : 0
}
},
"version" : "1.1"
},
"event" : {
"ingested" : "2021-12-03T05:46:02.766382193Z",
"timezone" : "+08:00",
"created" : "2021-12-03T05:46:01.739Z",
"kind" : "event",
"module" : "nginx",
"category" : [
"web"
],
"type" : [
"access"
],
"dataset" : "nginx.access",
"outcome" : "success"
},
"user_agent" : {
"original" : "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36",
"os" : {
"name" : "Windows",
"version" : "10",
"full" : "Windows 10"
},
"name" : "Chrome",
"device" : {
"name" : "Other"
},
"version" : "93.0.4577.82"
}
}
},
{
"_index" : "filebeat-7.12.1",
"_type" : "_doc",
"_id" : "LIXUfn0BLb2Jb20VBPp8",
"_score" : 1.0,
"_source" : {
"agent" : {
"hostname" : "node0",
"name" : "node0",
"id" : "1043e956-ee86-47e0-8edd-084cab906fd9",
"type" : "filebeat",
"ephemeral_id" : "934e20db-a77a-40ca-af7b-6c9556d7b1af",
"version" : "7.12.1"
},
"nginx" : {
"access" : {
"remote_ip_list" : [
"192.168.134.1"
]
}
},
"log" : {
"file" : {
"path" : "/usr/local/nginx/logs/access.log"
},
"offset" : 2510
},
"source" : {
"address" : "192.168.134.1",
"ip" : "192.168.134.1"
},
"fileset" : {
"name" : "access"
},
"url" : {
"original" : "/"
},
"input" : {
"type" : "log"
},
"@timestamp" : "2021-12-03T05:46:02.000Z",
"ecs" : {
"version" : "1.8.0"
},
"related" : {
"ip" : [
"192.168.134.1"
]
},
"service" : {
"type" : "nginx"
},
"host" : {
"name" : "node0"
},
"http" : {
"request" : {
"method" : "GET"
},
"response" : {
"status_code" : 304,
"body" : {
"bytes" : 0
}
},
"version" : "1.1"
},
"event" : {
"ingested" : "2021-12-03T05:46:03.760599179Z",
"timezone" : "+08:00",
"created" : "2021-12-03T05:46:02.757Z",
"kind" : "event",
"module" : "nginx",
"category" : [
"web"
],
"type" : [
"access"
],
"dataset" : "nginx.access",
"outcome" : "success"
},
"user_agent" : {
"original" : "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36",
"os" : {
"name" : "Windows",
"version" : "10",
"full" : "Windows 10"
},
"name" : "Chrome",
"device" : {
"name" : "Other"
},
"version" : "93.0.4577.82"
}
}
}
]
}
}