一.介绍
轻量型数据采集器Beats是一个免费且开放的平台，集合了多种单一用途数据采集器。它们从成百上千或成千上万台机器和系统向 Logstash 或 Elasticsearch 发送数据。
如果采集数据不需要任何处理，那么可以直接发送到Elasticsearch中。
如果采集的数据需要处理，那么可以发送到Logstash中，处理完成后再发送到Elasticsearch。最后通过Kibana对数据进行一系列的可视化展示。
二. Filebeat
2.1介绍
Filebeat是一款轻量型日志采集器，用于监控、收集服务器日志文件
2.2架构
首先Filebeat指定一些日志文件为数据输入源，之后使用Harvester（收割机）源源不断的读取日志，最后通过Spooler（卷轴）将日志数据传送到对应的目的地。
2.3安装
- 使用rz工具将Filebeat压缩文件上传到Linux虚拟机
-
解压：
tar -zxvf filebeat-7.12.1-linux-x86_64.tar.gz -C /opt/
2.4入门案例
接下来我们使用filebeat读取一个普通的日志文件
-
创建一个文本文件
vim /usr/local/mylog.log # 为该文件随便添加一句话
Filebeat入门案例 - 在filebeat中创建配置文件，配置文本文件的读取参数
# 进入filebeat文件夹下
cd /opt/filebeat-7.12.1-linux-x86_64/
# 创建配置文件
vim mylogconfig.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /opt/log/mylog.log
output.console:
pretty: true
enable: true - 基于配置文件启动filebeat
./filebeat -e -c mylogconfig.yml 参数说明：
-e：标准输出，输出到控制台
-c：指定配置文件
4. 向文本文件追加数据，测试filebeat是否能为增量数据生成日志数据
# 打开另一个会话窗口，进入文本文件的目录下
cd /opp/
# 向文本文件中追加内容，再次查看filebeat的控制台
echo '科比是我的偶像' >> mylog.log 2.5自定义字段
Filebeat读取日志文件后会生成json格式的日志，我们还可以为生成的日志添加一些自定义字段：
# 修改配置文件：
filebeat.inputs:
- type: log
enabled: true
paths:
- /opt/log/mylog.log
tags: ["mylog","test"]#添加自定义标签，便于后续处理
fields:
from: mylog
fields_under_root: true #true为添加到根节点，false为添加到子节点中
output.console:
pretty: true
enable: true
# 重启
filebeat ./filebeat -e -c mylogconfig.yml
# 向文本文件追加数据
echo '科比是我的偶像' >> mylog.log 我们可以看到生成的日志数据多了两个字段
2.6收集Nginx日志
- 安装Nginx
tar -zxvf nginx-1.21.1.tar.gz -C /usr/local/
# 安装依赖包
yum -y install gcc gcc-c++ pcre pcre-devel zlib zlib-devel openssl openssl-devel
# 安装Nginx
./configure
make & make install
# 启动Nginx
/usr/local/nginx/sbin/nginx
./nginx Nginx的日志文件在/usr/local/nginx/logs中，正常日志存在access.log中，异常日志存在error.log
中。
- 读取Nginx日志的配置文件
# 在filebeat中创建配置文件
cd /usr/local/filebeat-7.12.1-linux-x86_64/
vim nginxlogconfig.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /usr/local/nginx/logs/*.log
tags: ["nginx"]
output.console:
pretty: true
enable: true
# 启动filebeat，如果filebeat还在启动，关闭已启动的filebeat
./filebeat -e -c nginxlogconfig.yml 2.72.7 Filebeat模板
- 配置Nginx读取模板：
# 查看Filebeat的日志处理模板
./filebeat modules list
[root@node0 filebeat]# ./filebeat modules list
Enabled:
nginx
Disabled:
activemq
apache
auditd
aws
azure
barracuda
bluecoat
cef
checkpoint
cisco
coredns
crowdstrike
cyberark
cylance
elasticsearch
envoyproxy
f5
fortinet
gcp
google_workspace
googlecloud
gsuite
haproxy
ibmmq
icinga
iis
imperva
infoblox
iptables
juniper
kafka
kibana
logstash
microsoft
misp
mongodb
mssql
mysql
mysqlenterprise
nats
netflow
netscout
o365
okta
oracle
osquery
panw
pensando
postgresql
proofpoint
rabbitmq
radware
redis
santa
snort
snyk
sonicwall
sophos
squid
suricata
system
threatintel
tomcat
traefik
zeek
zoom
zscaler
# 启用模板
./filebeat modules enable nginx
cd modules.d/
vim nginx.yml
# 配置日志处理模板
- module: nginx
# Access logs
access:
enabled: true
var.paths: ["/usr/local/nginx/logs/access.log"]
# Error logs
error:
enabled: true
var.paths: ["/usr/local/nginx/logs/error.log"] - 修改配置文件：
vim nginxlogconfig.yml
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: true
output.console:
pretty: true
enable: true - 启动filebeat，如果filebeat还在启动，关闭已启动的filebeat
./filebeat -e -c nginxlogconfig.yml {
"@timestamp": "2021-12-03T04:57:55.131Z",
"@metadata": {
"beat": "filebeat",
"type": "_doc",
"version": "7.12.1",
"pipeline": "filebeat-7.12.1-nginx-access-pipeline"
},
"event": {
"dataset": "nginx.access",
"module": "nginx",
"timezone": "+08:00"
},
"fileset": {
"name": "access"
},
"ecs": {
"version": "1.8.0"
},
"host": {
"name": "node0"
},
"log": {
"offset": 1746,
"file": {
"path": "/usr/local/nginx/logs/access.log"
}
},
"message": "192.168.134.1 - - [03/Dec/2021:12:57:51 +0800] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36\"",
"input": {
"type": "log"
},
"service": {
"type": "nginx"
},
"agent": {
"id": "1043e956-ee86-47e0-8edd-084cab906fd9",
"name": "node0",
"type": "filebeat",
"version": "7.12.1",
"hostname": "node0",
"ephemeral_id": "dda18f29-2c37-49e9-b1b0-755f61cf47ea"
}
} 2.8 将数据输出到ES中
- 启动Elasticsearch
- 启动Kibana，连接Elasticsearch
- 修改Filebeat配置文件：
vim nginxlogconfig.yml
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: true
output.elasticsearch:
hosts: ["node0:9200"] - 启动filebeat，如果filebeat还在启动，关闭已启动的filebeat
./filebeat -e -c nginxlogconfig.yml - 进入Kibana查看数据
GET /filebeat-7.12.1/_search
{
"query": {
"match_all": {}
}
} {
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 2,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "filebeat-7.12.1",
"_type" : "_doc",
"_id" : "K4XUfn0BLb2Jb20VAvp9",
"_score" : 1.0,
"_source" : {
"agent" : {
"hostname" : "node0",
"name" : "node0",
"id" : "1043e956-ee86-47e0-8edd-084cab906fd9",
"ephemeral_id" : "934e20db-a77a-40ca-af7b-6c9556d7b1af",
"type" : "filebeat",
"version" : "7.12.1"
},
"nginx" : {
"access" : {
"remote_ip_list" : [
"192.168.134.1"
]
}
},
"log" : {
"file" : {
"path" : "/usr/local/nginx/logs/access.log"
},
"offset" : 2319
},
"source" : {
"address" : "192.168.134.1",
"ip" : "192.168.134.1"
},
"fileset" : {
"name" : "access"
},
"url" : {
"original" : "/"
},
"input" : {
"type" : "log"
},
"@timestamp" : "2021-12-03T05:46:00.000Z",
"ecs" : {
"version" : "1.8.0"
},
"related" : {
"ip" : [
"192.168.134.1"
]
},
"service" : {
"type" : "nginx"
},
"host" : {
"name" : "node0"
},
"http" : {
"request" : {
"method" : "GET"
},
"response" : {
"status_code" : 304,
"body" : {
"bytes" : 0
}
},
"version" : "1.1"
},
"event" : {
"ingested" : "2021-12-03T05:46:02.766382193Z",
"timezone" : "+08:00",
"created" : "2021-12-03T05:46:01.739Z",
"kind" : "event",
"module" : "nginx",
"category" : [
"web"
],
"type" : [
"access"
],
"dataset" : "nginx.access",
"outcome" : "success"
},
"user_agent" : {
"original" : "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36",
"os" : {
"name" : "Windows",
"version" : "10",
"full" : "Windows 10"
},
"name" : "Chrome",
"device" : {
"name" : "Other"
},
"version" : "93.0.4577.82"
}
}
},
{
"_index" : "filebeat-7.12.1",
"_type" : "_doc",
"_id" : "LIXUfn0BLb2Jb20VBPp8",
"_score" : 1.0,
"_source" : {
"agent" : {
"hostname" : "node0",
"name" : "node0",
"id" : "1043e956-ee86-47e0-8edd-084cab906fd9",
"type" : "filebeat",
"ephemeral_id" : "934e20db-a77a-40ca-af7b-6c9556d7b1af",
"version" : "7.12.1"
},
"nginx" : {
"access" : {
"remote_ip_list" : [
"192.168.134.1"
]
}
},
"log" : {
"file" : {
"path" : "/usr/local/nginx/logs/access.log"
},
"offset" : 2510
},
"source" : {
"address" : "192.168.134.1",
"ip" : "192.168.134.1"
},
"fileset" : {
"name" : "access"
},
"url" : {
"original" : "/"
},
"input" : {
"type" : "log"
},
"@timestamp" : "2021-12-03T05:46:02.000Z",
"ecs" : {
"version" : "1.8.0"
},
"related" : {
"ip" : [
"192.168.134.1"
]
},
"service" : {
"type" : "nginx"
},
"host" : {
"name" : "node0"
},
"http" : {
"request" : {
"method" : "GET"
},
"response" : {
"status_code" : 304,
"body" : {
"bytes" : 0
}
},
"version" : "1.1"
},
"event" : {
"ingested" : "2021-12-03T05:46:03.760599179Z",
"timezone" : "+08:00",
"created" : "2021-12-03T05:46:02.757Z",
"kind" : "event",
"module" : "nginx",
"category" : [
"web"
],
"type" : [
"access"
],
"dataset" : "nginx.access",
"outcome" : "success"
},
"user_agent" : {
"original" : "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36",
"os" : {
"name" : "Windows",
"version" : "10",
"full" : "Windows 10"
},
"name" : "Chrome",
"device" : {
"name" : "Other"
},
"version" : "93.0.4577.82"
}
}
}
]
}
}