實驗簡介:
由logstash收集nginx日志寫入kafka中,在由另一台主機logstash讀取kafka日志寫入elasticsearch
一 logstash收集日志寫入kafka
1.1.1 編寫logstash配置檔案
[root@localhost ~]# cat /etc/logstash/conf.d/nginx-kafka.conf
input {
file {
path => "/opt/vhosts/fatai/logs/access_json.log"
start_position => "beginning"
type => "nginx-accesslog"
codec => "json"
stat_interval => "2"
}
}
output {
kafka {
bootstrap_servers => "192.168.10.10:9092"
topic_id => 'nginx-access-kafkaceshi'
codec => "json"
}
}
1.1.2 驗證并重新開機logstash
[root@localhost ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/nginx-kafka.conf -t
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
Configuration OK
[root@localhost ~]# systemctl restart logstash.service
1.1.3 kafka端驗證主題
[root@DNS-Server tools]# /tools/kafka/bin/kafka-topics.sh --list --zookeeper 192.168.10.10:2181,192.168.10.167:2181,192.168.10.171:2181
nginx-access-kafkaceshi
二 logstash收集kafka日志并寫入elk
[root@Docker ~]# cat /etc/logstash/conf.d/nginx_kafka.conf
input {
kafka {
bootstrap_servers => "192.168.10.10:9092" #kafka位址
topics => "nginx-access-kafkaceshi" #定義主題
group_id => "nginx-access-kafkaceshi" #自定義
codec => "json" #指定編碼
consumer_threads => 1 #消費者線程
decorate_events => true #要不要加kafka标記
}
}
output {
if [type] == "nginx-accesslog"{ #type 是收集時候logstash定義的
elasticsearch {
hosts => ["192.168.10.10:9200"]
index=> "nginx-accesslog-kafka-test-%{+YYYY.MM.dd}"
}
}
}
1.1.2 檢測并重新開機
[root@Docker ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/nginx_kafka.conf -t
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
Configuration OK
[root@Docker ~]# systemctl restart logstash.service
1.1.3 elasticsearch驗證
![](https://img.laitimes.com/img/_0nNw4CM6IyYiwiM6ICdiwiIn5GcucTOxgDO3QTM30COxETNyUzM1EDOwgDM4EDMy0SN0MTN0QTMvwFOwgTMwIzLcVDNzUDN0EzLcd2bsJ2Lc12bj5ycn9Gbi52YugTMwIzcldWYtl2Lc9CX6MHc0RHaiojIsJye.png)
作者:闫世成
出處:http://cnblogs.com/yanshicheng
本文版權歸作者和部落格園共有,歡迎轉載,但未經作者同意必須保留此段聲明,且在文章頁面明顯位置給出原文連接配接。如有問題或建議,請聯系上述郵箱,非常感謝。