通過日志我們可以及時發現軟體所遇到的問題，但是日志位于伺服器上，不便于觀察，可視化的實時收集日志并分析十分重要，而ELK棧為另外們提供了很好的解決方案。這裡，我們使用Logstash收集Nginx日志，并輸出到Elasticsearch，用Kibana來顯示。
依賴
下載下傳與配置
1. 下載下傳與解壓ELK
cd /usr/local/src/
wget -c https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.1.1.zip
wget -c https://artifacts.elastic.co/downloads/kibana/kibana-6.1.1-linux-x86_64.tar.gz
wget -c https://artifacts.elastic.co/downloads/logstash/logstash-6.1.1.tar.gz
tar -xf kibana-6.1.1-linux-x86_64.tar.gz -C /usr/local/src/
tar -xf logstash-6.1.1.tar.gz -C /usr/local/src/
unzip elasticsearch-6.1.1.zip -d /usr/local/src/
### 建立軟連接配接
ln -sv /usr/local/src/logstash-6.1.1 /usr/local/logstash
ln -sv /usr/local/src/kibana-6.1.1-linux-x86_64/ /usr/local/kibana
ln -sv /usr/local/src/elasticsearch-6.1.1/ /usr/local/elasticsearch
vim /etc/profile.d/els.sh
# 内容如下
PATH=$PATH:/usr/local/elasticsearch/bin/
PATH=$PATH:/usr/local/kibana/bin/
PATH=$PATH:/usr/local/logstash/bin/
#
source /etc/profile.d/els.sh
2. 配置elasticsearch需要的系統配置
ulimit -n 65536
vim /etc/security/limits.conf
#limits.conf内容如下
* soft nofile 65536
* hard nofile 65536
- ELK安裝X-pack擴充(可選)
logstash-plugin install x-pack
elasticsearch-plugin install -xpack
kibana-plugin install x-pack
設定ELK棧的密碼
/usr/local/elasticsearch/bin/x-pack/setup-passwords interactive
4. 配置nginx的日志格式與日志路徑
log_format main '{"@timestamp":"$time_iso8601",'
'"host":"$server_addr",'
' "clientip" : "$remote_addr",'
' "size" : "$body_bytes_sent" ,'
'"respnsetime":"$request_time",'
'"upstremtime":"$upstream_response_time",'
'"upstremhost":"$upstream_addr",'
'"httphost":"$host",'
'"referer":"$http_referer",'
'"xff":"$http_x_forwarded_for",'
'"agent":"$http_user_agent",'
'"clientip":"$remote_addr",'
'"request":"$request",'
'"uri":"$uri",'
'"status":"$status"}';
access_log /var/logs/nginx/access.log main;
error_log /var/logs/nginx/error.log error;
5 . 寫Logstash的日志收集檔案
input {
file {
type => "nginx-error-log"
path => "/usr/local/openresty/nginx/logs/error.log"
}
file {
type => "nginx-access-log"
path => "/usr/local/openresty/nginx/logs/access.log"
codec => json
}
}
filter {
if [type] =~ "nginx-error-log"
{
grok {
match => {
"message" => "(?<datetime>\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2}) \[(?<errtype>\w+)\] \S+: \*\d+ (?<errmsg>[^,]+), (?<errinfo>.*$)"
}
}
mutate {
rename => {
"message" => "z_message"
"host" => "fromhost"
}
}
} else if [type] =~ "nginx-access-log" {
mutate {
split => {"upstremtime" => ","}
}
mutate {
convert => { "upstremtime" => "float"}
}
}
if [errinfo]
{
ruby {
code => "
new_event = LogStash::Event.new(Hash[event.get('errinfo').split(', ').map{ |l| l.split(': ') }])
new_event.remove('@timestamp')
event.append(new_event)
"
}
grok {
match => {
"request" => '"%{WORD:verb} %{URIPATHPARAM:urlpathparam}?(?: HTTP/%{NUMBER:httpversion})"'
}
patterns_dir => ["/home/data/logstash/patterns/"]
#patterns_dir => ["/usr/local/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/"]
remove_field => [ "errinfo","request" ]
}
}
}
output {
#elasticsearch { host => localhost }
stdout { codec => rubydebug }
if [type] =~ "nginx-error-log" {
#redis{
# data_type => "channel"
# key => "logstash-nginx-%{+yyyy.MM.dd}"
#}
elasticsearch {
hosts => [ "${ELASTIC_HOST:'http://127.0.0.1:9200'}"]
index => "logstash-nginx-error-log-%{+YYYY.MM.dd}"
document_type => "data"
user => "xxx"
password => "xxx"
}
}else if [type] =~ "nginx-access-log" {
elasticsearch {
hosts => [ "${ELASTIC_HOST:'http://127.0.0.1:9200'}"]
index => "logstash-nginx-access-log-%{+YYYY.MM.dd}"
document_type => "data"
user => "xxx"
password => "xxx"
}
}
}
可以命名為
logstash.conf
6 . 如部署到伺服器，可使用FTP,scp,rsync,ansible等等工具上傳到伺服器(可選)
7 . 運作logstash進行測試，測試前，因為我們的logstash檔案有使用到環境變量
ELASTIC_HOST
，主機位址可以自己進行定義。
export ELASTIC_HOST=[某伺服器位址]
logstash -f -t logstash.conf ##檢查配置是否正确
logstash -f logstash.conf ## 運作logstash
8. 如果想以服務方式運作logstash
cd /usr/local/logstash/bin/
./system-install --help
Usage: system-install [OPTIONSFILE] [STARTUPTYPE] [VERSION]
NOTE: These arguments are ordered, and co-dependent
OPTIONSFILE: Full path to a startup.options file
OPTIONSFILE is required if STARTUPTYPE is specified, but otherwise looks first
in /usr/local/logstash/config/startup.options and then /etc/logstash/startup.options
Last match wins
STARTUPTYPE: e.g. sysv, upstart, systemd, etc.
OPTIONSFILE is required to specify a STARTUPTYPE.
VERSION: The specified version of STARTUPTYPE to use. The default is usually
preferred here, so it can safely be omitted.
Both OPTIONSFILE & STARTUPTYPE are required to specify a VERSION.
我使用的為CentOS7系統，使用Systemd。如果想要在Systemd中定義環境變量那麼。
cd /etc/systemd/system/
vim logstash.service
### 定義變量的行
Environment="KEY=VALUE"
ExecStart=/usr/local/logstash/bin/logstash "--path.settings" "/usr/share/logstash/config" ###修改服務執行腳本
9 . 添加系統的Logstash使用者
useradd -s /sbin/nologin -M logstash
chown -R logstash.logstash /usr/local/logstash ## 改變logstash的屬主屬組，否則服務啟動會報錯。
10. 啟動logstash
systemctrl start logstash
journalctl -u logstash ## 檢視logstash的日志
額外
想要監控logstash的情況的話
在logstash的配置檔案中
cd /usr/local/logstash/config
xpack.monitoring.elasticsearch.url : "Elasticsearch伺服器位址"
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: xxx
最後
Logstash的功能很強大，還有一些其餘的方案可以嘗試，可以配合ELK與Docker一起使用，達到更靈活的效果
![Cloud Studio初體驗[圖]](data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=)