多行日志事件

類似于mysql slow log 這一類的日志并非一次一行，而是多行

Logstash 也可以處理，隻是目前此功能還比較弱

配置如下

[root@h102 etc]# cat logstash-multiline.conf
input {
  stdin {
    codec => multiline {
      pattern => "^# User@Host:"
      negate => true
      what => previous
    }
  }
}

output {
  elasticsearch { hosts => ["localhost:9200"] }
  stdout { codec => rubydebug }
}
[root@h102 etc]# time /opt/logstash/bin/logstash -f logstash-multiline.conf -t 
Configuration OK

real	0m18.807s
user	0m30.841s
sys	0m2.290s
[root@h102 etc]#

複制

pattern 為正則比對
negate 為反轉，隻能為 true 或 false , 預設為 false ，代表不反轉
what 為處理行為，隻能為 previous 或 next ，為 previous 時，代表比對此模式的行屬于前面的事件内容，為 next 時，代表比對此模式的行屬于後面的事件内容

上面的配置表明，如果不以 # User@Host: 開頭的行都屬于前面的事件内容

開啟 Logstash 進行測試

[root@h102 etc]# time /opt/logstash/bin/logstash -f logstash-multiline.conf 
Settings: Default filter workers: 1
Logstash startup completed
# Time: 150710 16:37:53
# User@Host: root[root] @ localhost []
{
    "@timestamp" => "2016-01-05T14:01:57.953Z",
       "message" => "# Time: 150710 16:37:53",
      "@version" => "1",
          "host" => "h102.temp"
}
# Thread_id: 113  Schema: mysqlslap  Last_errno: 0  Killed: 0
# Query_time: 1.134132  Lock_time: 0.000029  Rows_sent: 1  Rows_examined: 1  Rows_affected: 0  Rows_read: 1
# Bytes_sent: 2168
SET timestamp=1436517473;
SELECT intcol1,intcol2,intcol3,intcol4,intcol5,charcol1,charcol2,charcol3,charcol4,charcol5,charcol6,charcol7,charcol8,charcol9,charco
l10 FROM t1 WHERE id =  '31';
# User@Host: root[root] @ localhost []
{
    "@timestamp" => "2016-01-05T14:02:03.773Z",
       "message" => "# User@Host: root[root] @ localhost []\n# Thread_id: 113  Schema: mysqlslap  Last_errno: 0  Killed: 0\n# Query_time: 1.134132  Lock_time: 0.000029  Rows_sent: 1  Rows_examined: 1  Rows_affected: 0  Rows_read: 1\n# Bytes_sent: 2168\nSET timestamp=1436517473;\nSELECT intcol1,intcol2,intcol3,intcol4,intcol5,charcol1,charcol2,charcol3,charcol4,charcol5,charcol6,charcol7,charcol8,charcol9,charco\nl10 FROM t1 WHERE id =  '31';",
      "@version" => "1",
          "tags" => [
        [0] "multiline"
    ],
          "host" => "h102.temp"
}
# Thread_id: 110  Schema: mysqlslap  Last_errno: 0  Killed: 0
# Query_time: 1.385901  Lock_time: 0.000037  Rows_sent: 1  Rows_examined: 1  Rows_affected: 0  Rows_read: 1
# Bytes_sent: 2167
SET timestamp=1436517473;
SELECT intcol1,intcol2,intcol3,intcol4,intcol5,charcol1,charcol2,charcol3,charcol4,charcol5,charcol6,charcol7,charcol8,charcol9,charco
l10 FROM t1 WHERE id =  '43';
# User@Host: root[root] @ localhost []
{
    "@timestamp" => "2016-01-05T14:02:51.114Z",
       "message" => "# User@Host: root[root] @ localhost []\n# Thread_id: 110  Schema: mysqlslap  Last_errno: 0  Killed: 0\n# Query_time: 1.385901  Lock_time: 0.000037  Rows_sent: 1  Rows_examined: 1  Rows_affected: 0  Rows_read: 1\n# Bytes_sent: 2167\nSET timestamp=1436517473;\nSELECT intcol1,intcol2,intcol3,intcol4,intcol5,charcol1,charcol2,charcol3,charcol4,charcol5,charcol6,charcol7,charcol8,charcol9,charco\nl10 FROM t1 WHERE id =  '43';",
      "@version" => "1",
          "tags" => [
        [0] "multiline"
    ],
          "host" => "h102.temp"
}

複制

發現在輸入 # User@Host: 之前，所有的行都被進行壓棧處理，輸入此條資訊後，前面的資訊進行了一個完結，又重新等待新的輸入，直到遇到又一個 # User@Host:

Tip: 暫時沒有很好的辦法處理諸如 # Time: 150710 16:37:53 的行，這樣的行被算在了前一條的事件日志中

指令彙總

java -version
/opt/logstash/bin/logstash -e 'input { stdin { } } output { stdout {} }'
cat first-pipeline.conf
/opt/logstash/bin/logstash -f first-pipeline.conf -t
/opt/logstash/bin/logstash -f first-pipeline.conf
curl -XGET 'localhost:9200/logstash-2015.12.23/_search?q=response=404'
curl -XGET 'localhost:9200/logstash-2015.12.23/_search?q=response=304&pretty'
curl -XGET 'localhost:9200/logstash-2015.12.23/_search?q=geoip.city_name=Buffalo&pretty'
grep -v "#" /etc/filebeat/filebeat.yml | grep -v "^$"
/etc/init.d/filebeat start
/etc/init.d/filebeat status
cat logstash-filebeat-es-simple.conf
/opt/logstash/bin/logstash -f logstash-filebeat-es-simple.conf
curl localhost:9200/_cat/indices?v
curl -XGET 'localhost:9200/filebeat-2016.01.05/_search?q=message=2935&pretty'
cat logstash-syslog.conf
/opt/logstash/bin/logstash -f logstash-syslog.conf
telnet localhost 5000
curl -XGET 'localhost:9200/logstash-2016.12.23/_search?q=message=louis&pretty'
cat logstash-multiline.conf
time /opt/logstash/bin/logstash -f logstash-multiline.conf

原文位址

Logstash 基礎12指令彙總

多行日志事件

指令彙總