nginx通路日志 logstash 配置檔案執行個體1

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

<code>日志格式：</code>

<code>log_format usgateway '$http_clientip\t$http_ServiceName\t$http_uid\t$http_sid\t[$time_local]'</code>

<code>                                     </code><code>'\t$request\t$status\t$body_bytes_sent\t$connection_requests'</code>

<code>                                     </code><code>'\t$remote_addr\t$http_referer\t$http_user_agent'</code>

<code>                                     </code><code>'\t$request_body\t$request_time\t$msec';</code>

<code>日志執行個體：</code>

<code>10.10.45.152---[23/Jun/2017:17:37:42 +0800]POST /sg HTTP/1.14055765910.10.130.100-Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0){\x22mbrNetName\x22:\x22\xE9\x87\x91\xE6\x98\x9F\x22,\x22nameCn\x22:\x22\xE8\x8A\x92\xE6\x9E\x9C\xE7\xBD\x91\x22,\x22gender\x22:\x2211\x22,\x22birthday\x22:\x222010-10-01\x22,\x22mbrId\x22:\x2235954629\x22,\x22emailAddr\x22:\[email protected]\x22,\x22mobileNo\x22:\x2215902074059\x22}0.0011498210662.427</code>

<code>logstash配置檔案：</code>

<code>input {  </code>

<code>        </code><code>file {  </code>

<code>                </code><code>type =&gt; "uSG_gateway_access"  </code>

<code>                </code><code>path =&gt; ["/usr/local/elk/elklog/nginxlog/log0/uSG_gateway_elk.log"]  </code>

<code>        </code><code>}</code>

<code>} </code>

<code>filter {</code>

<code>ruby {</code>

<code>init =&gt; "@kname = ['http_clientip','http_ServiceName','http_uid','http_sid','time_local','request','status','body_bytes_sent','connection_requests','remote_addr','http_referer','http_user_agent','request_body','request_time','msec']"</code>

<code>code =&gt; "new_event = LogStash::Event.new(Hash[@kname.zip(event.get('message').split(''))])</code>

<code>new_event.remove('@timestamp')</code>

<code>event.append(new_event)"</code>

<code>}</code>

<code>if [request] {</code>

<code>init =&gt; "@kname = ['method','uri','verb']"</code>

<code>code =&gt; "new_event = LogStash::Event.new(Hash[@kname.zip(event.get('request').split(' '))])</code>

<code>event.append(new_event)</code>

<code>"</code>

<code>mutate {</code>

<code>convert =&gt; ["body_bytes_sent" , "integer", "content_length", "integer", "upstream_response_time", "float","request_time", "float"]</code>

<code>        </code><code>grok {</code>

<code>match =&gt; [ "message", "%{IPORHOST:clientip}%{USER}%{USER}%{USER}\[%{HTTPDATE:timestamp}\]"]</code>

<code>date {</code>

<code>match =&gt; [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ]</code>

<code>locale =&gt; "en"</code>

<code>        </code><code>geoip {</code>

<code>source =&gt; "clientip"</code>

<code>    </code><code>}</code>

<code>useragent {</code>

<code>    </code><code>source =&gt; "http_user_agent"</code>

<code>    </code><code>target =&gt; "useragent"</code>

<code>  </code><code>}</code>

<code>output { </code>

<code>elasticsearch {</code>

<code>hosts =&gt; "10.10.45.200:8201"</code>

<code>        </code><code>index =&gt; "logstash-gateway-frontend-%{+YYYY.MM.dd}"</code>

<code>注意：日志分隔符為table鍵。       </code>     本文轉自1321385590 51CTO部落格，原文連結http://blog.51cto.com/linux10000/1941411：，如需轉載請自行聯系原作者