介紹
日志管理系統,大家普遍知道的都是ELK的解決方案,但是ELK要實作認證和一些狀态監控,需要安裝x-pack插件包,但是x-pack是要收費的,當然可以選擇破解,但是比較麻煩。而且ELK是一個解決方案,在其中包含很多軟體,
不單elasticsearch,kibana,logstash,還需要redis或kafaka,收集日志還需要不同的beats,整個結構非常複雜,且占用較多資源,要想完全搞懂需要較長時間。但是很多時候,使用ELK的成本太高,可以使用一些替代方案,
除了ELK還有很多日志管理工具,其中的一個很不錯的日志方案:Graylog,Graylog是一個可以跟ELK相提并論的日志管理的後起之秀,一個開源的 log 收容器,背後的儲存是搭配 mongodb,而搜尋引擎則由 elasticsearch 提供,
自身內建web端,不需要單獨部署,目前最新為3.0版本。
graylog官網官方操作文檔如下:
https://docs.graylog.org/en/3.0/pages/extended_search.html
結構組成
單機
日志收集 Graylog 叢集
日志收集 Graylog 完整架構圖
日志收集 Graylog Docker 安裝
請先安裝Docker
https://mp.csdn.net/mp_blog/creation/editor/115546221
docker-compose部署使用
version: '3'
services:
# MongoDB: https://hub.docker.com/_/mongo/
mongo:
image: mongo:3
networks:
- graylog
# Elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/6.x/docker.html
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.8.10
environment:
- http.host=0.0.0.0
- transport.host=localhost
- network.host=0.0.0.0
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
deploy:
resources:
limits:
memory: 1g
networks:
- graylog
# Graylog: https://hub.docker.com/r/graylog/graylog/
graylog:
image: graylog/graylog:3.3
environment:
# CHANGE ME (must be at least 16 characters)!
- GRAYLOG_PASSWORD_SECRET=somepasswordpepper
# Password: admin
- GRAYLOG_ROOT_PASSWORD_SHA2=8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
- GRAYLOG_HTTP_EXTERNAL_URI=http://127.0.0.1:9000/
networks:
- graylog
depends_on:
- mongo
- elasticsearch
ports:
# Graylog web interface and REST API
- 9000:9000
# Syslog TCP
- 1514:1514
# Syslog UDP
- 1514:1514/udp
# GELF TCP
- 12201:12201
# GELF UDP
- 12201:12201/udp
networks:
graylog:
driver: bridge |
示範
以admin 登入http://IP_adress:9000/ 注:IP_adress是伺服器真實的IP位址
賬号:admin 密碼:admin
日志收集 Graylog graylog 功能組成
日志收集 Graylog http方式測試
curl -XPOST http://IP_adress/gelf -p0 -d '{"short_message":"Hello there", "host":"example.org", "facility":"test", "_foo":"bar"}'
示例:
curl -XPOST http://192.168.179.191:12201/gelf -p0 -d '{"short_message":"Hello there", "host":"example.org", "facility":"test", "_foo":"bar"}' |
日志收集 Graylog logback.xml 接入測試
引入pom
<!--logback gelf日志收集-->
<dependency>
<groupId>biz.paluch.logging</groupId>
<artifactId>logstash-gelf</artifactId>
<version>1.11.1</version>
</dependency> |
logback.xml增加如下配置
<springProperty scope="context" name="APP_NAME" source="spring.application.name"/>
<springProperty scope="context" name="GROYLOG_IP" source="groylog.ip"/>
<springProperty scope="context" name="GROYLOG_PORT" source="groylog.port"/>
<springProperty scope="context" name="GROYLOG_MAXIMUMMESSAGESIZE" source="groylog.maximumMessageSize"/>
<appender name="GELF" class="biz.paluch.logging.gelf.logback.GelfLogbackAppender">
<host>udp:${GROYLOG_IP}</host><!--graylog 伺服器ip-->
<port>${GROYLOG_PORT}</port><!--graylog udp端口-->
<version>1.1</version>
<facility>${APP_NAME}</facility>
<extractStackTrace>true</extractStackTrace>
<filterStackTrace>true</filterStackTrace>
<mdcProfiling>true</mdcProfiling>
<timestampPattern>yyyy-MM-dd HH:mm:ss,SSSS</timestampPattern>
<maximumMessageSize>${GROYLOG_MAXIMUMMESSAGESIZE}</maximumMessageSize>
<!-- This are fields using MDC -->
<mdcFields>mdcField1,mdcField2</mdcFields>
<dynamicMdcFields>mdc.*,(mdc|MDC)fields</dynamicMdcFields>
<includeFullMdc>true</includeFullMdc>
</appender>
<root level="INFO">
<appender-ref ref="GELF"/>
</root> |
application.properties 配置檔案增加如下配置:
groylog.ip=192.168.179.191
groylog.port=12201
groylog.maximumMessageSize=8192 |
示例:
日志收集 Graylog log4j 接入測試
引入pom
<dependency>
<groupId>org.graylog2.log4j2</groupId>
<artifactId>log4j2-gelf</artifactId>
<version>1.3.1</version>
</dependency> |
log4j2-spring.xml 配置
<GELF name="gelf" server="192.168.179.191" port="12201" hostName="cc-test" protocol="UDP">
<PatternLayout pattern="%logger{36} - %msg%n"/>
<Filters>
<Filter type="MarkerFilter" marker="FLOW" onMatch="DENY" onMismatch="NEUTRAL"/>
<Filter type="MarkerFilter" marker="EXCEPTION" onMatch="DENY" onMismatch="ACCEPT"/>
</Filters>
<!-- Additional fields -->
<KeyValuePair key="foo" value="bar"/>
<KeyValuePair key="jvm" value="${java:vm}"/>
</GELF>
<Root level="info">
<AppenderRef ref="gelf"/>
</Root> |