集中式日志分析系統ELK安裝部署

ELK安裝

安裝Elasticsearch

安裝Logstash

安裝Kibana

安裝Filebeat

（步驟略，參考官網即可https://www.elastic.co/guide/en/elasticsearch/reference/6.0/getting-started.html）

ELK叢集配置

elasticsearch node-1

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
cluster.name: my-application

# ------------------------------------ Node ------------------------------------
node.name: node-1

# ----------------------------------- Memory -----------------------------------
bootstrap.memory_lock: true
bootstrap.system_call_filter: false

# ---------------------------------- Network -----------------------------------
network.host: 0.0.0.0
http.port: 9200

# --------------------------------- Discovery ----------------------------------
discovery.zen.ping.unicast.hosts: ["192.168.20.132","192.168.20.131"]
discovery.zen.minimum_master_nodes: 1


#form www.elastic.co  ->  Docs  ->  X-Pack -> install
#action.auto_create_index: .security,.security-6,monitoring*,.watches,.triggered_watches,.watcher-history*,.ml*

#install ssl 
#xpack.ssl.key: /opt/elasticsearch-6.1.0/config/certs/mynetstation/mynetstation.key 
#xpack.ssl.certificate: /opt/elasticsearch-6.1.0/config/certs/mynetstation/mynetstation.crt 
#xpack.ssl.certificate_authorities: /opt/elasticsearch-6.1.0/config/certs/ca/ca.crt 
#xpack.security.transport.ssl.enabled: false

           

elasticsearch node-2

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
cluster.name: my-application

# ------------------------------------ Node ------------------------------------
node.name: node-2

# ----------------------------------- Memory -----------------------------------
bootstrap.memory_lock: true
bootstrap.system_call_filter: false

# ---------------------------------- Network -----------------------------------
network.host: 0.0.0.0
http.port: 9200

# --------------------------------- Discovery ----------------------------------
discovery.zen.ping.unicast.hosts: ["192.168.20.132", "192.168.20.131"]
discovery.zen.minimum_master_nodes: 1
           

logstash

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    id => "esUser"
    hosts => ["192.168.20.132:9200"]
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}
           

filebeat

###################### Filebeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common
# options. The filebeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html

# For more available modules and options, please see the filebeat.reference.yml sample
# configuration file.
filebeat:
- idle_timeout : "5s" 
#=========================== Filebeat prospectors ============================= 
filebeat.prospectors:
- type: log
  enabled: true
  paths:
    - /var/log/*.log

#============================= Filebeat modules ===============================
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 10s

#==================== Elasticsearch template setting ==========================
setup.template.settings:
  index.number_of_shards: 3

#============================== Kibana =====================================
setup.kibana:
  host: "192.168.20.132:5601"

#----------------------------- Logstash output --------------------------------
output.logstash:
  hosts: ["192.168.20.132:5044"]
           

kibana

# Kibana is served by a back end server. This setting specifies the port to use.
#server.port: 5601

# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "0.0.0.0"
elasticsearch.url: "http://192.168.20.132:9200"
           

注意事項

es不推薦用root使用者啟動，需要建立一個使用者并授予啟動es權限

es啟動會自動檢查啟動使用者配置，還需要使用者設定Lock記憶體大小

vi /etc/security/limits.conf

esUser - nofile 65536
esUser soft memlock unlimited
esUser hard memlock unlimited