EFK 監控postgresql日志

服務端（192.168.100.226）
https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.13.0-linux-x86_64.tar.gz
https://artifacts.elastic.co/downloads/kibana/kibana-7.13.0-linux-x86_64.tar.gz



## elasticsearch
vi /usr/local/elasticsearch/config/elasticsearch.yml
cluster.name: rao-cluster
node.name: node1
path.data: /usr/local/elasticsearch/data
path.logs: /usr/local/elasticsearch/logs
network.host: 192.168.100.226
http.port: 9200
discovery.seed_hosts: ["192.168.100.226"]
cluster.initial_master_nodes: ["node1"]
# 允許跨域通路，head通路時需要開啟
http.cors.enabled: true
http.cors.allow-origin: "*"

useradd es
echo es|passwd --stdin es

解決伺服器記憶體過小而導緻啟動報錯：
vi /etc/security/limits.conf
* soft nproc 65536
* hard nproc 65536
* soft nofile 65536
* hard nofile 65536
root soft nproc 65536
root hard nproc 65536
root soft nofile 65536
root hard nofile 65536

# 檢視目前值
ulimit -Hn



vi /etc/sysctl.conf 
vm.max_map_count=655360
生效：sysctl -p


vi /usr/local/elasticsearch/bin/start.sh
su - es -c "nohup /usr/local/elasticsearch/bin/elasticsearch &"


結束程序：  ps aux|grep elasticsearch|awk '{print $2}'|xargs kill

通路：http://192.168.100.226:9200/
{
  "name" : "node1",
  "cluster_name" : "rao-cluster",
  "cluster_uuid" : "tugYCx9bT_KVqnFWZihC4A",
  "version" : {
    "number" : "7.13.0",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "5ca8591c6fcdb1260ce95b08a8e023559635c6f3",
    "build_date" : "2021-05-19T22:22:26.081971330Z",
    "build_snapshot" : false,
    "lucene_version" : "8.8.2",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}


## kibana
vi /usr/local/kibana/config/kibana.yml
server.port: 5601
server.host: "192.168.100.226"
server.name: "192.168.100.226"
elasticsearch.hosts: ["http://192.168.100.226:9200"]
i18n.locale: "zh-CN"



vi /usr/local/kibana/bin/start.sh
su - es -c "nohup /usr/local/kibana/bin/kibana &"

通路： http://192.168.100.226:5601/app/home#/



## filebeat 
postgresql資料庫節點（192.68.100.220）
https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.13.0-linux-x86_64.tar.gz


主要包含兩個主要元件：input和harvesters(收割者)

harvester: harvester用于按行讀取單個檔案的内容。每個檔案都會啟動一個harvester，harvester負責打開和關閉檔案。filebeat中還有一個Registrar元件用于記錄檔案的偏移量，即上一次讀取的位置，下一次打開檔案時會從Registrar讀取偏移量然後繼續讀取資料。

input：負責管理harvester并且找到所有符合讀取條件的檔案。如果輸入類型為log，則input會在驅動器上找到與定義的路徑符合的檔案，并會給每個檔案都啟動一個harvester.


vi /usr/local/filebeat/filebeat.yml
setup.template.settings:
  index.number_of_shards: 1
  # 因為es是單節點，是以将副本分片設定為0.否則會報黃
  index.number_of_replicas: 0
output.elasticsearch:
  hosts: ["192.168.100.226:9200"]
  username: "es"
  password: "es"
setup.kibana:
  host: "192.168.100.226:5601"


啟用子產品：
cd /usr/local/filebeat/ && ./filebeat modules enable postgresql


編輯子產品：
vi /usr/local/filebeat/modules.d/postgresql.yml
- module: postgresql

  log:
    enabled: true
    var.paths: ["/data/postgresql/data/pg_log/*.csv"]


加載kibana儀表盤Dashboard，elasticsearch裡生成 Index patterns，Index template,索引生命周期管理政策： /usr/local/filebeat/filebeat setup
測試顯示推送的内容 ：  cd /usr/local/filebeat/  && ./filebeat -e -c filebeat.yml
啟動方式：vi /usr/lib/systemd/system/filebeat.service
[Unit]
Description=filebeat
After=network-online.target
Wants=network-online.target

[Service]
Restart=always
ExecStart=/usr/local/filebeat/filebeat -c /usr/local/filebeat/filebeat.yml

[Install]
WantedBy=multi-user.target

# sysemctl start filebeat

打開加載索引： http://192.168.100.226:5601/app/management/data/index_management/indices