EFK教程(3) - ElasticSearch冷熱資料分離目錄用途架構192.168.1.51 elasticsearch-data部署雙執行個體192.168.1.52 elasticsearch-data部署雙執行個體192.168.1.53 elasticsearch-data部署雙執行個體測試

基于ElasticSearch多執行個體架構，實作資源合理配置設定、冷熱資料分離

作者：“發颠的小狼”，歡迎轉載與投稿

用途

前情提要：

▷ 在第一篇《EFK教程 - 快速入門指南》中，闡述了EFK的安裝部署，其中ES的架構為三節點，即master、ingest、data角色同時部署在三台伺服器上。

▷ 在第二篇《EFK教程 - ElasticSearch高性能高可用架構》中，闡述了EFK的data/ingest/master角色的用途及分别部署三節點，在實作性能最大化的同時保障高可用

前兩篇文章，ES叢集中隻存在一個執行個體，而在本文中，将在一個叢集中部署多個ES執行個體，來實作資源合理配置設定。例如data伺服器存在SSD與SAS硬碟，可以将熱資料存放到SSD，而冷資料存放到SAS，實作資料冷熱分離。

在本文中，将為data伺服器建立2個執行個體，分别基于SSD和基于SAS硬碟，将nginx的9月份索引放在SAS盤上，其它的全放在SSD盤上

架構

架構圖

伺服器配置

192.168.1.51 elasticsearch-data部署雙執行個體

索引遷移（此步不能忽略）：将192.168.1.51上的索引放到其它2台data節點上

curl -X PUT "192.168.1.31:9200/*/_settings?pretty" -H 'Content-Type: application/json' -d'
{
  "index.routing.allocation.include._ip": "192.168.1.52,192.168.1.53"
}'

确認目前索引存儲位置：确認所有索引不在192.168.1.51節點上

curl "http://192.168.1.31:9200/_cat/shards?h=n"

停掉192.168.1.51的程序，修改目錄結構及配置：請自行按SSD和SAS硬碟挂載好資料盤

# 安裝包下載下傳和部署請參考第一篇《EFK教程 - 快速入門指南》
cd /opt/software/
tar -zxvf elasticsearch-7.3.2-linux-x86_64.tar.gz
mv /opt/elasticsearch /opt/elasticsearch-SAS
mv elasticsearch-7.3.2 /opt/
mv /opt/elasticsearch-7.3.2 /opt/elasticsearch-SSD
chown elasticsearch.elasticsearch /opt/elasticsearch-* -R
rm -rf /data/SAS/*
chown elasticsearch.elasticsearch /data/* -R
mkdir -p /opt/logs/elasticsearch-SAS
mkdir -p /opt/logs/elasticsearch-SSD
chown elasticsearch.elasticsearch /opt/logs/* -R

SAS執行個體/opt/elasticsearch-SAS/config/elasticsearch.yml配置

cluster.name: my-application
node.name: 192.168.1.51-SAS
path.data: /data/SAS
path.logs: /opt/logs/elasticsearch-SAS
network.host: 192.168.1.51

http.port: 9200
transport.port: 9300
# discovery.seed_hosts和cluster.initial_master_nodes 一定要帶上端口号，不然會走http.port和transport.port端口
discovery.seed_hosts: ["192.168.1.31:9300","192.168.1.32:9300","192.168.1.33:9300"]
cluster.initial_master_nodes: ["192.168.1.31:9300","192.168.1.32:9300","192.168.1.33:9300"]
http.cors.enabled: true
http.cors.allow-origin: "*"

node.master: false
node.ingest: false
node.data: true

# 本機隻允行啟2個執行個體
node.max_local_storage_nodes: 2

SSD執行個體/opt/elasticsearch-SSD/config/elasticsearch.yml配置

cluster.name: my-application
node.name: 192.168.1.51-SSD
path.data: /data/SSD
path.logs: /opt/logs/elasticsearch-SSD
network.host: 192.168.1.51

http.port: 9201
transport.port: 9301
# discovery.seed_hosts和cluster.initial_master_nodes 一定要帶上端口号，不然會走http.port和transport.port端口
discovery.seed_hosts: ["192.168.1.31:9300","192.168.1.32:9300","192.168.1.33:9300"]
cluster.initial_master_nodes: ["192.168.1.31:9300","192.168.1.32:9300","192.168.1.33:9300"]
http.cors.enabled: true
http.cors.allow-origin: "*"

node.master: false
node.ingest: false
node.data: true

# 本機隻允行啟2個執行個體
node.max_local_storage_nodes: 2

SAS執行個體和SSD執行個體啟動方式

sudo -u elasticsearch /opt/elasticsearch-SAS/bin/elasticsearch
sudo -u elasticsearch /opt/elasticsearch-SSD/bin/elasticsearch

确認SAS和SSD已啟2執行個體

curl "http://192.168.1.31:9200/_cat/nodes?v"

192.168.1.52 elasticsearch-data部署雙執行個體

索引遷移（此步不能忽略）：将192.168.1.52上的索引放到其它2台data節點上

curl -X PUT "192.168.1.31:9200/*/_settings?pretty" -H 'Content-Type: application/json' -d'
{
  "index.routing.allocation.include._ip": "192.168.1.51,192.168.1.53"
}'

确認目前索引存儲位置：确認所有索引不在192.168.1.52節點上

curl "http://192.168.1.31:9200/_cat/shards?h=n"

停掉192.168.1.52的程序，修改目錄結構及配置：請自行按SSD和SAS硬碟挂載好資料盤

# 安裝包下載下傳和部署請參考第一篇《EFK教程 - 快速入門指南》
cd /opt/software/
tar -zxvf elasticsearch-7.3.2-linux-x86_64.tar.gz
mv /opt/elasticsearch /opt/elasticsearch-SAS
mv elasticsearch-7.3.2 /opt/
mv /opt/elasticsearch-7.3.2 /opt/elasticsearch-SSD
chown elasticsearch.elasticsearch /opt/elasticsearch-* -R
rm -rf /data/SAS/*
chown elasticsearch.elasticsearch /data/* -R
mkdir -p /opt/logs/elasticsearch-SAS
mkdir -p /opt/logs/elasticsearch-SSD
chown elasticsearch.elasticsearch /opt/logs/* -R

cluster.name: my-application
node.name: 192.168.1.52-SAS
path.data: /data/SAS
path.logs: /opt/logs/elasticsearch-SAS
network.host: 192.168.1.52

http.port: 9200
transport.port: 9300
# discovery.seed_hosts和cluster.initial_master_nodes 一定要帶上端口号，不然會走http.port和transport.port端口
discovery.seed_hosts: ["192.168.1.31:9300","192.168.1.32:9300","192.168.1.33:9300"]
cluster.initial_master_nodes: ["192.168.1.31:9300","192.168.1.32:9300","192.168.1.33:9300"]
http.cors.enabled: true
http.cors.allow-origin: "*"

node.master: false
node.ingest: false
node.data: true

# 本機隻允行啟2個執行個體
node.max_local_storage_nodes: 2

cluster.name: my-application
node.name: 192.168.1.52-SSD
path.data: /data/SSD
path.logs: /opt/logs/elasticsearch-SSD
network.host: 192.168.1.52

http.port: 9201
transport.port: 9301
# discovery.seed_hosts和cluster.initial_master_nodes 一定要帶上端口号，不然會走http.port和transport.port端口
discovery.seed_hosts: ["192.168.1.31:9300","192.168.1.32:9300","192.168.1.33:9300"]
cluster.initial_master_nodes: ["192.168.1.31:9300","192.168.1.32:9300","192.168.1.33:9300"]
http.cors.enabled: true
http.cors.allow-origin: "*"

node.master: false
node.ingest: false
node.data: true

# 本機隻允行啟2個執行個體
node.max_local_storage_nodes: 2

sudo -u elasticsearch /opt/elasticsearch-SAS/bin/elasticsearch
sudo -u elasticsearch /opt/elasticsearch-SSD/bin/elasticsearch

curl "http://192.168.1.31:9200/_cat/nodes?v"

192.168.1.53 elasticsearch-data部署雙執行個體

索引遷移（此步不能忽略）：一定要做這步，将192.168.1.53上的索引放到其它2台data節點上

curl -X PUT "192.168.1.31:9200/*/_settings?pretty" -H 'Content-Type: application/json' -d'
{
  "index.routing.allocation.include._ip": "192.168.1.51,192.168.1.52"
}'

确認目前索引存儲位置：确認所有索引不在192.168.1.52節點上

curl "http://192.168.1.31:9200/_cat/shards?h=n"

停掉192.168.1.53的程序，修改目錄結構及配置：請自行按SSD和SAS硬碟挂載好資料盤

# 安裝包下載下傳和部署請參考第一篇《EFK教程 - 快速入門指南》
cd /opt/software/
tar -zxvf elasticsearch-7.3.2-linux-x86_64.tar.gz
mv /opt/elasticsearch /opt/elasticsearch-SAS
mv elasticsearch-7.3.2 /opt/
mv /opt/elasticsearch-7.3.2 /opt/elasticsearch-SSD
chown elasticsearch.elasticsearch /opt/elasticsearch-* -R
rm -rf /data/SAS/*
chown elasticsearch.elasticsearch /data/* -R
mkdir -p /opt/logs/elasticsearch-SAS
mkdir -p /opt/logs/elasticsearch-SSD
chown elasticsearch.elasticsearch /opt/logs/* -R

cluster.name: my-application
node.name: 192.168.1.53-SAS
path.data: /data/SAS
path.logs: /opt/logs/elasticsearch-SAS
network.host: 192.168.1.53

http.port: 9200
transport.port: 9300
# discovery.seed_hosts和cluster.initial_master_nodes 一定要帶上端口号，不然會走http.port和transport.port端口
discovery.seed_hosts: ["192.168.1.31:9300","192.168.1.32:9300","192.168.1.33:9300"]
cluster.initial_master_nodes: ["192.168.1.31:9300","192.168.1.32:9300","192.168.1.33:9300"]
http.cors.enabled: true
http.cors.allow-origin: "*"

node.master: false
node.ingest: false
node.data: true

# 本機隻允行啟2個執行個體
node.max_local_storage_nodes: 2

cluster.name: my-application
node.name: 192.168.1.53-SSD
path.data: /data/SSD
path.logs: /opt/logs/elasticsearch-SSD
network.host: 192.168.1.53

http.port: 9201
transport.port: 9301
# discovery.seed_hosts和cluster.initial_master_nodes 一定要帶上端口号，不然會走http.port和transport.port端口
discovery.seed_hosts: ["192.168.1.31:9300","192.168.1.32:9300","192.168.1.33:9300"]
cluster.initial_master_nodes: ["192.168.1.31:9300","192.168.1.32:9300","192.168.1.33:9300"]
http.cors.enabled: true
http.cors.allow-origin: "*"

node.master: false
node.ingest: false
node.data: true

# 本機隻允行啟2個執行個體
node.max_local_storage_nodes: 2

sudo -u elasticsearch /opt/elasticsearch-SAS/bin/elasticsearch
sudo -u elasticsearch /opt/elasticsearch-SSD/bin/elasticsearch

curl "http://192.168.1.31:9200/_cat/nodes?v"

測試

将所有索引移到SSD硬碟上

# 下面的參數會在後面的文章講解，此處照抄即可
curl -X PUT "192.168.1.31:9200/*/_settings?pretty" -H 'Content-Type: application/json' -d'
{
  "index.routing.allocation.include._host_ip": "",
  "index.routing.allocation.include._host": "",
  "index.routing.allocation.include._name": "",
  "index.routing.allocation.include._ip": "",
  "index.routing.allocation.require._name": "*-SSD"
}'

确認所有索引全在SSD硬碟上

curl "http://192.168.1.31:9200/_cat/shards?h=n"

将nginx9月份的日志索引遷移到SAS硬碟上

curl -X PUT "192.168.1.31:9200/nginx_*_2019.09/_settings?pretty" -H 'Content-Type: application/json' -d'
{
  "index.routing.allocation.require._name": "*-SAS"
}'

确認nginx9月份的日志索引遷移到SAS硬碟上

curl "http://192.168.1.31:9200/_cat/shards"

EFK教程(3) - ElasticSearch冷熱資料分離目錄用途架構192.168.1.51 elasticsearch-data部署雙執行個體192.168.1.52 elasticsearch-data部署雙執行個體192.168.1.53 elasticsearch-data部署雙執行個體測試

目錄

用途

架構

192.168.1.51 elasticsearch-data部署雙執行個體

192.168.1.52 elasticsearch-data部署雙執行個體

192.168.1.53 elasticsearch-data部署雙執行個體

測試

繼續閱讀

使用underscore的template自定義模闆

underscore模闆功能的使用和學習

niginx的location的location詳細規則和優先級關系以及測試案例2.1比對文法2.2優先級判斷2.3練習3.1處理邏輯如何執行3.2功能測試3.3流程解析

Puppet自動化Nginx+Mongrel負載均衡配置

[HTML5]自定義屬性 data-* 和 jQuery.data 詳解

Ubuntu修改nginx将多個域名分别配置到伺服器不同端口

七牛雲-C#SDK-上傳-前期準備

修改or隐藏nginx版本号

Nginx安全改動：隐藏Nginx版本号背景解決serve_tokens指令參考

版本号隐藏

Nginx服務優化（版本隐藏、版本号僞裝、逾時管理、程序管理、日志分割）Nginx服務優化未完待續

nginx location中斜線的位置的重要性

vue-cli簡介（中文翻譯）

Ajax發送和擷取json資料到Spring mvc 1.spring mvc後端2.web前段

JSONObject包導入異常 java.lang.NoClassDefFoundErrorweb項目的導入包的問題