elasticsearch 叢集unassign shards 故障
2019年8月7号 zabbix收到告警 Elasticsearch Cluster 發送unassign shards故障,active shards percent: 99.3%,健康狀态呈現yellow狀态。
阿裡雲 故障: hn-elasticsearch1-2-183.test.com發生Elasticsearch Cluster unassign shards: 2故障!
告警時間: 2019.08.07 08:01:59
問題詳情: cluster unassigned shards:2
第一時間檢視叢集狀态
一、叢集狀态
curl -sXGET “http://127.0.0.1:9200/_cat/health?v”
[[email protected] elasticsearch-6.2.0]# curl -sXGET "http://127.0.0.1:9200/_cat/health?v"
epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1565142862 09:54:22 skye yellow 3 3 302 152 0 0 2 0 - 99.3%
二、檢視節點狀态
[[email protected] elasticsearch-6.2.0]# curl -sXGET "http://127.0.0.1:9200/_cat/nodes?v"
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
172.16.2.183 66 98 18 1.86 2.74 2.75 mdi * skye-1
172.16.2.185 71 99 28 2.65 2.66 2.54 mdi - skye-2
172.16.2.186 74 99 19 4.13 3.42 3.29 mdi - skye-3
三、檢視索引狀态
[[email protected] logs]# curl -sXGET "http://127.0.0.1:9200/_cat/indices?v"
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open zipkin:span-2019-08-05 qjXVX9gZQcuBonkgmX4Fpg 3 1 147002111 0 59.4gb 29.6gb
green open .monitoring-kibana-6-2019.08.03 kTXaQqRXReqSe17oJjGLpQ 1 1 8617 0 3.9mb 1.9mb
green open skye-2019-07-23 N2F-9aQ9SDCZILS56H5XaA 3 1 161178660 0 57.6gb 28.8gb
green open .monitoring-kibana-6-2019.07.31 k0GzT46mTbiO2yJ6NvfucQ 1 1 8624 0 3.9mb 1.9mb
green open zipkin:span-2019-08-02 VHgMjVleQYCduiSOrGszPA 3 1 144939021 0 58.7gb 29.2gb
green open .monitoring-kibana-6-2019.08.06 c7ZLWgYTSNe4xH0mgzSptw 1 1 8611 0 3.9mb 1.9mb
green open zipkin:span-2019-07-29 pGMU2_1NR-GI3q5gSBszGw 3 1 158112743 0 64.6gb 32.2gb
green open skye-2019-08-06 hc96CFOSRN6LCwGrXIW2-Q 3 1 177958560 0 67.9gb 34gb
green open .monitoring-kibana-6-2019.08.05 Ro_6EKUcS-e6mHlBvWMwEw 1 1 8627 0 3.8mb 1.9mb
green open zipkin:span-2019-07-25 NymvMYw4TRqF2iA5blKLxA 3 1 148087275 0 60.8gb 30.3gb
green open zipkin:span-2019-07-26 ApRta_N6SoCKUq8W_S2bSg 3 1 161429650 0 66.2gb 33gb
green open zipkin:span-2019-08-07 go5xslJbTaWL3jCtT1X3Yg 3 1 77256783 0 31.4gb 15.4gb
green open zipkin:span-2019-07-21 aRMfujcrR9CmLA_HP6lf9w 3 1 127307986 0 52.1gb 26gb
green open .monitoring-es-6-2019.08.04 p6dLV707TZmsGItAGwjorg 1 1 607948 3757 901.3mb 450.9mb
green open zipkin:span-2019-08-06 8a4MD7o4SaWZdvvhGlJeJg 3 1 153623960 0 62gb 30.9gb
green open skye-2019-08-05 v5DHlT6DT-eYsITwqpEm6Q 3 1 174864066 0 66.5gb 33.3gb
green open skye-2019-08-04 qGhmvoCxS7KxX3Ub25G8zw 3 1 149314826 0 56.7gb 28.3gb
green open .monitoring-es-6-2019.08.03 X6v5ah-NTdmnGARRWSOcqA 1 1 603559 1212 884.7mb 435.7mb
green open .monitoring-es-6-2019.08.05 u5sXmyf_SuqVCWnEHy0QYA 1 1 606158 3308 893mb 446.1mb
green open skye-2019-07-30 qn-1N_WkQF2UG6Qp2wvBwA 3 1 154401320 0 60.6gb 30.3gb
green open .monitoring-es-6-2019.07.31 n9Z6shIhRvODPibglD6vyw 1 1 600603 3023 888.7mb 441.3mb
green open .monitoring-alerts-6 BLtjLSrNS8KK6ItTui2VnA 1 1 6 2 75.3kb 37.6kb
yellow open .monitoring-es-6-2019.08.07 0V4gD6VOQA2tzS64MbDpqw 1 1 57442 1216 46.4mb 46.4mb
green open .monitoring-kibana-6-2019.08.01 s7j8urxmTkmDjlgfrGVZEg 1 1 8618 0 4mb 2mb
green open .watches QfQvZJSNR06Rgm-pDQ17SA 1 1 0 0 543b 271b
green open zipkin:span-2019-08-04 hns-FMLtRFq1QnWQrVHebA 3 1 127823259 0 51.7gb 25.8gb
green open skye-2019-07-20 QJCPI3ysRa6IwvrCmJgGow 3 1 143328678 0 51.1gb 25.5gb
green open .monitoring-kibana-6-2019.08.02 2Tnhw3qdTH6dizvO_lm2Sg 1 1 8613 0 3.8mb 1.9mb
green open skye-2019-07-28 taY9hVf6Syqei4visZGusg 3 1 177000766 0 65.4gb 32.7gb
green open skye-2019-07-22 9dx2r1ZVRJOW3jrc2ZGWoQ 3 1 150277882 0 56.1gb 28gb
green open zipkin:span-2019-07-19 QdTh9GH4QYKvvoXMdn8X1g 3 1 134742803 0 55.4gb 27.6gb
green open zipkin:span-2019-07-17 YvjEz5HwRTm4u4ISzlJw4A 3 1 123873409 0 50.7gb 25.3gb
green open skye-2019-08-07 SSDibXpfQkOfid0QVCzVaw 3 1 80598990 0 30.5gb 15gb
green open zipkin:span-2019-08-01 qt0QfsTIS4WSPBOoYp3fGg 3 1 148785022 0 60.4gb 30.1gb
green open skye-2019-07-27 eqs7tzs6R-OmTh69o1tMMw 3 1 172635156 0 63.2gb 31.6gb
green open zipkin:span-2019-07-23 2XEA9WlqTwG-MMxfgzkr_w 3 1 152261287 0 62.4gb 31.2gb
green open skye-2019-07-19 gCSnEtfdSziK3HRn_TYhkg 3 1 145913556 0 51.9gb 25.9gb
green open skye-2019-07-21 At3xLHmNQ4-pm0qPOgrNSg 3 1 136676726 0 48.3gb 24.2gb
green open .monitoring-kibana-6-2019.08.04 4ClbUy-NSGK3lGqTHQRL_w 1 1 8622 0 3.8mb 1.9mb
green open skye-2019-07-29 F8GtVvj0Q0mFJvUlcNzqmA 3 1 185472138 0 69.8gb 34.9gb
green open skye-2019-07-31 GTVYUuegRzGUQWKtWHqkbg 3 1 149953344 0 58gb 29gb
green open skye-2019-07-26 n68YQcv5SBG_jy41D7iRag 3 1 185373586 0 68gb 34gb
green open zipkin:span-2019-07-24 2yH2ZaBoSqyrM0Db3zLAIw 3 1 148338152 0 60.6gb 30.2gb
green open skye-2019-08-03 zUd684kSTOOrfckNISRwNw 3 1 154055266 0 58.9gb 29.4gb
green open skye-2019-07-16 UG-yC1WFRbieV8EMoqxB5g 3 1 153397842 0 55.4gb 27.7gb
green open zipkin:span-2019-07-16 fI6LfNPfQPaLGxr62wCIaA 3 1 129074886 0 52.8gb 26.4gb
green open zipkin:span-2019-07-30 Bs_PDcp2QvCMOPdwjK_5uA 3 1 90024904 0 37gb 18.4gb
green open skye-2019-08-02 ZzSg4juMTZ6Yezqz6zWdJQ 3 1 169878668 0 64.7gb 32.4gb
green open .triggered_watches OPRctrGfTgu84yLc2P_4SQ 1 1 5 272014 71.1mb 35.5mb
green open skye-2019-07-17 8xgPl9P5RO-5QXsXonykHQ 3 1 151114338 0 54.6gb 27.3gb
green open zipkin:span-2019-07-28 Esve3R7aSjyEo58A4cS1Fg 3 1 152251699 0 62.3gb 31.1gb
green open .monitoring-es-6-2019.08.01 nm4uNMy2T0Cu4eE57cc5Zg 1 1 600620 3800 884.8mb 444.9mb
green open zipkin:span-2019-07-27 8aeQTZ5zTYKlYbG-E9d2pw 3 1 147761607 0 60.3gb 30.1gb
green open skye-2019-08-01 4nPzPpMlQn2uDHFHJd2-Hw 3 1 170813994 0 66.2gb 33.1gb
green open zipkin:span-2019-07-31 dvSt_7EPTw-jA_1Y3-f3og 3 1 131738094 0 53.5gb 26.7gb
green open zipkin:span-2019-07-20 Whvmdk2gRZeZ-Xx4bybOng 3 1 133067045 0 55.1gb 27.5gb
green open skye-2019-07-24 FRc14JMGR-u0tZpkFdrmsQ 3 1 161645004 0 58.6gb 29.3gb
yellow open .monitoring-kibana-6-2019.08.07 Y_fBCiy9SQmn2ARN7qEU9Q 1 1 812 0 204.3kb 204.3kb
green open zipkin:span-2019-08-03 mh-X1yrISOiv-3icl377nA 3 1 131723700 0 53.5gb 26.7gb
green open zipkin:span-2019-07-22 uhfexnVQSDOkYibQvPpGVQ 3 1 138862797 0 57.4gb 28.6gb
green open .monitoring-es-6-2019.08.06 aH9hQpdDQaS3w_LTbTWr8g 1 1 606657 3665 905.6mb 448mb
green open skye-2019-07-25 hqosW99TR4WIZyjD-GPCYA 3 1 166565088 0 61.8gb 30.9gb
green open .kibana DX3TAVELRhO6NUONpVU9mQ 1 1 12 5 150.6kb 75.3kb
green open .monitoring-es-6-2019.08.02 NDdWX3SXQu-MamnFSVRroA 1 1 600423 3608 876.6mb 438.4mb
[[email protected] logs]# curl -sXGET "http://127.0.0.1:9200/_cat/indices?v&pretty"
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open zipkin:span-2019-08-05 qjXVX9gZQcuBonkgmX4Fpg 3 1 147002111 0 59.4gb 29.6gb
green open .monitoring-kibana-6-2019.08.03 kTXaQqRXReqSe17oJjGLpQ 1 1 8617 0 3.9mb 1.9mb
green open skye-2019-07-23 N2F-9aQ9SDCZILS56H5XaA 3 1 161178660 0 57.6gb 28.8gb
green open .monitoring-kibana-6-2019.07.31 k0GzT46mTbiO2yJ6NvfucQ 1 1 8624 0 3.9mb 1.9mb
green open zipkin:span-2019-08-02 VHgMjVleQYCduiSOrGszPA 3 1 144939021 0 58.7gb 29.2gb
green open .monitoring-kibana-6-2019.08.06 c7ZLWgYTSNe4xH0mgzSptw 1 1 8611 0 3.9mb 1.9mb
green open zipkin:span-2019-07-29 pGMU2_1NR-GI3q5gSBszGw 3 1 158112743 0 64.6gb 32.2gb
green open skye-2019-08-06 hc96CFOSRN6LCwGrXIW2-Q 3 1 177958560 0 67.9gb 34gb
green open .monitoring-kibana-6-2019.08.05 Ro_6EKUcS-e6mHlBvWMwEw 1 1 8627 0 3.8mb 1.9mb
green open zipkin:span-2019-07-25 NymvMYw4TRqF2iA5blKLxA 3 1 148087275 0 60.8gb 30.3gb
green open zipkin:span-2019-07-26 ApRta_N6SoCKUq8W_S2bSg 3 1 161429650 0 66.2gb 33gb
green open zipkin:span-2019-08-07 go5xslJbTaWL3jCtT1X3Yg 3 1 77391731 0 31.1gb 15.5gb
green open zipkin:span-2019-07-21 aRMfujcrR9CmLA_HP6lf9w 3 1 127307986 0 52.1gb 26gb
green open .monitoring-es-6-2019.08.04 p6dLV707TZmsGItAGwjorg 1 1 607948 3757 901.3mb 450.9mb
green open zipkin:span-2019-08-06 8a4MD7o4SaWZdvvhGlJeJg 3 1 153623960 0 62gb 30.9gb
green open skye-2019-08-05 v5DHlT6DT-eYsITwqpEm6Q 3 1 174864066 0 66.5gb 33.3gb
green open skye-2019-08-04 qGhmvoCxS7KxX3Ub25G8zw 3 1 149314826 0 56.7gb 28.3gb
green open .monitoring-es-6-2019.08.03 X6v5ah-NTdmnGARRWSOcqA 1 1 603559 1212 884.7mb 435.7mb
green open .monitoring-es-6-2019.08.05 u5sXmyf_SuqVCWnEHy0QYA 1 1 606158 3308 893mb 446.1mb
green open skye-2019-07-30 qn-1N_WkQF2UG6Qp2wvBwA 3 1 154401320 0 60.6gb 30.3gb
green open .monitoring-es-6-2019.07.31 n9Z6shIhRvODPibglD6vyw 1 1 600603 3023 888.7mb 441.3mb
green open .monitoring-alerts-6 BLtjLSrNS8KK6ItTui2VnA 1 1 6 2 75.3kb 37.6kb
yellow open .monitoring-es-6-2019.08.07 0V4gD6VOQA2tzS64MbDpqw 1 1 57723 1520 46.5mb 46.5mb
green open .monitoring-kibana-6-2019.08.01 s7j8urxmTkmDjlgfrGVZEg 1 1 8618 0 4mb 2mb
green open .watches QfQvZJSNR06Rgm-pDQ17SA 1 1 0 0 543b 271b
green open zipkin:span-2019-08-04 hns-FMLtRFq1QnWQrVHebA 3 1 127823259 0 51.7gb 25.8gb
green open skye-2019-07-20 QJCPI3ysRa6IwvrCmJgGow 3 1 143328678 0 51.1gb 25.5gb
green open .monitoring-kibana-6-2019.08.02 2Tnhw3qdTH6dizvO_lm2Sg 1 1 8613 0 3.8mb 1.9mb
green open skye-2019-07-28 taY9hVf6Syqei4visZGusg 3 1 177000766 0 65.4gb 32.7gb
green open skye-2019-07-22 9dx2r1ZVRJOW3jrc2ZGWoQ 3 1 150277882 0 56.1gb 28gb
green open zipkin:span-2019-07-19 QdTh9GH4QYKvvoXMdn8X1g 3 1 134742803 0 55.4gb 27.6gb
green open zipkin:span-2019-07-17 YvjEz5HwRTm4u4ISzlJw4A 3 1 123873409 0 50.7gb 25.3gb
green open skye-2019-08-07 SSDibXpfQkOfid0QVCzVaw 3 1 80744382 0 31gb 15.2gb
green open zipkin:span-2019-08-01 qt0QfsTIS4WSPBOoYp3fGg 3 1 148785022 0 60.4gb 30.1gb
green open skye-2019-07-27 eqs7tzs6R-OmTh69o1tMMw 3 1 172635156 0 63.2gb 31.6gb
green open zipkin:span-2019-07-23 2XEA9WlqTwG-MMxfgzkr_w 3 1 152261287 0 62.4gb 31.2gb
green open skye-2019-07-19 gCSnEtfdSziK3HRn_TYhkg 3 1 145913556 0 51.9gb 25.9gb
green open skye-2019-07-21 At3xLHmNQ4-pm0qPOgrNSg 3 1 136676726 0 48.3gb 24.2gb
green open .monitoring-kibana-6-2019.08.04 4ClbUy-NSGK3lGqTHQRL_w 1 1 8622 0 3.8mb 1.9mb
green open skye-2019-07-29 F8GtVvj0Q0mFJvUlcNzqmA 3 1 185472138 0 69.8gb 34.9gb
green open skye-2019-07-31 GTVYUuegRzGUQWKtWHqkbg 3 1 149953344 0 58gb 29gb
green open skye-2019-07-26 n68YQcv5SBG_jy41D7iRag 3 1 185373586 0 68gb 34gb
green open zipkin:span-2019-07-24 2yH2ZaBoSqyrM0Db3zLAIw 3 1 148338152 0 60.6gb 30.2gb
green open skye-2019-08-03 zUd684kSTOOrfckNISRwNw 3 1 154055266 0 58.9gb 29.4gb
green open skye-2019-07-16 UG-yC1WFRbieV8EMoqxB5g 3 1 153397842 0 55.4gb 27.7gb
green open zipkin:span-2019-07-16 fI6LfNPfQPaLGxr62wCIaA 3 1 129074886 0 52.8gb 26.4gb
green open zipkin:span-2019-07-30 Bs_PDcp2QvCMOPdwjK_5uA 3 1 90024904 0 37gb 18.4gb
green open skye-2019-08-02 ZzSg4juMTZ6Yezqz6zWdJQ 3 1 169878668 0 64.7gb 32.4gb
green open .triggered_watches OPRctrGfTgu84yLc2P_4SQ 1 1 5 272014 71.1mb 35.5mb
green open skye-2019-07-17 8xgPl9P5RO-5QXsXonykHQ 3 1 151114338 0 54.6gb 27.3gb
green open zipkin:span-2019-07-28 Esve3R7aSjyEo58A4cS1Fg 3 1 152251699 0 62.3gb 31.1gb
green open .monitoring-es-6-2019.08.01 nm4uNMy2T0Cu4eE57cc5Zg 1 1 600620 3800 884.8mb 444.9mb
green open zipkin:span-2019-07-27 8aeQTZ5zTYKlYbG-E9d2pw 3 1 147761607 0 60.3gb 30.1gb
green open skye-2019-08-01 4nPzPpMlQn2uDHFHJd2-Hw 3 1 170813994 0 66.2gb 33.1gb
green open zipkin:span-2019-07-31 dvSt_7EPTw-jA_1Y3-f3og 3 1 131738094 0 53.5gb 26.7gb
green open zipkin:span-2019-07-20 Whvmdk2gRZeZ-Xx4bybOng 3 1 133067045 0 55.1gb 27.5gb
green open skye-2019-07-24 FRc14JMGR-u0tZpkFdrmsQ 3 1 161645004 0 58.6gb 29.3gb
yellow open .monitoring-kibana-6-2019.08.07 Y_fBCiy9SQmn2ARN7qEU9Q 1 1 816 0 275.8kb 275.8kb
green open zipkin:span-2019-08-03 mh-X1yrISOiv-3icl377nA 3 1 131723700 0 53.5gb 26.7gb
green open zipkin:span-2019-07-22 uhfexnVQSDOkYibQvPpGVQ 3 1 138862797 0 57.4gb 28.6gb
green open .monitoring-es-6-2019.08.06 aH9hQpdDQaS3w_LTbTWr8g 1 1 606657 3665 905.6mb 448mb
green open skye-2019-07-25 hqosW99TR4WIZyjD-GPCYA 3 1 166565088 0 61.8gb 30.9gb
green open .kibana DX3TAVELRhO6NUONpVU9mQ 1 1 12 5 150.6kb 75.3kb
green open .monitoring-es-6-2019.08.02 NDdWX3SXQu-MamnFSVRroA 1 1 600423 3608 876.6mb 438.4mb
四、問題分析,檢視日志
[2019-08-07T10:24:26,053][INFO ][o.e.c.r.a.DiskThresholdMonitor] [skye-1] low disk watermark [85%] exceeded on [sCr8-8UdSI2Di06LAdnpbg][skye-3][/data/elasticsearch-6.2.0/data/nodes/0] free: 134.7gb[13.6%], replicas will not be assigned to this node
[2019-08-07T10:24:56,058][INFO ][o.e.c.r.a.DiskThresholdMonitor] [skye-1] low disk watermark [85%] exceeded on [5uby7FwmQ2uk41lirnyiNw][skye-1][/data/elasticsearch-6.2.0/data/nodes/0] free: 132.2gb[13.4%], replicas will not be assigned to this node
[2019-08-07T10:24:56,058][INFO ][o.e.c.r.a.DiskThresholdMonitor] [skye-1] low disk watermark [85%] exceeded on [sCr8-8UdSI2Di06LAdnpbg][skye-3][/data/elasticsearch-6.2.0/data/nodes/0] free: 134.7gb[13.6%], replicas will not be assigned to this node
[2019-08-07T10:24:56,058][INFO ][o.e.c.r.a.DiskThresholdMonitor] [skye-1] low disk watermark [85%] exceeded on [ElF4oNnnTX2tOHmY6R4XXQ][skye-2][/data/elasticsearch-6.2.0/data/nodes/0] free: 132.9gb[13.5%], replicas will not be assigned to this node
[2019-08-07T10:25:26,358][INFO ][o.e.c.r.a.DiskThresholdMonitor] [skye-1] low disk watermark [85%] exceeded on [5uby7FwmQ2uk41lirnyiNw][skye-1][/data/elasticsearch-6.2.0/data/nodes/0] free: 132.2gb[13.4%], replicas will not be assigned to this node
[2019-08-07T10:25:26,358][INFO ][o.e.c.r.a.DiskThresholdMonitor] [skye-1] low disk watermark [85%] exceeded on [ElF4oNnnTX2tOHmY6R4XXQ][skye-2][/data/elasticsearch-6.2.0/data/nodes/0] free: 133gb[13.5%], replicas will not be assigned tothis node
[2019-08-07T10:25:26,358][INFO ][o.e.c.r.a.DiskThresholdMonitor] [skye-1] low disk watermark [85%] exceeded on [sCr8-8UdSI2Di06LAdnpbg][skye-3][/data/elasticsearch-6.2.0/data/nodes/0] free: 134.7gb[13.6%], replicas will not be assigned to this node
[2019-08-07T10:25:56,363][INFO ][o.e.c.r.a.DiskThresholdMonitor] [skye-1] low disk watermark [85%] exceeded on [5uby7FwmQ2uk41lirnyiNw][skye-1][/data/elasticsearch-6.2.0/data/nodes/0] free: 131.9gb[13.4%], replicas will not be assigned to this node
[2019-08-07T10:25:56,363][INFO ][o.e.c.r.a.DiskThresholdMonitor] [skye-1] low disk watermark [85%] exceeded on [sCr8-8UdSI2Di06LAdnpbg][skye-3][/data/elasticsearch-6.2.0/data/nodes/0] free: 134.6gb[13.6%], replicas will not be assigned to this node
[2019-08-07T10:25:56,363][INFO ][o.e.c.r.a.DiskThresholdMonitor] [skye-1] low disk watermark [85%] exceeded on [ElF4oNnnTX2tOHmY6R4XXQ][skye-2][/data/elasticsearch-6.2.0/data/nodes/0] free: 132.9gb[13.5%], replicas will not be assigned to this node
[2019-08-07T10:26:07,015][INFO ][o.e.c.m.MetaDataDeleteIndexService] [skye-1] [skye-2019-07-22/9dx2r1ZVRJOW3jrc2ZGWoQ] deleting index
[2019-08-07T10:26:07,015][INFO ][o.e.c.m.MetaDataDeleteIndexService] [skye-1] [zipkin:span-2019-07-22/uhfexnVQSDOkYibQvPpGVQ] deleting index
[2019-08-07T10:26:26,368][INFO ][o.e.c.r.a.DiskThresholdMonitor] [skye-1] rerouting shards: [one or more nodes has gone under the high or low watermark]
[2019-08-07T10:27:03,048][INFO ][o.e.c.r.a.AllocationService] [skye-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.monitoring-es-6-2019.08.07][0]] ...]).
[2019-08-07T10:27:26,648][INFO ][o.e.c.r.a.DiskThresholdMonitor] [skye-1] rerouting shards: [one or more nodes has gone under the high or low watermark]
[2019-08-07T10:28:26,922][INFO ][o.e.c.r.a.DiskThresholdMonitor] [skye-1] rerouting shards: [one or more nodes has gone under the high or low watermark]
五、臨時解決，删除舊資料
[[email protected] logs]# curl -XDELETE "http://127.0.0.1:9200/*2019-07-22"
{"acknowledged":true}
[[email protected] logs]#
臨時恢複
[[email protected] logs]# curl -sXGET "http://127.0.0.1:9200/_cat/indices?v&pretty"
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open zipkin:span-2019-08-05 qjXVX9gZQcuBonkgmX4Fpg 3 1 147002111 0 59.4gb 29.6gb
green open .monitoring-kibana-6-2019.08.03 kTXaQqRXReqSe17oJjGLpQ 1 1 8617 0 3.9mb 1.9mb
green open skye-2019-07-23 N2F-9aQ9SDCZILS56H5XaA 3 1 161178660 0 57.6gb 28.8gb
green open .monitoring-kibana-6-2019.07.31 k0GzT46mTbiO2yJ6NvfucQ 1 1 8624 0 3.9mb 1.9mb
green open zipkin:span-2019-08-02 VHgMjVleQYCduiSOrGszPA 3 1 144939021 0 58.7gb 29.2gb
green open .monitoring-kibana-6-2019.08.06 c7ZLWgYTSNe4xH0mgzSptw 1 1 8611 0 3.9mb 1.9mb
green open zipkin:span-2019-07-29 pGMU2_1NR-GI3q5gSBszGw 3 1 158112743 0 64.6gb 32.2gb
green open skye-2019-08-06 hc96CFOSRN6LCwGrXIW2-Q 3 1 177958560 0 67.9gb 34gb
green open .monitoring-kibana-6-2019.08.05 Ro_6EKUcS-e6mHlBvWMwEw 1 1 8627 0 3.8mb 1.9mb
green open zipkin:span-2019-07-25 NymvMYw4TRqF2iA5blKLxA 3 1 148087275 0 60.8gb 30.3gb
green open zipkin:span-2019-07-26 ApRta_N6SoCKUq8W_S2bSg 3 1 161429650 0 66.2gb 33gb
green open zipkin:span-2019-08-07 go5xslJbTaWL3jCtT1X3Yg 3 1 82658442 0 43.4gb 22.1gb
green open zipkin:span-2019-07-21 aRMfujcrR9CmLA_HP6lf9w 3 1 127307986 0 52.1gb 26gb
green open .monitoring-es-6-2019.08.04 p6dLV707TZmsGItAGwjorg 1 1 607948 3757 901.3mb 450.9mb
green open zipkin:span-2019-08-06 8a4MD7o4SaWZdvvhGlJeJg 3 1 153623960 0 62gb 30.9gb
green open skye-2019-08-05 v5DHlT6DT-eYsITwqpEm6Q 3 1 174864066 0 66.5gb 33.3gb
green open skye-2019-08-04 qGhmvoCxS7KxX3Ub25G8zw 3 1 149314826 0 56.7gb 28.3gb
green open .monitoring-es-6-2019.08.03 X6v5ah-NTdmnGARRWSOcqA 1 1 603559 1212 884.7mb 435.7mb
green open .monitoring-es-6-2019.08.05 u5sXmyf_SuqVCWnEHy0QYA 1 1 606158 3308 893mb 446.1mb
green open skye-2019-07-30 qn-1N_WkQF2UG6Qp2wvBwA 3 1 154401320 0 60.6gb 30.3gb
green open .monitoring-es-6-2019.07.31 n9Z6shIhRvODPibglD6vyw 1 1 600603 3023 888.7mb 441.3mb
green open .monitoring-alerts-6 BLtjLSrNS8KK6ItTui2VnA 1 1 6 2 75.3kb 37.6kb
green open .monitoring-es-6-2019.08.07 0V4gD6VOQA2tzS64MbDpqw 1 1 71678 876 113.7mb 56mb
green open .monitoring-kibana-6-2019.08.01 s7j8urxmTkmDjlgfrGVZEg 1 1 8618 0 4mb 2mb
green open .watches QfQvZJSNR06Rgm-pDQ17SA 1 1 0 0 543b 271b
green open zipkin:span-2019-08-04 hns-FMLtRFq1QnWQrVHebA 3 1 127823259 0 51.7gb 25.8gb
green open skye-2019-07-20 QJCPI3ysRa6IwvrCmJgGow 3 1 143328678 0 51.1gb 25.5gb
green open .monitoring-kibana-6-2019.08.02 2Tnhw3qdTH6dizvO_lm2Sg 1 1 8613 0 3.8mb 1.9mb
green open skye-2019-07-28 taY9hVf6Syqei4visZGusg 3 1 177000766 0 65.4gb 32.7gb
green open zipkin:span-2019-07-19 QdTh9GH4QYKvvoXMdn8X1g 3 1 134742803 0 55.4gb 27.6gb
green open zipkin:span-2019-07-17 YvjEz5HwRTm4u4ISzlJw4A 3 1 123873409 0 50.7gb 25.3gb
green open skye-2019-08-07 SSDibXpfQkOfid0QVCzVaw 3 1 86754294 0 36gb 19.5gb
green open zipkin:span-2019-08-01 qt0QfsTIS4WSPBOoYp3fGg 3 1 148785022 0 60.4gb 30.1gb
green open skye-2019-07-27 eqs7tzs6R-OmTh69o1tMMw 3 1 172635156 0 63.2gb 31.6gb
green open zipkin:span-2019-07-23 2XEA9WlqTwG-MMxfgzkr_w 3 1 152261287 0 62.4gb 31.2gb
green open skye-2019-07-19 gCSnEtfdSziK3HRn_TYhkg 3 1 145913556 0 51.9gb 25.9gb
green open skye-2019-07-21 At3xLHmNQ4-pm0qPOgrNSg 3 1 136676726 0 48.3gb 24.2gb
green open .monitoring-kibana-6-2019.08.04 4ClbUy-NSGK3lGqTHQRL_w 1 1 8622 0 3.8mb 1.9mb
green open skye-2019-07-29 F8GtVvj0Q0mFJvUlcNzqmA 3 1 185472138 0 69.8gb 34.9gb
green open skye-2019-07-31 GTVYUuegRzGUQWKtWHqkbg 3 1 149953344 0 58gb 29gb
green open skye-2019-07-26 n68YQcv5SBG_jy41D7iRag 3 1 185373586 0 68gb 34gb
green open zipkin:span-2019-07-24 2yH2ZaBoSqyrM0Db3zLAIw 3 1 148338152 0 60.6gb 30.2gb
green open skye-2019-08-03 zUd684kSTOOrfckNISRwNw 3 1 154055266 0 58.9gb 29.4gb
green open skye-2019-07-16 UG-yC1WFRbieV8EMoqxB5g 3 1 153397842 0 55.4gb 27.7gb
green open zipkin:span-2019-07-16 fI6LfNPfQPaLGxr62wCIaA 3 1 129074886 0 52.8gb 26.4gb
green open zipkin:span-2019-07-30 Bs_PDcp2QvCMOPdwjK_5uA 3 1 90024904 0 37gb 18.4gb
green open skye-2019-08-02 ZzSg4juMTZ6Yezqz6zWdJQ 3 1 169878668 0 64.7gb 32.4gb
green open .triggered_watches OPRctrGfTgu84yLc2P_4SQ 1 1 5 272014 71.1mb 35.5mb
green open skye-2019-07-17 8xgPl9P5RO-5QXsXonykHQ 3 1 151114338 0 54.6gb 27.3gb
green open zipkin:span-2019-07-28 Esve3R7aSjyEo58A4cS1Fg 3 1 152251699 0 62.3gb 31.1gb
green open .monitoring-es-6-2019.08.01 nm4uNMy2T0Cu4eE57cc5Zg 1 1 600620 3800 884.8mb 444.9mb
green open zipkin:span-2019-07-27 8aeQTZ5zTYKlYbG-E9d2pw 3 1 147761607 0 60.3gb 30.1gb
green open skye-2019-08-01 4nPzPpMlQn2uDHFHJd2-Hw 3 1 170813994 0 66.2gb 33.1gb
green open zipkin:span-2019-07-31 dvSt_7EPTw-jA_1Y3-f3og 3 1 131738094 0 53.5gb 26.7gb
green open zipkin:span-2019-07-20 Whvmdk2gRZeZ-Xx4bybOng 3 1 133067045 0 55.1gb 27.5gb
green open skye-2019-07-24 FRc14JMGR-u0tZpkFdrmsQ 3 1 161645004 0 58.6gb 29.3gb
green open .monitoring-kibana-6-2019.08.07 Y_fBCiy9SQmn2ARN7qEU9Q 1 1 998 0 613.5kb 335.5kb
green open zipkin:span-2019-08-03 mh-X1yrISOiv-3icl377nA 3 1 131723700 0 53.5gb 26.7gb
green open .monitoring-es-6-2019.08.06 aH9hQpdDQaS3w_LTbTWr8g 1 1 606657 3665 905.6mb 448mb
green open skye-2019-07-25 hqosW99TR4WIZyjD-GPCYA 3 1 166565088 0 61.8gb 30.9gb
green open .kibana DX3TAVELRhO6NUONpVU9mQ 1 1 12 5 150.6kb 75.3kb
green open .monitoring-es-6-2019.08.02 NDdWX3SXQu-MamnFSVRroA 1 1 600423 3608 876.6mb 438.4mb
[[email protected] ~]# df -TH
Filesystem Type Size Used Avail Use% Mounted on
/dev/vda1 ext4 43G 6.1G 34G 16% /
devtmpfs devtmpfs 8.4G 0 8.4G 0% /dev
tmpfs tmpfs 8.4G 0 8.4G 0% /dev/shm
tmpfs tmpfs 8.4G 472k 8.4G 1% /run
tmpfs tmpfs 8.4G 0 8.4G 0% /sys/fs/cgroup
tmpfs tmpfs 1.7G 0 1.7G 0% /run/user/0
/dev/mapper/vdb--vg-vdb--lv ext4 1.1T 914G 144G 87% /data
tmpfs tmpfs 1.7G 0 1.7G 0% /run/user/1002
[[email protected] logs]# df -TH
Filesystem Type Size Used Avail Use% Mounted on
/dev/vda1 ext4 43G 6.1G 34G 16% /
devtmpfs devtmpfs 8.4G 0 8.4G 0% /dev
tmpfs tmpfs 8.4G 0 8.4G 0% /dev/shm
tmpfs tmpfs 8.4G 443k 8.4G 1% /run
tmpfs tmpfs 8.4G 0 8.4G 0% /sys/fs/cgroup
tmpfs tmpfs 1.7G 0 1.7G 0% /run/user/0
/dev/mapper/vdb--vg-vdb--lv ext4 1.1T 877G 181G 83% /data
tmpfs tmpfs 1.7G 0 1.7G 0% /run/user/1002
六、後續解決方案:
elasticsearch.yml配置檔案中
cluster.routing.allocation.disk.threshold_enabled
預設為true。設定為false禁用磁盤配置設定決策程式。
cluster.routing.allocation.disk.watermark.low
控制磁盤使用的低水位線。它預設為85%，這意味着Elasticsearch不會将分片配置設定給使用磁盤超過85％的節點。它也可以設定為絕對位元組值（如500mb），以防止Elasticsearch在小于指定的可用空間量時配置設定分片。此設定不會影響新建立的索引的主分片，或者特别是之前從未配置設定過的任何分片。
cluster.routing.allocation.disk.watermark.high
控制高水印。它預設為90%，意味着Elasticsearch将嘗試從磁盤使用率超過90％的節點重新定位分片。它也可以設定為絕對位元組值（類似于低水印），以便在節點小于指定的可用空間量時将其從節點重新定位。此設定會影響所有分片的配置設定，無論先前是否配置設定。
cluster.routing.allocation.disk.watermark.flood_stage
控制洪水階段水印。它預設為95％，這意味着Elasticsearch index.blocks.read_only_allow_delete對每個索引強制執行隻讀索引塊（），該索引在至少有一個磁盤超過泛洪階段的節點上配置設定了一個或多個分片。這是防止節點耗盡磁盤空間的最後手段。一旦有足夠的可用磁盤空間允許索引操作繼續，就必須手動釋放索引塊。
api 修改高低水位
PUT _cluster/settings
{
“transient”: {
“cluster.routing.allocation.disk.watermark.low”: “80%”,
“cluster.routing.allocation.disk.watermark.high”: “50gb”,
}
}
使用百分比或者gb數都可以，注意low的值一般要比high的大。
七、參考:
官方文檔：
https://www.elastic.co/guide/en/elasticsearch/reference/current/disk-allocator.html
https://blog.csdn.net/weikaixxxxxx/article/details/86499060
https://blog.csdn.net/xiangcheng001/article/details/75331056
![ES分詞插件IK Analyzer安裝[圖]](data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=)