Elasticsearch集群数据备份与恢复 Snapshot & Restore

简介

• Elasticsearch 拥有副本机制来保障集群的高可用，然而无法解决如下情况的数据丢失:

• • 主副本所在机器存储全部损坏。
  • 误删除索引数据。
  • 升级失败，数据无法回滚。

• 定期对数据做备份，按需恢复可以很好的解决如上问题。
• Elasticsearch 提供了 Snapshot 和 Restore API 用于对集群数据完成备份与恢复。
• 数据备份的过程可以简单理解成将本地数据文件同步到远程仓库(repository) 的过程。
• 支持全量和增量备份。
• repository 常见有如下类型: fs / S3 / HDFS / Azure / Google Cloud Storage。

实战

1 创建原始索引并导入数据

PUT _template/blog-template
{
  "index_patterns": [
    "blog-*"
  ],
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 1
  },
  "mappings": {
    "dynamic_templates": [
      {
        "integers": {
          "match": "int_*",
          "mapping": {
            "type": "integer"
          }
        }
      },
      {
        "strings": {
          "match_mapping_type": "string",
          "mapping": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      }
    ],
    "properties": {
      "title": {
        "type": "text"
      }
    }
  }
}
PUT blog-test
POST blog-test/_bulk
{"index":{}}
{"title":"elasticsearch best practice","author_firstname":"tony","author_lastname":"song","tags":"elasticsearch ,logstash","int_age":18,"locale":"zh,en"}
{"index":{}}
{"title":"elastic stack release V5","author_firstname":"shay","author_lastname":"banon","tags":"elk ,elasticsearch,release","int_age":28,"locale":"zhc,en"}
{"index":{}}
{"title":"kibana tutorial","author_firstname":"zen","author_lastname":"wei","tags":"","int_age":38,"locale":"zhc,en"}
GET blog-test
GET blog-test/_search      

2 修改elasticsearch配置文件

修改 elasticsearch.yml 配置文件，增加如下配置:

path.repo: ["/home/elastic/backup"]      

重启elasticsearch进程，查看创建的repo：

GET _cluster/settings?include_defaults&filter_path=*.path.repo
#输出结果：
{
  "defaults" : {
    "path" : {
      "repo" : [
        "/home/elastic/backup"
      ]
    }
  }
}      

3 创建关联 repository

#创建关联 repository
PUT /_snapshot/my_fs_backup
{
    "type": "fs",
    "settings": {
        "location": "/home/elastic/backup/my_fs_backup",
        "compress": true
    }
}      

查看创建的关联：

GET _snapshot/my_fs_backup
#输出结果:
{
  "my_fs_backup" : {
    "type" : "fs",
    "settings" : {
      "compress" : "true",
      "location" : "/home/elastic/backup/my_fs_backup"
    }
  }
}      

查看在哪个node上创建了关联：

POST _snapshot/my_fs_backup/_verify
#输出结果
{
  "nodes" : {
    "pMrJwVGSQcSgeTZdh61QRw" : {
      "name" : "node1"
    }
  }
}      

4 执行 snapshot

• indices

  ：做快照的索引。

• wait_for_completion=true

  ：是否等待完成快照后再响应，如果为true会等快照完成后才响应。（默认为false，不等快照完成立即响应）

• ignore_unavailable

  : 设置为true时，当创建快照时忽略不存在的索引。

• include_global_state

  : 设置为false时，当某个索引所有的主分片不是全部的都可用时，可以完成快照。

#创建snapshot_1对blog-test索引做快照
PUT /_snapshot/my_fs_backup/snapshot_1?wait_for_completion=true
{
  "indices": "blog-test",
  "ignore_unavailable": true,
  "include_global_state": false
}      

查看创建的快照：

GET /_snapshot/my_fs_backup/snapshot_*
#GET /_snapshot/my_fs_backup/_all 看所有的
#输出结果：
{
  "snapshots" : [
    {
      "snapshot" : "snapshot_1",
      "uuid" : "FEbAt3BiR1SAiBkO7pfoZg",
      "version_id" : 7020199,
      "version" : "7.2.1",
      "indices" : [
        "blog-test"
      ],
      "include_global_state" : false,
      "state" : "SUCCESS",
      "start_time" : "2021-02-06T03:28:40.001Z",
      "start_time_in_millis" : 1612582120001,
      "end_time" : "2021-02-06T03:28:40.213Z",
      "end_time_in_millis" : 1612582120213,
      "duration_in_millis" : 212,
      "failures" : [ ],
      "shards" : {
        "total" : 3,
        "failed" : 0,
        "successful" : 3
      }
    }
  ]
}      

查看快照状态：

GET _snapshot/my_fs_backup/snapshot_1/_status
#输出结果:
{
  "snapshots" : [
    {
      "snapshot" : "snapshot_1",
      "repository" : "my_fs_backup",
      "uuid" : "FEbAt3BiR1SAiBkO7pfoZg",
      "state" : "SUCCESS",
      "include_global_state" : false,
      "shards_stats" : {
        "initializing" : 0,
        "started" : 0,
        "finalizing" : 0,
        "done" : 3,
        "failed" : 0,
        "total" : 3
      },
      "stats" : {
        "incremental" : {
          "file_count" : 12,
          "size_in_bytes" : 15608
        },
        "total" : {
          "file_count" : 12,
          "size_in_bytes" : 15608
        },
        "start_time_in_millis" : 1612582120074,
        "time_in_millis" : 108
      },
      "indices" : {
        "blog-test" : {
          "shards_stats" : {
            "initializing" : 0,
            "started" : 0,
            "finalizing" : 0,
            "done" : 3,
            "failed" : 0,
            "total" : 3
          },
          "stats" : {
            "incremental" : {
              "file_count" : 12,
              "size_in_bytes" : 15608
            },
            "total" : {
              "file_count" : 12,
              "size_in_bytes" : 15608
            },
            "start_time_in_millis" : 1612582120074,
            "time_in_millis" : 108
          },
          "shards" : {
            "0" : {
              "stage" : "DONE",
              "stats" : {
                "incremental" : {
                  "file_count" : 4,
                  "size_in_bytes" : 5104
                },
                "total" : {
                  "file_count" : 4,
                  "size_in_bytes" : 5104
                },
                "start_time_in_millis" : 1612582120143,
                "time_in_millis" : 39
              }
            },
            "1" : {
              "stage" : "DONE",
              "stats" : {
                "incremental" : {
                  "file_count" : 4,
                  "size_in_bytes" : 5265
                },
                "total" : {
                  "file_count" : 4,
                  "size_in_bytes" : 5265
                },
                "start_time_in_millis" : 1612582120074,
                "time_in_millis" : 25
              }
            },
            "2" : {
              "stage" : "DONE",
              "stats" : {
                "incremental" : {
                  "file_count" : 4,
                  "size_in_bytes" : 5239
                },
                "total" : {
                  "file_count" : 4,
                  "size_in_bytes" : 5239
                },
                "start_time_in_millis" : 1612582120113,
                "time_in_millis" : 21
              }
            }
          }
        }
      }
    }
  ]
}      

5 恢复 snapshot

• rename_pattern

  : 正则匹配原索引名

• rename_replacement

  : 将匹配到的字段用于重命名新索引

POST _snapshot/my_fs_backup/snapshot_1/_restore
{
  "indices": "blog-test",
  "ignore_unavailable": true,
  "rename_pattern": "(.+)",
  "rename_replacement": "restored_$1"
}      

查询新索引的数据：

GET restored_blog-test/_search