21 Backup and Migration

21 Backup and Migration #

备份和迁移是数据管理中重要的任务。在处理数据时，我们需要确保数据的安全性，并且能够在需要时迁移数据到其他系统或设备上。

在备份方面，我们可以使用各种不同的方法。其中一种方法是创建数据的副本并将其存储在不同的位置，以防止主要数据丢失。这可以通过使用外部存储设备，例如硬盘或云存储来实现。在备份数据时，我们还可以使用压缩和加密技术，以减少存储空间和保护数据安全。

迁移数据可以是将数据从一台设备迁移到另一台设备，或者将数据从一个系统迁移到另一个系统。迁移数据的方法取决于数据的类型和规模。对于小规模数据，我们可以使用手动方法，例如复制和粘贴。对于大规模数据，我们可能需要使用专业的数据迁移工具。

在进行备份和迁移之前，我们需要评估数据的重要性和价值，以确定备份和迁移的频率和方式。我们还应该制定恢复策略，以确保在数据丢失或意外情况发生时能够及时恢复数据。

总之，备份和迁移是数据管理中至关重要的一环。通过使用合适的方法和工具，我们可以确保数据的安全和可用性，并且能够在需要时成功迁移数据。

Solution #

Offline Solution #

  • Snapshot
  • Reindex
  • Logstash
  • ElasticSearch-dump
  • ElasticSearch-Exporter

Incremental Backup Solution #

  • logstash

Backup Using Snapshots #

Configuration Information #

Before registering, make sure to add the configuration information to the elasticsearch.yml file:

path.repo: ["/opt/elasticsearch/backup"]

Create a Repository #

Register a repository to store the snapshots. Please note that this does not generate a snapshot, it only registers a repository.

curl -XPUT 'http://10.11.60.5:9200/_snapshot/repo_backup_1' -H 'Content-Type: application/json' -d '{
    "type": "fs",
    "settings": {
        "location": "/opt/elasticsearch/backup",
        "max_snapshot_bytes_per_sec": "20mb",
        "max_restore_bytes_per_sec": "20mb",
        "compress": true
    }
}'

To check the repository information:

curl -XGET 'http://10.11.60.5:9200/_snapshot/repo_backup_1?pretty'

The response will be:

[root@STOR-ES elasticsearch]# curl -XGET 'http://10.11.60.5:9200/_snapshot/repo_backup_1?pretty'
{
  "repo_backup_1" : {
    "type" : "fs",
    "settings" : {
      "location" : "/opt/elasticsearch/backup",
      "max_restore_bytes_per_sec" : "20mb",
      "compress" : "true",
      "max_snapshot_bytes_per_sec" : "20mb"
    }
  }
}

Create a Snapshot # 

curl -XPUT 'http://10.11.60.5:9200/_snapshot/repo_backup_1/snapshot_1?wait_for_completion=true&pretty' -H 'Content-Type: application/json' -d '{
    "indices": "bro-2019-09-14,bro-2019-09-15,wmi-2019-09-14,wmi-2019-09-15,syslog-2019-09-14,sylog-2019-09-15",
    "rename_pattern": "bro_(.+)",
    "rename_replacement": "dev_bro_$1",
    "ignore_unavailable": true,
    "include_global_state": true
}'

Execute:

{
  "snapshot" : {
    "snapshot" : "snapshot_1",
    "version_id" : 2040399,
    "version" : "2.4.3",
    "indices" : [ "bro-2019-09-14", "bro-2019-09-15", "wmi-2019-09-15", "syslog-2019-09-14", "wmi-2019-09-14" ],
    "state" : "SUCCESS",
    "start_time" : "2019-09-18T05:58:08.860Z",
    "start_time_in_millis" : 1568786288860,
    "end_time" : "2019-09-18T06:02:18.037Z",
    "end_time_in_millis" : 1568786538037,
    "duration_in_millis" : 249177,
    "failures" : [ ],
    "shards" : {
      "total" : 25,
      "failed" : 0,
      "successful" : 25
    }
  }
}

Restore the Data #

Scenario #

This section introduces the scenarios in which the solution can be applied.

Considerations for Migration #

  • Version: Migrating data from lower versions to higher versions.
  • Multi-tenancy adaptation.

Data from multiple factories enters different indices. The data from the original index “bro-2019-09-15” needs to be moved to “factorycode-bro-2019-09-15”.

  • Migrating data multiple times or in batches.
  • Enriching data during migration.
  • Separating FieldMapping and data information?