通过curl命令或者kibana快照备份，恢复的方式进行数据迁移
环境介绍
之前创建的ELK 因为VPC环境的问题，需要对ELK从新部署，但是还需要保留现有的数据，于是便有了这篇文档。
10.0.20.74 old es
10.0.10.229 new es
方法一、命令行操作
一、创建快照存储库
1.修改老的es配置文件，用于快照存储库。
[root@elk-server /home]# cat /etc/elasticsearch/elasticsearch.yml |grep path.repo
path.repo: /home/esdata
如果你也是集群模式的话 必须要创建并挂载共享目录
2、分配权限并重启es
[root@elk-server /home]# chown -R elasticsearch:elasticsearch esdata [root@elk-server /home]# systemctl restart elaseicesarch [root@elk-server /home]# ll |grep esdata drwxr-xr-x 3 elasticsearch elasticsearch 134 Jun 8 08:13 esdata
3、创建ES本地存储库
curl -XPUT -uxuewenlong:bsh@123 http://10.0.20.74:9200/_snapshot/my_back -H "Content-Type: application/json" -d ' { "type": "fs", "settings": { "location": "/home/esdata/", #存储库位置，要和配置文件里一致用户组是es "max_restore_bytes_per_sec":"10mb", #恢复最大速率 "compress":"true", #是否压缩 "max_snapshot_bytes_per_sec":"10mb", #创建最大速率 "chunk_size":"100mb" #压缩块大小 } }'
4、验证存储库
curl -XGET -uxuewenlong:bsh@123 http://10.0.20.74:9200/_snapshot/ { "my_back": { "type": "fs", "settings": { "chunk_size": "100M", "location": "/home/esdata/", "max_restore_bytes_per_sec": "10mb", "compress": "true", "max_snapshot_bytes_per_sec": "10mb" } } }
二、备份索引
1、备份指定的索引
curl -XPUT -uxuewenlong:bsh@123 http://10.0.20.74:9200/_snapshot/my_back/esdata?wait_for_completion=true -H "Content-Type: application/json" -d ' { "indices": "miniprogram-prod*" #写入需要备份的索引,多个所以以,分隔 如"index1,index2" 支持正则 }'
2、备份当前库所有索引
curl -XPUT -uxuewenlong:bsh@123 http://10.0.20.74:9200/_snapshot/my_back/esdata?wait_for_completion=true -H "Content-Type: application/json"
3、验证备份快照
curl -XGET -uxuewenlong:bsh@123 http://10.0.20.74:9200/_snapshot/my_back/esdata { "snapshots": [{ "snapshot": "esdata", "uuid": "t3BNDuiCRwK_uAIz71RCbQ", "version_id": 7050099, "version": "7.5.0", "indices": ["miniprogram-prod-aspect-az.log.2020.01", "miniprogram-prod-aspect-az.log.2020.03", "miniprogram-prod-aspect-az.log.2020.06", "miniprogram-prod-access-az.log.2019.12", "miniprogram-prod-aspect-az.log.2020.02", "miniprogram-prod-aspect-bz.log.2020.02", "miniprogram-prod-access-az.log.2020.06", "miniprogram-prod-aspect-az.log.2020.05", "miniprogram-prod-aspect-az.log.2020.04", "miniprogram-prod-access-az.log.2020.04", "miniprogram-prod-access-az.log.2020.03", "miniprogram-prod-aspect-bz.log.2020.05", "miniprogram-prod-access-az.log.2020.01", "miniprogram-prod-access-az.log.2020.05", "miniprogram-prod-access-bz.log.2020.03", "miniprogram-prod-aspect-bz.log.2020.04", "miniprogram-prod-access-bz.log.2020.06", "miniprogram-prod-aspect-bz.log.2020.03", "miniprogram-prod-access-bz.log.2020.02", "miniprogram-prod-access-az.log.2020.02", "miniprogram-prod-aspect-bz.log.2020.01", "miniprogram-prod-access-bz.log.2020.01", "miniprogram-prod-access-bz.log.2020.05", "miniprogram-prod-access-bz.log.2020.04", "miniprogram-prod-aspect-bz.log.2020.06"], "include_global_state": true, "metadata": { "policy": "miniesdata" }, "state": "SUCCESS", "start_time": "2020-06-08T07:48:00.006Z", "start_time_in_millis": 1591602480006, "end_time": "2020-06-08T08:01:57.507Z", "end_time_in_millis": 1591603317507, "duration_in_millis": 837501, "failures": [], "shards": { "total": 25, "failed": 0, "successful": 25 } }] }
三、迁移到新的es集群中
[root@elk-server /home]# zip -r esdata.zip esdata [root@elk-server /home]# scp /home/esdata.zip xuewenlong@10.0.10.229:/home
1、登录新es集群操作
解压es数据，分配权限
[root@ip-10-0-10-229 home]# unzip esdata.zip [root@ip-10-0-10-229 home]# ll drwxr-xr-x. 9 root root 115 Jun 9 00:58 bsh drwx------. 3 ec2-user ec2-user 113 Jun 8 08:25 ec2-user drwxr-xr-x. 3 xuewenlong xuewenlong 134 Jun 9 01:31 esdata [root@ip-10-0-10-229 home]#chown -R elasticsearch:elasticsearch esdata
2、修改es配置文件
[root@ip-10-0-10-229 home]# cat /etc/elasticsearch/elasticsearch.yml |grep path.repo path.repo: /home/esdata
3、创建一样的存储库
**路径以及权限一定要一样
curl -XPUT -uxuewenlong:bsh@123 http://10.0.10.229:9200/_snapshot/my_back -H "Content-Type: application/json" -d ' { "type": "fs", "settings": { "location": "/home/esdata/", #存储库位置，要和配置文件里一致用户组是es "max_restore_bytes_per_sec":"10mb", #恢复最大速率 "compress":"true", #是否压缩 "max_snapshot_bytes_per_sec":"10mb", #创建最大速率 "chunk_size":"100mb" #压缩块大小 } }'
4、验证备份是否存在
curl -XGET -uelastic:bsh@123 http://10.0.10.229:9200/_snapshot/my_back/esdata "snapshots": [{ "snapshot": "esdata", "uuid": "t3BNDuiCRwK_uAIz71RCbQ", "version_id": 7050099, "version": "7.5.0", "indices": ["miniprogram-prod-aspect-az.log.2020.01", "miniprogram-prod-aspect-az.log.2020.03", "miniprogram-prod-aspect-az.log.2020.06", "miniprogram-prod-access-az.log.2019.12", "miniprogram-prod-aspect-az.log.2020.02", "miniprogram-prod-aspect-bz.log.2020.02", "miniprogram-prod-access-az.log.2020.06", "miniprogram-prod-aspect-az.log.2020.05", "miniprogram-prod-aspect-az.log.2020.04", "miniprogram-prod-access-az.log.2020.04", "miniprogram-prod-access-az.log.2020.03", "miniprogram-prod-aspect-bz.log.2020.05", "miniprogram-prod-access-az.log.2020.01", "miniprogram-prod-access-az.log.2020.05", "miniprogram-prod-access-bz.log.2020.03", "miniprogram-prod-aspect-bz.log.2020.04", "miniprogram-prod-access-bz.log.2020.06", "miniprogram-prod-aspect-bz.log.2020.03", "miniprogram-prod-access-bz.log.2020.02", "miniprogram-prod-access-az.log.2020.02", "miniprogram-prod-aspect-bz.log.2020.01", "miniprogram-prod-access-bz.log.2020.01", "miniprogram-prod-access-bz.log.2020.05", "miniprogram-prod-access-bz.log.2020.04", "miniprogram-prod-aspect-bz.log.2020.06"], "include_global_state": true, "metadata": { "policy": "miniesdata" }, "state": "SUCCESS", "start_time": "2020-06-08T07:48:00.006Z", "start_time_in_millis": 1591602480006, "end_time": "2020-06-08T08:01:57.507Z", "end_time_in_millis": 1591603317507, "duration_in_millis": 837501, "failures": [], "shards": { "total": 25, "failed": 0, "successful": 25 } }] }
四.数据恢复到新集群中
1、恢复单条索引
curl -XPOST -uxuewenlong:bsh@123 http://10.0.10.229:9200/_snapshot/my_back/esdata/_restore -H "Content-Type: application/json" -d ' { "miniprogram-prod*" }'
2、恢复全部索引
curl -XPOST -uxuewenlong:bsh@123 http://10.0.10.229:9200/_snapshot/my_back/esdata/_restore -H "Content-Type: application/json"
3、删除快照
curl -XDELETE -uxuewenlong:bsh@123 -uxuewenlong:bsh@123 http://10.0.10.229:9200/_snapshot/my_back/esdata/
方法二、kibana操作
1.修改老的es配置文件，用于快照存储库。
[root@elk-server /home]# cat /etc/elasticsearch/elasticsearch.yml |grep path.repo
path.repo: /home/esdata
一、创建快照存储库
1、在旧的kibana上创建快照存储库
2、在旧的kibana验证存储库
二、创建策略备份索引
1、在旧的kibana创建策略
支持cron定时任务 还是比较方便的
2、在旧的kibana进行数据备份
保存完策略之后点击小按钮就会立马进行备份
3、在旧的kibana备份完查看备份
三、迁移到新的es集群中
1、登陆服务器old es 服务器讲备份数据打包 移至新的es服务器
[root@elk-server /home]# zip -r esdata.zip esdata [root@elk-server /home]# scp /home/esdata.zip xuewenlong@10.0.10.229:/home
2、登录新es集群操作解压es数据，分配权限
[root@ip-10-0-10-229 home]# unzip esdata.zip [root@ip-10-0-10-229 home]# ll drwxr-xr-x. 9 root root 115 Jun 9 00:58 bsh drwx------. 3 ec2-user ec2-user 113 Jun 8 08:25 ec2-user drwxr-xr-x. 3 xuewenlong xuewenlong 134 Jun 9 01:31 esdata [root@ip-10-0-10-229 home]#chown -R elasticsearch:elasticsearch esdata
3、在新的kibana创建存储库
4、在新的kibana刷新浏览器 查看恢复快照
5、在新的kibana查看恢复状态
四、验证es数据
在新的kibana 上创建索引，数据是存在的
至此es数据迁移成功