Elasticsearch如何备份到HDFS

es备份到hdfs

简介

elasticsearch副本提供了高可靠性;它可以保证节点丢失而不会中断服务,但是副本不能做到容灾备份,所以需要把elasticsearch的数据被分到hdfs中。

测试环境

elasticsearch 6.3.2
Hadoop 2.9.1

操作步骤

  1. 安装repository-hdfs

     进入ES的目录,执行命令:bin/elasticsearch-plugin install repository-hdfs
     如果需要移除插件,执行命令:bin/elasticsearch-plugin remove repository-hdfs
    
  2. 建立仓库命令

    curl -H "Content-Type: application/json" -XPUT 'http://192.168.2.227:9200/_snapshot/backup' -d '{"type":"hdfs", "settings":{ "path":"/elasticsearch/respositories/my_hdfs_repository", "uri":"hdfs://192.168.2.202:9000" }}’
    

    备注:一个仓库可以包含多个快照

  3. 查看仓库

      curl -XGET  'http://192.168.2.227:9200/_snapshot/backup?pretty'
    
  4. 快照特定的索引

      curl -H "Content-Type: application/json" -XPUT 'http://192.168.2.227:9200/_snapshot/backup/snapshot_1' -d '{"indices":"blog"}'
    
  5. 快照多个索引

      curl -H "Content-Type: application/json" -XPUT 'http://192.168.2.227:9200/_snapshot/backup/snapshot_1' -d '{"indices":"blog1,blog2"}'
    

    经过这一步操作可以查看到hdfs里多了备份文件

  6. 查看特定快照信息

      curl -XGET 'http://192.168.2.227:9200/_snapshot/backup/snapshot_1?pretty'
    

    INITIALIZING:分片在检查集群状态看看自己是否可以被快照。这个一般是非常快的。
    STARTED:数据正在被传输到仓库。
    FINALIZING:数据传输完成;分片现在在发送快照元数据。
    DONE:快照完成!
    FAILED:快照处理的时候碰到了错误,这个分片/索引/快照不可能完成了。检查你的日志获取更多信息。

  7. 恢复特定索引

      curl -XPOST 'http://192.168.2.227:9200/_snapshot/backup/snapshot_1/_restore?pretty'
    

    删除快照

      curl -XDELETE 'http://192.168.2.227:9200/_snapshot/backup/snapshot_1’
    

    监控快照

      curl -XGET 'http://192.168.2.227:9200/_snapshot/backup/snapshot_1/_status'
    
  8. 执行脚本

     #!/bin/bash
     current_time=$(date +%Y%m%d%H%M%S)
     command_prefix="http://192.168.2.227:9200/_snapshot/backup/all_"
     command=$command_prefix$current_time
     curl -H "Content-Type: application/json" -XPUT $command -d '{"indices":"index*,logstash*,nginx*,magicianlog*,invokelog*,outside*"}'
    
  9. crontab定时任务,每天备份一次

    0 0 * * * /root/shell/snapshot_all_hdfs.sh >>/root/shell/logs/snapshot_all_day.log 2>&1
    

常见问题处理

    failed to create snapshot","caused_by":{"type":"access_control_exception","reason":"Permission denied: user=elasticsearch, access=WRITE, inode=\"/elasticsearch/respositories/my_hdfs_repository

修改hdfs-site.xml,把dfs.permissions改成false
提前在hdfs中新建备份的文件目录, /elasticsearch/respositories/my_hdfs_repository
并修改hdfs文件权限,bin/hdfs dfs -chmod 777

参考资料

https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html
https://blog.csdn.net/ysl1242157902/article/details/79219061

你可能感兴趣的:(Elasticsearch如何备份到HDFS)