异地备份mongo

为了提升逼格，我决定使用英文版的的这句作为开头，尽管我最初想到的是中文版orz

天灾人祸这种事，总归是有的。尽管服务器的灾难性故障不常发生，概率也比遇到鬼或是遇到爱情来得高些，将风险考虑在内总归是好的，所谓有备无患。如果你已经在自己的架构中将单点故障考虑进去了，就不必往下拉啦，继续舔酸奶盖去吧~如果你的云服务商已经考虑了单点故障，你也没啥好做啦。

如果两者都没有，你应该请我喝杯星冰乐再往下拉。

ps：我最近多是在处理性能优化，数据迁移，安全策略，系统稳定性这类事，简直是伪装成开发的运维orz

策略

采用异地备份来应对单点故障导致的数据丢失

最初的思路

首先明确我们需要备份哪些数据,关于edX数据，可以参考我此前的这篇文章

一番深(hu)思(si)熟(luan)虑(xiang)之后，你会发现，我们关心的数据无非是数据库(mongodb/mysql)里存的，以及静态文件(/edx/var/edxapp/staticfiles)

思路很简单，每天定时备份数据库和静态文件，将其同步到远程服务器。只保存最近3，5天的备份文件，逾期的删除

这里需要注意的问题可能有三点：

• 不要影响到用户使用，备份是消耗服务器资源的，会造成服务器压力，写个定时任务，让其在凌晨做
• 采用增量同步的方式节省带宽
• 任务完成后，给自己发个邮件，确定成功执行了

那么写个shell脚本基本就完事啦，主要可能涉及的工具：

• mongodump
• mysqldump
• tar
• scp/rsync
• crontab

backup闪亮登场

就在我shell脚本快写完的时候，在github上发现了这个东西：backup

一言话概括就是

Easy full stack backup operations on UNIX-like systems

稍微扫一眼Overview,分分钟爱不释手啊。

从文档中不难发现，backup本质上是对以上我们提到的工具做了封装。

能不写shell，我是尽量不写shell的。

先读下文档吧。

install backup

backup是ruby写的，使用gem安装。不得不说ruby的DSL能力真强啊

sudo gem install backup

哦，顺便说下，我在ruby1.8.7和ruby1.9.3下都没成功，之后在ubuntu12.04上手动编译了ruby2.0.0才安装好backup。也是蛮诡异的，可能是我机器的问题，文档里说ruby1.8.7和ruby1.9.3应该都是没有问题的

backup demo

从一个简单的案例开始。

任务描述：备份我的home目录下的所有文件，同步到远程服务器的~/backup_my_home目录，只保留最近的5个备份

在shell里执行

backup generate:model --trigger backup_my_home \

  --archives --storages='scp'  --compressor='gzip'

会生成:Generated model file: '/home/USERNAME/Backup/models/backup_my_home.rb'

编辑以上文件就行

# encoding: utf-8

Model.new(:backup_my_home, 'Description for backup_my_home') do archive :my_archive do |archive| # Run the `tar` command using `sudo` # archive.use_sudo archive.add "/home/USERNAME" end # SCP (Secure Copy) [Storage] # store_with SCP do |server| server.username = "my_username" server.password = "my_password" server.ip = "123.45.678.90" server.port = 22 server.path = "~/backup_my_home/" server.keep = 5 end # Gzip [Compressor] compress_with Gzip end 

几乎无需解释对吧:)

开始备份:backup perform -t backup_my_home

开始备份edX

静态文件

backup generate:model --trigger edx_staticfile_backup \

  --archives --storages='scp'  --compressor='gzip'

# encoding: utf-8

Model.new(:edx_staticfile_backup, 'Description for edx_staticfile_backup') do # archive.root '/path/to/archive/root' # archive :my_archive do |archive| # Run the `tar` command using `sudo` # archive.use_sudo #archive.add "/path/to/a/file.rb" archive.add "/edx/var/edxapp/staticfiles" #archive.exclude "/path/to/a/excluded_file.rb" #archive.exclude "/path/to/a/excluded_folder" end ## # SCP (Secure Copy) [Storage] #你需要先去目标服务器建立~/edx_backups/staticfile目录 store_with SCP do |server| server.username = "my_username" server.password = "my_password" server.ip = "123.45.678.90" server.port = 22 server.path = "~/edx_backups/staticfile" server.keep = 5 # Additional options for the SSH connection. # server.ssh_options = {} end ## # Gzip [Compressor] # compress_with Gzip end 

备份mongo

databases

>mongo

>show dbs

admin                            

cs_comments_service_development  

edxapp                           

local                            

为mongo添加用户

>mongo

>use edxapp

>db.addUser(“wwj”, “xxx”)

backup generate:model --trigger edx_mongodb_backup \

  --databases="mongodb" --storages='scp'  --compressor='gzip' --notifiers='mail'

# encoding: utf-8

Model.new(:edx_mongodb_backup, 'Description for edx_mongodb_backup') do ## # MongoDB [Database] # database MongoDB do |db| db.name = "edxapp" #db.name = :all db.username = "wwj" db.password = "xxx" db.host = "localhost" db.port = 27017 db.ipv6 = false #db.only_collections = ["only", "these", "collections"] #db.additional_options = [] db.lock = false db.oplog = false end ## # SCP (Secure Copy) [Storage] # store_with SCP do |server| server.username = "my_username" server.password = "my_password" server.ip = "123.45.678.90" server.port = 22 server.path = "~/edx_backups/mongo" server.keep = 5 # Additional options for the SSH connection. # server.ssh_options = {} end ## # Gzip [Compressor] # compress_with Gzip notify_by Mail do |mail| mail.on_success = true mail.on_warning = true mail.on_failure = true mail.from = "[email protected]" mail.to = "[email protected]" mail.address = "smtp.gmail.com" mail.port = 587 mail.domain = "your.host.name" mail.user_name = "[email protected]" mail.password = "my_password" mail.authentication = "plain" mail.encryption = :starttls end end 

备份mysql

databases

text

>mysql -u root >show databases; +--------------------+ | Database | +--------------------+ | information_schema | | edxapp | | mysql | | ora | | performance_schema | | test | | xqueue | +--------------------+ 

backup generate:model --trigger edx_mysql_backup \

  --databases="mysql" --storages='scp'  --compressor='gzip'

# encoding: utf-8



Model.new(:edx_mysql_backup, 'Description for edx_mysql_backup') do ## # MySQL [Database] # database MySQL do |db| # To dump all databases, set `db.name = :all` (or leave blank) #db.name = "my_database_name" db.name = "edxapp" db.username = "root" db.password = "" db.host = "localhost" db.port = 3306 #mysql.sock 位置有变 #db.socket = "/tmp/mysql.sock" # Note: when using `skip_tables` with the `db.name = :all` option, # table names should be prefixed with a database name. # e.g. ["db_name.table_to_skip", ...] #db.skip_tables = ["skip", "these", "tables"] #db.only_tables = ["only", "these", "tables"] db.additional_options = ["--quick", "--single-transaction"] end ## # SCP (Secure Copy) [Storage] # store_with SCP do |server| server.username = "my_username" server.password = "my_password" server.ip = "123.45.678.90" server.port = 22 server.path = "~/edx_backups/mysql" server.keep = 5 # Additional options for the SSH connection. # server.ssh_options = {} end ## # Gzip [Compressor] # compress_with Gzip end 

开始备份

backup perform -t edx_staticfile_backup

backup perform -t edx_mongodb_backup

backup perform -t edx_mysql_backup

定时任务

每天凌晨3:00

sudo crontab -e写入

00 03 * * * /usr/local/bin/backup perform -t edx_staticfile_backup

00 03 * * * /usr/local/bin/backup perform -t edx_mongodb_backup

00 03 * * * /usr/local/bin/backup perform -t edx_mysql_backup

后续

之后我们还是采用rsync的做法吧，不然数据大到一定程度，太浪费带宽了

只要更改store_with就行

注意：数据服务器到备份服务器应当配置好ssh免登陆

  compress_with Gzip do |gzip| gzip.rsyncable = true end store_with RSync do |storage| time = Time.now storage.mode = :ssh storage.host = "123.45.67.89" storage.port = 22 storage.ssh_user = "wwj" #免密码ssh #每周为一轮，使用的是同名覆盖 path = time.strftime '%A' storage.path = "~/edx_backups/staticfile/#{ path }" end 

注意

~/Backup/log/backup.log 中的时间是ruby的时间，与本地差（晚）8个小时。时区的问题