LDAP + Gerrit搭建code review系统（四） --- backup备份
对于任何一个正式使用的系统，备份都是非常重要的任务。在这个LDAP+Gerrit的系统中，有三部分的内容需要备份：LDAP中的用户和群组定义，Gerrit中的git目录，以及mysql中的code review记录。
1. LDAP备份（LDAP Replication）
这是一个使用provider-consumer模式的增量备份方案，provider的任何变动都会立刻备份到consumer上。具体内容详见https://help.ubuntu.com/lts/serverguide/openldap-server.html
Provider Configuration
Begin by configuring the Provider.
-
Create an LDIF file with the following contents and name it provider_sync.ldif:
# Add indexes to the frontend db. dn: olcDatabase={1}hdb,cn=config changetype: modify add: olcDbIndex olcDbIndex: entryCSN eq - add: olcDbIndex olcDbIndex: entryUUID eq #Load the syncprov and accesslog modules. dn: cn=module{0},cn=config changetype: modify add: olcModuleLoad olcModuleLoad: syncprov - add: olcModuleLoad olcModuleLoad: accesslog # Accesslog database definitions dn: olcDatabase={2}hdb,cn=config objectClass: olcDatabaseConfig objectClass: olcHdbConfig olcDatabase: {2}hdb olcDbDirectory: /var/lib/ldap/accesslog olcSuffix: cn=accesslog olcRootDN: cn=admin,dc=example,dc=com olcDbIndex: default eq olcDbIndex: entryCSN,objectClass,reqEnd,reqResult,reqStart # Accesslog db syncprov. dn: olcOverlay=syncprov,olcDatabase={2}hdb,cn=config changetype: add objectClass: olcOverlayConfig objectClass: olcSyncProvConfig olcOverlay: syncprov olcSpNoPresent: TRUE olcSpReloadHint: TRUE # syncrepl Provider for primary db dn: olcOverlay=syncprov,olcDatabase={1}hdb,cn=config changetype: add objectClass: olcOverlayConfig objectClass: olcSyncProvConfig olcOverlay: syncprov olcSpNoPresent: TRUE # accesslog overlay definitions for primary db dn: olcOverlay=accesslog,olcDatabase={1}hdb,cn=config objectClass: olcOverlayConfig objectClass: olcAccessLogConfig olcOverlay: accesslog olcAccessLogDB: cn=accesslog olcAccessLogOps: writes olcAccessLogSuccess: TRUE # scan the accesslog DB every day, and purge entries older than 7 days olcAccessLogPurge: 07+00:00 01+00:00Change the rootDN in the LDIF file to match the one you have for your directory.
-
The apparmor profile for slapd will not need to be adjusted for the accesslog database location since/etc/apparmor.d/local/usr.sbin.slapd contains:
/var/lib/ldap/ r, /var/lib/ldap/** rwk,
Create a directory, set up a databse config file, and reload the apparmor profile:
sudo -u openldap mkdir /var/lib/ldap/accesslog sudo -u openldap cp /var/lib/ldap/DB_CONFIG /var/lib/ldap/accesslog sudo service apparmor reload
-
Add the new content and, due to the apparmor change, restart the daemon:
sudo ldapadd -Q -Y EXTERNAL -H ldapi:/// -f provider_sync.ldif sudo service slapd restart
The Provider is now configured.
Consumer Configuration
And now configure the Consumer.
-
Install the software by going through Installation. Make sure the slapd-config databse is identical to the Provider's. In particular, make sure schemas and the databse suffix are the same.
-
Create an LDIF file with the following contents and name it consumer_sync.ldif:
dn: cn=module{0},cn=config changetype: modify add: olcModuleLoad olcModuleLoad: syncprov dn: olcDatabase={1}hdb,cn=config changetype: modify add: olcDbIndex olcDbIndex: entryUUID eq - add: olcSyncRepl olcSyncRepl: rid=0 provider=ldap://ldap01.example.com bindmethod=simple binddn="cn=admin,dc=example,dc=com" credentials=secret searchbase="dc=example,dc=com" logbase="cn=accesslog" logfilter="(&(objectClass=auditWriteObject)(reqResult=0))" schemachecking=on type=refreshAndPersist retry="60 +" syncdata=accesslog - add: olcUpdateRef olcUpdateRef: ldap://ldap01.example.comEnsure the following attributes have the correct values:
-
provider (Provider server's hostname -- ldap01.example.com in this example -- or IP address)
-
binddn (the admin DN you're using)
-
credentials (the admin DN password you're using)
-
searchbase (the database suffix you're using)
-
olcUpdateRef (Provider server's hostname or IP address)
-
rid (Replica ID, an unique 3-digit that identifies the replica. Each consumer should have at least one rid)
-
-
Add the new content:
sudo ldapadd -Q -Y EXTERNAL -H ldapi:/// -f consumer_sync.ldif
You're done. The two databases (suffix: dc=example,dc=com) should now be synchronizing.
Testing
Once replication starts, you can monitor it by running
ldapsearch -z1 -LLLQY EXTERNAL -H ldapi:/// -s base -b dc=example,dc=com contextCSN dn: dc=example,dc=com contextCSN: 20120201193408.178454Z#000000#000#000000
on both the provider and the consumer. Once the output (20120201193408.178454Z#000000#000#000000 in the above example) for both machines match, you have replication. Every time a change is done in the provider, this value will change and so should the one in the consumer(s).
If your connection is slow and/or your ldap database large, it might take a while for the consumer's contextCSN match the provider's. But, you will know it is progressing since the consumer's contextCSN will be steadly increasing.
If the consumer's contextCSN is missing or does not match the provider, you should stop and figure out the issue before continuing. Try checking the slapd (syslog) and the auth log files in the provider to see if the consumer's authentication requests were successful or its requests to retrieve data (they look like a lot of ldapsearch statements) return no errors.
To test if it worked simply query, on the Consumer, the DNs in the database:
sudo ldapsearch -Q -LLL -Y EXTERNAL -H ldapi:/// -b dc=example,dc=com dn
You should see the user 'john' and the group 'miners' as well as the nodes 'People' and 'Groups'.
备份机首先需要遵照前面三个步骤正常安装Gerrit，注意在安装过程中使用同样的参数，除了将ldap和mysql的server设为localhost，然后将gerrit server目录copy到备份机，不能覆盖的文件是gerrit/etc目录中以ssh_开头的几个密钥文件。Gerrit的war包中自带一个replication的插件，通过对这个插件的配置，可以实现将任何版本库的change推送到备份机上。如果这个插件在开始配置的时候没有安装，可以通过命令行： java -jar gerrit.war init -d review_site --install_plugin=replication来重新安装。
配置replication需要在gerrit目录的etc子目录中创建文件replication.config，其内容类似以下面的部分：
[remote "sdc-mirror"]
url = ssh://19005113@xrs-linux-server:29418/${name}.git
replicationDelay = 0
[remote "roc-mirror"]
url = ssh://[email protected]:29418/${name}.git
projects = ImageSuite
push = +refs/heads/*:refs/heads/*
push = +refs/tags/*:refs/tags/*
[gerrit]
defaultForceUpdate = true
在上面的config中，${name}.git是占位符，gerrit会自动将其替逐一换成现有的所有git版本库。本例中，笔者定义了两处备份，第一处sdc-mirror将备份所有的git库，并且此处replicationDelay置为0，也就是gerrit收到的任何改动都会立刻自动备份到sdc-mirror，缺省值是15s，也就是有15秒的延时。第二处roc-mirror只备份ImageSuite.git这一个版本库，而且备份的内容只为分支和标签。
如果我们打开一个git版本库目录，eg. gerrit/git/SandBox.git，查看refs子目录，里面的目录内容非常有趣，除了heads和tags这样常规的git目录用来记录分支和标记之外，还有changes目录，用来记录review，meta目录用来记录权限配置，如果开启了reviewnotes plugin，还有notes目录用来记录review comments.如果没有像上例中roc-mirror那样详细定义备份哪些分支，缺省值为refs/*:refs/*，也就是全部内容都做备份。
replication配置好之后，除了重启gerrit server，还可以通过远程的gerrit命令来加载。Gerrit server支持一批gerrit命令，但是这些命令只能从client端通过ssh连接执行，可以运行下面的命令行检验你是否有正确的权限
ssh -p 29418 xrs-gerrit-server gerrit ls-projects
加载plugin的命令为：
ssh -p 29418 xrs-gerrit-server gerrit plugin reload replication
如果命令执行失败，需要检查你的gerrit server是否允许远程调用plugin，在gerrit.config文件中加入：
[plugins]
allowRemoteAdmin = true
远程加载成功后，可以查看gerrit/logs/replication_log检查运行结果。这里需要注意的是：
a. 如果出现hostkey reject错，需要在gerrit用户的.ssh目录下创建config文件，添加远端的备份机如下：
Host xrs-linux-server
IdentityFile ~/.ssh/id_rsa
PreferredAuthentications publickey
Host rocgit01.corp.carestreamhealth.com
IdentityFile ~/.ssh/id_rsa
PreferredAuthentications publickey
b. 在备份机上运行命令"ssh -p 29418 username@host" 检查ssh权限是否开通，如果是第一次连接，回答yes接受gerrit server的public key。特别注意，gerrit server上的备份用户应该上拥有管理员权限，这样同步之后，该用户才能在备份机的gerrit上也拥有管理员权限。c. 注意为备份用户在gerrit web管理页面上添加Create references权限，这是因为每个change都是新添加的reference，如果没有这个权限，会出现gerrit prohibit错，提示你不能将reference推送到远端不存在的分支上去。同样需要开通的还有push的forge author identity和forge committer identity权限，因为所有的refs/changes/*是由gerrit用户创建的，与备份用户身份不同。
3. mysql备份
mysql备份采取master-slave的方式，具体配置方法详见https://dev.mysql.com/doc/refman/5.1/en/replication-howto.html
master将数据存成bin log的方式，slave获得master的bin log后存为relay bin log，然后导入数据，master和slave通过不同的server-id标示。同步开始前，需要将master的数据用dump的方式导入slave，并且在slave端通过change master to语句将对应的master的bin log位置也设置正确。Start Slave开始同步。运行show master status;和show slave status\G;分别查看master端和slave端的数据位置。
4. Gerrit的刷新
即使以上备份都正常进行，在备份机的gerrit页面上也看不到最新的更新，这是因为gerrit使用了lucene进行二次index,这个索引就在gerrit/index目录下。如果要刷新gerrit以反应最新的数据情况，先运行bin/gerrit.sh stop，然后在gerrit根目录中运行java -jar bin/gerrit.war reindex重建索引，之后bin/gerrit.sh start重启gerrit就能看到所有的更新了。