Mongodb Replica Set高可用

概述

对于mongodb的相关介绍，在此不多累赘，如需了解请见:

http://www.mongodb.org/display/DOCS/Home 

本文目的是说明如下几个功能：

1.在测试环境构建Replica Set方案

2.在服务器端和客户端测试failover

3.测试天然的读写分离，减轻服务器压力

4.添加权限认证功能

部署Replica Sets 方案 

Relica Sets使用的是n个mongod节点，构建具备自动的容错功能(auto-failover),自动恢复的(auto-recovery)的高可用方案,理论上需要三个mongodb实例，在这个测试环境中，我采用的方案是2个mongod+1个arbiter。

(0)基础环境

mongod1 : deploy 
ip : 10.12.7.107 
port : 27031 
$ mkdir -p /data/db/0 
./mongod  --dbpath /data/db/0 --port 27031 --replSet myset 

mongod2 : doc 
ip : 10.12.7.108 
port : 27032 
$ mkdir -p /data/db/1 
./mongod  --dbpath /data/db/1 --port 27032 --replSet myset 

mongod3 : deploy 
ip : 10.12.7.107 
port : 27033 
$ mkdir -p /data/db/2 
./mongod  --dbpath /data/db/2 --port 27033 --replSet myset 

(1)实例化Replica Sets

任选一个mongod节点，mongo shell 登陆进去,执行如下内容：

> config = {_id: 'myset', members: [ 
        {_id: 0, host: '10.12.7.107:27031'}, 
        {_id: 1, host: '10.12.7.108:27032'}, 
        {_id: 2, host: '10.12.7.107:27033', arbiterOnly: true}]} 
> rs.initiate(config) 
> rs.conf() #查看配置信息 
> rs.staus() 

到此，Replica Sets配置就算完成了，相关的配置信息保存在local数据库中。

 

(2)执行流程说明

在同一时刻，每组 Replica Sets 只有一个 Primary，用于接受写操作。而后会异步复制到其他成员数据库中。一旦 primary 死掉，会自动投票选出接任的 primary 来，原服务器恢复后成为普通成员。如果数据尚未从先前的 primary 复制到成员服务器，有可能会丢失数据。

在服务器端和客户端测试failover

(1)服务器端

在 mongo 中向 primary (27031) 插入数据：

> use test 
switched to db test 
> db.users.insert({name:"terrylc"}) 

会在mongodb的输出信息中，看到如下信息：

#27031 
Tue Oct 18 08:44:07 [FileAllocator] allocating new datafile /data/db/0/test.ns, filling with zeroes... 
Tue Oct 18 08:44:07 [FileAllocator] done allocating datafile /data/db/0/test.ns, size: 16MB,  took 0.054 secs 
Tue Oct 18 08:44:07 [FileAllocator] allocating new datafile /data/db/0/test.0, filling with zeroes... 
Tue Oct 18 08:44:07 [FileAllocator] done allocating datafile /data/db/0/test.0, size: 64MB,  took 0.153 secs 
Tue Oct 18 08:44:07 [FileAllocator] allocating new datafile /data/db/0/test.1, filling with zeroes... 
Tue Oct 18 08:44:07 [conn3] building new index on { _id: 1 } for test.users 
Tue Oct 18 08:44:07 [conn3] done for 0 records 0secs 
Tue Oct 18 08:44:07 [conn3] insert test.users 213ms 
Tue Oct 18 08:44:09 [FileAllocator] done allocating datafile /data/db/0/test.1, size: 128MB,  took 1.527 secs 
 
#27032 
Tue Oct 18 23:43:02 [FileAllocator] allocating new datafile /data/db/1/test.ns, filling with zeroes... 
Tue Oct 18 23:43:02 [FileAllocator] done allocating datafile /data/db/1/test.ns, size: 16MB,  took 0.054 secs 
Tue Oct 18 23:43:02 [FileAllocator] allocating new datafile /data/db/1/test.0, filling with zeroes... 
Tue Oct 18 23:43:03 [FileAllocator] done allocating datafile /data/db/1/test.0, size: 64MB,  took 0.556 secs 
Tue Oct 18 23:43:03 [FileAllocator] allocating new datafile /data/db/1/test.1, filling with zeroes... 
Tue Oct 18 23:43:03 [replica set sync] building new index on { _id: 1 } for test.users 
Tue Oct 18 23:43:03 [replica set sync] done for 0 records 0secs 
Tue Oct 18 23:43:03 [FileAllocator] done allocating datafile /data/db/1/test.1, size: 128MB,  took 0.166 secs 
 
#27033 
Tue Oct 18 08:42:22 [ReplSetHealthPollTask] replSet info 10.12.7.107:27031 is up 
Tue Oct 18 08:42:22 [ReplSetHealthPollTask] replSet member 10.12.7.107:27031 PRIMARY 
Tue Oct 18 08:42:22 [ReplSetHealthPollTask] replSet info 10.12.7.108:27032 is up 
Tue Oct 18 08:42:22 [ReplSetHealthPollTask] replSet member 10.12.7.108:27032 RECOVERING 
Tue Oct 18 08:42:34 [ReplSetHealthPollTask] replSet member 10.12.7.108:27032 SECONDARY 

停止27031服务，观察现象： 

#27032 
Tue Oct 18 23:47:47 [conn2] end connection 10.12.7.107:58587 
Tue Oct 18 23:47:47 [replica set sync] replSet syncThread: 10278 dbclient error communicating with server: 10.12.7.107:27031 
Tue Oct 18 23:47:48 [ReplSetHealthPollTask] DBClientCursor::init call() failed 
Tue Oct 18 23:47:48 [ReplSetHealthPollTask] replSet info 10.12.7.107:27031 is down (or slow to respond): DBClientBase::findOne: transport error: 10.12.7.107:27031 query: { replSetHeartbeat: "myset", v: 1, pv: 1, checkEmpty: false, from: "10.12.7.108:27032" } 
Tue Oct 18 23:47:48 [rs Manager] replSet info electSelf 1 
Tue Oct 18 23:47:48 [rs Manager] replSet couldn't elect self, only received -9999 votes 
Tue Oct 18 23:47:54 [rs Manager] replSet info electSelf 1 
Tue Oct 18 23:47:54 [rs Manager] replSet PRIMARY 
 
#27033 
Tue Oct 18 08:48:52 [conn2] end connection 10.12.7.107:43768 
Tue Oct 18 08:48:53 [conn3] 10.12.7.108:27032 is trying to elect itself but 10.12.7.107:27031 is already primary and more up-to-date 
Tue Oct 18 08:48:54 [ReplSetHealthPollTask] DBClientCursor::init call() failed 
Tue Oct 18 08:48:54 [ReplSetHealthPollTask] replSet info 10.12.7.107:27031 is down (or slow to respond): DBClientBase::findOne: transport error: 10.12.7.107:27031 query: { replSetHeartbeat: "myset", v: 1, pv: 1, checkEmpty: false, from: "10.12.7.107:27033" } 
Tue Oct 18 08:48:59 [conn3] replSet info voting yea for 1 
Tue Oct 18 08:49:00 [ReplSetHealthPollTask] replSet member 10.12.7.108:27032 PRIMARY 
}}} 

Mongod 27032 被选为 Primary,在对应的终端下，执行shell mongo 

$ ./mongo localhost:27032 
> rs.status() 
{ 
    "set" : "myset", 
    "date" : ISODate("2011-10-18T15:53:01Z"), 
    "myState" : 1, 
    "members" : [ 
        { 
            "_id" : 0, 
            "name" : "10.12.7.107:27031", 
            "health" : 0, 
            "state" : 1, 
            "stateStr" : "(not reachable/healthy)", 
            "uptime" : 0, 
            "optime" : { 
                "t" : 1318952647000, 
                "i" : 1 
            }, 
            "optimeDate" : ISODate("2011-10-18T15:44:07Z"), 
            "lastHeartbeat" : ISODate("2011-10-18T15:47:46Z"), 
            "errmsg" : "socket exception" 
        }, 
        { 
            "_id" : 1, 
            "name" : "10.12.7.108:27032", 
            "health" : 1, 
            "state" : 1, 
            "stateStr" : "PRIMARY", 
            "optime" : { 
                "t" : 1318952647000, 
                "i" : 1 
            }, 
            "optimeDate" : ISODate("2011-10-18T15:44:07Z"), 
            "self" : true 
        }, 
        { 
            "_id" : 2, 
            "name" : "10.12.7.107:27033", 
            "health" : 1, 
            "state" : 7, 
            "stateStr" : "ARBITER", 
            "uptime" : 705, 
            "optime" : { 
                "t" : 0, 
                "i" : 0 
            }, 
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"), 
            "lastHeartbeat" : ISODate("2011-10-18T15:53:00Z") 
        } 
    ], 
    "ok" : 1 
} 
 
}}} 

查询先前插入的记录正常:

myset:PRIMARY> db.users.findOne() 
{ "_id" : ObjectId("4e9d9ec76df8f2f72bd9c45a"), "name" : "terrylc" } 

如果重新启动27031对应的mongod，此实例将会变成secondary

Tue Oct 18 23:56:46 [ReplSetHealthPollTask] replSet info 10.12.7.107:27031 is up 
Tue Oct 18 23:56:46 [ReplSetHealthPollTask] replSet member 10.12.7.107:27031 SECONDARY 
Tue Oct 18 23:56:47 [conn6] I am already primary, 10.12.7.107:27031 can try again once I've stepped down 
Tue Oct 18 23:56:48 [initandlisten] connection accepted from 10.12.7.107:44777 #7 
Tue Oct 18 23:56:49 [slaveTracking] building new index on { _id: 1 } for local.slaves 
Tue Oct 18 23:56:49 [slaveTracking] done for 0 records 0secs 
Tue Oct 18 23:57:03 [conn5] end connection 127.0.0.1:58555 

(2)客户端测试

从客户端连接 Replica Sets，需要 drivers 支持，本例中，我采用pymongo2.0.1

* 测试功能可用性

目标：当一个主节点出现故障时，备用节点是否可以转化为主节点。

import pymongo 
import sys 
def testInsert(): 
    flag = 0 
    while True: 
        if flag == 4: 
            print "try to connect server 4 times but failed,so give up" 
            break 
        try: 
            conn = pymongo.Connection(host=["10.12.7.107:27031", "10.12.7.108:27032", "10.12.7.107:27033"]) 
            db = conn.test 
            post = {"name1": "terrylc"} 
            db.users.insert(post) 
            conn.disconnect() 
            return sys.exit(0) 
        except Exception, e: 
            print e 
            flag += 1 
 
if __name__ == '__main__': 
    testInsert() 
}}} 

结论：

在服务器端的某个实例当掉后，客户端依然能够正常工作。

 * 测试大规模写数据完整性

需求：当进行大规模写的时候，如果几个节点在特定时间交叉宕机后，能否确保数据完成性

目标：用python客户端向db中连续插入5000条记录，期望在插入完成后db中增加5000条记录：

def testInsert2(index): 
    try: 
        conn = pymongo.Connection(host=["10.12.7.107:27031", "10.12.7.108:27032", "10.12.7.107:27033"]) 
        db = conn.test 
        while index <= 5000: 
            post = {"testTAG": index} 
            time.sleep(1) 
            objectId = db.users.save(post, safe=True,w=2) 
            if objectId is not None: 
                index += 1 
        conn.disconnect() 
    except Exception, e: 
        print e 
        print "you index is :" + str(index) 
        time.sleep(2) 
        testInsert2(index) 
 
if __name__ == '__main__': 
    testInsert2(1) 
}}} 

结论:

在测试过程，分别模拟几台mongodb节点宕机现象，客户端在进行重新连接后能够继续进行写操作，并且最后数据一致且都能查询到。

测试天然的读写分离，减轻服务器压力

 其实很简单的方式是在客户端的请求中添加如下请求字段即可：

conn = pymongo.Connection(host=["10.12.7.107:27031", "10.12.7.108:27032", "10.12.7.107:27033"], 
                                      slave_okay=True) 
 
slave_okay=True  # 这个是关键，可以将读写分离 

添加权限认证功能

Authentication was added in 1.7.5(mongodb)

Replica Sets 的认证跟原本的单台的认证是不同的，最直接的表现在使用replica Sets时，启动实例方式如下： 

$ echo "this is my super secret key" > mykey 
$ chmod 600 mykey 
$ mongod --keyFile mykey # other options... 
 
$ echo "this is my super secret key" > mykey 
$ chmod 600 mykey 
$ mongod --keyFile mykey # other options... 
 
$ echo "this is my super secret key" > mykey 
$ chmod 600 mykey 
$ mongod --keyFile mykey # other options... 

然后在对应的数据库中添加用户即可。

启动方式不再使用--auth

补充

如果后续需要考虑写服务器的负载问题，可以在现在的组件外层加上Sharding层。

来自http://www.cnblogs.com/terrylc/archive/2011/12/13/2285955.html