MongoDB复制集架构

    

mongodb的复制至少需要两个节点。其中一个是主节点,负责处理客户端请求,其余的都是从节点,负责复制主节点上的数据。

mongodb各个节点常见的搭配方式为:一主一从、一主多从。

主节点记录在其上的所有操作oplog,从节点定期轮询主节点获取这些操作,然后对自己的数据副本执行这些操作,从而保证从节点的数据与主节点一致。

MongoDB复制结构图如下所示:


副本集特征:

  • N 个节点的集群

  • 任何节点可作为主节点

  • 所有写入操作都在主节点上

  • 自动故障转移

  • 自动恢复


    

Replica Set角色

Replica Set 的成员是一堆有着同样的数据内容 mongod 的实例集合,包含以下三类角色:

  1. 主节点(Primary)
    是 Replica Set 中唯一的接收写请求的节点,并将写入指令记录到 oplog 上。副本节点通过复制 oplog 的写入指令同步主节点的数据。Secondary。一个 Replica Set 有且只有Primary 节点,当Primar挂掉后,其他 Secondary 或者 Arbiter 节点会重新选举出来一个主节点。应用程序的默认读取请求也是发到 Primary节点处理的。

  2. 副本节点(Secondary)
    通过复制主节点 oplog 中的指令与主节点保持同样的数据集,当主节点挂掉的时候,参与主节点选举。

  3. 仲裁者(Arbiter)
    不存储实际应用数据,只投票参与选取主节点,但不会被选举成为主节点。



复制集架构配置

架构:

Arbite:127.0.0.1:27017  

primary:127.0.0.1:27018

secondary: 127.0.0.1:27019


配置文件内容如下,只要修改logpath、dbpath、pidfilepath和端口即可

arbiter.conf 

# syslog配置
logpath=/var/log/mongo/logs/mongod1.log
logappend=true
timeStampFormat=iso8601-utc
# storage 存储配置
dbpath=/data/mongo/repldb1
directoryperdb=true
# processManagement 进程管理配置
fork=true
# 进程 PID 文件保存目录
pidfilepath=/var/run/mongod1.pid
# net 网络配置
bind_ip=127.0.0.1
port=27017 
# security 配置
#auth=true
# 复制集
replSet=test-set


primary.conf

# syslog配置
logpath=/var/log/mongo/logs/mongod2.log
logappend=true
timeStampFormat=iso8601-utc
# storage 存储配置
dbpath=/data/mongo/repldb2
directoryperdb=true
# processManagement 进程管理配置
fork=true
# 进程 PID 文件保存目录
pidfilepath=/var/run/mongod2.pid
# net 网络配置
bind_ip=127.0.0.1
port=27018
# security 配置
#auth=true
# 复制集
replSet=test-set


secondary.conf

# syslog配置
logpath=/var/log/mongo/logs/mongod3.log
logappend=true
timeStampFormat=iso8601-utc
# storage 存储配置
dbpath=/data/mongo/repldb3
directoryperdb=true
# processManagement 进程管理配置
fork=true
# 进程 PID 文件保存目录
pidfilepath=/var/run/mongod3.pid
# net 网络配置
bind_ip=127.0.0.1
port=27019
# security 配置
#auth=true
# 复制集
replSet=test-set


分别启动这三个实例

mongod --config arbiter.conf 
mongod --config primary.conf 
mongod --config secondary.conf


初始化:

使用mongodb客户端登陆两个常规节点(primary和secondary)中的任何一个,执行如下命令

mongod 127.0.0.1:27018

登陆后执行初始化命令


> rs.initiate(
... {"_id":"test-set",
... "members":[
... {"_id":1,"host":"127.0.0.1:27018"},
... {"_id":2,"host":"127.0.0.1:27019"}
... ]
... });
{ "ok" : 1 }
test-set:SECONDARY> rs.addArb("127.0.0.1:27017")
{ "ok" : 1 }


查看集群状态:

test-set:PRIMARY> rs.status()
{
        "set" : "test-set",
        "date" : ISODate("2018-04-19T07:22:22.813Z"),
        "myState" : 1,
        "term" : NumberLong(1),
        "heartbeatIntervalMillis" : NumberLong(2000),
        "optimes" : {
                "lastCommittedOpTime" : {
                        "ts" : Timestamp(1524122535, 1),
                        "t" : NumberLong(1)
                },
                "appliedOpTime" : {
                        "ts" : Timestamp(1524122535, 1),
                        "t" : NumberLong(1)
                },
                "durableOpTime" : {
                        "ts" : Timestamp(1524122535, 1),
                        "t" : NumberLong(1)
                }
        },
        "members" : [
                {
                        "_id" : 1,
                        "name" : "127.0.0.1:27018",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 1303,
                        "optime" : {
                                "ts" : Timestamp(1524122535, 1),
                                "t" : NumberLong(1)
                        },
                        "optimeDate" : ISODate("2018-04-19T07:22:15Z"),
                        "electionTime" : Timestamp(1524121903, 1),
                        "electionDate" : ISODate("2018-04-19T07:11:43Z"),
                        "configVersion" : 2,
                        "self" : true
                },
                {
                        "_id" : 2,
                        "name" : "127.0.0.1:27019",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 649,
                        "optime" : {
                                "ts" : Timestamp(1524122535, 1),
                                "t" : NumberLong(1)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(1524122535, 1),
                                "t" : NumberLong(1)
                        },
                        "optimeDate" : ISODate("2018-04-19T07:22:15Z"),
                        "optimeDurableDate" : ISODate("2018-04-19T07:22:15Z"),
                        "lastHeartbeat" : ISODate("2018-04-19T07:22:21.472Z"),
                        "lastHeartbeatRecv" : ISODate("2018-04-19T07:22:21.472Z"),
                        "pingMs" : NumberLong(0),
                        "syncingTo" : "127.0.0.1:27018",
                        "configVersion" : 2
                },
                {
                        "_id" : 3,
                        "name" : "127.0.0.1:27017",
                        "health" : 1,
                        "state" : 7,
                        "stateStr" : "ARBITER",
                        "uptime" : 69,
                        "lastHeartbeat" : ISODate("2018-04-19T07:22:21.471Z"),
                        "lastHeartbeatRecv" : ISODate("2018-04-19T07:22:18.484Z"),
                        "pingMs" : NumberLong(0),
                        "configVersion" : 2
                }
        ],
        "ok" : 1
}


登录127.0.0.1:27019  角色显示为Secondary

mongo 127.0.0.1:27019
2018-04-19T07:00:44.075Z I CONTROL  [initandlisten] 
test-set:SECONDARY> 
test-set:SECONDARY> 
test-set:SECONDARY>


在primary上创建数据,测试secondary上是否会同步

test-set:PRIMARY> show dbs
admin  0.000GB
local  0.000GB
test-set:PRIMARY> use testdb
switched to db testdb
test-set:PRIMARY> show collections
test-set:PRIMARY> db.test.insert({"name":"user1"});
WriteResult({ "nInserted" : 1 })


登录secondary,查看

test-set:SECONDARY> rs.slaveOk()
test-set:SECONDARY> show dbs
admin   0.000GB
local   0.000GB
testdb  0.000GB
test-set:SECONDARY> use testdb
switched to db testdb
test-set:SECONDARY> show collections
test
test-set:SECONDARY> db.test.find()
{ "_id" : ObjectId("5ad844efa46a748c5a90fe41"), "name" : "user1" }


secondary上插入数据报错,说明secondary上没有写的权限

test-set:SECONDARY>  db.test.insert({"name":"user2"});
WriteResult({ "writeError" : { "code" : 10107, "errmsg" : "not master" } })


验证复制集架构容灾功能

kill掉primary的进程,登录127.0.0.1:27019

test-set:PRIMARY> show dbs
admin   0.000GB
local   0.000GB
testdb  0.000GB
test-set:PRIMARY> rs.status()
{
        "set" : "test-set",
        "date" : ISODate("2018-04-19T07:44:28.912Z"),
        "myState" : 1,
        "term" : NumberLong(2),
        "heartbeatIntervalMillis" : NumberLong(2000),
        "optimes" : {
                "lastCommittedOpTime" : {
                        "ts" : Timestamp(1524123775, 1),
                        "t" : NumberLong(1)
                },
                "appliedOpTime" : {
                        "ts" : Timestamp(1524123862, 1),
                        "t" : NumberLong(2)
                },
                "durableOpTime" : {
                        "ts" : Timestamp(1524123862, 1),
                        "t" : NumberLong(2)
                }
        },
        "members" : [
                {
                        "_id" : 1,
                        "name" : "127.0.0.1:27018",
                        "health" : 0,
                        "state" : 8,
                        "stateStr" : "(not reachable/healthy)",
                        "uptime" : 0,
                        "optime" : {
                                "ts" : Timestamp(0, 0),
                                "t" : NumberLong(-1)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(0, 0),
                                "t" : NumberLong(-1)
                        },
                        "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
                        "optimeDurableDate" : ISODate("1970-01-01T00:00:00Z"),
                        "lastHeartbeat" : ISODate("2018-04-19T07:44:27.023Z"),
                        "lastHeartbeatRecv" : ISODate("2018-04-19T07:42:59.830Z"),
                        "pingMs" : NumberLong(0),
                        "lastHeartbeatMessage" : "Connection refused",
                        "configVersion" : -1
                },
                {
                        "_id" : 2,
                        "name" : "127.0.0.1:27019",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 2625,
                        "optime" : {
                                "ts" : Timestamp(1524123862, 1),
                                "t" : NumberLong(2)
                        },
                        "optimeDate" : ISODate("2018-04-19T07:44:22Z"),
                        "infoMessage" : "could not find member to sync from",
                        "electionTime" : Timestamp(1524123790, 1),
                        "electionDate" : ISODate("2018-04-19T07:43:10Z"),
                        "configVersion" : 2,
                        "self" : true
                },
                {
                        "_id" : 3,
                        "name" : "127.0.0.1:27017",
                        "health" : 1,
                        "state" : 7,
                        "stateStr" : "ARBITER",
                        "uptime" : 1395,
                        "lastHeartbeat" : ISODate("2018-04-19T07:44:27.009Z"),
                        "lastHeartbeatRecv" : ISODate("2018-04-19T07:44:28.662Z"),
                        "pingMs" : NumberLong(0),
                        "configVersion" : 2
                }
        ],
        "ok" : 1
}


可以看到127.0.0.1:27018 有一个 "stateStr" : "(not reachable/healthy)"的报错,而127.0.0.1:27019已经变成了primary角色


启动原来的primary.conf配置的实例,角色由原来的primary变成了secondary

test-set:SECONDARY> show dbs
2018-04-19T15:50:04.113+0800 E QUERY    [thread1] Error: listDatabases failed:{
        "ok" : 0,
        "errmsg" : "not master and slaveOk=false",
        "code" : 13435,
        "codeName" : "NotMasterNoSlaveOk"
} :
_getErrorWithCode@src/mongo/shell/utils.js:25:13
Mongo.prototype.getDBs@src/mongo/shell/mongo.js:62:1
shellHelper.show@src/mongo/shell/utils.js:782:19
shellHelper@src/mongo/shell/utils.js:672:15
@(shellhelp2):1:1
test-set:SECONDARY>


其他命令rs.help()

 test-set:PRIMARY> rs.help()
        rs.status()                                { replSetGetStatus : 1 } checks repl set status
        rs.initiate()                              { replSetInitiate : null } initiates set with default settings
        rs.initiate(cfg)                           { replSetInitiate : cfg } initiates set with configuration cfg
        rs.conf()                                  get the current configuration object from local.system.replset
        rs.reconfig(cfg)                           updates the configuration of a running replica set with cfg (disconnects)
        rs.add(hostportstr)                        add a new member to the set with default attributes (disconnects)
        rs.add(membercfgobj)                       add a new member to the set with extra attributes (disconnects)
        rs.addArb(hostportstr)                     add a new member which is arbiterOnly:true (disconnects)
        rs.stepDown([stepdownSecs, catchUpSecs])   step down as primary (disconnects)
        rs.syncFrom(hostportstr)                   make a secondary sync from the given member
        rs.freeze(secs)                            make a node ineligible to become primary for the time specified
        rs.remove(hostportstr)                     remove a host from the replica set (disconnects)
        rs.slaveOk()                               allow queries on secondary nodes
        rs.printReplicationInfo()                  check oplog size and time range
        rs.printSlaveReplicationInfo()             check replica set members and replication lag
        db.isMaster()                              check who is primary

rs.slaveOk():secondary默认不允许读写,执行此命令后,从节点有可读权限


rs.conf():查看配置情况

test-set:SECONDARY> rs.conf()
{
        "_id" : "test-set",
        "version" : 2,
        "protocolVersion" : NumberLong(1),
        "members" : [
                {
                        "_id" : 1,
                        "host" : "127.0.0.1:27018",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {
                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                },
                {
                        "_id" : 2,
                        "host" : "127.0.0.1:27019",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {
                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                },
                {
                        "_id" : 3,
                        "host" : "127.0.0.1:27017",
                        "arbiterOnly" : true,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {
                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                }
        ],
        "settings" : {
                "chainingAllowed" : true,
                "heartbeatIntervalMillis" : 2000,
                "heartbeatTimeoutSecs" : 10,
                "electionTimeoutMillis" : 10000,
                "catchUpTimeoutMillis" : 60000,
                "getLastErrorModes" : {
                },
                "getLastErrorDefaults" : {
                        "w" : 1,
                        "wtimeout" : 0
                },
                "replicaSetId" : ObjectId("5ad841247ffb05dbb6b4a13b")
        }
}


rs.add():新增一个节点到副本集中,注意只能在primary上操作,secondary角色执行该命令会报错,例如:

test-set:SECONDARY> rs.add("127.0.0.1:27020")

{
        "ok" : 0,
        "errmsg" : "replSetReconfig should only be run on PRIMARY, but my state is SECONDARY; use the \"force\" argument to override",
        "code" : 10107,
        "codeName" : "NotMaster"
}


测试:

登录primary,新建一个third.conf的配置文件,并启动实例,端口为127.0.0.1:27020

 mongod --config third.conf 
 # netstat -tunlp | grep 27020
tcp        0      0 127.0.0.1:27020             0.0.0.0:*                   LISTEN      13528/mongod


添加新的secondary角色

test-set:PRIMARY>  rs.add("127.0.0.1:27020")
{ "ok" : 1 }
test-set:PRIMARY> rs.status()
{
        "set" : "test-set",
        "date" : ISODate("2018-04-19T08:16:36.910Z"),
        "myState" : 1,
        "term" : NumberLong(2),
        "heartbeatIntervalMillis" : NumberLong(2000),
        "optimes" : {
                "lastCommittedOpTime" : {
                        "ts" : Timestamp(1524125792, 1),
                        "t" : NumberLong(2)
                },
                "appliedOpTime" : {
                        "ts" : Timestamp(1524125792, 1),
                        "t" : NumberLong(2)
                },
                "durableOpTime" : {
                        "ts" : Timestamp(1524125792, 1),
                        "t" : NumberLong(2)
                }
        },
        "members" : [
                {
                        "_id" : 1,
                        "name" : "127.0.0.1:27018",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 1661,
                        "optime" : {
                                "ts" : Timestamp(1524125792, 1),
                                "t" : NumberLong(2)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(1524125792, 1),
                                "t" : NumberLong(2)
                        },
                        "optimeDate" : ISODate("2018-04-19T08:16:32Z"),
                        "optimeDurableDate" : ISODate("2018-04-19T08:16:32Z"),
                        "lastHeartbeat" : ISODate("2018-04-19T08:16:36.730Z"),
                        "lastHeartbeatRecv" : ISODate("2018-04-19T08:16:35.732Z"),
                        "pingMs" : NumberLong(0),
                        "syncingTo" : "127.0.0.1:27019",
                        "configVersion" : 3
                },
                {
                        "_id" : 2,
                        "name" : "127.0.0.1:27019",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 4553,
                        "optime" : {
                                "ts" : Timestamp(1524125792, 1),
                                "t" : NumberLong(2)
                        },
                        "optimeDate" : ISODate("2018-04-19T08:16:32Z"),
                        "electionTime" : Timestamp(1524123790, 1),
                        "electionDate" : ISODate("2018-04-19T07:43:10Z"),
                        "configVersion" : 3,
                        "self" : true
                },
                {
                        "_id" : 3,
                        "name" : "127.0.0.1:27017",
                        "health" : 1,
                        "state" : 7,
                        "stateStr" : "ARBITER",
                        "uptime" : 3323,
                        "lastHeartbeat" : ISODate("2018-04-19T08:16:36.730Z"),
                        "lastHeartbeatRecv" : ISODate("2018-04-19T08:16:36.713Z"),
                        "pingMs" : NumberLong(0),
                        "configVersion" : 3
                },
                {
                        "_id" : 4,
                        "name" : "127.0.0.1:27020",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 110,
                        "optime" : {
                                "ts" : Timestamp(1524125792, 1),
                                "t" : NumberLong(2)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(1524125792, 1),
                                "t" : NumberLong(2)
                        },
                        "optimeDate" : ISODate("2018-04-19T08:16:32Z"),
                        "optimeDurableDate" : ISODate("2018-04-19T08:16:32Z"),
                        "lastHeartbeat" : ISODate("2018-04-19T08:16:36.729Z"),
                        "lastHeartbeatRecv" : ISODate("2018-04-19T08:16:35.914Z"),
                        "pingMs" : NumberLong(0),
                        "syncingTo" : "127.0.0.1:27019",
                        "configVersion" : 3
                }
        ],
        "ok" : 1
}


rs.remove(hostportstr):删除一个secondary节点

test-set:PRIMARY> rs.remove("127.0.0.1:27020")
{ "ok" : 1 }
test-set:PRIMARY> rs.status()
{
        "set" : "test-set",
        "date" : ISODate("2018-04-19T08:21:46.241Z"),
        "myState" : 1,
        "term" : NumberLong(2),
        "heartbeatIntervalMillis" : NumberLong(2000),
        "optimes" : {
                "lastCommittedOpTime" : {
                        "ts" : Timestamp(1524126097, 1),
                        "t" : NumberLong(2)
                },
                "appliedOpTime" : {
                        "ts" : Timestamp(1524126097, 1),
                        "t" : NumberLong(2)
                },
                "durableOpTime" : {
                        "ts" : Timestamp(1524126097, 1),
                        "t" : NumberLong(2)
                }
        },
        "members" : [
                {
                        "_id" : 1,
                        "name" : "127.0.0.1:27018",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 1971,
                        "optime" : {
                                "ts" : Timestamp(1524126097, 1),
                                "t" : NumberLong(2)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(1524126097, 1),
                                "t" : NumberLong(2)
                        },
                        "optimeDate" : ISODate("2018-04-19T08:21:37Z"),
                        "optimeDurableDate" : ISODate("2018-04-19T08:21:37Z"),
                        "lastHeartbeat" : ISODate("2018-04-19T08:21:45.652Z"),
                        "lastHeartbeatRecv" : ISODate("2018-04-19T08:21:42.657Z"),
                        "pingMs" : NumberLong(0),
                        "configVersion" : 4
                },
                {
                        "_id" : 2,
                        "name" : "127.0.0.1:27019",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 4863,
                        "optime" : {
                                "ts" : Timestamp(1524126097, 1),
                                "t" : NumberLong(2)
                        },
                        "optimeDate" : ISODate("2018-04-19T08:21:37Z"),
                        "electionTime" : Timestamp(1524123790, 1),
                        "electionDate" : ISODate("2018-04-19T07:43:10Z"),
                        "configVersion" : 4,
                        "self" : true
                },
                {
                        "_id" : 3,
                        "name" : "127.0.0.1:27017",
                        "health" : 1,
                        "state" : 7,
                        "stateStr" : "ARBITER",
                        "uptime" : 3632,
                        "lastHeartbeat" : ISODate("2018-04-19T08:21:45.656Z"),
                        "lastHeartbeatRecv" : ISODate("2018-04-19T08:21:42.656Z"),
                        "pingMs" : NumberLong(0),
                        "configVersion" : 4
                }
        ],
        "ok" : 1
}


编辑修改集群配置(只能在primary节点上操作)

登录到MongoDB的primary实例

执行cfg=rs.conf()命令,将集群配置保存在cfg变量中

test-set:PRIMARY> cfg=rs.conf()
{
        "_id" : "test-set",
        "version" : 4,
        "protocolVersion" : NumberLong(1),
        "members" : [
                {
                        "_id" : 1,
                        "host" : "127.0.0.1:27018",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {
                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                },
                {
                        "_id" : 2,
                        "host" : "127.0.0.1:27019",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {
                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                },
                {
                        "_id" : 3,
                        "host" : "127.0.0.1:27017",
                        "arbiterOnly" : true,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {
                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                }
        ],
        "settings" : {
                "chainingAllowed" : true,
                "heartbeatIntervalMillis" : 2000,
                "heartbeatTimeoutSecs" : 10,
                "electionTimeoutMillis" : 10000,
                "catchUpTimeoutMillis" : 60000,
                "getLastErrorModes" : {
                },
                "getLastErrorDefaults" : {
                        "w" : 1,
                        "wtimeout" : 0
                },
                "replicaSetId" : ObjectId("5ad841247ffb05dbb6b4a13b")
        }
}


查看集群第一个成员的配置

test-set:PRIMARY> cfg.members[0]
{
        "_id" : 1,
        "host" : "127.0.0.1:27018",
        "arbiterOnly" : false,
        "buildIndexes" : true,
        "hidden" : false,
        "priority" : 1,
        "tags" : {
        },
        "slaveDelay" : NumberLong(0),
        "votes" : 1
}


修改第一个成员的优先级配置

test-set:PRIMARY> cfg.members[0].priority=2
2

##修改后,再次查看配置,优先级已经被修改
test-set:PRIMARY> cfg.members[0]
{
        "_id" : 1,
        "host" : "127.0.0.1:27018",
        "arbiterOnly" : false,
        "buildIndexes" : true,
        "hidden" : false,
        "priority" : 2,
        "tags" : {
        },
        "slaveDelay" : NumberLong(0),
        "votes" : 1
}


保存配置使生效

test-set:PRIMARY> rs.reconfig(cfg)
{ "ok" : 1 }