Mongodb副本集:
NoSQL的产生就是为了解决大数据量、高扩展性、高性能、灵活数据模型、高可用性。但是光通过主从模式的架构远远达不到上面几点,由此MongoDB设计了副本集和分片的功能,先来用用副本集。
Mongodb副本集的同步机制:
数据复制的目的是使数据得到最大的可用性,避免单点故障引起的整站不能访问的情况的发生,Mongodb的副本集在同一时刻只有一台服务器是可以写的,副本集的主从复制也是一个异步同步的过程,是slave端从primary端获取日志,然后在自己身上完全顺序的执行日志所记录的各种操作(该日志是不记录查询操作的),这个日志就是local数据库中的oplog.rs表,默认在64位机器上这个表是比较大的,占磁盘大小的5%,oplog.rs的大小可以在启动参数中设定:--oplogSize 1000,单位是M。
创建/etc/yum.repos.d/mongodb-enterprise.repo文件,这样就可以使用yum来安装mongodb了。
[mongodb-org-3.0]
name=MongoDB Repository baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/3.0/x86_64/ gpgcheck=1 enabled=1 gpgkey=https://www.mongodb.org/static/pgp/server-3.0.asc |
每一个release版本的.repo文件可以在repository找到,记住奇数镜像版本(比如e.g.2.5)是开发版本,放在生成环境是不太合适的。
在官方网站可以下载到比较新的版本,下载地址:
https://www.mongodb.org/dr/fastdl.mongodb.org/linux/mongodb-linux-x86_64-3.2.3.tgz/download;
我这里下载的是稳定版本:mongodb-linux-x86_64-3.0.3.tgz,安装也是用的3.0.3; 3.0.3的下载地址为:http://download.csdn.net/detail/mchdba/9546256
tar -xvf mongodb-linux-x86_64-3.0.3.tgz -C /usr/local/ |
安装mongodbreplica sets,至少需要3台linux服务器,而初始化副本集合,本来应该是3台服务器,但是这里只是测试环境,启用了3个mongodb端口进程来来模拟3个mongodb的服务器达到一样的效果,在实际中,可以部署在多台mongodb服务器上,如下表所示:
Mongodb 主机 |
服务器端口地址 |
默认角色 |
mongodb主机 1 |
localhost:27017 |
primary |
mongodb主机 2 |
localhost:27018 |
secondary |
mongodb主机 3 |
localhost:27019 |
secondary |
开始启动3个mongodb服务:
# 启动第一个mongodb 27017,默认为primary服务器 mkdir -p /data/mongodb/log27017 mkdir -p /data/mongodb/data27017 /usr/local/mongodb-linux-x86_64-3.0.3/bin/mongod --replSet repset --port 27017 --dbpath /data/mongodb/data27017 --oplogSize 2048 --logpath /data/mongodb/log27017/mongod.log &
# 启动第二个mongodb 27018 mkdir -p /data/mongodb/log27018 mkdir -p /data/mongodb/data27018 /usr/local/mongodb-linux-x86_64-3.0.3/bin/mongod --replSet repset --port 27018 --dbpath /data/mongodb/data27018 --oplogSize 2048 --logpath /data/mongodb/log27018/mongod.log &
# 启动第三个 mongodb 27019 mkdir -p /data/mongodb/log27019 mkdir -p /data/mongodb/data27019 /usr/local/mongodb-linux-x86_64-3.0.3/bin/mongod --replSet repset --port 27019 --dbpath /data/mongodb/data27019 --oplogSize 2048 --logpath /data/mongodb/log27019/mongod.log & |
在任何一台mongodb实例上登录,进入admin库,执行config命令:
> use admin switched to db admin > config = { _id:"repset", members:[ ... {_id:0,host:"mongodb_server:27017"}, ... {_id:1,host:"mongodb_server:27018"}, ... {_id:2,host:"mongodb_server:27019"}] ... };
> config = { _id:"repset", members:[ ... ... {_id:0,host:"mongodb_server:27017"}, ... ... {_id:1,host:"mongodb_server:27018"}, ... ... {_id:2,host:"mongodb_server:27019"}] ... ... }; { "_id" : "repset", "members" : [ { "_id" : 0, "host" : "mongodb_server:27017" }, { "_id" : 1, "host" : "mongodb_server:27018" }, { "_id" : 2, "host" : "mongodb_server:27019" } ] } > |
# 初始化副本集配置,使用rs.initiate(config);命令
rs.initiate(config);
> rs.initiate(config);
{ "ok" : 1 }
repset:SECONDARY>
看到config配置里面的第一个默认为primary,后面的都为secondary,然后查看"optime": Timestamp(1465641737, 1),如果集合中的mongodb的optime一致,表是集群中各个mongodb数据是保持一致性的:
# 随意登录一个mongodb服务 [root@oracle_master ~]# /usr/local/mongodb-linux-x86_64-3.0.3/bin/mongo localhost:27017 MongoDB shell version: 3.0.3 connecting to: localhost:27017/test Server has startup warnings: 2016-06-10T07:56:33.333-0700 I CONTROL [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended. 2016-06-10T07:56:33.333-0700 I CONTROL [initandlisten] repset:PRIMARY> use admin switched to db admin repset:PRIMARY>
# 查看状态如下: repset:PRIMARY> rs.status(); { "set" : "repset", "date" : ISODate("2016-06-10T15:04:54.045Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "mongodb_server:27017", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 501, "optime" : Timestamp(1465571046, 1), "optimeDate" : ISODate("2016-06-10T15:04:06Z"), "electionTime" : Timestamp(1465571050, 1), "electionDate" : ISODate("2016-06-10T15:04:10Z"), "configVersion" : 1, "self" : true }, { "_id" : 1, "name" : "mongodb_server:27018", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 47, "optime" : Timestamp(1465571046, 1), "optimeDate" : ISODate("2016-06-10T15:04:06Z"), "lastHeartbeat" : ISODate("2016-06-10T15:04:52.493Z"), "lastHeartbeatRecv" : ISODate("2016-06-10T15:04:52.612Z"), "pingMs" : 1, "configVersion" : 1 }, { "_id" : 2, "name" : "mongodb_server:27019", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 47, "optime" : Timestamp(1465571046, 1), "optimeDate" : ISODate("2016-06-10T15:04:06Z"), "lastHeartbeat" : ISODate("2016-06-10T15:04:52.642Z"), "lastHeartbeatRecv" : ISODate("2016-06-10T15:04:52.642Z"), "pingMs" : 0, "lastHeartbeatMessage" : "could not find member to sync from", "configVersion" : 1 } ], "ok" : 1 } repset:PRIMARY> |
可以看到27107的mongodb默认成为了primary,而27018和27019的成为了secondary了:
[root@oracle_master ~]# tail -f /data/mongodb/log27017/mongod.log 2016-06-11T03:42:19.664-0700 I REPL [ReplicationExecutor] replSet election succeeded, assuming primary role 2016-06-11T03:42:19.664-0700 I REPL [ReplicationExecutor] transition to PRIMARY 2016-06-11T03:42:19.686-0700 I REPL [ReplicationExecutor] Member mongodb_server:27019 is now in state STARTUP2 2016-06-11T03:42:19.738-0700 I REPL [rsSync] transition to primary complete; database writes are now permitted 2016-06-11T03:42:22.798-0700 I NETWORK [initandlisten] connection accepted from 192.168.136.128:18197 #6 (4 connections now open) 2016-06-11T03:42:22.812-0700 I NETWORK [initandlisten] connection accepted from 192.168.136.128:18198 #7 (5 connections now open) 2016-06-11T03:42:22.849-0700 I NETWORK [conn6] end connection 192.168.136.128:18197 (4 connections now open) 2016-06-11T03:42:22.906-0700 I NETWORK [conn7] end connection 192.168.136.128:18198 (3 connections now open) 2016-06-11T03:42:23.668-0700 I REPL [ReplicationExecutor] Member mongodb_server:27018 is now in state SECONDARY 2016-06-11T03:42:23.689-0700 I REPL [ReplicationExecutor] Member mongodb_server:27019 is now in state SECONDARY |
(1)先进去主库primary的mongodb上,录入数据:
# 登录primary主库 [root@oracle_master mongodb]# /usr/local/mongodb-linux-x86_64-3.0.3/bin/mongo localhost:27017 MongoDB shell version: 3.0.3 connecting to: localhost:27017/test Server has startup warnings: 2016-06-11T03:41:07.938-0700 I CONTROL [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended. 2016-06-11T03:41:07.938-0700 I CONTROL [initandlisten] repset:PRIMARY> # 录入数据 repset:PRIMARY> use yes_db; switched to db yes_db repset:PRIMARY> db.yes_users.insert({"uid":"20160611_0001"},{"uname":"tim.man"}); WriteResult({ "nInserted" : 1 }) repset:PRIMARY> |
(2)去第一个备库27018上,验证数据,有了yes_users表的数据,如下:
[root@oracle_master mongodb]# /usr/local/mongodb-linux-x86_64-3.0.3/bin/mongo localhost:27018 MongoDB shell version: 3.0.3 connecting to: localhost:27018/test Server has startup warnings: 2016-06-11T03:41:14.305-0700 I CONTROL [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended. 2016-06-11T03:41:14.305-0700 I CONTROL [initandlisten] repset:SECONDARY> use yes_db; switched to db yes_db repset:SECONDARY> show tables; 2016-06-11T04:03:02.088-0700 E QUERY Error: listCollections failed: { "note" : "from execCommand", "ok" : 0, "errmsg" : "not master" } at Error (<anonymous>) at DB._getCollectionInfosCommand (src/mongo/shell/db.js:646:15) at DB.getCollectionInfos (src/mongo/shell/db.js:658:20) at DB.getCollectionNames (src/mongo/shell/db.js:669:17) at shellHelper.show (src/mongo/shell/utils.js:625:12) at shellHelper (src/mongo/shell/utils.js:524:36) at (shellhelp2):1:1 at src/mongo/shell/db.js:646 # mongodb默认是从主节点读写数据的,副本节点上不允许读,需要设置副本节点可以读。 repset:SECONDARY> db.getMongo().setSlaveOk(); repset:SECONDARY> show tables; system.indexes yes_users repset:SECONDARY> db.yes_users.find(); # 看到有数据了 { "_id" : ObjectId("575bef8126150b8de92029eb"), "uid" : "20160611_0001" } repset:SECONDARY> |
(3)去第二个secondary备库27019上,检查数据是否已经复制过来:
[root@oracle_master mongodb]# /usr/local/mongodb-linux-x86_64-3.0.3/bin/mongo localhost:27019 MongoDB shell version: 3.0.3 connecting to: localhost:27019/test Server has startup warnings: 2016-06-11T03:41:20.363-0700 I CONTROL [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended. 2016-06-11T03:41:20.363-0700 I CONTROL [initandlisten] repset:SECONDARY> use yes_db switched to db yes_db repset:SECONDARY> db.getMongo().setSlaveOk();# mongodb默认是从主节点读写数据的,副本节点上不允许读,需要设置副本节点可以读。 repset:SECONDARY> db.yes_users.find();# 看到有数据了,数据已经复制过来。 { "_id" : ObjectId("575bef8126150b8de92029eb"), "uid" : "20160611_0001" } repset:SECONDARY> |
官方介绍:https://docs.mongodb.com/master/core/replica-set-high-availability/
Replica Set High Availability
Replica sets provide high availability using automatic failover. Failover allows a secondary member to become primary if the current primary becomes unavailable. |
当primary不可用,则自动将合适的secondary成员切换成新的primary,接下来的(1)-> (5)就来验证这个:
(1) 关闭27017的mongod服务,即停止primary主库:
[root@oracle_master mongodb]# /usr/local/mongodb-linux-x86_64-3.0.3/bin/mongo localhost:27017 MongoDB shell version: 3.0.3 connecting to: localhost:27017/test Server has startup warnings: 2016-06-11T03:41:07.938-0700 I CONTROL [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended. 2016-06-11T03:41:07.938-0700 I CONTROL [initandlisten] repset:PRIMARY> use admin switched to db admin repset:PRIMARY> db.shutdownServer(); 2016-06-11T04:18:24.148-0700 I NETWORK DBClientCursor::init call() failed server should be down... 2016-06-11T04:18:24.151-0700 I NETWORK trying reconnect to localhost:27017 (127.0.0.1) failed 2016-06-11T04:18:24.152-0700 I NETWORK reconnect localhost:27017 (127.0.0.1) ok 2016-06-11T04:18:24.437-0700 I NETWORK Socket recv() errno:104 Connection reset by peer 127.0.0.1:27017 2016-06-11T04:18:24.437-0700 I NETWORK SocketException: remote: 127.0.0.1:27017 error: 9001 socket exception [RECV_ERROR] server [127.0.0.1:27017] 2016-06-11T04:18:24.437-0700 I NETWORK DBClientCursor::init call() failed 2016-06-11T04:18:24.440-0700 I NETWORK trying reconnect to localhost:27017 (127.0.0.1) failed 2016-06-11T04:18:24.441-0700 W NETWORK Failed to connect to 127.0.0.1:27017, reason: errno:111 Connection refused 2016-06-11T04:18:24.441-0700 I NETWORK reconnect localhost:27017 (127.0.0.1) failed failed couldn't connect to server localhost:27017 (127.0.0.1), connection attempt failed > |
(2) 登录原来的主库primary库的mongodb,就会报错:
[root@oracle_master mongodb]# /usr/local/mongodb-linux-x86_64-3.0.3/bin/mongo localhost:27017 MongoDB shell version: 3.0.3 connecting to: localhost:27017/test 2016-06-11T04:20:40.597-0700 W NETWORK Failed to connect to 127.0.0.1:27017, reason: errno:111 Connection refused 2016-06-11T04:20:40.614-0700 E QUERY Error: couldn't connect to server localhost:27017 (127.0.0.1), connection attempt failed at connect (src/mongo/shell/mongo.js:181:14) at (connect):1:6 at src/mongo/shell/mongo.js:181 exception: connect failed [root@oracle_master mongodb]# |
(3) primary自动切换到下一个mongodb服务,即27018端口的mongodb为primary服务:
[root@oracle_master mongodb]# /usr/local/mongodb-linux-x86_64-3.0.3/bin/mongo localhost:27018 MongoDB shell version: 3.0.3 connecting to: localhost:27018/test Server has startup warnings: 2016-06-11T03:41:14.305-0700 I CONTROL [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended. 2016-06-11T03:41:14.305-0700 I CONTROL [initandlisten] repset:PRIMARY> rs.status(); { "set" : "repset", "date" : ISODate("2016-06-11T11:20:50.730Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "mongodb_server:27017", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : Timestamp(0, 0), "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2016-06-11T11:20:50.467Z"), "lastHeartbeatRecv" : ISODate("2016-06-11T11:18:22.616Z"), "pingMs" : 6, "lastHeartbeatMessage" : "Failed attempt to connect to mongodb_server:27017; couldn't connect to server mongodb_server:27017 (192.168.136.128), connection attempt failed", "configVersion" : -1 }, { "_id" : 1, "name" : "mongodb_server:27018", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2377, "optime" : Timestamp(1465642881, 2), "optimeDate" : ISODate("2016-06-11T11:01:21Z"), "electionTime" : Timestamp(1465643906, 1), "electionDate" : ISODate("2016-06-11T11:18:26Z"), "configVersion" : 1, "self" : true }, { "_id" : 2, "name" : "mongodb_server:27019", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2312, "optime" : Timestamp(1465642881, 2), "optimeDate" : ISODate("2016-06-11T11:01:21Z"), "lastHeartbeat" : ISODate("2016-06-11T11:20:50.177Z"), "lastHeartbeatRecv" : ISODate("2016-06-11T11:20:50.718Z"), "pingMs" : 0, "configVersion" : 1 } ], "ok" : 1 } repset:PRIMARY> |
(4)在27018的后台log里面,也可以看到这一切换记录:
2016-06-11T04:18:25.647-0700 I REPL [ReplicationExecutor] Standing for election 2016-06-11T04:18:25.649-0700 W NETWORK [ReplExecNetThread-6] Failed to connect to 192.168.136.128:27017, reason: errno:111 Connection refused 2016-06-11T04:18:25.649-0700 I REPL [ReplicationExecutor] replSet possible election tie; sleeping 848ms until 2016-06-11T04:18:26.497-0700 2016-06-11T04:18:25.650-0700 I REPL [ReplicationExecutor] Error in heartbeat request to mongodb_server:27017; Location18915 Failed attempt to connect to mongodb_server:27017; couldn't connect to server mongodb_server:27017 (192.168.136.128), connection attempt faile d 2016-06-11T04:18:25.650-0700 W NETWORK [ReplExecNetThread-6] Failed to connect to 192.168.136.128:27017, reason: errno:111 Connection refused 2016-06-11T04:18:25.650-0700 I REPL [ReplicationExecutor] Error in heartbeat request to mongodb_server:27017; Location18915 Failed attempt to connect to mongodb_server:27017; couldn't connect to server mongodb_server:27017 (192.168.136.128), connection attempt faile d 2016-06-11T04:18:26.497-0700 I REPL [ReplicationExecutor] Standing for election 2016-06-11T04:18:26.500-0700 I REPL [ReplicationExecutor] replSet info electSelf 2016-06-11T04:18:26.502-0700 I REPL [ReplicationExecutor] replSet election succeeded, assuming primary role 2016-06-11T04:18:26.502-0700 I REPL [ReplicationExecutor] transition to PRIMARY 2016-06-11T04:18:27.333-0700 I REPL [rsSync] transition to primary complete; database writes are now permitted |
(5)再启动原来旧的primary库27017端口的mongodb(如果是实际上的坏了就是修复后启动),27017就会自动变成secondary状态。
看到后台mongodb日志:
2016-06-11T04:28:33.979-0700 I NETWORK [initandlisten] connection accepted from 192.168.136.128:11453 #170 (2 connections now open) 2016-06-11T04:28:33.979-0700 I NETWORK [conn170] end connection 192.168.136.128:11453 (1 connection now open) 2016-06-11T04:28:33.992-0700 I NETWORK [initandlisten] connection accepted from 192.168.136.128:11454 #171 (2 connections now open) 2016-06-11T04:28:35.422-0700 I REPL [ReplicationExecutor] Member mongodb_server:27017 is now in state SECONDARY 2016-06-11T04:28:55.672-0700 I NETWORK [conn169] end connection 192.168.136.128:11424 (1 connection now open) 2016-06-11T04:28:55.673-0700 I NETWORK [initandlisten] connection accepted from 192.168.136.128:11459 #172 (2 connections now open) |
再查看replica sets状态:
[root@oracle_master mongodb]# /usr/local/mongodb-linux-x86_64-3.0.3/bin/mongo localhost:27018 MongoDB shell version: 3.0.3 connecting to: localhost:27018/test Server has startup warnings: 2016-06-11T03:41:14.305-0700 I CONTROL [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended. 2016-06-11T03:41:14.305-0700 I CONTROL [initandlisten] repset:PRIMARY> rs.status(); { "set" : "repset", "date" : ISODate("2016-06-11T11:30:00.192Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "mongodb_server:27017", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 85, "optime" : Timestamp(1465642881, 2), "optimeDate" : ISODate("2016-06-11T11:01:21Z"), "lastHeartbeat" : ISODate("2016-06-11T11:29:58.752Z"), "lastHeartbeatRecv" : ISODate("2016-06-11T11:29:58.263Z"), "pingMs" : 0, "configVersion" : 1 }, { "_id" : 1, "name" : "mongodb_server:27018", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2927, "optime" : Timestamp(1465642881, 2), "optimeDate" : ISODate("2016-06-11T11:01:21Z"), "electionTime" : Timestamp(1465643906, 1), "electionDate" : ISODate("2016-06-11T11:18:26Z"), "configVersion" : 1, "self" : true }, { "_id" : 2, "name" : "mongodb_server:27019", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2862, "optime" : Timestamp(1465642881, 2), "optimeDate" : ISODate("2016-06-11T11:01:21Z"), "lastHeartbeat" : ISODate("2016-06-11T11:29:59.858Z"), "lastHeartbeatRecv" : ISODate("2016-06-11T11:29:59.804Z"), "pingMs" : 0, "configVersion" : 1 } ], "ok" : 1 } repset:PRIMARY> |
至此,验证完成,和理论上示意图上面说的效果是一致的,mongodb的replica sets可以实现高可用解决方案。
Secondary库启动报错,如下所示:
[root@oracle_master log27018]# more /data/mongodb/log27018/mongod.log 2016-06-11T02:44:32.326-0700 I JOURNAL [initandlisten] journal dir=/data/mongodb/data27018/journal 2016-06-11T02:44:32.326-0700 I JOURNAL [initandlisten] recover : no journal files present, no recovery needed 2016-06-11T02:44:32.326-0700 I JOURNAL [initandlisten] 2016-06-11T02:44:32.326-0700 E JOURNAL [initandlisten] Insufficient free space for journal files 2016-06-11T02:44:32.326-0700 I JOURNAL [initandlisten] Please make at least 3379MB available in /data/mongodb/data27018/journal or use --smallfiles 2016-06-11T02:44:32.326-0700 I JOURNAL [initandlisten] 2016-06-11T02:44:32.327-0700 I STORAGE [initandlisten] exception in initAndListen: 15926 Insufficient free space for journals, terminating 2016-06-11T02:44:32.327-0700 W REPL [initandlisten] ReplicationCoordinatorImpl::shutdown() called before startReplication() finished. Shutting down without cleaning up the replication system 2016-06-11T02:44:32.327-0700 I CONTROL [initandlisten] now exiting 2016-06-11T02:44:32.327-0700 I NETWORK [initandlisten] shutdown: going to close listening sockets... 2016-06-11T02:44:32.327-0700 I NETWORK [initandlisten] removing socket file: /tmp/mongodb-27018.sock 2016-06-11T02:44:32.327-0700 I NETWORK [initandlisten] shutdown: going to flush diaglog... 2016-06-11T02:44:32.327-0700 I NETWORK [initandlisten] shutdown: going to close sockets... 2016-06-11T02:44:32.327-0700 I STORAGE [initandlisten] shutdown: waiting for fs preallocator... 2016-06-11T02:44:32.327-0700 I STORAGE [initandlisten] shutdown: final commit... 2016-06-11T02:44:32.327-0700 I STORAGE [initandlisten] shutdown: closing all files... 2016-06-11T02:44:32.327-0700 I STORAGE [initandlisten] closeAllFiles() finished 2016-06-11T02:44:32.327-0700 I CONTROL [initandlisten] dbexit: rc: 100 [root@oracle_master log27018]# |
解决:
看到Please make at least3379MB available in /data/mongodb/data27018/journal or use –smallfiles,所以清理下磁盘空间,磁盘至少要保证有30G以上的空间才行。
网友段书峰blog地址:blog.itpub.net/15498