mongodb 分片与副本集

  一般线上mongodb服务器都好多台，可以自由搭建mongodb分片集群的，但是不能随意测试，如果想自己随意测试新功能怎么办呢？可以在本地搭建，在一台服务器mongodb1（hch_test_dbm1_121_62）上大家mongodb分片集群，下面分片是示意图(https://docs.mongodb.com/manual/_images/sharded-cluster-production-architecture.png)：

 

OK，接下来就准备开始搭建mongodb分片集群，从安装mongodb开始，然后逐渐部署config servers、mongos servers、shard servers一步步来开始准备。

 

1，下载安装：

https://www.mongodb.org/dr/fastdl.mongodb.org/linux/mongodb-linux-x86_64-3.2.3.tgz/download，我这里下载的是稳定版本：mongodb-linux-x86_64-3.0.3.tgz

 

下载后解压缩

tar -xvf mongodb-linux-x86_64-3.0.3.tgz -C/usr/local/

mkdir /data

mkdir/usr/local/mongodb-linux-x86_64-3.0.3/log/

 

mkdir /data/mongodb

chown -R mongodb.mongodb /data/mongodb

chown -R mongodb.mongodb/usr/local/mongodb-linux-x86_64-3.0.3

 

chown -R mongodb:mongodb/usr/local/mongodb-linux-x86_64-3.0.3 -R

chown -R mongodb:mongodb /data/mongodb -R

 

2、在mongodb1上准备好3个config servers

创建数据目录日志目录：

mkdir  /data/mongodb/config20001 -p

mkdir /data/mongodb/config20002 -p

mkdir /data/mongodb/config20003 -p

mkdir /data/mongodb/logs/ -p

 

启动config servers 进程：

/usr/local/mongodb-linux-x86_64-3.0.3/bin/mongod--configsvr --dbpath /data/mongodb/config20001 --port 20001 --logpath/data/mongodb/logs/configsvr_20001.log --logappend --fork

/usr/local/mongodb-linux-x86_64-3.0.3/bin/mongod--configsvr --dbpath /data/mongodb/config20002 --port 20002 --logpath/data/mongodb/logs/configsvr_20002.log --logappend --fork

/usr/local/mongodb-linux-x86_64-3.0.3/bin/mongod--configsvr --dbpath /data/mongodb/config20003 --port 20003 --logpath/data/mongodb/logs/configsvr_20003.log --logappend --fork

 

 

3、启动mongos路由服务

在mongodb1上启动mongos服务器（可以启动一个也可以启动多个）:

/usr/local/mongodb-linux-x86_64-3.0.3/bin/mongos--configdb mongodb1:20001,mongodb1:20002,mongodb1:20003  --port 40000 --chunkSize 64 --logpath/data/mongodb/logs/mongos40004.log --logappend --fork

 

 

4、开始准备shard servers

 

# 第一个shard1分片副本集

# 创建目录

mkdir /data/mongodb/shard37017

mkdir /data/mongodb/shard37027

mkdir /data/mongodb/shard37037

 

# 开始启动shard servers进程

/usr/local/mongodb-linux-x86_64-3.0.3/bin/mongod--shardsvr --replSet shard1 --port 37017 --dbpath /data/mongodb/shard37017--oplogSize 2048 --logpath /data/mongodb/logs/shard_m11_37017.log --logappend--fork

/usr/local/mongodb-linux-x86_64-3.0.3/bin/mongod--shardsvr --replSet shard1 --port 37027 --dbpath /data/mongodb/shard37027--oplogSize 2048 --logpath /data/mongodb/logs/shard_m1s1_37027.log --logappend--fork

/usr/local/mongodb-linux-x86_64-3.0.3/bin/mongod--shardsvr --replSet shard1 --port 37037 --dbpath /data/mongodb/shard37037--oplogSize 2048 --logpath /data/mongodb/logs/shard_m11_37037.log --logappend--fork

 

# 第二个shard2分片副本集

# 创建目录

mkdir /data/mongodb/shard37018

mkdir /data/mongodb/shard37028

mkdir /data/mongodb/shard37038

 

# 启动shard severs进程

/usr/local/mongodb-linux-x86_64-3.0.3/bin/mongod--shardsvr --replSet shard2 --port 37018 --dbpath /data/mongodb/shard37018--oplogSize 2048 --logpath /data/mongodb/logs/shard_m11_37018.log --logappend--fork

/usr/local/mongodb-linux-x86_64-3.0.3/bin/mongod--shardsvr --replSet shard2 --port 37028 --dbpath /data/mongodb/shard37028--oplogSize 2048 --logpath /data/mongodb/logs/shard_m1s1_37028.log --logappend--fork

/usr/local/mongodb-linux-x86_64-3.0.3/bin/mongod--shardsvr --replSet shard2 --port 37038 --dbpath /data/mongodb/shard37038--oplogSize 2048 --logpath /data/mongodb/logs/shard_m11_37038.log --logappend--fork

 

 

 

 

 

5、开始设置分片副本集shard1

 

# 设置第一个分片副本集，必须使用admin数据库，登录命令如下

/usr/local/mongodb-linux-x86_64-3.0.3/bin/mongomongodb1:37017/admin

 

#定义副本集命令

> config = { _id:"shard1",members:[

{_id:0,host:"mongodb1:37017",priority:1},

{_id:1,host:"mongodb1:37027",priority:2},

{_id:2,host:"mongodb1:37037",arbiterOnly:true}

]

};

 

# 初始化副本集命令

> rs.initiate(config);

 

# 执行过程如下：

[mongodb@dnadb_1 mongodb]$ /usr/local/mongodb-linux-x86_64-3.0.3/bin/mongo localhost:27017/admin MongoDB shell version: 3.0.3 connecting to: localhost:27017/admin Server has startup warnings: 2016-04-07T23:41:07.703+0800 I CONTROL  [initandlisten] 2016-04-07T23:41:07.703+0800 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'. 2016-04-07T23:41:07.703+0800 I CONTROL  [initandlisten] **        We suggest setting it to 'never' 2016-04-07T23:41:07.703+0800 I CONTROL  [initandlisten] 2016-04-07T23:41:07.703+0800 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'. 2016-04-07T23:41:07.703+0800 I CONTROL  [initandlisten] **        We suggest setting it to 'never' 2016-04-07T23:41:07.703+0800 I CONTROL  [initandlisten] > config = { _id:"shard1", members:[ ... {_id:0,host:"mongodb1:37017",priority:1}, ... {_id:1,host:"mongodb1:37027",priority:2}, ... {_id:2,host:"mongodb1:37037",arbiterOnly:true} ... ] ... }; {          "_id" : "shard1",          "members" : [                    {                             "_id" : 0,                             "host" : "mongodb1:37017",                             "priority" : 1                    },                    {                             "_id" : 1,                             "host" : "mongodb1:37027",                             "priority" : 2                    },                    {                             "_id" : 2,                             "host" : "mongodb1:37037",                             "arbiterOnly" : true                    }          ] } >  rs.initiate(config); { "ok" : 1 } shard1:OTHER>

 

 

6、开始设置分片副本集shard2

# 开始设置第二个分片副本集

> config = { _id:"shard2",members:[

{_id:0,host:"mongodb1:37018",priority:1},

{_id:1,host:"mongodb1:37028",priority:2},

{_id:2,host:"mongodb1:37038",arbiterOnly:true}

]

};

 

# 初始化副本集

> rs.initiate(config);

 

 

 

执行过程如下：

[mongodb@dnadb_1 mongodb]$ /usr/local/mongodb-linux-x86_64-3.0.3/bin/mongo localhost:27018/admin MongoDB shell version: 3.0.3 connecting to: localhost:27018/admin Server has startup warnings: 2016-04-07T23:42:18.944+0800 I CONTROL  [initandlisten] 2016-04-07T23:42:18.944+0800 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'. 2016-04-07T23:42:18.944+0800 I CONTROL  [initandlisten] **        We suggest setting it to 'never' 2016-04-07T23:42:18.944+0800 I CONTROL  [initandlisten] 2016-04-07T23:42:18.944+0800 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'. 2016-04-07T23:42:18.944+0800 I CONTROL  [initandlisten] **        We suggest setting it to 'never' 2016-04-07T23:42:18.944+0800 I CONTROL  [initandlisten] > config = { _id:"shard2", members:[ ... {_id:0,host:"mongodb1:37018",priority:1}, ... {_id:1,host:"mongodb1:37028",priority:2}, ... {_id:2,host:"mongodb1:37038",arbiterOnly:true} ... ] ... }; {          "_id" : "shard2",          "members" : [                    {                             "_id" : 0,                             "host" : "mongodb1:37018",                             "priority" : 1                    },                    {                             "_id" : 1,                             "host" : "mongodb1:37028",                             "priority" : 2                    },                    {                             "_id" : 2,                             "host" : "mongodb1:37038",                             "arbiterOnly" : true                    }          ] } > rs.initiate(config); { "ok" : 1 } shard2:OTHER>

 

 

7、检查下副本状态：

[mongodb@hch_test_dbm1_121_62 ~]$ /usr/local/mongodb-linux-x86_64-3.0.3/bin/mongomongodb1:37028/admin

MongoDB shell version: 3.0.3

connecting to: mongodb1:37028/admin

Server has startup warnings:

2016-07-09T19:08:58.250+0800 I CONTROL  [initandlisten]

2016-07-09T19:08:58.250+0800 I CONTROL  [initandlisten] ** WARNING: You are runningon a NUMA machine.

2016-07-09T19:08:58.250+0800 I CONTROL  [initandlisten] **          We suggest launching mongod like thisto avoid performance problems:

2016-07-09T19:08:58.250+0800 I CONTROL  [initandlisten] **              numactl --interleave=all mongod[other options]

2016-07-09T19:08:58.251+0800 I CONTROL  [initandlisten]

2016-07-09T19:08:58.251+0800 I CONTROL  [initandlisten] ** WARNING:/sys/kernel/mm/transparent_hugepage/enabled is 'always'.

2016-07-09T19:08:58.251+0800 I CONTROL  [initandlisten] **        We suggest setting it to 'never'

2016-07-09T19:08:58.251+0800 I CONTROL  [initandlisten]

2016-07-09T19:08:58.251+0800 I CONTROL  [initandlisten] ** WARNING:/sys/kernel/mm/transparent_hugepage/defrag is 'always'.

2016-07-09T19:08:58.251+0800 I CONTROL  [initandlisten] **        We suggest setting it to 'never'

2016-07-09T19:08:58.251+0800 I CONTROL  [initandlisten]

shard2:PRIMARY> rs.status();

{

         "set": "shard2",

         "date": ISODate("2016-07-09T11:35:36.680Z"),

         "myState": 1,

         "members": [

                   {

                            "_id": 0,

                            "name": "mongodb1:37018",

                            "health": 1,

                            "state": 2,

                            "stateStr": "SECONDARY",

                            "uptime": 62,

                            "optime": Timestamp(1468064073, 1),

                            "optimeDate": ISODate("2016-07-09T11:34:33Z"),

                            "lastHeartbeat": ISODate("2016-07-09T11:35:35.736Z"),

                            "lastHeartbeatRecv": ISODate("2016-07-09T11:35:35.739Z"),

                            "pingMs": 0,

                            "configVersion": 1

                   },

                   {

                            "_id": 1,

                            "name": "mongodb1:37028",

                            "health": 1,

                            "state": 1,

                            "stateStr": "PRIMARY",

                            "uptime": 1598,

                            "optime": Timestamp(1468064073, 1),

                            "optimeDate": ISODate("2016-07-09T11:34:33Z"),

                            "electionTime": Timestamp(1468064110, 1),

                            "electionDate": ISODate("2016-07-09T11:35:10Z"),

                            "configVersion": 1,

                            "self": true

                   },

                   {

                            "_id": 2,

                            "name": "mongodb1:37038",

                            "health": 1,

                            "state": 7,

                            "stateStr": "ARBITER",

                            "uptime": 62,

                            "lastHeartbeat": ISODate("2016-07-09T11:35:35.747Z"),

                            "lastHeartbeatRecv": ISODate("2016-07-09T11:35:35.745Z"),

                            "pingMs": 0,

                            "configVersion": 1

                   }

         ],

         "ok": 1

}

shard2:PRIMARY>

 

8、设置自动分片生效

目前搭建了mongodb配置服务器、路由服务器，各个分片服务器，不过应用程序连接到 mongos 路由服务器并不能使用分片机制，还需要在程序里设置分片配置，让分片生效。

 

# 串联路由服务器与分配副本集命令

db.runCommand( { addshard :"shard1/mongodb1:37017,mongodb1:37027,mongodb1:37037"});

db.runCommand( { addshard :"shard2/mongodb1:37018,mongodb1:37028,mongodb1:37038"});

 

# 执行过程

[mongodb@db_m1_slave_1 logs]$ /usr/local/mongodb-linux-x86_64-3.0.3/bin/mongolocalhost:40000/admin

MongoDB shell version: 3.0.3

connecting to: localhost:30000/admin

mongos> db.runCommand( { addshard :"shard1/mongodb1:37017,mongodb1:37027,mongodb1:37037"});

{ "shardAdded" :"shard1", "ok" : 1 }

mongos> db.runCommand( { addshard :"shard2/mongodb1:37018,mongodb1:37028,mongodb1:37038"});

{ "shardAdded" :"shard2", "ok" : 1 }

mongos>

 

 

9、查看分片服务器配置

db.runCommand({listshards : 1 });

命令输出结果

mongos> db.runCommand({listshards : 1});

{

         "shards": [

                   {

                            "_id": "shard1",

                            "host": "shard1/mongodb1:37017,mongodb1:37027"

                   },

                   {

                            "_id": "shard2",

                            "host": "shard2/mongodb1:37018,mongodb1:37028"

                   }

         ],

         "ok": 1

}

mongos>

 

PS：因为37037、37038是每个分片副本集的仲裁节点，所以在上面结果没有列出来。

 

 

查看均衡器，默认是打开的：

mongos> sh.getBalancerState();

true

mongos>

 

 

10、如何让录入的数据自动分片

目前配置服务、路由服务、分片服务、副本集服务都已经串联起来了，但我们的目的是希望插入数据，数据能够自动分片，那么还需要做如下工作：

 

连接在mongos上，准备让指定的数据库、指定的集合分片生效。

 

# 指定分片生效

db.runCommand( { enablesharding:"app"});

db.runCommand( { enablesharding:"im"});

db.runCommand( { enablesharding:"parking"});

db.runCommand( { enablesharding:"pv"});

db.runCommand( { enablesharding:"report"});

db.runCommand( { enablesharding:"screen"});

db.runCommand( { enablesharding:"search"});

db.runCommand( { enablesharding :"traffice"});

db.runCommand( { enablesharding:"wifi"});

 

执行过程如下：

mongos> db.runCommand( { enablesharding:"app"});

});

db.runCommand( { enablesharding:"search"});

db.runCommand( { enablesharding:"traffice"});

db.runCommand( { enablesharding:"wifi"});

{ "ok" : 1 }

mongos> db.runCommand( { enablesharding:"im"});

{ "ok" : 1 }

mongos> db.runCommand( { enablesharding:"parking"});

{ "ok" : 1 }

mongos> db.runCommand( { enablesharding:"pv"});

{ "ok" : 1 }

mongos> db.runCommand( { enablesharding:"report"});

{ "ok" : 1 }

mongos> db.runCommand( { enablesharding:"screen"});

{ "ok" : 1 }

mongos> db.runCommand( { enablesharding:"search"});

{ "ok" : 1 }

mongos> db.runCommand( { enablesharding:"traffice"});

{ "ok" : 1 }

mongos> db.runCommand( { enablesharding:"wifi"});

{ "ok" : 1 }

mongos>

mongos>

 

 

# 指定数据库里需要分片的集合和片键

db.runCommand( { shardcollection :"uba.table1",key : {id: 1} } )

我们设置uba的 table1 表需要分片，根据 id 自动分片到 shard1 ，shard2，shard3 上面去。要这样设置是因为不是所有 mongodb 的数据库和表都需要分片！

 

--1 app

mongos> use bg;

switched to db bg

mongos> db.card.save({x:1});

WriteResult({ "nInserted" : 1 })

mongos>

 sh.enableBalancing("bg");

 

db.runCommand({enablesharding:"app"});

db.runCommand( { shardcollection :"app.download",key : {_id: 1} } );

 

执行过程如下：

         mongos>db.runCommand({enablesharding:"app"});

         {"ok" : 0, "errmsg" : "already enabled" }

         mongos>use admin

         switchedto db admin

         mongos>db.runCommand({enablesharding:"app"});

         {"ok" : 0, "errmsg" : "already enabled" }

         mongos>db.runCommand( { shardcollection : "app.download",key : {_id: 1} } );

         {"collectionsharded" : "app.download", "ok" : 1 }

         mongos>

 

 

11、配置片键

需要进入admin数据库里面去执行配置命令：

use admin

db.runCommand({"shardcollection":"201.customer_user_mst",key:{"name":1}})        

db.runCommand({"shardcollection":"201.user_attr_mst",key:{"name":1}})       

 

db.runCommand({"shardcollection":"pv.pv1",key:{"_id":1}})       

 

db.pv1.insert({ "_id" :ObjectId("4b8ed00a1d42d47b3afa3c41"), "x" : 999,"note" : "in db2" });