MongoDB Sharding集群部署
和Replica Set类似，都需要一个仲裁节点，但是Sharding还需要配置节点和路由节点。就三种集群搭建方式来说，这种是最复杂的。部署图如下：
MongoDB Sharding Cluster需要三种角色：
（1）Shard Server：即存储实际数据的分片，每个Shard可以是一个mongod实例，也可以是一组mongod实例构成的Replica Set。
（2）Config Server：用来存储所有shard节点的配置信息、每个chunk的shard key范围、chunk在各shard的分布情况、该集群中所有DB和collection的sharding配置信息。
（3）Route Process：这是一个前端路由，客户端由此接入，然后询问Config Servers需要到哪个shard上查询或保存记录，再连接相应的shard进行操作，最后将结果返回给客户端，而这一切对客户端是透明的，客户端不用关心所操作的记录存储在哪个shard上。
下面我们配置测试一下Sharding：
一、机器环境
1.Shard Server
shard_a
192.168.110.71：10000
192.168.100.90：10000
192.168.100.110：10000 （Arbiter Server）
shard_b
192.168.110.71：10001
192.168.100.90：10001
192.168.100.110：10001 （Arbiter Server）
2.Config Server
192.168.110.71：20000
3.Route Process
192.168.100.110：30000
二、配置
1.创建数据目录
在这3台机器的mongodb安装目录下创建日志目录和数据目录
mkdir -p logs
mkdir -p conf
mkdir -p data/shard_a
mkdir -p data/shard_b
在192.168.110.71创建config数据目录
mkdir -p data/config
2.创建配置文件
shard_a.conf
port=10000 pidfilepath=/home/slim/mongodb-2.6.8/data/shard_a.pid dbpath=/home/slim/mongodb-2.6.8/data/shard_a directoryperdb=true logpath=/home/slim/mongodb-2.6.8/logs/shard_a.log logappend=true fork=true profile=1 slowms = 5 noprealloc=false replSet=shard_a oplogSize=100 shardsvr=trueshard_b.conf
port=100001 pidfilepath=/home/slim/mongodb-2.6.8/data/shard_b.pid dbpath=/home/slim/mongodb-2.6.8/data/shard_b directoryperdb=true logpath=/home/slim/mongodb-2.6.8/logs/shard_b.log logappend=true fork=true profile=1 slowms = 5 noprealloc=false replSet=shard_b oplogSize=100 shardsvr=true
config.conf
port=20000 pidfilepath=/home/slim/mongodb-2.6.8/data/config.pid dbpath=/home/slim/mongodb-2.6.8/data/config directoryperdb=true logpath=/home/slim/mongodb-2.6.8/logs/config.log logappend=true fork=true profile=0 configsvr=truemongos.conf
port=30000 logpath=/home/slim/mongodb-2.6.8/logs/mongos.log logappend=true fork=true maxConns=1000 chunkSize=1 configdb=192.168.110.71:20000
这里面暂不配置
bind_ip，方便配置文件维护。
一般来说一个数据节点对应一个配置节点，仲裁节点则不需要对应的配置节点。
参数说明：
bind_ip:mongodb所绑定的ip地址
port:端口
pidfilepath：pid文件目录
dbpath:数据库存储文件目录
directoryperdb:设置每个数据库将被保存在一个单独的目录
logpath:日志文件路径
logappend:以追加方式写入,false:重新启动覆盖文件
fork:以守护进程的方式运行，创建服务器进程
profile:设置日志级别
#0 - 关闭性能分析，测试环境可以打开，生成环境关闭，对性能有很大影响;
#1 - 开启慢查询日志，执行时间大于100毫秒的语句
#2 - 开启所有操作日志
slowms:慢查询的时间设置
noprealloc:预分配方式,默认false,true禁用预分配的数据文件
replSet:指定复制集名称
shardsvr:确定是否分片
oplogSize:操作日志文件的最大大小。单位为Mb，默认为硬盘剩余空间的5%
configsvr:声明这是一个集群的config服务,默认端口27019，默认目录/data/configdb
configdb:监听的配置服务器,只能有1个或者3个
chunkSize:单位 mb 生成环境请使用 100 或删除,删除后默认是64
maxConns:最大连接数,默认值：取决于系统（即的ulimit和文件描述符）限制,能设置该值大于20000。
二、配置Replica Set
启动每台机器的
Shard Server服务.
./bin/mongod -f conf/shard_a.conf &
./bin/mongod -f conf/shard_b.conf &
待全部启动完毕后，登录任意一台mongodb配置
replSet，如下：
1.配置shard_a
登录
192.168.100.90:10000
./bin/mongo 192.168.100.90:10000
MongoDB shell version: 2.0.6
connecting to: 192.168.100.90:10000/test
> use admin;
switched to db admin
> config_shard_a={_id : 'shard_a', members:[{_id:0, host:'192.168.110.71:10000'},{_id:1, host:'192.168.100.90:10000'},{_id:2,host:'192.168.100.110:10000',arbiterOnly:true}]};
{
"_id" : "shard_a",
"members" : [
{
"_id" : 0,
"host" : "192.168.110.71:10000"
},
{
"_id" : 1,
"host" : "192.168.100.90:10000"
},
{
"_id" : 2,
"host" : "192.168.100.110:10000",
"arbiterOnly" : true
}
]
}
> rs.initiate(config_shard_a);
{
"info" : "Config now saved locally. Should come online in about a minute.",
"ok" : 1
}
> rs.status();
{
"set" : "shard_a",
"date" : ISODate("2015-03-17T05:10:38Z"),
"myState" : 1,
"members" : [
{
"_id" : 0,
"name" : "192.168.110.71:10000",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 88,
"optime" : {
"t" : 1426568948000,
"i" : 1
},
"optimeDate" : ISODate("2015-03-17T05:09:08Z"),
"lastHeartbeat" : ISODate("2015-03-17T05:10:37Z"),
"lastHeartbeatRecv" : ISODate("2015-03-17T05:10:37Z"),
"pingMs" : 0,
"syncingTo" : "192.168.100.90:10000"
},
{
"_id" : 1,
"name" : "192.168.100.90:10000",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 6289,
"optime" : {
"t" : 1426568948000,
"i" : 1
},
"optimeDate" : ISODate("2015-03-17T05:09:08Z"),
"electionTime" : {
"t" : 1426568958000,
"i" : 1
},
"electionDate" : ISODate("2015-03-17T05:09:18Z"),
"self" : true
},
{
"_id" : 2,
"name" : "192.168.100.110:10000",
"health" : 1,
"state" : 7,
"stateStr" : "ARBITER",
"uptime" : 88,
"lastHeartbeat" : ISODate("2015-03-17T05:10:37Z"),
"lastHeartbeatRecv" : ISODate("2015-03-17T05:10:37Z"),
"pingMs" : 0
}
],
"ok" : 1
}
2.配置shard_b
登录192.168.100.90:10001
./bin/mongo 192.168.100.90:10001
MongoDB shell version: 2.0.6
connecting to: 192.168.110.71:10001/test
> use admin;
switched to db admin
> config_shard_b={_id : 'shard_b', members:[{_id:0, host:'192.168.110.71:10001'},{_id:1, host:'192.168.100.90:10001'},{_id:2,host:'192.168.100.110:10001',arbiterOnly:true}]};
{
"_id" : "shard_b",
"members" : [
{
"_id" : 0,
"host" : "192.168.110.71:10001"
},
{
"_id" : 1,
"host" : "192.168.100.90:10001"
},
{
"_id" : 2,
"host" : "192.168.100.110:10001",
"arbiterOnly" : true
}
]
}
> rs.initiate(config_shard_b);
{
"info" : "Config now saved locally. Should come online in about a minute.",
"ok" : 1
}
STARTUP2> rs.status();
{
"set" : "shard_b",
"date" : ISODate("2015-03-17T05:17:48Z"),
"myState" : 1,
"members" : [
{
"_id" : 0,
"name" : "192.168.110.71:10001",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 128,
"optime" : {
"t" : 1426569457000,
"i" : 1
},
"optimeDate" : ISODate("2015-03-17T05:17:37Z"),
"electionTime" : {
"t" : 1426569463000,
"i" : 1
},
"electionDate" : ISODate("2015-03-17T05:17:43Z"),
"self" : true
},
{
"_id" : 1,
"name" : "192.168.100.90:10001",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 11,
"optime" : {
"t" : 1426569457000,
"i" : 1
},
"optimeDate" : ISODate("2015-03-17T05:17:37Z"),
"lastHeartbeat" : ISODate("2015-03-17T05:17:47Z"),
"lastHeartbeatRecv" : ISODate("2015-03-17T05:17:48Z"),
"pingMs" : 1,
"lastHeartbeatMessage" : "syncing to: 192.168.110.71:10001",
"syncingTo" : "192.168.110.71:10001"
},
{
"_id" : 2,
"name" : "192.168.100.110:10001",
"health" : 1,
"state" : 7,
"stateStr" : "ARBITER",
"uptime" : 11,
"lastHeartbeat" : ISODate("2015-03-17T05:17:47Z"),
"lastHeartbeatRecv" : ISODate("2015-03-17T05:17:47Z"),
"pingMs" : 0
}
],
"ok" : 1
}
注：配置shard时不能连接ARBITER配置，否则会抛出：
"errmsg" : "couldn't initiate : initiation and reconfiguration of a replica set must be sent to a node that can become primary"
三、配置Sharding
1.启动配置服务
./bin/mongod -f conf/config.conf &
2.启动路由服务
./bin/mongos -f conf/mongos.conf &
3. 路由节点配置
./bin/mongo 192.168.100.110:30000
MongoDB shell version: 2.0.6
connecting to: 192.168.100.110:30000/test
mongos> use admin;
switched to db admin
mongos> db.runCommand({addshard:"shard_a/192.168.110.71:10000,192.168.100.90:10000",name:"shard_a"});
{ "shardAdded" : "shard_a", "ok" : 1 }
mongos> db.runCommand({addshard:"shard_b/192.168.110.71:10001,192.168.100.90:10001",name:"shard_b"});
{ "shardAdded" : "shard_b", "ok" : 1 }
mongos> db.adminCommand({listshards:1});
{
"shards" : [
{
"_id" : "shard_a",
"host" : "shard_a/192.168.100.90:10000,192.168.110.71:10000"
},
{
"_id" : "shard_b",
"host" : "shard_b/192.168.100.90:10001,192.168.110.71:10001"
}
],
"ok" : 1
}replica set + shard功能就配置好了，注意：虽然配置好了，但是还要声明库和表要进行分片。
注：
启动路由服务时，若报如下错误：
BadValue need either 1 or 3 configdbs
try './bin/mongos --help' for more information
原因以及解决办法：
mongos.conf文件中configdb参数有误，在这里路由个数，只能为 1或3。configdb =192.168.110.71:20000
4.声明库和表要分片
添加分片存储的数据库
mongos> db.runCommand({enablesharding:"shard"});
{ "ok" : 1 }设置分片的集合名称，且必须指定Shard Key：
mongos> db.runCommand({shardcollection:"shard.user", key:{_id:1}});
{ "collectionsharded" : "shard.user", "ok" : 1 }
查看sharding状态：
mongos> db.printShardingStatus();
--- Sharding Status ---
sharding version: {
"_id" : 1,
"version" : 4,
"minCompatibleVersion" : 4,
"currentVersion" : 5,
"clusterId" : ObjectId("5507bbdfd9b4a799a27e0120")
}
shards:
{ "_id" : "shard_a", "host" : "shard_a/192.168.100.90:10000,192.168.110.71:10000" }
{ "_id" : "shard_b", "host" : "shard_b/192.168.100.90:10001,192.168.110.71:10001" }
databases:
{ "_id" : "admin", "partitioned" : false, "primary" : "config" }
{ "_id" : "shard", "partitioned" : true, "primary" : "shard_b" }
shard.user chunks:
shard_b 1
{ "_id" : { $minKey : 1 } } -->> { "_id" : { $maxKey : 1 } } on : shard_b { "t" : 1000, "i" : 0 }查看数据库状态：
mongos> use shard;
mongos> db.stats();
{
"raw" : {
"shard_b/192.168.100.90:10001,192.168.110.71:10001" : {
"db" : "shard",
"collections" : 4,
"objects" : 6,
"avgObjSize" : 122.66666666666667,
"dataSize" : 736,
"storageSize" : 1073152,
"numExtents" : 4,
"indexes" : 1,
"indexSize" : 8176,
"fileSize" : 67108864,
"nsSizeMB" : 16,
"dataFileVersion" : {
"major" : 4,
"minor" : 5
},
"extentFreeList" : {
"num" : 0,
"totalSize" : 0
},
"ok" : 1
}
},
"objects" : 6,
"avgObjSize" : 122,
"dataSize" : 736,
"storageSize" : 1073152,
"numExtents" : 4,
"indexes" : 1,
"indexSize" : 8176,
"fileSize" : 67108864,
"extentFreeList" : {
"num" : 0,
"totalSize" : 0
},
"ok" : 1
}查看表状态：
mongos> db.user.stats();
{
"sharded" : true,
"systemFlags" : 1,
"userFlags" : 1,
"ns" : "shard.user",
"count" : 0,
"numExtents" : 1,
"size" : 0,
"storageSize" : 8192,
"totalIndexSize" : 8176,
"indexSizes" : {
"_id_" : 8176
},
"avgObjSize" : 0,
"nindexes" : 1,
"nchunks" : 1,
"shards" : {
"shard_b" : {
"ns" : "shard.user",
"count" : 0,
"size" : 0,
"storageSize" : 8192,
"numExtents" : 1,
"nindexes" : 1,
"lastExtentSize" : 8192,
"paddingFactor" : 1,
"systemFlags" : 1,
"userFlags" : 1,
"totalIndexSize" : 8176,
"indexSizes" : {
"_id_" : 8176
},
"ok" : 1
}
},
"ok" : 1
}注：
1. 报以下错误，原因是库还不允许分片
{ "ok" : 0, "errmsg" : "sharding not enabled for db" }
2.报以下错误，原因是Shard Key设置有误
"errmsg" : "Unsupported shard key pattern. Pattern must either be a single hashed field, or a list of ascending fields."
四、测试
登录mongos服务,执行如下脚本：
for (var i = 1; i <= 20000; i++) db.user.save({id:i,name:"jack"+i,sex:"male",age:27,value:"test"});
查看结果：
mongos> db.user.stats();
{
"sharded" : true,
"systemFlags" : 0,
"userFlags" : 1,
"ns" : "shard.user",
"count" : 20000,
"numExtents" : 9,
"size" : 2240000,
"storageSize" : 3489792,
"totalIndexSize" : 670432,
"indexSizes" : {
"_id_" : 670432
},
"avgObjSize" : 112,
"nindexes" : 1,
"nchunks" : 4,
"shards" : {
"shard_a" : {
"ns" : "shard.user",
"count" : 15058,
"size" : 1686496,
"avgObjSize" : 112,
"storageSize" : 2793472,
"numExtents" : 5,
"nindexes" : 1,
"lastExtentSize" : 2097152,
"paddingFactor" : 1,
"systemFlags" : 0,
"userFlags" : 1,
"totalIndexSize" : 498736,
"indexSizes" : {
"_id_" : 498736
},
"ok" : 1
},
"shard_b" : {
"ns" : "shard.user",
"count" : 4942,
"size" : 553504,
"avgObjSize" : 112,
"storageSize" : 696320,
"numExtents" : 4,
"nindexes" : 1,
"lastExtentSize" : 524288,
"paddingFactor" : 1,
"systemFlags" : 1,
"userFlags" : 1,
"totalIndexSize" : 171696,
"indexSizes" : {
"_id_" : 171696
},
"ok" : 1
}
},
"ok" : 1
}从结果可以看到：
shard_a：15058
shard_b：4942