mongodb 分片群集配置 及数据迁移
Sharding cluster介绍
这是一种可以水平扩展的模式，在数据量很大时特给力，实际大规模应用一般会采用这种架构去构建monodb系统。
要构建一个 MongoDB Sharding Cluster，需要三种角色：
Shard Server: mongod 实例，用于存储实际的数据块，实际生产环境中一个shard server角色可由几台机器组个一个relica set承担，防止主机单点故障
Config Server: mongod 实例，存储了整个 Cluster Metadata，其中包括 chunk 信息。
Route Server: mongos 实例，前端路由，客户端由此接入，且让整个集群看上去像单一数据库，前端应用可以透明使用。
Sharding架构图:
host ip 端口信息
-------------------------------------------------
mds1 192.168.20.11 mongod shard11:28011
mongod shard12:28012
mongod config1:28010
mongs1:28000
mds2 192.168.20.13 mongod shard12:28013
mongod shard22:28014
mongod config2:28010
mongs2:28000
mds3 192.168.20.15 mongod shard13:28015
mongod shard23:28016
mongod config3:28010
mongs3:28000
一、LINUX 环境准备工作
1.1 先配置ULIMIT
修改文件：
/etc/security/limits.conf
比如添加以下内容：
mongo soft nproc 30000
mongo hard nproc 65536
mongo soft nofile 30000
mongo hard nofile 65536
(这个在配置MONGODB 最大连接数是有用，因为MONGODB 默认的最大连接数为819,还要减掉各节点间的config等服务器的连接，如果你是在生产环境下，
相信一会儿就不够，并且mongo 安装用户如果不是root,比如我的mongo,也会报错，linux 默认一个非root 用户只能打开1024个文件。
)
软件准备
1. 创建用户
useradd mongo
2. 安装monodb软件
su – mongo
tar zxvf mongodb-linux-x86_64-2.6.4.tar
3. 创建数据目录
根据本例sharding架构图所示，在各台sever上创建shard数据文件目录
mds1:
su – mono
mkdir -p /home/mongodb/mongodb2.6.4/logs
mkdir -p /home/mongodb/mongodb2.6.4/config
mkdir -p /opt/mongodb/data/shard11
mkdir -p /opt/mongodb/data/shard21
mds2:
su – mono
mkdir -p /home/mongodb/mongodb2.6.4/logs
mkdir -p /home/mongodb/mongodb2.6.4/config
mkdir -p /opt/mongodb/data/shard12
mkdir -p /opt/mongodb/data/shard22
mds3:
su – mono
mkdir -p /home/mongodb/mongodb2.6.4/logs
mkdir -p /home/mongodb/mongodb2.6.4/config
mkdir -p /opt/mongodb/data/shard13
mkdir -p /opt/mongodb/data/shard23
配置REPLICAT SETS
1.配置SHARD1 所在的REPLICA SETS
（在实际配置中，我都使用了建立一个sh 文件，并对应--config 文件运行的方式，这样方便以后每次执行而不会出错）。
vi mongod_shard11_start.sh
/home/mongo/mongodb2.6.4/bin/mongod --shardsvr --replSet shard1 --port 28011 --dbpath /opt/mongodb/data/shard11 --oplogSize 1000 --logpath /home/mongo/mongodb2.6.4/logs/shard11.log --logappend --maxConn 30000 --fork
mds2
vi mongod_shard12_start.sh
/home/mongo/mongodb2.6.4/bin/mongod –shardsvr –replSet shard1 –port 28013 –dbpath /opt/mongodb/data/shard12 –oplogSize 1000 –logpath /home/mongo/mongodb2.6.4/logs/shard12.log –logappend --maxConn 30000 –fork
mds3
vi mongod_shard13_start.sh
/home/mongo/mongodb2.6.4/bin/mongod –shardsvr –replSet shard1 –port 28015 –dbpath /opt/mongodb/data/shard13 –oplogSize 1000 –logpath /home/mongo/mongodb2.6.4/logs/shard13.log –logappend --maxConn 30000 –fork
并启动 对应的
mongod_shard11_start.sh
mongod_shard12_start.sh
mongod_shard13_start.sh
初始化replica set shard1
用mongo 连接其中一个节点： 比如：shard11 执行以下：
> config = {_id: 'shard1', members: [
{_id: 0, host:'198.168.20.11:28011'},
{_id: 1, host:'198.168.20.13:28013'},
{_id: 2, host:'198.168.20.15:28015'}]
}
> rs.initiate(config);
此步的具体操作如下：
mongo -port 28011
config = {_id: 'shard1', members: [
{_id: 0, host:'198.168.20.11:28011'},
{_id: 1, host:'198.168.20.13:28013'},
{_id: 2, host:'198.168.20.15:28015'}]
}
> rs.initiate(config)
{
"ok" : 0,
"errmsg" : "couldn't initiate : can't find self in the replset config my port: 28011"
}
> cfg={_id:"shard1",members:[{_id:0,host:"192.168.20.11:28011"},
... {_id:1,host:"192.168.20.13:28013"}]}
{
"_id" : "shard1",
"members" : [
{
"_id" : 0,
"host" : "192.168.20.11:28011"
},
{
"_id" : 1,
"host" : "192.168.20.13:28013"
}
]
}
> rs.initiate(cfg)
{
"info" : "Config now saved locally. Should come online in about a minute.",
"ok" : 1
}
shard1:OTHER> rs.status()
{
"set" : "shard1",
"date" : ISODate("2014-11-09T08:57:38Z"),
"myState" : 1,
"members" : [
{
"_id" : 0,
"name" : "192.168.20.11:28011",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 2710,
"optime" : Timestamp(1415523264, 1),
"optimeDate" : ISODate("2014-11-09T08:54:24Z"),
"electionTime" : Timestamp(1415523272, 1),
"electionDate" : ISODate("2014-11-09T08:54:32Z"),
"self" : true
},
{
"_id" : 1,
"name" : "192.168.20.13:28013",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 194,
"optime" : Timestamp(1415523264, 1),
"optimeDate" : ISODate("2014-11-09T08:54:24Z"),
"lastHeartbeat" : ISODate("2014-11-09T08:57:36Z"),
"lastHeartbeatRecv" : ISODate("2014-11-09T08:57:36Z"),
"pingMs" : 0,
"syncingTo" : "192.168.20.11:28011"
}
],
"ok" : 1
}
shard1:PRIMARY> rs.add({_id:2,host:"192.168.20.15:28015"})
{ "ok" : 1 }
shard1:PRIMARY> rs.status()
{
"set" : "shard1",
"date" : ISODate("2014-11-09T08:58:44Z"),
"myState" : 1,
"members" : [
{
"_id" : 0,
"name" : "192.168.20.11:28011",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 2776,
"optime" : Timestamp(1415523521, 1),
"optimeDate" : ISODate("2014-11-09T08:58:41Z"),
"electionTime" : Timestamp(1415523272, 1),
"electionDate" : ISODate("2014-11-09T08:54:32Z"),
"self" : true
},
{
"_id" : 1,
"name" : "192.168.20.13:28013",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 260,
"optime" : Timestamp(1415523521, 1),
"optimeDate" : ISODate("2014-11-09T08:58:41Z"),
"lastHeartbeat" : ISODate("2014-11-09T08:58:44Z"),
"lastHeartbeatRecv" : ISODate("2014-11-09T08:58:44Z"),
"pingMs" : 0,
"syncingTo" : "192.168.20.11:28011"
},
{
"_id" : 2,
"name" : "192.168.20.15:28015",
"health" : 1,
"state" : 5,
"stateStr" : "STARTUP2",
"uptime" : 3,
"optime" : Timestamp(0, 0),
"optimeDate" : ISODate("1970-01-01T00:00:00Z"),
"lastHeartbeat" : ISODate("2014-11-09T08:58:43Z"),
"lastHeartbeatRecv" : ISODate("2014-11-09T08:58:43Z"),
"pingMs" : 0
}
],
"ok" : 1
}
shard1:PRIMARY>
配置SHARD2
mds1
vi shard21_start.sh
/home/mongo/mongodb2.6.4/bin/mongod --shardsvr --replSet shard2 –port 28012 –dbpath /opt/mongodb/data/shard21 –oplogSize 1000 –logpath /home/mongo/mongodb2.6.4/logs/shard21.log –logappend –fork
mds2
vi mongod_shard22_start.sh
/home/mongo/mongodb2.6.4/bin/mongod –shardsvr –replSet shard2 –port 28014 –dbpath /opt/mongodb/data/shard22 –oplogSize 1000 –logpath /home/mongo/mongodb2.6.4/logs/shard22.log –logappend –fork
mds3
vi mongod_shard23_start.sh
/home/mongo/mongodb2.6.4/bin/mongod –shardsvr –replSet shard2 –port 28016 –dbpath /opt/mongodb/data/shard23 –oplogSize 1000 –logpath /home/mongo/mongodb2.6.4/logs/shard23.log –logappend –fork
运行以上sh
初始化replica set shard2
用mongo 连接其中一个节点： 比如：shard21 执行以下：
> config = {_id: ’shard2′, members: [
{_id: 0, host: '198.168.20.11:28012'},
{_id: 1, host: '198.168.20.13:28014'},
{_id: 2, host: '198.168.20.15:28016'}]
}
> rs.initiate(config);
> cfg={_id:"shard2",members:[{_id:0,host:"192.168.20.11:28012"},
... {_id:1,host:"192.168.20.13:28014"},
... {_id:2,host:"192.168.20.15:28016"}]}
{
"_id" : "shard2",
"members" : [
{
"_id" : 0,
"host" : "192.168.20.11:28012"
},
{
"_id" : 1,
"host" : "192.168.20.13:28014"
},
{
"_id" : 2,
"host" : "192.168.20.15:28016"
}
]
}
> rs.initiate(cfg)
{
"info" : "Config now saved locally. Should come online in about a minute.",
"ok" : 1
}
shard2:OTHER> rs.status()
{
"set" : "shard2",
"date" : ISODate("2014-11-09T09:05:12Z"),
"myState" : 2,
"members" : [
{
"_id" : 0,
"name" : "192.168.20.11:28012",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 1431,
"optime" : Timestamp(1415523903, 1),
"optimeDate" : ISODate("2014-11-09T09:05:03Z"),
"self" : true
},
{
"_id" : 1,
"name" : "192.168.20.13:28014",
"health" : 1,
"state" : 5,
"stateStr" : "STARTUP2",
"uptime" : 9,
"optime" : Timestamp(0, 0),
"optimeDate" : ISODate("1970-01-01T00:00:00Z"),
"lastHeartbeat" : ISODate("2014-11-09T09:05:11Z"),
"lastHeartbeatRecv" : ISODate("2014-11-09T09:05:11Z"),
"pingMs" : 0
},
{
"_id" : 2,
"name" : "192.168.20.15:28016",
"health" : 1,
"state" : 5,
"stateStr" : "STARTUP2",
"uptime" : 7,
"optime" : Timestamp(0, 0),
"optimeDate" : ISODate("1970-01-01T00:00:00Z"),
"lastHeartbeat" : ISODate("2014-11-09T09:05:11Z"),
"lastHeartbeatRecv" : ISODate("2014-11-09T09:05:11Z"),
"pingMs" : 0
}
],
"ok" : 1
}
shard2:SECONDARY>
配置三台config server
mds1:
./mongod –configsvr –dbpath /home/mongo/mongodb2.6.4/config –port 28010 –logpath /home/mongo/mongodb2.6.4/logs/config.log –logappend –fork
#config server也需要dbpath
mds2:
./mongod –configsvr –dbpath /home/mongo/mongodb2.6.4/config –port 28010 –logpath /home/mongo/mongodb2.6.4/logs/config.log –logappend –fork
mds3:
./mongod –configsvr –dbpath /home/mongo/mongodb2.6.4/config –port 28010 –logpath /home/mongo/mongodb2.6.4/logs/config.log –logappend –fork
配置mongs
在server1,server2,server3上分别执行：
./mongos –configdb 192.168.20.11:28010,192.168.20.13:28010,192.168.20.15:28010 –port 28000 –chunkSize 128 –logpath /home/mongo/mongodb2.6.4/logs/mongos.log –logappend –fork
#mongs不需要dbpath
开始运行出现以下问题：
about to fork child process, waiting until server is ready for connections.
forked process: 5969
日志显示：
waited 209s for distributed lock configUpgrade for upgrading config database to new format v5
后各节点设置时间同步后，问题解决。
配置集群
连接到其中一个mongos进程，并切换到admin数据库做以下配置
1. 连接到mongs，并切换到admin
./mongo 192.168.20.11:28000/admin
>db
Admin
2. 加入shards
命令格式：
replicaSetName/[:port] [,serverhostname2[:port],…]
执行如下：
>db.runCommand({addshard:"shard1/192.168.20.11:28011,192.168.20.13:28011,192.168.20.15:28011"});
>db.runCommand({addshard:"shard2/192.168.20.11:28012,192.168.20.13:28012,192.168.20.15:28012"});
>db.runCommand({addshard:"shard3/192.168.20.11:28013,192.168.20.13:28013,192.168.20.15:28013"});
3. 可选参数
Name:用于指定每个shard的名字，不指定的话系统将自动分配
maxSize:指定各个shard可使用的最大磁盘空间，单位megabytes
修改maxsize:
use config
db.shards.update({_id:"s1"},{$set:{maxSize:2000000}})
4. Listing shards
>db.runCommand( { listshards : 1 } )
如果列出了以上二个你加的shards，表示shards已经配置成功
mongos> db.runCommand( { listshards : 1 } )
{
"shards" : [
{
"_id" : "s1",
"host" : "shard1/192.168.20.11:28011,192.168.20.13:28013,192.168.20.15:28015"
},
{
"_id" : "s2",
"host" : "shard2/192.168.20.11:28012,192.168.20.13:28014,192.168.20.15:28016"
}
],
"ok" : 1
}
mongos> use config
mongos> db.shards.find()
{ "_id" : "s1", "host" : "shard1/192.168.20.11:28011,192.168.20.13:28013,192.168.20.15:28015", "maxSize" : 2000000 }
{ "_id" : "s2", "host" : "shard2/192.168.20.11:28012,192.168.20.13:28014,192.168.20.15:28016", "maxSize" : 2000000 }
5. 激活数据库分片
命令：
> db.runCommand( { enablesharding :"database_name"} );
通 过执行以上命令，可以让数据库跨shard，如果不执行这步，数据库只会存放在一个shard，一旦激活数据库分片，数据库中不同的collection 将被存放在不同的shard上，但一个collection仍旧存放在同一个shard上，要使单个collection也分片，还需单独对 collection作些操作
用客户端连接Mongos
Mongo --port 50000
>sh.enableSharding("database name")
或者
>db.runCommand( { enableSharding: <database> } )
这样即可。
如果你想对一个大的collection进行sharding可以执行如下命令
sh.shardCollection("people.addresses", { "state": 1, "_id": 1 } )
示例：
mongos> use admin
switched to db admin
mongos> db.runCommand({enablesharding:"test"})
{ "ok" : 1 }
mongos> use admin
switched to db admin
mongos> db.runCommand({shardcollection:"test.test",key:{_id:1}})
{ "collectionsharded" : "test.test", "ok" : 1 }
mongos>
mongos>for (var i =2; i <= 1200000; i++) db.user.save({_id:i,name:"teststaasdasdfasdfsa name",address:"adresss adfasdfasdfasdfasdfasdf",country:"china",province:"shanghai",city:"minhang"});
mongos>db.test.stats()
数据迁移：
---------------------------------------------------------------------------------------------------------------------------------
不然导入完成后，发现数据库，表没有同步到另外一个集群。
我这里做了从原单机数据库数据导出后，再分片集群库中导入操作。
当设置了要分片的库，及表后，系统会自动帮你添加库及表。
设置完成后，就可以导入数据了。
use admin
db.runCommand({enablesharding:"ip"})
db.runCommand({shardcollection:"ip.ipaddress",key:{ip1:1}})
./mongorestore -h 192.168.20.11 --port 28000 -d ip /opt/backup/ip20141109/ip
./mongorestore -h 192.168.20.11 --port 28000 -d dmp /opt/backup/dmp/dmp
参考：http://www.open-open.com/lib/view/open1329745095312.html
======================================================
测试后感觉这种架构不太适合我，只有两个分片，而却有3个节点的集群，磁盘空间的使用率不高。后决定修改架构：
集群调整
3台服务器，做成3个分片，每个分片有一个集群，
集群中有2台数据存储服务器+1台仲裁服务器
把仲裁服务器分别放在3台服务器中
host ip 端口信息 arbiterOnly
--------------------------------------------------------------------
mds1 192.168.20.11 mongod shard1:28011
mongod shard2:28012
mongod shard3:28013 ------------ True
mongod config1:28010
mongs1:28000
mds2 192.168.20.13 mongod shard1:28011
mongod shard2:28012 ------------ True
mongod shard3:28013
mongod config2:28010
mongs2:28000
mds3 192.168.20.15 mongod shard1:28011 ------------ True
mongod shard2:28012
mongod shard2:28013
mongod config3:28010
mongs3:28000
对应的数据保存目录各为：
/opt/mongodb/data/shard1
/opt/mongodb/data/shard2
/opt/mongodb/data/shard3
集群配置：
shard1
> config = {_id: 'shard1', members: [
{_id: 0, host:'192.168.20.11:28011'},
{_id: 1, host:'192.168.20.13:28011'},
{_id: 2, host:'192.168.20.15:28011',"arbiterOnly":true}]
}
> rs.initiate(config);
shard2
> config = {_id: 'shard2', members: [
{_id: 0, host:'192.168.20.11:28012'},
{_id: 1, host:'192.168.20.13:28012',"arbiterOnly":true},
{_id: 2, host:'192.168.20.15:28012'}]
}
> rs.initiate(config);
rs.addArb("192.168.20.13:28012");
rs.add({_id: 1, host:'192.168.20.13:28012',"arbiterOnly":true})
rs.add({_id: 2, host:'192.168.20.15:28012'})
shard3
> cfg = {_id: 'shard3', members: [
{_id: 0, host:'192.168.20.13:28013'},
{_id: 1, host:'192.168.20.15:28013'},
{_id: 2, host:'192.168.20.11:28013',"arbiterOnly":true}
]
}
> rs.initiate(cfg);
修改优先权：
MongoDB replica sets中通过设置priority的值来决定优先权的大小，这个值的范围是0--100，值越大，优先权越高。
如果值是0，那么不能成为primay。
cfg = rs.conf()
cfg.members[0].priority = 1
cfg.members[1].priority = 2
cfg.members[2].priority = 3
rs.reconfig(cfg)
说明：cfg.members[0].priority =
括号中的数字是执行rs.conf()得出的节点顺序，第一个节点在这里写0，第二个节点写1，依次类推。
config server 配置：
配置服务器的dbpath 放在软件安装目录（SSD硬盘）
mds1:
./mongod –configsvr –dbpath /home/mongo/mongodb2.6.4/config –port 28010 –logpath /home/mongo/mongodb2.6.4/logs/config.log –logappend –fork
#config server也需要dbpath
mds2:
./mongod –configsvr –dbpath /home/mongo/mongodb2.6.4/config –port 28010 –logpath /home/mongo/mongodb2.6.4/logs/config.log –logappend –fork
mds3:
./mongod –configsvr –dbpath /home/mongo/mongodb2.6.4/config –port 28010 –logpath /home/mongo/mongodb2.6.4/logs/config.log –logappend –fork
配置mongs
在server1,server2,server3上分别执行：
./mongos –configdb 192.168.20.11:28010,192.168.20.13:28010,192.168.20.15:28010 –port 28000 –chunkSize 96 –logpath /home/mongo/mongodb2.6.4/logs/mongos.log –logappend –fork
分片配置：
>db.runCommand({addshard:"shard1/192.168.20.11:28011,192.168.20.13:28011,192.168.20.15:28011"});
>db.runCommand({addshard:"shard2/192.168.20.11:28012,192.168.20.13:28012,192.168.20.15:28012"});
>db.runCommand({addshard:"shard3/192.168.20.11:28013,192.168.20.13:28013,192.168.20.15:28013"});