MongoDB中的副本集(Replica Set)是一组维护相同数据集的Mongod服务。副本集可提供冗余和高可用性,是所有生产部署的基础。
也可以说,副本集类似有自动故障恢复功能的主从集群。通俗的讲就是用多台机器进行同一数据的异步同步,从而使多台机器拥有同一数据的多个副本,并且当主库宕掉时不需要用户干预的情况下自动切换其他备份服务器做主库。而且还可以利用副本服务器做只读服务器,实现读写分离,提高负载。
副本集有两种类型三种角色
在一台服务器上做多实例,一主一从一仲裁
mkdir -p /usr/local/mongodb/replica_sets/myrs_27017/log
mkdir -p /usr/local/mongodb/replica_sets/myrs_27017/data/db
vim /usr/local/mongodb/replica_sets/myrs_27017/mongod.conf
systemLog:
destination:file
path:"/usr/local/mongodb/replica_sets/myrs_27017/log/mongod.log"
logAppend: true
storage:
dbpath:"/usr/local/mongodb/replica_sets/myrs_27017/data/db"
journal:
# 启用持久化日志
enabled: true
processManagement:
fork:true
pidFilePath:"/usr/local/mongodb/replica_sets/myrs_27017/log/mongod.pid"
net:
bindIp:localhost,192.168.64.129
port:27017
replication:
replSetName:myrs
mongod -f /usr/local/mongodb/replica_sets/myrs_27017/mongod.conf
mkdir -p /usr/local/mongodb/replica_sets/myrs_27018/log
mkdir -p /usr/local/mongodb/replica_sets/myrs_27018/data/db
vim /usr/local/mongodb/replica_sets/myrs_27018/mongod.conf
systemLog:
destination:file
path:"/usr/local/mongodb/replica_sets/myrs_27018/log/mongod.log"
logAppend: true
storage:
dbpath:"/usr/local/mongodb/replica_sets/myrs_27018/data/db"
journal:
# 启用持久化日志
enabled: true
processManagement:
fork:true
pidFilePath:"/usr/local/mongodb/replica_sets/myrs_27018/log/mongod.pid"
net:
bindIp:localhost,192.168.64.129
port:27018
replication:
replSetName:myrs
mongod -f /usr/local/mongodb/replica_sets/myrs_27018/mongod.conf
mkdir -p /usr/local/mongodb/replica_sets/myrs_27019/log
mkdir -p /usr/local/mongodb/replica_sets/myrs_27019/data/db
vim /usr/local/mongodb/replica_sets/myrs_27019/mongod.conf
systemLog:
destination:file
path:"/usr/local/mongodb/replica_sets/myrs_27019/log/mongod.log"
logAppend: true
storage:
dbpath:"/usr/local/mongodb/replica_sets/myrs_27019/data/db"
journal:
# 启用持久化日志
enabled: true
processManagement:
fork:true
pidFilePath:"/usr/local/mongodb/replica_sets/myrs_27019/log/mongod.pid"
net:
bindIp:localhost,192.168.64.129
port:27019
replication:
replSetName:myrs
mongod -f /usr/local/mongodb/replica_sets/myrs_27019/mongod.conf
mongo --host=localhost --port=27107
> rs.initiate()
myrs:SECONDARY>
myrs:PRIMARY>
# 命令提示符发生变化,变成了一个从节点角色,此时默认不能读写,稍等片刻,回车,变成主节点
myrs:PRIMARY> rs.conf()
rs.add("192.168.64.129:27018")
# 添加副本节点
rs.addArb("192.168.64.129:27019")
# 添加仲裁节点
rs.slaveOk()
mongo --port=27107
use articledb
db.comment.insert(
{
"articleid":"10000",
"content":"今天我们来学习mongodb",
"userid":"1001",
"nickname":"www",
"createdatetime":new Date(),
"likenum":NumberInt(10),
"state":null
}
)
mongo --port=27108
use articledb
db.comment.insert(
{
"articleid":"10000",
"content":"今天我们来学习mongodb",
"userid":"1001",
"nickname":"www",
"createdatetime":new Date(),
"likenum":NumberInt(10),
"state":null
}
)
show tables
mongo --port=27109
rs.slaveOk()
show dbs
# 也会报错
MongoDB在副本集中,会自动进行主节点的选举,主节点选举的出发条件
主节点故障
主节点网络不可达(默认心跳信息为10秒)
人工干预(rs.stepDown(600))
primary直接降级在600s内不会把自己选为primary
一旦触发选举,就要根据一定规则来选择主节点。
选举规则是根据票数来决定谁获胜
在获得票数的时候,优先级(priority)参数影响重大。
可以通过设置优先级来设置额外票数。优先级即权重,取值为0-1000,相当于增加0-1000的票数,优先级的值越大,就越可能获得多数成员的投票。指定较高的值可使成员更有资格成为主要成员,更低的值可使成员更不符合条件
默认情况下,优先级的值是1
分片(sharding)是一种跨多台机器分布数据的方法,MongoDB使用分片来支持具有非常大的数据集和高吞吐量操作的部署。分片是将数据拆分,散到不同的机器上,不需要功能强大的大型计算机就可以存储更多的数据,处理更多的负载。有时也叫分区(partitioning)。
具有大型数据集或高吞吐量应用程序的数据库系统可能会挑战单个服务器的容量。例如,高查询率会耗尽服务器的CPU容量。工作集大小大于系统的RAM会强调磁盘驱动的I/O容量。
有两种解决系统增长的方法:垂直扩展和水平扩展。
垂直扩展是增加单个服务器的容量,例如使用更强大的CPU,添加更多RAM或增加存储空间量。可用技术的局限性可能会限制单个机器对于给定工作负载而言足够强大。此外,基于云的提供商基于可用的硬件配置具有硬性上限,垂直缩放有实际的最大值。
水平扩展是划分系统数据集并分散加载到多个服务器上,添加其他服务器以根据需要增加容量。虽然单个机器的总体速度或容量可能不高,但每台机器处理整个工作负载的子集,可能提供比单个高速大容量服务器更高的效率。扩展部署容量只需要根据需要添加额外的服务器,这可能比单个机器的高端硬件的总成本更低。但是这样基础架构和部署维护的复杂性会增加。
两个分片节点副本集(3+3)+一个配置节点副本集(3)+两个路由节点(2),共11个服务节点。
基于分片切分后的数据块称为chunk,一个分片后的集合会包含多个chunk,每个chunk位于哪个分片(shard)则记录在Config Server(配置服务器)上。
Mongos在操作分片集合时,会自动根据分片键找到对应的chunk,并向该chunk所在的分片发起操作请求。
数据是根据分片策略来进行切分的,而分片策略则由分片键(ShardKey)+分片算法(ShardStrategy)组成。
MongoDB支持两种分片算法:哈希分片和范围分片
哈希分片使用哈希索引来在分片集群中对数据进行划分。哈希索引计算某一个字段的哈希值作为索引键,这个值被用作片键。
数据是分布在不同的chunk上的,而chunk则会分配到不同的分片上,那么如何保证分配上的数据(chunk)是均衡的呢?
mkdir -p /usr/local/mongodb/sharded_cluster/myshardrs01_27018/log
mkdir -p /usr/local/mongodb/sharded_cluster/myshardrs01_27018/data/db
mkdir -p /usr/local/mongodb/sharded_cluster/myshardrs01_27118/log
mkdir -p /usr/local/mongodb/sharded_cluster/myshardrs01_27118/data/db
mkdir -p /usr/local/mongodb/sharded_cluster/myshardrs01_27218/log
mkdir -p /usr/local/mongodb/sharded_cluster/myshardrs01_27218/data/db
vim /usr/local/mongodb/sharded_cluster/myshardrs01_27018/mongod.conf
systemLog:
destination:file
path:"/usr/local/mongodb/replica_sets/myrs_27018/log/mongod.log"
logAppend: true
storage:
dbpath:"/usr/local/mongodb/replica_sets/myrs_27018/data/db"
journal:
# 启用持久化日志
enabled: true
processManagement:
fork:true
pidFilePath:"/usr/local/mongodb/replica_sets/myrs_27018/log/mongod.pid"
net:
bindIp:localhost,192.168.64.129
port:27018
replication:
replSetName: myshardrs01
sharding:
clusterRole: shardsvr
# 分片角色,shardsvr为分片节点,configsvr配置节点
mongod -f /usr/local/mongodb/sharded_cluster/myshardrs01_27018/mongod.conf
mongod -f /usr/local/mongodb/sharded_cluster/myshardrs01_27118/mongod.conf
mongod -f /usr/local/mongodb/sharded_cluster/myshardrs01_27218/mongod.conf
ps -ef | grep mongod
mongo --port 27018
rs.initiate()
rs.status()
rs.add("192.168.64.129:27118")
rs.addArb("192.168.64.129:27218")
rs.conf()
mkdir -p /usr/local/mongodb/sharded_cluster/myshardrs02_27318/log
mkdir -p /usr/local/mongodb/sharded_cluster/myshardrs02_27318/data/db
mkdir -p /usr/local/mongodb/sharded_cluster/myshardrs02_27418/log
mkdir -p /usr/local/mongodb/sharded_cluster/myshardrs02_27418/data/db
mkdir -p /usr/local/mongodb/sharded_cluster/myshardrs02_27518/log
mkdir -p /usr/local/mongodb/sharded_cluster/myshardrs02_27518/data/db
vim /usr/local/mongodb/sharded_cluster/myshardrs01_27318/mongod.conf
systemLog:
destination:file
path:"/usr/local/mongodb/replica_sets/myrs_27318/log/mongod.log"
logAppend: true
storage:
dbpath:"/usr/local/mongodb/replica_sets/myrs_27318/data/db"
journal:
# 启用持久化日志
enabled: true
processManagement:
fork:true
pidFilePath:"/usr/local/mongodb/replica_sets/myrs_27318/log/mongod.pid"
net:
bindIp:localhost,192.168.64.129
port:27318
replication:
replSetName: myshardrs02
sharding:
clusterRole: shardsvr
# 分片角色,shardsvr为分片节点,configsvr配置节点
mongod -f /usr/local/mongodb/sharded_cluster/myshardrs01_27318/mongod.conf
mongod -f /usr/local/mongodb/sharded_cluster/myshardrs01_27418/mongod.conf
mongod -f /usr/local/mongodb/sharded_cluster/myshardrs01_27518/mongod.conf
ps -ef | grep mongod
mongo --port 27318
rs.initiate()
rs.status()
rs.add("192.168.64.129:27418")
rs.addArb("192.168.64.129:27518")
rs.conf()
mkdir -p /usr/local/mongodb/sharded_cluster/myconfigrs_27019/log
mkdir -p /usr/local/mongodb/sharded_cluster/myconfigrs_27019/data/db
mkdir -p /usr/local/mongodb/sharded_cluster/myconfigrs_27119/log
mkdir -p /usr/local/mongodb/sharded_cluster/myconfigrs_27119/data/db
mkdir -p /usr/local/mongodb/sharded_cluster/myconfigrs_27219/log
mkdir -p /usr/local/mongodb/sharded_cluster/myconfigrs_27219/data/db
vim /usr/local/mongodb/sharded_cluster/myshardrs01_27019/mongod.conf
systemLog:
destination:file
path:"/usr/local/mongodb/replica_sets/myrs_27019/log/mongod.log"
logAppend: true
storage:
dbpath:"/usr/local/mongodb/replica_sets/myrs_27019/data/db"
journal:
# 启用持久化日志
enabled: true
processManagement:
fork:true
pidFilePath:"/usr/local/mongodb/replica_sets/myrs_27019/log/mongod.pid"
net:
bindIp:localhost,192.168.64.129
port:27019
replication:
replSetName: myconfigr
sharding:
clusterRole: configsvr
# 分片角色,shardsvr为分片节点,configsvr配置节点
mongod -f /usr/local/mongodb/sharded_cluster/myshardrs01_27019/mongod.conf
mongod -f /usr/local/mongodb/sharded_cluster/myshardrs01_27119/mongod.conf
mongod -f /usr/local/mongodb/sharded_cluster/myshardrs01_27219/mongod.conf
ps -ef | grep mongod
mongo --port 27019
rs.initiate()
rs.status()
rs.add("192.168.64.129:27119")
rs.add("192.168.64.129:27219")
rs.conf()
mkdir -p /usr/local/mongodb/sharded_cluster/mymongos_27017/log
vim /usr/local/mongodb/sharded_cluster/mymongos_27017/mongod.conf
systemLog:
destination:file
path:"/usr/local/mongodb/replica_sets/mymongos_27017/log/mongod.log"
logAppend: true
processManagement:
fork:true
pidFilePath:"/usr/local/mongodb/replica_sets/mymongos_27017/log/mongod.pid"
net:
bindIp:localhost,192.168.64.129
port:27017
sharding:
configDB: myconfigrs/192.168.64.129:27019,192.168.64.129:27119,192.168.64.129:27219
# 分片角色,shardsvr为分片节点,configsvr配置节点
mongod -f /usr/local/mongodb/sharded_cluster/mymongos_27017/mongod.conf
mongo --port 27017
mongos> use add
mongos> db.aa.insert({aa:"aa"})
mongos> sh.addShard("myshardrs01/192.168.64.129:27018,192.168.64.129:27118,192.168.64.129:27218")
mongos> sh.status()
mongos> sh.addShard("myshardrs02/192.168.175.10:27318,192.168.175.10:27418,192.168.175.10:27518")
mongos> sh.status()
use admin
db.runCommand({removeShard:"myshardrs02"})
mongos> db.runCommand({removeShard:"myshardrs02"})
mongos> sh.status()
# 可以看到转移数据过程
mongos> sh.enableSharding("articledb")
mongos> sh.shardCollection(namespace,key,unique)
对集合进行分片时,需要选择一个片键(shard Key),shard Key是每条记录都必须包含的,且建立了索引的单个字段或复合字段,MongoDB按照片键将数据划分到不同的数据块中,并将数据块均衡地分布到所有分片中。为了按照片键划分分数块,MongoDB使用基于哈希的分片方式(随机平均分配)或者基于范围的分片方式(数值大小分配)。用什么字段当片键都可以,如:nickname作为片键,但一定是必填字段。
sh.shardCollection("articledb.comment",{"nickname":"hashed"})
# 对comment这个集合使用hash方式分片
sh.status()
# 可以看到转移数据过程
mongos> sh.shardCollection("articledb.author",{"age":1})
# 如使用做着年龄字段作为片键,按照年龄的值进行分片
mongos> use articledb
switched to db articledb
mongos> for(var i=1;i<=1000;i++) {db.comment.insert({_id:i+"",nickname:"Test"+i})}
WriteResult({ "nInserted" : 1 })
mongos> db.comment.count()
1000
js的语法,因为mongo的shell是一个JavaScript的shell
从路由上插入的数据,必须包含片键,否则无法插入
分别登录两个片的主节点,统计文档数量
myshardrs01:
mongo --port 27018
PRIMARY> use articledb
switched to db articledb
myshardrs01:PRIMARY> db.comment.count()
505
myshardrs02:
mongo --port 27318
PRIMARY> use articledb
switched to db articledb
myshardrs02:PRIMARY> db.comment.count()
495
myshardrs02:PRIMARY> db.comment.find()
{ "_id" : "1", "nickname" : "Test1" }
{ "_id" : "3", "nickname" : "Test3" }
{ "_id" : "5", "nickname" : "Test5" }
{ "_id" : "6", "nickname" : "Test6" }
{ "_id" : "7", "nickname" : "Test7" }
{ "_id" : "10", "nickname" : "Test10" }
{ "_id" : "11", "nickname" : "Test11" }
{ "_id" : "12", "nickname" : "Test12" }
{ "_id" : "14", "nickname" : "Test14" }
{ "_id" : "17", "nickname" : "Test17" }
{ "_id" : "22", "nickname" : "Test22" }
{ "_id" : "23", "nickname" : "Test23" }
{ "_id" : "24", "nickname" : "Test24" }
{ "_id" : "28", "nickname" : "Test28" }
{ "_id" : "29", "nickname" : "Test29" }
{ "_id" : "30", "nickname" : "Test30" }
{ "_id" : "34", "nickname" : "Test34" }
{ "_id" : "37", "nickname" : "Test37" }
{ "_id" : "39", "nickname" : "Test39" }
{ "_id" : "44", "nickname" : "Test44" }
mongos> use articledb
switched to db articledb
mongos> for (var i=1;i<=2000;i++) {db.author.save({"name":"test"+i,"age":NumberInt(i%120)})}
WriteResult({ "nInserted" : 1 })
mongos> db.author.count()
2000
myshardrs02:PRIMARY> db.author.count()
2000
发现所有的数据都集中在了一个分片副本上
如果发现没有分片:
use config
db.settings.save({_id:"chunksize",value:1})
# 改成1M
db.settings.save({_id:"chunksize",value:64})
集合使用哈希策略后,在其集合上插入数据会发现大致在两个分片集群上能够均匀分布
集合使用范围策略后,数据会首先分配到其中的一个分片集群上
mkdir -p /usr/local/mongodb/sharded_cluster/mymongos_27117/log
vim /usr/local/mongodb/sharded_cluster/mymongos_27117/mongod.conf
systemLog:
destination:file
path:"/usr/local/mongodb/replica_sets/mymongos_27117/log/mongod.log"
logAppend: true
processManagement:
fork:true
pidFilePath:"/usr/local/mongodb/replica_sets/mymongos_27117/log/mongod.pid"
net:
bindIp:localhost,192.168.64.129
port:27117
sharding:
configDB: myconfigrs/192.168.64.129:217019,192.168.64.129:27119,192.168.64.129:27219
# 分片角色,shardsvr为分片节点,configsvr配置节点
mongod -f /usr/local/mongodb/sharded_cluster/mymongos_27117/mongod.conf
mongo --port 27117
mongos> db.status()