Mongodb Sharding 分片

Mongodb Sharding

  

介绍 Sharding in MongoDB 

分片组建：

Shards(建议用Replica)、Query Routers、Config Servers。

 

Shards store the data. To provide high availability and dataconsistency, in a production sharded cluster, each shard is a replica set [1].For more information on replica sets, see ReplicaSets.

Query Routers, or mongos instances, interface with clientapplications and direct operations to the appropriate shard or shards. Thequery router processes and targets operations to shards and then returnsresults to the clients. A sharded cluster can contain more than one queryrouter to divide the client request load. A client sends requests to one queryrouter. Most sharded clusters have many query routers.

Config servers store thecluster’s metadata. This data contains a mapping of the cluster’s data set tothe shards. The query router uses this metadata to target operations tospecific shards. Production sharded clusters have exactly 3 config servers.

[1]	For development and testing purposes only, each shard can be a single mongod instead of a replica set. Donot deploy production clusters without 3 config servers.

  

分片类型:Range和Hash 

比较总结:线性用range散列用hash。

详细的就不从官网复制了。

环境准备:

分片结构分布如下：

 ShardServer 1：27020

ShardServer 2：27021

ShardServer 3：27022

ShardServer 4：27023

ConfigServer ：27100

RouteProcess：40000

ShardServer 5：27024(模拟新增服务节点)

 

步骤一：启动Shard Server 

mongod--port 27020 --dbpath=F:\DingSai\Mongodb\shard\rs1\data --logpath=F:\DingSai\Mongodb\shard\rs1\logs\mongodb.log --logappend

 mongod--port 27021 --dbpath=F:\DingSai\Mongodb\shard\rs2\data --logpath=F:\DingSai\Mongodb\shard\rs2\logs\mongodb.log --logappend 

mongod--port 27022 --dbpath=F:\DingSai\Mongodb\shard\rs3\data --logpath=F:\DingSai\Mongodb\shard\rs3\logs\mongodb.log --logappend

mongod--port 27023 --dbpath=F:\DingSai\Mongodb\shard\rs4\data  --logpath=F:\DingSai\Mongodb\shard\rs4\logs\mongodb.log--logappend

 

步骤二： 启动ConfigServer 

mongod--port 27100 --dbpath=F:\DingSai\Mongodb\shard\config\data--logpath=F:\DingSai\Mongodb\shard\config\logs\mongodb.log --logappend

 注意：这里我们完全可以像启动普通mongodb服务一样启动，不需要添加—shardsvr和configsvr参数。因为这两个参数的作用就是改变启动端口的，所以我们自行指定了端口就可以。

 

步骤三： 启动RouteProcess

mongos --port 40000 --configdblocalhost:27100 --logpath=F:\DingSai\Mongodb\shard\RouteProcess\logs\route.log --chunkSize 500

mongos启动参数中，chunkSize这一项是用来指定chunk的大小的，单位是MB，默认大小为200MB.

 

步骤四： 配置Sharding

接下来，我们使用MongoDB Shell登录到mongos，添加Shard节点

bin/mongoadmin --port 40000

MongoDBshell version: 2.0.7

connectingto: 127.0.0.1:40000/admin

use admin

mongos>db.runCommand({ addshard:"localhost:27020" })

mongos>db.runCommand({ addshard:"localhost:27021" })

mongos>db.runCommand({ addshard:"localhost:27022" })

mongos>db.runCommand({ addshard:"localhost:27023" })

 

--可以添加节点名db.runCommand({addshard : "192.168.253.212:27017",  "name" : "XXX Server" });

 

--对数据库 ding 启用分片

mongos>db.runCommand({ enablesharding:"ding" })

 

 

--对数据库  ding 下面的  c2  表，启用分片--按照 _id 列进行分片

mongos>db.runCommand({shardcollection: "ding.c2", key: {_id:1}}

  

步骤五：测试

插入数据:

mongoadmin --port 40000

--切换数据库

useding

##插入5万行

mongos>for(var i=0;i<500;i++){ db.c2.insert({name:'dingsai'+i,seq:i}) }

##查看记录

mongos>  db.c2.find().count()

 

--查看分片状态

 db.printShardingStatus()

或者sh.status();

 mongos> db.printShardingStatus()

---Sharding Status ---

  sharding version: {

        "_id" : 1,

        "version" : 4,

        "minCompatibleVersion" : 4,

        "currentVersion" : 5,

        "clusterId" :ObjectId("54dffdc6d33e0feb326a8f90")

}

  shards:

        { "_id" : "shard0000", "host" : "localhost:27020" }

        { "_id" : "shard0001", "host" : "localhost:27021" }

        { "_id" : "shard0002", "host" : "localhost:27022" }

        { "_id" : "shard0003", "host" : "localhost:27023" }

  databases:

         { "_id" : "ding", "partitioned" : true, "primary" : "shard0001" }

             ding.c2

                        shard key: {"_id" : 1 }

                        chunks:

                                shard0000       1

                                shard0001       1

                                shard0002       1

                        { "_id" : {"$minKey" : 1 } } -->> { "_id" :ObjectId("54e009d6752aea9c8fc3d25a") } on :shard0000 Timestamp(2, 0)

                        { "_id" :ObjectId("54e009d6752aea9c8fc3d25a") } -->> { "_id" :ObjectId("54e00a76752aea9c8fc3e684") } on :shard0001 Timestamp(3, 1)

                        { "_id" :ObjectId("54e00a76752aea9c8fc3e684") } -->> { "_id" :{ "$maxKey" : 1 } } on :shard0002 Timestamp(3, 0)

 

}

 

数据存储在三个节点上

新增分片节点开启服务:

bin/mongod--port 27024 --dbpath=F:\DingSai\Mongodb\shard\rs5\data --logpath=F:\DingSai\Mongodb\shard\rs5\logs\mongodb.log --logappend

登录RouteProcess:

mongoadmin --port 40000

useadmin

 

增加27024

mongos>db.runCommand({addshard : "localhost:27024"});

删除分片节点

删除sharding节点时,需要一定的时间，多次执行删除语句可查看当前删除中的状态

 

删除27021-shard0001分片开始删除

mongos>  db.runCommand({removeShard :"shard0001" });

{

        "msg" : "drainingstarted successfully",

        "state" : "started",

        "shard" :"shard0001",

        "note" : "you need todrop or movePrimary these databases",

        "dbsToMove" : [

                "ding"

        ],

        "ok" : 1

}

 

 查看当前删除状态 删除中

 

mongos>db.runCommand({removeShard : "shard0001" });

{

        "msg" : "draining ongoing",

        "state" : "ongoing",

        "remaining" : {

                "chunks" :NumberLong(0),

                "dbs" : NumberLong(1)

        },

        "note" : "you need todrop or movePrimary these databases",

        "dbsToMove" : [

                "ding"

        ],

        "ok" : 1

}

 

查看当前删除状态 删除完成

mongos>db.runCommand({removeShard : "shard0001" });

{

        "ok" : 0,

        "errmsg" : "removeShardmay only be run against the admin database.",

        "code" : 13

}

 

再次查看parding状态 

ongos>db.printShardingStatus()

--Sharding Status ---

 sharding version: {

       "_id" : 1,

       "version" : 4,

       "minCompatibleVersion" : 4,

       "currentVersion" : 5,

       "clusterId" :ObjectId("54dffdc6d33e0feb326a8f90")

 

 shards:

       { "_id" : "shard0000", "host" : "localhost:27020" }

       { "_id" : "shard0001", "host" : "localhost:27021",  "draining" :true }

       { "_id" : "shard0002", "host" : "localhost:27022" }

       { "_id" : "shard0003", "host" : "localhost:27023" }

       { "_id" : "shard0004", "host" : "localhost:27024" }

 databases:

        { "_id" : "ding", "partitioned" : true, "primary" : "shard0001" }

                ding.c2

                       shard key: {"_id" : 1 }

                       chunks:

                               shard0000       1

                               shard0003       1

                               shard0002       1

                       { "_id" : {"$minKey" : 1 } } -->> { "_id" :ObjectId("54e009d6752aea9c8fc3d25a") } on :shard0000 Timestamp(2, 0)

                       { "_id" :ObjectId("54e009d6752aea9c8fc3d25a") } -->> { "_id" :ObjectId("54e00a76752aea9c8fc3e684") } on :shard0003 Timestamp(4, 0)

                       { "_id" :ObjectId("54e00a76752aea9c8fc3e684") } -->> { "_id" :{ "$maxKey" : 1 } } on :shard0002 Timestamp(3, 0)

 

ongos>

 

发现：shard0001号机器状态为draining:true

数据也已经从shard0001移到别的节点上。

 

 

 

节点损坏查询:

1.  关闭27002端口节点 

2.查看损坏节点信息：mongos>use config

switchedto db config

mongos>db.mongos.find()

{"_id" : "hpo-PC:40000", "ping" :ISODate("2015-02-15T03:36:40.670Z"), "up" : 5762,"waiting" : false, "mongoVersion" : "2.6.6" }

mongos>

 

3.重启27002后正常使用 

步骤六：数据备份 

导出ding数据库 到ding文件夹

mongodump-db ding -out F:\DingSai\Mongodb\bin\ding\ -h localhost:40000

 

总结：

什么时候使用Sharding

1.  复制所有的写入操作到主节点

2.  延迟的敏感数据会在主节点查询

3.  单个副本集限制在12个节点

4.  当请求量巨大时会出现内存不足。

5.  本地磁盘不足

6.  垂直扩展价格昂贵 

以上只是简单的实现，官网推荐使用副本集(Replica Set)作为分片节点。解决系统单点问题。

改天有时间再实现。 

问题：

Windows32位有2G文件限制，使用的时候注意。 

 

IMPORTANT

Ittakes time and resources to deploy sharding. If your system has already reachedor exceeded its capacity, it will be difficult to deploy sharding withoutimpacting your application.

 

Asa result, if you think you will need to partition your database in the future,do not wait until your system is over capacity to enable sharding. 

有一篇不错的文章：http://www.cnblogs.com/skyme/p/3459765.html