MongDB集群部署和升级方案

******集群部署部分******

主机信息mongodb集群
192.168.1.46 
192.168.1.47 
192.168.1.48 
192.168.1.49 
192.168.1.50 
192.168.1.51 

规划
sharding 1
  192.168.1.46 
  192.168.1.47 
  192.168.1.48
  端口 28111

sharding 2
  192.168.1.49 
  192.168.1.50 
  192.168.1.51
  端口 28112

config 副本集
 192.168.1.46 
 192.168.1.47 
 192.168.1.48
  端口 28600

mongos 集群
 192.168.1.46 
 192.168.1.47 
 192.168.1.48
 端口 28700

做一个LSB 各个负载均衡IP+port
mongo:192.167.63.22:40005

[
  目前DB使用的lvs主机:
  192.167.3.169
  192.167.3.170
]

数据库版本
 3.4.24

下载
 wget https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-rhel70-3.4.24.tgz
 wget https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-rhel70-3.6.17.tgz[升级需要的版本]

目录规划 mkdir -p src/{a,b,c,d}
 mkdir -p /DATA/mongo_user_sale/{data,etc,logs,config,mongos}
 

部署
第一部分
1.先部署sharding副本集
  参数
  # for documentation of all options, see:
#   http://docs.mongodb.org/manual/reference/configuration-options/

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  logRotate: reopen
  path: /DATA/mongo_user_sale/logs/mongo_usersale.log

# Where and how to store data.
storage:
  dbPath: /DATA/mongo_user_sale/data
  journal:
    enabled: true
  directoryPerDB: true
  engine: wiredTiger
#########storage.wiredTiger Options
  wiredTiger:
    engineConfig:
      cacheSizeGB: 24
      journalCompressor: "zlib"
      directoryForIndexes: true
    collectionConfig:
      blockCompressor: "zlib"
    indexConfig:
      prefixCompression: true

##########operationProfiling Options
operationProfiling:
   slowOpThresholdMs: 100
   mode: "all"

########ProcessManagement Options
processManagement:
  fork: true
  pidFilePath: /DATA/mongo_user_sale/data/mongod.pid

# network interfaces
net:
  port: 28111
  bindIp: 0.0.0.0
  maxIncomingConnections: 18500
  #serviceExecutor: adaptive #4.2版本
#security:
  #keyFile: /DATA/mongo_user_sale/etc/keyfile

#operationProfiling:

##########replication Options
replication:
  replSetName: usersale1Rs
  oplogSizeMB: 50240
  #secondaryIndexPrefetch: all

sharding:
  clusterRole: shardsvr    #configsvr or shardsvr

## Enterprise-Only Options

#auditLog:

#snmp:

setParameter:
  enableLocalhostAuthBypass: true
  replWriterThreadCount: 32
  wiredTigerConcurrentReadTransactions: 128
  wiredTigerConcurrentWriteTransactions: 128

第一次部署将security注释
注意:其他sharding部署注意 副本集名称和端口

2.在第一个sharding 192.168.1.46操作
  (1)编辑参数# pwd
    /DATA/mongo_user_sale/etc
     vim mongo_usersale.conf
  (2)启动
      [root@node10-133-1-46 etc]# /usr/local/mongodb-3.4.24/bin/mongod -f /DATA/mongo_user_sale/etc/mongo_usersale.conf 
    about to fork child process, waiting until server is ready for connections.
    forked process: 47385
    child process started successfully, parent exiting
      说明启动成功

  (3)配置单机副本集
    # /usr/local/mongodb-3.4.24/bin/mongo 192.168.1.46:28111/admin
    > config={_id:"usersale1Rs",members:[{_id:1,host:"192.168.1.46:28111",priority:136,tags:{'use':'user-sale-rs1-1'}}]}
    > rs.initiate(config)
    { "ok" : 1 }
    usersale1Rs:SECONDARY> 
    usersale1Rs:PRIMARY> 

  (4)创建账号
    a. 管理账号:
    usersale1Rs:PRIMARY> use admin 
      switched to db admin
    db.createUser(  
    {  
        user: "admin",  
        pwd: "tk123456",  
        roles:  
        [  
          {role: "userAdminAnyDatabase", db: "admin"},
          { role: "readAnyDatabase", db: "admin" },
          { role: "dbOwner", db: "admin" },
          { role: "userAdmin", db: "admin" },
          { role: "root", db: "admin" },
          { role: "clusterMonitor", db: "admin" },
          { role: "dbAdmin", db: "admin" },
        ]  
      }  
    )
 
  

  b. mongodb只读账号
    use admin
    db.createUser(  
    {  
        user: "readany",  
        pwd: "tk123456",  
        roles:  
        [  
          { role: "readAnyDatabase", db: "admin" },
        ]  
      }  
    )

  c. 监控用户
    use admin  
    db.createUser(  
    {  
        user: "mongodb_exporter",  
        pwd: "tk123456",  
        roles:  
        [  
          { role: "readAnyDatabase", db: "admin" },
          { role: "dbAdmin", db: "admin" }
        ]  
      }  
    )

  d. 授予抓取Oplog权限
    db.createRole({role:'sysadmin',roles:[], privileges:[ {resource:{anyResource:true},actions:['anyAction']}]})
    db.grantRolesToUser( "admin" , [ { role: "sysadmin", db: "admin" } ]) 
    db.grantRolesToUser( "readany" , [ { role: "sysadmin", db: "admin" } ]) 
       第一个admin和readany是用户名
    db.system.users.find().pretty() 查询看一下是否授权成功

  (5)重启并开启认证模块
     a. 添加认证文件
       vim keyfile
      MIIEowIBAAKCAQEAxJ4E9nfpSaEtBUUP/ypkVmzr2ORD4BUSzQXp+EMXv0oeIeW0
      vU6Jv/mvAPD0+9veCGK/LFmZGVSQZbjyYnFxx1xLCgzqkkvgX8HqaChPKySta3wi
      YHQLgBDJ4boXZoLIFlGO9UWjJTMXZSbD0Tdu4g2DL3qZLGENDLFam4gt0t2HnZ+g
      iY/9C9PKHnKqUpjfJNnWU8h09Zc4erZdM50KWxEdC4iFMKWTSOZr/AxzXC5cOYC/
      y8ALfBLlJstavp1KMG2XoQoGRRnLSX3bWqIRt60oFUq35x8slknHxOKkXxhX7583
      BfLHfWxrCU8viOp9z1yb0URUR6fotykpUz9/QwIDAQABAoIBADkCADpeRjvdCaqv
      Fl3BDttoQ5Munt7pPNljSvwxnee3ZWBRwmQqhQo85OEYcIRsXMoG2ceW71HgTlwK
      fpHLXyDIksUNUNujcbX9152/a94Fd4FU45GpcxojDwPfS+pG8GObOvNopSZzEivS
      Az21uTZ51zxwv5Jwi0cs3vDLq9/HmkPGUFpIuRq/CdZGmOh4m0HWWaKohPFHQPJ8
      lfMjAaECgYEA7RWeg6nnRzMHhNiMRw2bBURdYhDu13gorWY2wr+Xse8+oxHDrz0d
      baHbF0UWkUlTpeiQK30ygWy82/bm8v6lds12iElkY13Ua+xMxHdZkY1h2k69n/cE
      en18ESaapcX1DjWNzWCtP89g9x2p/MMljDJ/tr2xt6KxiEcxVpToo0sCgYEA1E3e
      pCLflIGbN2x3fUJkoM2aVpbaffsbLHkf8SglV9OJI7ROAvZ29xmXa7sYkN7OdXMu

        # chmod 600 keyfile 权限必须是600
     b. 将mongod shutdown
     # /usr/local/mongodb-3.4.24/bin/mongod -f /DATA/mongo_user_sale/etc/mongo_usersale.conf  --shutdown
      将如下注释去掉
      security:
        keyFile: /DATA/mongo_user_sale/etc/keyfile
     c. 启动 /usr/local/mongodb-3.4.24/bin/mongod -f /DATA/mongo_user_sale/etc/mongo_usersale.conf
      进入后需要认证
      db.auth('admin','tk123456')
      usersale1Rs:PRIMARY> db.auth('admin','tk123456')
      1
      usersale1Rs:PRIMARY> rs.status() 查看状态
       "members" : [
                {
                        "_id" : 1,
                        "name" : "192.168.1.46:28111",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 363,
                        "optime" : {
                                "ts" : Timestamp(1586659765, 1),
                                "t" : NumberLong(2)
                        },
                        "optimeDate" : ISODate("2020-04-12T02:49:25Z"),
                        "syncingTo" : "",
                        "syncSourceHost" : "",
                        "syncSourceId" : -1,
                        "infoMessage" : "",
                        "electionTime" : Timestamp(1586659413, 1),
                        "electionDate" : ISODate("2020-04-12T02:43:33Z"),
                        "configVersion" : 1,
                        "self" : true,
                        "lastHeartbeatMessage" : ""
                }
        ],
        "ok" : 1
}

  (6)复制1.46的参数文件和认证文件到47/48主机，并启动
    /usr/local/mongodb-3.4.24/bin/mongod -f /DATA/mongo_user_sale/etc/mongo_usersale.conf 

  (7)添加节点
  rs.add({_id:2, host:"192.168.1.47:28111", priority:100,tags:{'use':'user-sale-rs1-2'}})
  rs.add({_id:3, host:"192.168.1.48:28111", priority:190,tags:{'use':'user-sale-rs1-3'}}) #添加完，该节点为primary
  注意，我设置了优先级，可以不设置

3.日志切分--每天生成一个
mongodb日志切换
mongodb日志切换路径  保留日志30天
vim /etc/logrotate.d/mongo_28111
/DATA1/mongo_28116/logs/mongo_28111.log {
    daily
    rotate 30
    dateext
    compress
    missingok
    notifempty
    sharedscripts
    copytruncate
    postrotate
        /bin/kill -SIGUSR1 `cat /DATA/mongo_user_sale/data/mongod.lock 2> /dev/null` 2> /dev/null || true
    endscript
}

启动
logrotate -vf /etc/logrotate.conf

4.压测
./ycsb load mongodb -s -P /DATA/soft/ycsb-0.12.0/workloads/workloada -threads 1024 -p recordcount=0 -p operationcount=0 -p insertstart=0 -p insertcount=330000000 -p readproportion=0.1 -p updateproportion=0.9 -p insertproportion=0 -p fieldcount=8 -p fieldlength=250 -p mongodb.url=mongodb://admin:[email protected]:28111/admin?waitQueueMultiple=100 -p table=userinfo > /dev/null  2> /DATA/soft/lsbwrite &

 
 监控
 /usr/local/mongodb-3.4.24/bin//mongostat --host=192.168.1.48  --port=28111 -u admin -p "tk123456" --authenticationDatabase admin 

5.部署第二个sharding
  参考第一个部署步骤
  端口和副本集修改
  > config={_id:"usersale2Rs",members:[{_id:1,host:"192.168.1.49:28112",priority:136,tags:{'use':'user-sale-rs2-1'}}]}
    > rs.initiate(config)
  rs.add({_id:2, host:"192.168.1.50:28112", priority:100,tags:{'use':'user-sale-rs2-2'}})
  rs.add({_id:3, host:"192.168.1.51:28112", priority:90,tags:{'use':'user-sale-rs2-3'}}) 

第二部分
 部署config副本集
1.配置参数
# fordocumentation of all options, see:
#  http://docs.mongodb.org/manual/reference/configuration-options/

# where to writelogging data.
systemLog:
  destination: file
  logAppend: true
  logRotate: reopen
  path: /DATA/mongo_user_sale/logs/configsvr.log

# Where and howto store data.
storage:
  dbPath: /DATA/mongo_user_sale/config
  journal:
    enabled: true
  directoryPerDB: true
  engine: wiredTiger
#########storage.wiredTigerOptions
  wiredTiger:
    engineConfig:
      cacheSizeGB: 4
      directoryForIndexes: true
    indexConfig:
      prefixCompression: true

##########operationProfilingOptions
#operationProfiling:
   #slowOpThresholdMs: 50
   #mode: "all"

########ProcessManagementOptions
processManagement:
  fork: true
  pidFilePath: /DATA/mongo_user_sale/config/config.pid

# networkinterfaces
net:
  port: 28600
  bindIp: 0.0.0.0
  maxIncomingConnections: 15000
security:
  keyFile: /DATA/mongo_user_sale/etc/keyfile

#operationProfiling:

##########replicationOptions
replication:
  replSetName: configRS
  oplogSizeMB: 20480
  #secondaryIndexPrefetch: all

sharding:
  clusterRole: configsvr    #configsvr or shardsvr

2.启动
 # /usr/local/mongodb-3.4.24/bin/mongod -f /DATA/mongo_user_sale/etc/config.conf
3.配置副本集
> rs.initiate( {_id: "configRS",
...    configsvr: true,
...    members: [
...       { _id: 1, host: "192.168.1.46:28600", priority:100, tags:{'use':'config-1' }}
...    ]
... } )
{ "ok" : 1 }
configRS:SECONDARY> 
configRS:PRIMARY> 

3.创建账号
 管理账号:
    usersale1Rs:PRIMARY> use admin 
      switched to db admin
    db.createUser(  
    {  
        user: "admin",  
        pwd: "tk123456",  
        roles:  
        [  
          {role: "userAdminAnyDatabase", db: "admin"},
          { role: "readAnyDatabase", db: "admin" },
          { role: "dbOwner", db: "admin" },
          { role: "userAdmin", db: "admin" },
          { role: "root", db: "admin" },
          { role: "clusterMonitor", db: "admin" },
          { role: "dbAdmin", db: "admin" },
        ]  
      }  
    )

4.启停止后，将认证参数打开
5.添加节点
rs.add({_id:2, host:"192.168.1.47:28600", priority:90, tags:{'use':'config-2'}})
rs.add({_id:3, host:"192.168.1.48:28600", priority:30, tags:{'use':'config-3'}})

【或者
 rs.initiate( {_id: "configReplSet",
   configsvr: true,
   members: [
      { _id: 1, host: "192.168.1.46:28600", priority:100, tags:{'use':'config-1' }},
      { _id: 2, host: "192.168.1.47:28600", priority:90, tags:{'use':'config-2' }},
      { _id: 3, host: "192.168.1.48:28600", priority:30, tags:{'use':'config-3' }}
   ]
} )
】

第二部分
 部署mongos
1.参数配置
 systemLog:
  destination: file
  path: /DATA/mongo_user_sale/logs/mongos.log
  logAppend: true
processManagement:
  fork: true
  pidFilePath: /DATA/mongo_user_sale/mongos/mongos.pid
net:
  port: 28700
  bindIp: 0.0.0.0
sharding:
  configDB: configRS/192.168.1.46:28600,192.168.1.47:28600,192.168.1.48:28600
security:
  keyFile: "/DATA/mongo_user_sale/etc/keyfile"

2.启动
  /usr/local/mongodb-3.4.24/bin/mongos -f /DATA/mongo_user_sale/etc/mongos.conf

说明:
 mongos和config创建的账号密码一致，可以和sharding不一样

 /usr/local/mongodb-3.4.24/bin/mongo 192.168.1.46:28700/admin

集群添加副本集*****
>use admin 
 db.runCommand({addshard:"usersale1Rs/192.168.1.46:28111,192.168.1.47:28111,192.168.1.48:28111",name:"usersale1_tag"});
 db.runCommand({addshard:"usersale2Rs/192.168.1.49:28112,192.168.1.50:28112,192.168.1.51:28112",name:"usersale2_tag"});
 如果有同名的数据库则添加失败:
  "errmsg" : "can't add shard 'sale2Rs/192.168.1.49:28116,192.168.1.50:28116,192.168.1.51:28116' because a local database 'ycsb' exists in another sale1_tag",
  
  >sh.status()
  databases:
        {  "_id" : "ycsb",  "primary" : "usersale1_tag",  "partitioned" : false }

 >db.runCommand({listshards:1}) 

******集群升级部分******
2. 登录mognos禁用balance
  mongos> sh.stopBalancer()

mongos> sh.isBalancerRunning()
true
mongos> sh.getBalancerState()
true

mongos> sh.status()
--- Sharding Status --- 
  sharding version: {
        "_id" : 1,
        "minCompatibleVersion" : 5,
        "currentVersion" : 6,
        "clusterId" : ObjectId("5e92898736edafe6acf3f45c")
  }
  shards:
        {  "_id" : "usersale1_tag",  "host" : "usersale1Rs/192.168.1.46:28111,192.168.1.47:28111,192.168.1.48:28111",  "state" : 1 }
        {  "_id" : "usersale2_tag",  "host" : "usersale2Rs/192.168.1.49:28112,192.168.1.50:28112,192.168.1.51:28112",  "state" : 1 }
  active mongoses:
        "3.4.24" : 3
  autosplit:
        Currently enabled: yes
  balancer:
        Currently enabled:  yes
        Currently running:  yes
NaN
        Collections with active migrations: 
                ycsb.userinfo started at Sun Apr 12 2020 11:55:04 GMT+0800 (CST)
        Failed balancer rounds in last 5 attempts:  0
        Migration Results for the last 24 hours: 
                32 : Success
  databases:
        {  "_id" : "ycsb",  "primary" : "usersale1_tag",  "partitioned" : true }
                ycsb.userinfo
                        shard key: { "_id" : 1 }
                        unique: false
                        balancing: true
                        chunks:
                                usersale1_tag   309
                                usersale2_tag   32
                        too many chunks to print, use verbose if you want to force print

mongos> sh.stopBalancer()
{ "ok" : 1 }
mongos> sh.getBalancerState()
false

3. 升级config server

  3.1 从节点 先使用3.6版本依次替换3.4 版本,并依次重启
   主节点查看 configRS:PRIMARY> rs.status()

  3.2 关闭secondary config server
    # /usr/local/mongodb-3.4.24/bin/mongod -f /DATA/mongo_user_sale/etc/config.conf --shutdown
killing process with pid: 14654
[root@node10-133-1-48 operation]# /usr/local/mongodb-3.6.17/bin/mongod -f /DATA/mongo_user_sale/etc/config.conf
about to fork child process, waiting until server is ready for connections.
forked process: 25427
child process started successfully, parent exiting

  3.3 对于主节点，先切换成从节点
      rs.stepDown() 或调整优先级
      cfg = rs.conf()
cfg.members[1].priority = 136
rs.reconfig(cfg)

# /usr/local/mongodb-3.4.24/bin/mongod -f /DATA/mongo_user_sale/etc/config.conf --shutdown
killing process with pid: 14654
[root@node10-133-1-48 operation]# /usr/local/mongodb-3.6.17/bin/mongod -f /DATA/mongo_user_sale/etc/config.conf

4. 升级shard
  按复制集依次处理
  3.1 从节点 先使用3.6版本依次替换3.4 版本,并依次重启
  3.2 关闭secondary shard1 server
       设置为隐藏节点
cfg = rs.conf()
cfg.members[1].priority = 0
cfg.members[1].hidden = true
rs.reconfig(cfg)
使用mongostat查看是否select还有值>1
    [root@node10-133-1-46 operation]# /usr/local/mongodb-3.4.24/bin/mongod -f /DATA/mongo_user_sale/etc/mongo_usersale.conf --shutdown
killing process with pid: 48861
[root@node10-133-1-46 operation]# 
[root@node10-133-1-46 operation]# /usr/local/mongodb-3.6.17/bin/mongod -f /DATA/mongo_user_sale/etc/mongo_usersale.conf
cfg = rs.conf()
cfg.members[1].priority = 10
cfg.members[1].hidden = false
rs.reconfig(cfg)

  3.3 
    对于主节点，先切换成从节点
    先调整优先级，让其他一台secondary成为primaryk
    然后将该节点设置为隐藏节点
      

sharding2升级步骤参考sharding1即可

5. 升级mongos 实例
  5.1 使用kill 或者 kill -2 杀掉mognos实例
    kill -2 17739
  5.2  使用3.6代替3.4 重启
    //usr/local/mongodb-3.6.17/bin/mongos -f /DATA/mongo_user_sale/etc/mongos.conf
    其他mognos 实例类似
    
6. 登录任一mongos实例重启启用balancer
  /usr/local/mongodb-3.4.18/bin/mongo --port 5080
  mongos> sh.startBalancer()
  
7. 启用后台不兼容3.6 特性
  mongos> 
  mongos>  db.adminCommand( { setFeatureCompatibilityVersion: "3.6" } )
{ "ok" : 1 }

   mongos执行不输出任何: db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } )
   可以在sharding上执行和config执行

usersale2Rs:PRIMARY> db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } )
{
        "featureCompatibilityVersion" : {
                "version" : "3.4"
        },
        "ok" : 1,
        "$gleStats" : {
                "lastOpTime" : Timestamp(0, 0),
                "electionId" : ObjectId("7fffffff0000000000000004")
        },
        "$configServerState" : {
                "opTime" : {
                        "ts" : Timestamp(1586669968, 1),
                        "t" : NumberLong(3)
                }
        }
}
usersale2Rs:PRIMARY> db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } )
{
        "featureCompatibilityVersion" : {
                "version" : "3.6"
        },
        "ok" : 1,
        "operationTime" : Timestamp(1586670057, 521),
        "$gleStats" : {
                "lastOpTime" : Timestamp(0, 0),
                "electionId" : ObjectId("7fffffff0000000000000004")
        },
        "$configServerState" : {
                "opTime" : {
                        "ts" : Timestamp(1586670058, 589),
                        "t" : NumberLong(3)
                }
        },
        "$clusterTime" : {
                "clusterTime" : Timestamp(1586670060, 2502),
                "signature" : {
                        "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                        "keyId" : NumberLong(0)
                }
        }
}

重启 mongos 实例。
更改featureCompatibilityVersion后，需要重新启动所有mongos实例以获取因果一致性行为的更改。