mongodb源码分析(十五)replication replset模式的初始化
相对于主从模式,replset模式复杂得多,其中的主从对应于这里的primary,secondary概念,primary和
secondary之间可以切换,primary掉线后能够自动的选取一个secondary成为新的primary,当然这里也是有
限制的,本文将会分析到.首先来看replset模式用到的几个集合.
local.oplog.rs: 记录replset模式下的操作日志,master/slave模式下为local.oplog.$main.
local.system.replset replset模式的配置.就是rs.initiate,rs.add等设置的信息.
先来看看一个典型的replset 配置.
当我们写一个数据时如:
db.foo.insert({x:1})
db.runCommand({getLastError:1,w:"veryImportant"})只有当这次写被写到了veryImportant指定的三个地方,如ny sf cloud时,getLastError才会返回成功,否则其
会一直等待.这种方式可以确保一份数据被写到了不同的服务器上.来看看另一种的replset配置.
{_id:'myset', members:[{_id:0,host:'192.168.136.1:27040'},{_id:1,host:'192.168.136.1:27050',votes:0}]}
这里只有两台服务器,若端口为27050的服务器关闭,那么27040端口的服务器还是primary.并不会转成secondary
并且无法工作.但是如下配置:
{_id:'myset', members:[{_id:0,host:'192.168.136.1:27040'},{_id:1,host:'192.168.136.1:27050'}]}那么当27050关闭后27040将从primary转成secondary,整个replset将无法工作.原因在于这里的votes.mongodb的
replset规定在线的服务器的votes总和的两倍要大于所有replset中配置的服务器votes的总和,
2*online_votes>all_replset_config_vote,这时replset才能正常的工作,否则将无法正常的工作.如果不设置votes默
认其值为1.讨论另外一种情况,当27040的服务器掉线时那么27050的服务器将无法成为primary,系统将不再工作.
若一开始配置如下,27040的服务器成为primary,这个时候若27040掉线,27050将接管工作成为primary.但是若
27050掉线,那么服务器将变得不可用,因为votes值为0了.这里最好通过添加仲裁来解决问题,仲裁虽然只做投票,并
{_id:'myset', members:[{_id:0,host:'192.168.136.1:27040',votes:0},{_id:1,host:'192.168.136.1:27050'}]}
不会成为primary,secondary,但是其可以在一些服务器掉线时通过保证votes值让整个系统保持正常运行,所以
10gen也建议:
Deploy an arbiter to ensure that a replica set will have a sufficient number of members to elect a primary. While having replica sets with 2 members is not recommended for production environments, if you have just two members, deploy an arbiter. Also, for any replica set with an even number of members, deploy an arbiter.
继续看replset的流程.
1. 初始化时如果启动参数不配置replset那么启动时replset会不断的加载config.config的来源有三个.一是本地
local.system.replset集合中保存的数据,二是调用rs.initiate函数设置的config,三是来自其它replset集的心跳协
议传过来的.
2. 得到配置信息后初始化,和其它服务器建立心跳连接.
3. 启动同步线程,replset集都需要启动同步线程,但是只有secondary会去同步primary的数据.
4. 启动produce线程,这个线程负责向primary请求数据,同步线程从这个线程得到操作log然后在本地replay.
5. 启动时和后面的心态协议部分会调用msgCheckNewState更改服务器状态,从secondary转primary或者反之.
下面来看代码.首先从rs.initiate(cfg)初始化开始.初始化时执行replSetInitiate命令.直接转到该命令的执行.
virtual bool run(const string& , BSONObj& cmdObj, int, string& errmsg, BSONObjBuilder& result, bool fromRepl) {
if( cmdObj["replSetInitiate"].type() != Object ) {//配置数据来自于启动命令行
string name;
vector<HostAndPort> seeds;
set<HostAndPort> seedSet;
parseReplsetCmdLine(cmdLine._replSet, name, seeds, seedSet); // may throw...
bob b;
b.append("_id", name);
bob members;
members.append("0", BSON( "_id" << 0 << "host" << HostAndPort::me().toString() ));
result.append("me", HostAndPort::me().toString());
for( unsigned i = 0; i < seeds.size(); i++ )
members.append(bob::numStr(i+1), BSON( "_id" << i+1 << "host" << seeds[i].toString()));
b.appendArray("members", members.obj());
configObj = b.obj();
}
else {//得到配置
configObj = cmdObj["replSetInitiate"].Obj();
}
bool parsed = false;
ReplSetConfig newConfig(configObj);//从配置数据中得到配置结构.
parsed = true;
checkMembersUpForConfigChange(newConfig, result, true);//查看配置的服务器是否能够连接
createOplog();//建立local.system.replset集合.
Lock::GlobalWrite lk;
bo comment = BSON( "msg" << "initiating set");
newConfig.saveConfigLocally(comment);//将配置保存到local.system.replset
result.append("info", "Config now saved locally. Should come online in about a minute.");
ReplSet::startupStatus = ReplSet::SOON;
ReplSet::startupStatusMsg.set("Received replSetInitiate - should come online shortly.");
return true;
}run->ReplSetConfig
ReplSetConfig::ReplSetConfig(BSONObj cfg, bool force) :
_ok(false),_majority(-1)
{
_constructed = false;
clear();
from(cfg);//具体的读取配置,每一个服务器得到一个MemberCfg,解析可能的setting设置.
if( force ) {
version += rand() % 100000 + 10000;
}
if( version < 1 )
version = 1;
_ok = true;
_constructed = true;
}
run->checkMembersUpForConfigChange
void checkMembersUpForConfigChange(const ReplSetConfig& cfg, BSONObjBuilder& result, bool initial) {
int failures = 0, allVotes = 0, allowableFailures = 0;
int me = 0;
for( vector<ReplSetConfig::MemberCfg>::const_iterator i = cfg.members.begin(); i != cfg.members.end(); i++ ) {
allVotes += i->votes;//得到投票总数
}
allowableFailures = allVotes - (allVotes/2 + 1);//允许丢掉的投票数
vector<string> down;
for( vector<ReplSetConfig::MemberCfg>::const_iterator i = cfg.members.begin(); i != cfg.members.end(); i++ ) {
// we know we're up
if (i->h.isSelf()) {
continue;
}
BSONObj res;
{
bool ok = false;
{
int theirVersion = -1000;//心跳协议查看配置的服务器是否能够连接
ok = requestHeartbeat(cfg._id, "", i->h.toString(), res, -1, theirVersion, initial/*check if empty*/);
if( theirVersion >= cfg.version ) {
stringstream ss;
ss << "replSet member " << i->h.toString() << " has too new a config version (" << theirVersion << ") to reconfigure";
uasserted(13259, ss.str());
}
}
if( !ok && !res["rs"].trueValue() ) {//不能连接
down.push_back(i->h.toString());
bool allowFailure = false;
failures += i->votes;
if( !initial && failures <= allowableFailures ) {
const Member* m = theReplSet->findById( i->_id );
// it's okay if the down member isn't part of the config,
// we might be adding a new member that isn't up yet
allowFailure = true;
}
if( !allowFailure ) {//初始化时要求所有配置的服务器能够被连接
string msg = string("need all members up to initiate, not ok : ") + i->h.toString();
if( !initial )
msg = string("need most members up to reconfigure, not ok : ") + i->h.toString();
uasserted(13144, msg);
}
}
}
if( initial ) {
bool hasData = res["hasData"].Bool();
uassert(13311, "member " + i->h.toString() + " has data already, cannot initiate set. All members except initiator must be empty.",
!hasData || i->h.isSelf());
}
}
if (down.size() > 0) {
result.append("down", down);
}
}run->saveConfigLocally
void ReplSetConfig::saveConfigLocally(bo comment) {
checkRsConfig();
{
Lock::GlobalWrite lk; // TODO: does this really need to be a global lock?
Client::Context cx( rsConfigNs );
cx.db()->flushFiles(true);
//theReplSet->lastOpTimeWritten = ??;
//rather than above, do a logOp()? probably
BSONObj o = asBson();//得到实际的配置,下面的putSingletonGod将配置保存到local.system.replset中
Helpers::putSingletonGod(rsConfigNs.c_str(), o, false/*logOp=false; local db so would work regardless...*/);
if( !comment.isEmpty() && (!theReplSet || theReplSet->isPrimary()) )
logOpInitiate(comment);
cx.db()->flushFiles(true);
}
}
到这里初始化配置完成,下面看mongod启动时的初始化过程.启动部分是在repl.cpp startReplication
void startReplication() {//和master/slave一样,启动都是在这个函数,只是流程不一样
/* if we are going to be a replica set, we aren't doing other forms of replication. */
if( !cmdLine._replSet.empty() ) {//replset指定了--replSet xxx,这里不为空表面是启动replSet模式
newRepl();
replSet = true;
ReplSetCmdline *replSetCmdline = new ReplSetCmdline(cmdLine._replSet);//解析cmdline,cmdline可能是<setname>/<seedhost1>,<seedhost2>,那么启动的时候就指定了replSet的配置
boost::thread t( boost::bind( &startReplSets, replSetCmdline) );//开启一个线程来做replSet的初始化
return;
}
}
void startReplSets(ReplSetCmdline *replSetCmdline) {
Client::initThread("rsStart");
replLocalAuth();
(theReplSet = new ReplSet(*replSetCmdline))->go();//真正的初始化过程
cc().shutdown();//关闭这个线程的client
}
ReplSet::ReplSet(ReplSetCmdline& replSetCmdline) : ReplSetImpl(replSetCmdline) {}
ReplSetImpl::ReplSetImpl(ReplSetCmdline& replSetCmdline) :
elect(this),
_forceSyncTarget(0),
_blockSync(false),
_hbmsgTime(0),
_self(0),
_maintenanceMode(0),
mgr( new Manager(this) ),
ghost( new GhostSync(this) ),
_writerPool(replWriterThreadCount),
_prefetcherPool(replPrefetcherThreadCount),
_indexPrefetchConfig(PREFETCH_ALL) {
_cfg = 0;
memset(_hbmsg, 0, sizeof(_hbmsg));
strcpy( _hbmsg , "initial startup" );
lastH = 0;
changeState(MemberState::RS_STARTUP);
loadConfig();//加载replset的config,若config为空,则一直在其中循环加载,直到找到真正的config
// Figure out indexPrefetch setting
std::string& prefetch = cmdLine.rsIndexPrefetch;//通过--replIndexPrefetch启动设置,同步操作时首先预加载索引
if (!prefetch.empty()) {
IndexPrefetchConfig prefetchConfig = PREFETCH_ALL;
if (prefetch == "none")
prefetchConfig = PREFETCH_NONE;
else if (prefetch == "_id_only")
prefetchConfig = PREFETCH_ID_ONLY;
else if (prefetch == "all")
prefetchConfig = PREFETCH_ALL;
else
warning() << "unrecognized indexPrefetch setting: " << prefetch << endl;
setIndexPrefetchConfig(prefetchConfig);
}
}继续来看loadConfig的加载配置部分.
void ReplSetImpl::loadConfig() {
startupStatus = LOADINGCONFIG;
while( 1 ) {
{
vector<ReplSetConfig> configs;
configs.push_back( ReplSetConfig(HostAndPort::me()) );//从本地的local.system.replset查找配置,
//这里可能是上一次设置的或者是rs.initiate初始化时保存下来的设置
for( vector<HostAndPort>::const_iterator i = _seeds->begin(); i != _seeds->end(); i++ )
configs.push_back( ReplSetConfig(*i) );//从启动时设置的位置查找配置
{
scoped_lock lck( replSettings.discoveredSeeds_mx );
if( replSettings.discoveredSeeds.size() > 0 ) {//来自远端的心跳协议,通过心跳协议知道远端
//存在同一个replset集的服务器,从远端读取配置
for (set<string>::iterator i = replSettings.discoveredSeeds.begin();
i != replSettings.discoveredSeeds.end();
i++) {
configs.push_back( ReplSetConfig(HostAndPort(*i)) );
}
}
}
if (!replSettings.reconfig.isEmpty())//来自本地配置如rs.add等的新的配置
configs.push_back(ReplSetConfig(replSettings.reconfig, true));
int nok = 0;
int nempty = 0;
for( vector<ReplSetConfig>::iterator i = configs.begin(); i != configs.end(); i++ ) {
if( i->ok() )//成功的配置个数
nok++;
if( i->empty() )
nempty++;
}
if( nok == 0 ) {//没有配置是可用的
if( nempty == (int) configs.size() ) {
startupStatus = EMPTYCONFIG;
static unsigned once;
if( ++once == 1 ) {
log() << "replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done" << rsLog;
}
}
sleepsecs(10);
continue;
}
if( !_loadConfigFinish(configs) ) {
sleepsecs(20);
continue;
}
}
break;
}
startupStatus = STARTED;
}
继续看_loadConfigFinish,这个函数从可用配置中找出版本最高的一个配置,然后使用其做初始化.
bool ReplSetImpl::_loadConfigFinish(vector<ReplSetConfig>& cfgs) {
int v = -1;
ReplSetConfig *highest = 0;
int myVersion = -2000;
int n = 0;//选择一个版本最高的config,每当修改一次配置,如rs.add,rs.remove,version加一
for( vector<ReplSetConfig>::iterator i = cfgs.begin(); i != cfgs.end(); i++ ) {
ReplSetConfig& cfg = *i;
if( ++n == 1 ) myVersion = cfg.version;
if( cfg.ok() && cfg.version > v ) {
highest = &cfg;
v = cfg.version;
}
}
if( !initFromConfig(*highest) )//使用该config初始化replset
return false;
if( highest->version > myVersion && highest->version >= 0 ) {//保存该配置
highest->saveConfigLocally(BSONObj());//保存该config
}
return true;
}
_loadConfigFinish->initFromConfig,主要流程是对于每一个服务器建立一个MemberCfg的结构,并对其启动心跳协议.
bool ReplSetImpl::initFromConfig(ReplSetConfig& c, bool reconf/*=false*/) {
lock lk(this);
if( getLastErrorDefault || !c.getLastErrorDefaults.isEmpty() ) {
// see comment in dbcommands.cpp for getlasterrordefault
getLastErrorDefault = new BSONObj( c.getLastErrorDefaults );
}
list<ReplSetConfig::MemberCfg*> newOnes;
// additive short-cuts the new config setup. If we are just adding a
// node/nodes and nothing else is changing, this is additive. If it's
// not a reconfig, we're not adding anything
bool additive = reconf;
{
unsigned nfound = 0;
int me = 0;
for( vector<ReplSetConfig::MemberCfg>::iterator i = c.members.begin(); i != c.members.end(); i++ ) {
ReplSetConfig::MemberCfg& m = *i;
if( m.h.isSelf() ) {
me++;
}
if( reconf ) {//从新的配置
const Member *old = findById(m._id);
if( old ) {
nfound++;
if( old->config() != m ) {//同一台服务器配置配置更改了,如vote,priority更改
additive = false;
}
}
else {
newOnes.push_back(&m);
}
}
}//配置中没有本机地址，进入RS_SHUNNED状态,关闭所有连接,关闭心跳协议,重新进入加载配置状态
if( me == 0 ) { // we're not in the config -- we must have been removed
if (state().shunned()) {
// already took note of our ejection from the set
// so just sit tight and poll again
return false;
}
_members.orphanAll();
// kill off rsHealthPoll threads (because they Know Too Much about our past)
endOldHealthTasks();
// close sockets to force clients to re-evaluate this member
MessagingPort::closeAllSockets(0);
// take note of our ejection
changeState(MemberState::RS_SHUNNED);
loadConfig(); // redo config from scratch
return false;
}
// if we found different members that the original config, reload everything
if( reconf && config().members.size() != nfound )
additive = false;
}
_cfg = new ReplSetConfig(c);
_name = config()._id;
// this is a shortcut for simple changes
if( additive ) {//reconfig配置的路径
for( list<ReplSetConfig::MemberCfg*>::const_iterator i = newOnes.begin(); i != newOnes.end(); i++ ) {
ReplSetConfig::MemberCfg *m = *i;
Member *mi = new Member(m->h, m->_id, m, false);
/** we will indicate that new members are up() initially so that we don't relinquish our
primary state because we can't (transiently) see a majority. they should be up as we
check that new members are up before getting here on reconfig anyway.
*/
mi->get_hbinfo().health = 0.1;
_members.push(mi);//新添加的member,启动心跳协议
startHealthTaskFor(mi);
}
// if we aren't creating new members, we may have to update the
// groups for the current ones
_cfg->updateMembers(_members);//更新replset集中的member
return true;
}
// start with no members. if this is a reconfig, drop the old ones.
_members.orphanAll();//这里不只是初始化的配置,还可能是因为修改了某些member的配置来到这里
endOldHealthTasks();//所以结束所有心跳协议
int oldPrimaryId = -1;
{
const Member *p = box.getPrimary();
if( p )
oldPrimaryId = p->id();
}
forgetPrimary();//重置primary为空,后面primary将重新设置
// not setting _self to 0 as other threads use _self w/o locking
int me = 0;
string members = "";
for( vector<ReplSetConfig::MemberCfg>::const_iterator i = config().members.begin(); i != config().members.end(); i++ ) {
const ReplSetConfig::MemberCfg& m = *i;
Member *mi;
members += ( members == "" ? "" : ", " ) + m.h.toString();
if( m.h.isSelf() ) {//该member是自己,且自己在配置前是primary,则再次将自己设置为primary,初始化时primary并不在这里决定
mi = new Member(m.h, m._id, &m, true);
setSelfTo(mi);
if( (int)mi->id() == oldPrimaryId )
box.setSelfPrimary(mi);
}
else {
mi = new Member(m.h, m._id, &m, false);
_members.push(mi);
if( (int)mi->id() == oldPrimaryId )
box.setOtherPrimary(mi);
}
}
if( me == 0 ){
log() << "replSet warning did not detect own host in full reconfig, members " << members << " config: " << c << rsLog;
}
else {//启动心跳设置,每有一个member就需要一个线程与之通信,每2s启动一次连接
// Do this after we've found ourselves, since _self needs
// to be set before we can start the heartbeat tasks
for( Member *mb = _members.head(); mb; mb=mb->next() ) {
startHealthTaskFor( mb );
}
}
return true;
}
_loadConfigFinish->initFromConfig->startHealthTaskFor
void ReplSetImpl::startHealthTaskFor(Member *m) {
ReplSetHealthPollTask *task = new ReplSetHealthPollTask(m->h(), m->hbinfo());
healthTasks.insert(task);
task::repeat(task, 2000);//这里开启一个新的线程,并与m指定的服务器建立连接2000ms,执行一次replSetHeartbeat,查看远端服务器是否可达
}继续来看ReplSetHealthPollTask执行命令的函数ReplSetHealthPollTask::doWork
void doWork() {
HeartbeatInfo mem = m;
HeartbeatInfo old = mem;
try {
BSONObj info;
int theirConfigVersion = -10000;//心跳协议查看是否能够连接远端服务器
bool ok = _requestHeartbeat(mem, info, theirConfigVersion);
// weight new ping with old pings
// on the first ping, just use the ping value
if (old.ping != 0) {//设置ping一次的时间
mem.ping = (unsigned int)((old.ping * .8) + (mem.ping * .2));
}
if( ok ) {//远端服务器可达,则尝试将其加入到候选名单
up(info, mem);
}
else if (!info["errmsg"].eoo() &&//心跳协议显示该机有问题，从候选名单中删除
info["errmsg"].str() == "need to login") {//无法成为primary了
authIssue(mem);
}
else {//无法连接该机器
down(mem, info.getStringField("errmsg"));
}
}
catch(DBException& e) {
down(mem, e.what());
}
catch(...) {
down(mem, "replSet unexpected exception in ReplSetHealthPollTask");
}
m = mem;//更新该member的信息,包括状态如RS_STARTUP,RS_SECONDARY等
theReplSet->mgr->send( boost::bind(&ReplSet::msgUpdateHBInfo, theReplSet, mem) );
static time_t last = 0;
time_t now = time(0);
bool changed = mem.changed(old);
if( changed ) {
if( old.hbstate != mem.hbstate )
log() << "replSet member " << h.toString() << " is now in state " << mem.hbstate.toString() << rsLog;
}
if( changed || now-last>4 ) {//需要进行一次状态检查.
last = now;
theReplSet->mgr->send( boost::bind(&Manager::msgCheckNewState, theReplSet->mgr) );
}
}
_loadConfigFinish->initFromConfig->startHealthTaskFor->up
void up(const BSONObj& info, HeartbeatInfo& mem) {
HeartbeatInfo::numPings++;
mem.authIssue = false;
if( mem.upSince == 0 ) {
mem.upSince = mem.lastHeartbeat;
}
mem.health = 1.0;
mem.lastHeartbeatMsg = info["hbmsg"].String();
if( info.hasElement("opTime") )
mem.opTime = info["opTime"].Date();
// see if this member is in the electable set
if( info["e"].eoo() ) {
// for backwards compatibility
const Member *member = theReplSet->findById(mem.id());
if (member && member->config().potentiallyHot()) {//不是仲裁,且priority设置不为0,默认是1,为0则不可能成为primary
theReplSet->addToElectable(mem.id());
}
else {
theReplSet->rmFromElectable(mem.id());
}
}
// add this server to the electable set if it is within 10
// seconds of the latest optime we know of
else if( info["e"].trueValue() &&
mem.opTime >= theReplSet->lastOpTimeWritten.getSecs() - 10) {
unsigned lastOp = theReplSet->lastOtherOpTime().getSecs();
if (lastOp > 0 && mem.opTime >= lastOp - 10) {
theReplSet->addToElectable(mem.id());
}
}
else {
theReplSet->rmFromElectable(mem.id());
}
be cfg = info["config"];
if( cfg.ok() ) {//有新的config配置到来,更新配置
// received a new config
boost::function<void()> f =
boost::bind(&Manager::msgReceivedNewConfig, theReplSet->mgr, cfg.Obj().copy());
theReplSet->mgr->send(f);
}
}
void down(HeartbeatInfo& mem, string msg) {
mem.authIssue = false;//无法连接的服务器,将其标志为RS_DOWN,无法成为primary候选.
mem.health = 0.0;
mem.ping = 0;
if( mem.upSince || mem.downSince == 0 ) {
mem.upSince = 0;
mem.downSince = jsTime();
mem.hbstate = MemberState::RS_DOWN;
log() << "replSet info " << h.toString() << " is down (or slow to respond): " << msg << rsLog;
}
mem.lastHeartbeatMsg = msg;
theReplSet->rmFromElectable(mem.id());
}
回到initFromConfig,该函数执行完毕,继续回到ReplSetImpl,该对象构造完毕.回到startReplSets继续
执行
(theReplSet = new ReplSet(*replSetCmdline))->go();其执行的是ReplSetImpl::_go函数,继续来看这里的_go函数.
void ReplSetImpl::_go() {
loadLastOpTimeWritten();//得到最近一次写local.oplog.rs的时间,初始化时在saveConfigLocally时第一次写
changeState(MemberState::RS_STARTUP2);
startThreads();//开启同步线程,读取操作日志的线程.
newReplUp(); // oplog.cpp设置新的log函数
}
void ReplSetImpl::startThreads() {
task::fork(mgr);//这里启动管理服务,可以通过如下mgr->send让其执行send指定的函数,其内部是一个做服务的线程,接收执行任务,然后执行
mgr->send( boost::bind(&Manager::msgCheckNewState, theReplSet->mgr) );
if (myConfig().arbiterOnly) {//该服务器只执行仲裁动作
return;
}
boost::thread t(startSyncThread);//这个线程除了sync外还有一个功能将当前服务器设置为secondary,初始化时到这里其状态为RS_STARTUP2
replset::BackgroundSync* sync = replset::BackgroundSync::get();
boost::thread producer(boost::bind(&replset::BackgroundSync::producerThread, sync));//为syncThread获取同步数据
boost::thread notifier(boost::bind(&replset::BackgroundSync::notifierThread, sync));//为tags准备的,后面会有一篇文章专门讲到replset tags
task::fork(ghost);
// member heartbeats are started in ReplSetImpl::initFromConfig
}
本文就分析到这里,几个线程的作用以及状态的切换留待下文.总结:
本文分析replication replset模式初始化流程,初始化过程中主要是根据配置信息在各个服务器间建
立心跳协议,保证连接可达,根据连接信息更新各个服务器的状态,为下一步选取primary做准备.
本文链接:mongodb源码分析(十五)replication replset模式的初始化
作者:yhjj0108,杨浩