验证表明:mongo-connector工具支持MongoDB与ES之间的实时增insert、删delete、改update操作。
对于历史数据,mongo-connector工具不能同步到ES中,根因是本身工具不支持(初步界定),还是没有这种场景,待查(进一步研究后再更新)。
https://github.com/mongodb-labs/mongo-connector
mongo-connector工具创建一个从MongoDB簇到一个或多个目标系统的管道,目标系统包括:Solr,Elasticsearch,或MongoDB簇。
该工具在MongoDB与目标系统间同步数据,并跟踪MongoDB的oplog,保持操作与MongoDB的实时同步。
该工具已经在python2.6,2.7,3.3+下进行验证。
mongo-connector工具是基于python开发的实时同步服务工具。它要求mongo运行在replica-set模式,且需要 elastic2_doc_manager将数据写入ES。
这是Elastic2.x版本的文档管理器。对应Elastic1.x版本需要使用 elastic-doc-manager。
pip install mongo-connector
pip install elastic2-doc-manager
注意:
如果不安装(2)直接进入(3)、(4)则会报错:
[root@5b9dbaaa148a bin]# mongo-connector -m 10.8.5.99:27017 -t 10.8.5.101:9200 -d elastic2_doc_manager
Logging to mongo-connector.log.
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner
self.run()
MongoDB 必须开启复制集,如果已经开启请忽略这一步:
[root@b48eafd69929 bin]# ./mongod --replSet "rs0"
[root@b48eafd69929 bin]# ./mongo
MongoDB shell version: 3.2.4
connecting to: test
Server has startup warnings:
2016-07-05T09:49:01.330+0100 I CONTROL [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2016-07-05T09:49:01.330+0100 I CONTROL [initandlisten]
2016-07-05T09:49:01.331+0100 I CONTROL [initandlisten]
2016-07-05T09:49:01.331+0100 I CONTROL [initandlisten] ** WARNING: You are running on a NUMA machine.
2016-07-05T09:49:01.331+0100 I CONTROL [initandlisten] ** We suggest launching mongod like this to avoid performance problems:
2016-07-05T09:49:01.332+0100 I CONTROL [initandlisten] ** numactl --interleave=all mongod [other options]
2016-07-05T09:49:01.332+0100 I CONTROL [initandlisten]
2016-07-05T09:49:01.332+0100 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2016-07-05T09:49:01.332+0100 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2016-07-05T09:49:01.332+0100 I CONTROL [initandlisten]
2016-07-05T09:49:01.332+0100 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2016-07-05T09:49:01.332+0100 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2016-07-05T09:49:01.332+0100 I CONTROL [initandlisten]
> rs.initiate()
{
"info2" : "no configuration specified. Using a default configuration for the set",
"me" : "b48eafd69929:27017",
"ok" : 1
}
rs0:SECONDARY> rs.conf()
{
"_id" : "rs0",
"version" : 1,
"protocolVersion" : NumberLong(1),
"members" : [
{
"_id" : 0,
"host" : "b48eafd69929:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {
},
"slaveDelay" : NumberLong(0),
"votes" : 1
}
],
"settings" : {
"chainingAllowed" : true,
"heartbeatIntervalMillis" : 2000,
"heartbeatTimeoutSecs" : 10,
"electionTimeoutMillis" : 10000,
"getLastErrorModes" : {
},
"getLastErrorDefaults" : {
"w" : 1,
"wtimeout" : 0
},
"replicaSetId" : ObjectId("577b74bd0ba41a313110ad62")
}
}
rs0:PRIMARY> rs.status()
{
"set" : "rs0",
"date" : ISODate("2016-07-05T08:50:55.272Z"),
"myState" : 1,
"term" : NumberLong(1),
"heartbeatIntervalMillis" : NumberLong(2000),
"members" : [
{
"_id" : 0,
"name" : "b48eafd69929:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 115,
"optime" : {
"ts" : Timestamp(1467708606, 1),
"t" : NumberLong(1)
},
"optimeDate" : ISODate("2016-07-05T08:50:06Z"),
"infoMessage" : "could not find member to sync from",
"electionTime" : Timestamp(1467708605, 2),
"electionDate" : ISODate("2016-07-05T08:50:05Z"),
"configVersion" : 1,
"self" : true
}
],
"ok" : 1
}
[root@5b9dbaaa148a bin]# mongo-connector -m 10.8.5.99:27017 -t 10.8.5.101:9200 -d elastic2_doc_manager
Logging to mongo-connector.log.
参数含义:
-m: mongodb的地址与端口,端口默认为27017。
-t:ES的地址与端口,端口默认为9200。
-d:doc manager的名称,2.x版本为: elastic2-doc-manager。
#Mongo创建数据库(对应ES的Index)
rs0:PRIMARY> use zhang_index
switched to db zhang_index
#Mongo中插入数据(其中col_02对应ES中的Type)
rs0:PRIMARY> db.col_02.insert({name:"laoluo", birth:"1964-03-21", sex:"man", company:"chuizi"});
WriteResult({ "nInserted" : 1 })
rs0:PRIMARY> db.col_02.insert({name:"renzhengfei", birth:"1954-03-21", sex:"man", company:"huawei"});
[root@5b9dbaaa148a test_log]# curl -XGET http://10.8.5.101:9200/zhang_index/col_02/_search?pretty
{
"took" : 4,
"timed_out" : false,
"_shards" : {
"total" : 8,
"successful" : 8,
"failed" : 0
},
"hits" : {
"total" : 2,
"max_score" : 1.0,
"hits" : [ {
"_index" : "zhang_index",
"_type" : "col_02",
"_id" : "577b7d8ceb8e3dc2d1db12a9",
"_score" : 1.0,
"_source" : {
"company" : "huawei",
"name" : "renzhengfei",
"birth" : "1954-03-21",
"sex" : "man"
}
}, {
"_index" : "zhang_index",
"_type" : "col_02",
"_id" : "577b7d4aeb8e3dc2d1db12a7",
"_score" : 1.0,
"_source" : {
"company" : "chuizi",
"name" : "laoluo",
"birth" : "1964-03-21",
"sex" : "man"
}
} ]
}
}
rs0:PRIMARY> db.col_02.update({'name':'laoluo'}, {$set:{'name':'luoyonghao'}})
WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })
rs0:PRIMARY>
rs0:PRIMARY> db.col_02.find().pretty()
{
"_id" : ObjectId("577b7d4aeb8e3dc2d1db12a7"),
"name" : "luoyonghao",
"birth" : "1964-03-21",
"sex" : "man",
"company" : "chuizi"
}
{
"_id" : ObjectId("577b7d8ceb8e3dc2d1db12a9"),
"name" : "renzhengfei",
"birth" : "1954-03-21",
"sex" : "man",
"company" : "huawei"
}
[root@5b9dbaaa148a test_log]# curl -XGET http://10.8.5.101:9200/zhang_index/col_02/_search?pretty
{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 8,
"successful" : 8,
"failed" : 0
},
"hits" : {
"total" : 2,
"max_score" : 1.0,
"hits" : [ {
"_index" : "zhang_index",
"_type" : "col_02",
"_id" : "577b7d8ceb8e3dc2d1db12a9",
"_score" : 1.0,
"_source" : {
"company" : "huawei",
"name" : "renzhengfei",
"birth" : "1954-03-21",
"sex" : "man"
}
}, {
"_index" : "zhang_index",
"_type" : "col_02",
"_id" : "577b7d4aeb8e3dc2d1db12a7",
"_score" : 1.0,
"_source" : {
"company" : "chuizi",
"name" : "luoyonghao",
"birth" : "1964-03-21",
"sex" : "man"
}
} ]
}
}
rs0:PRIMARY> db.col_02.remove({'name':'renzhengfei'})
WriteResult({ "nRemoved" : 1 })
rs0:PRIMARY> db.col_02.find()
{ "_id" : ObjectId("577b7d4aeb8e3dc2d1db12a7"), "name" : "luoyonghao", "birth" : "1964-03-21", "sex" : "man", "company" : "chuizi" }
rs0:PRIMARY> db.col_02.find().pretty()
{
"_id" : ObjectId("577b7d4aeb8e3dc2d1db12a7"),
"name" : "luoyonghao",
"birth" : "1964-03-21",
"sex" : "man",
"company" : "chuizi"
}
结果表明,MongoDB删除的内容,ES端已经同步删除。
[root@5b9dbaaa148a test_log]# curl -XGET http://10.8.5.101:9200/zhang_index/col_02/_search?pretty
{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 8,
"successful" : 8,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 1.0,
"hits" : [ {
"_index" : "zhang_index",
"_type" : "col_02",
"_id" : "577b7d4aeb8e3dc2d1db12a7",
"_score" : 1.0,
"_source" : {
"company" : "chuizi",
"name" : "luoyonghao",
"birth" : "1964-03-21",
"sex" : "man"
}
} ]
}
}
https://docs.mongodb.com/manual/tutorial/deploy-replica-set/
https://www.linkedin.com/pulse/5-way-sync-data-from-mongodb-es-kai-hao
How to setup a MongoDB replica set for the connector?
https://docs.mongodb.com/manual/tutorial/deploy-replica-set/
——————————————————————————————————
更多ES相关实战干货经验分享,请扫描下方【铭毅天下】微信公众号二维码关注。
(每周至少更新一篇!)
和你一起,死磕Elasticsearch!
——————————————————————————————————
2016年7月6日 23:22 思于家中床前
作者:铭毅天下
转载请标明出处,原文地址:
http://blog.csdn.net/laoyang360/article/details/51842822
如果感觉本文对您有帮助,请点击‘顶’支持一下,您的支持是我坚持写作最大的动力,谢谢!