五种方式将数据从MongoDB同步到ElasticSearch

Elastic search(ES) is a pop-star for recording and analyzing data, andMongodb is a famous NoSQL database for storing and querying data. With our web infrastructure improving, how can we export data from mongodb to ES for searching or analyzing purposes? There are 5 possible solutions recommended for your choice.

1.synchronized by web server

We can use Mongoosastic module for storing-in-both-sides purpose when we use Nodejs as a web server container. When one document needs to be stored,  Mongoosastic can commit the changes to both mongo and ES. As the chart below:

Here is the reference link: Mongoosastic.

The advantage is that data can be stored in both mongo and ES simultaneously, and the downside is that overhead may be caused in CUD operation efficiency. And inconsistent data might be generated when one type of the db store failed. And the server framework is not flexible enough for db migrating.

2.Manually loading data from Mongo to ES

Transporter tool is a good choice to synchronize data once you want to export mongo data to another ES server. Transporter also can export data from or to other type of data store. Reference link is: Transporter.

It's important to know that the transporter synchronizing only once. When the job is done, the transporter comes to its end.

3. Plugin for ES

There is a plugin for ES named "elasticsearch-river-mongodb", and was widely used in ES 1.x, but now river mechanism for ES 2.x is deprecated. Reference link is elasticsearch-river-mongodb.

4. JDBC input plugin for logstash

We can take advantage of  buffering , inputting, outputting and filtering abilities from logstash by adding a mongo input and ES output plugin to get this job done.

JDBC input plugin is one of the choices, but it needs JDBC driver support. As I know there is no well-supported-free JDBC driver for mongo. Some trial versions can be found in Unity or Simba.

Reference link is : JDBC Plugin for Logstash

5. Mongo-ES connector

mongo-connector is a real-time sync service as a package ofPython. It creates a pipeline from a mongodb cluster to one or more target systems. It needs mongo to run in replica-set mode, sync data in mongo to the target then tails the mongo oplog. It needs a package named "elastic2_doc_manager" to write data to ES. Process chart below:

Reference link is : github or python.

To recapitulate it,  it is a must to remember: mongo replica set, an opened port and IP for ES, using elastic2_doc_manager if you use ES 2.x.

At present, I am not yet ready with any official support in Beats. it will be in the future.

So that's all, 5 ways to mongo-ES-sync.

注：本文转自https://www.linkedin.com/pulse/5-way-sync-data-from-mongodb-es-kai-hao