使用python，批量导入数据到elasticsearch中

使用python，批量导入数据到elasticsearch中

使用python官方模块，批量将文件中的数据导入到elasticsearch中。

1、文件内容

文件以{url:content}组成且以“\t”隔开，例子如下：

http://www.avsforum.com/forum/39-networking-media-servers-content-streaming/1624586-network-switch-types-there-differences.html	Join Date: Feb 2006 Mentioned: 1 Post(s) Tagged: 0 Thread(s) Quoted: 63 Post(s) Network switch types. Are there differences? Most people think that a network switch (the device you plug your network cables into) is a very simple device that helps split and share your network signals among multiple devices. At its most basic level that’s true. Switches though do much more. They assign what traffic goes to which device, they determine the best speeds of those devices and in more advanced switches can provide a host of features including adding QOS (Quality of Service), virtualization of your networks to isolate certain devices from one another (VLAN)
http://www.netgear.com/business/prod	us-switch.aspx Bob Silver Netgear AV Consultant Last edited by bobsilver; 07-31-2014 at 02:09 PM . bobsilver is offline   (LCD - Sony KDL -52 XBR4) (Receiver - Sony STR-DA4ES)(Blu Ray - Oppo BDP-83) (PS3)( Dish Hopper DVR With Sling) Speakers (L & R - Paradigm Studio 20) (Center -Paradigm CC-470) (Surrounds & Back Surrounds - Paradigm SA-15R in walls) (Subwoofer 1 - Sunfire HRS-12) (Subwoofer 2 - Paradigm PW-2100) Skytrooper is offline   Join Date: Feb 2006 Mentioned: 1 Post(s) Tagged: 0 Thread(s) Quoted: 63 Post(s) Originally Posted by Skytrooper Nice read. (Or ad) I currently have a Netgear unmanaged switch. If I would upgrade to the plus, would someone be able to just plug into one of my Ethernet jacks and be good to go? And could you elaborate in lay mans terms on the advantages of the plus switch? I don't want to complicate things. And why can't I just plug my 24 port switch directly into my cable modem without first going thru my 4 port Netgear Router in which only one output is used? Good questions. The Unmanaged switch will work the same as a Plus or managed switch as it relates to plugging in another device. No benefit getting another type of switch. The benefits of Plus switches are enhanced abilities to manage things like quality of service, security and general management. Do want to get to far in the weeds here. Your fine with an Unmanaged switch. Why you cant plug a switch into your modem instead of your router is this. Routers (be they wireless or wired) route the the internet connection translating its ip address into local addresses that

2、python代码

# coding=utf-8
from elasticsearch import Elasticsearch
from elasticsearch.helpers import bulk
import time
import argparse
import sys

import read_file

reload(sys)
sys.setdefaultencoding('utf-8')

#设置mapping
def set_mapping(es, index_name="content_engine", doc_type_name="en"):
    my_mapping = {
            "en": {
                "properties": {
                    "content": {
                        "type": "string"
                    },
                    "url": {
                        "type": "string"
                    }
                }
            }
    }

    #创建Index和mapping
    create_index = es.indices.create(index=index_name,body=my_mapping)		#{u'acknowledged': True}
    mapping_index = es.indices.put_mapping(index=index_name, doc_type=doc_type_name, body=my_mapping)		#{u'acknowledged': True}
    if create_index["acknowledged"]!=True or mapping_index["acknowledged"]!=True:
        print "Index creation failed..."

#将文件中的数据存储到es中
def set_date(es, input_file, index_name="content_engine", doc_type_name="en"):
    #读入数据
    line_list= read_file.read_file_by_line(input_file)

    #创建ACTIONS
    ACTIONS = []
    for line in line_list:
        fields = line.split("\t")
        # print fields[1]
        action = {
            "_index": index_name,
            "_type": doc_type_name,
            "_source": {
                  "url":fields[0],
                  "content":fields[1]}
        }
        ACTIONS.append(action)

    # 批量处理
    success, _ = bulk(es, ACTIONS, index=index_name, raise_on_error=True)
    print('Performed %d actions' % success)

#读取参数
def read_args():
    parser = argparse.ArgumentParser(description="Search Elastic Engine")
    parser.add_argument("-i", dest="input_file", action="store", help="input file1", required=True)
    #parser.add_argument("-o", dest="output_file", action="store", help="output file", required=True)
    return parser.parse_args()

if __name__ == '__main__':
	args = read_args()
	es = Elasticsearch(hosts=["192.168.1.20:9200"], timeout=5000)
	set_mapping(es)
	set_date(es,args.input_file)

3、执行结果

在Marvel中，使用命令GET  content_engine/_search搜索结果，结果如下：

{
   "took": 1,
   "timed_out": false,
   "_shards": {
      "total": 5,
      "successful": 5,
      "failed": 0
   },
   "hits": {
      "total": 2,
      "max_score": 1,
      "hits": [
         {
            "_index": "content_engine",
            "_type": "en",
            "_id": "AVCtwMsjvMY71Ei4hFMV",
            "_score": 1,
            "_source": {
               "url": "http://www.avsforum.com/forum/39-networking-media-servers-content-streaming/1624586-network-switch-types-there-differences.html",
               "content": "Join Date: Feb 2006 Mentioned: 1 Post(s) Tagged: 0 Thread(s) Quoted: 63 Post(s) Network switch types. Are there differences? Most people think that a network switch (the device you plug your network cables into) is a very simple device that helps split and share your network signals among multiple devices. At its most basic level that’s true. Switches though do much more. They assign what traffic goes to which device, they determine the best speeds of those devices and in more advanced switches can provide a host of features including adding QOS (Quality of Service), virtualization of your networks to isolate certain devices from one another (VLAN)"
            }
         },
         {
            "_index": "content_engine",
            "_type": "en",
            "_id": "AVCtwMsjvMY71Ei4hFMW",
            "_score": 1,
            "_source": {
               "url": "http://www.netgear.com/business/prod",
               "content": "us-switch.aspx Bob Silver Netgear AV Consultant Last edited by bobsilver; 07-31-2014 at 02:09 PM . bobsilver is offline   (LCD - Sony KDL -52 XBR4) (Receiver - Sony STR-DA4ES)(Blu Ray - Oppo BDP-83) (PS3)( Dish Hopper DVR With Sling) Speakers (L & R - Paradigm Studio 20) (Center -Paradigm CC-470) (Surrounds & Back Surrounds - Paradigm SA-15R in walls) (Subwoofer 1 - Sunfire HRS-12) (Subwoofer 2 - Paradigm PW-2100) Skytrooper is offline   Join Date: Feb 2006 Mentioned: 1 Post(s) Tagged: 0 Thread(s) Quoted: 63 Post(s) Originally Posted by Skytrooper Nice read. (Or ad) I currently have a Netgear unmanaged switch. If I would upgrade to the plus, would someone be able to just plug into one of my Ethernet jacks and be good to go? And could you elaborate in lay mans terms on the advantages of the plus switch? I don't want to complicate things. And why can't I just plug my 24 port switch directly into my cable modem without first going thru my 4 port Netgear Router in which only one output is used? Good questions. The Unmanaged switch will work the same as a Plus or managed switch as it relates to plugging in another device. No benefit getting another type of switch. The benefits of Plus switches are enhanced abilities to manage things like quality of service, security and general management. Do want to get to far in the weeds here. Your fine with an Unmanaged switch. Why you cant plug a switch into your modem instead of your router is this. Routers (be they wireless or wired) route the the internet connection translating its ip address into local addresses that"
            }
         }
      ]
   }
}

至此，数据导入完成！