python+elasticsearch集群基本使用

python+elasticsearch集群

官方文档：https://elasticsearch-py.readthedocs.io/en/master/
介绍
python提供了操作ElasticSearch 接口，因此要用python来操作ElasticSearch，首先要安装python的ElasticSearch包，用命令 pip install elasticsearch安装或下载安装：https://pypi.python.org/pypi/elasticsearch/6.3.1
基本使用(本人是在django项目中使用)

1.配置setteing

ES_SETTINGS = {
        "hosts": [{"host": '10.35.206.28', "port": 9200}, {"host": '10.35.206.27', "port": 9200},
                  {"host": '10.35.206.30', "port": 9200}],
        "index": record',
        "type": 'prometheus'
    }

hosts：配置多为多个实现集群

2.Elasticsearch客户端创建

from django.conf import settings
from elasticsearch import Elasticsearch

es = Elasticsearch(hosts=settings.ES_SETTINGS['hosts'])

3.使用helpers插入es

from elasticsearch import helpers
data = {
            "_index": "record",
            "_type": "prometheus",
            "_id": str(instance.id) + "prometheus",
            "_score": "",
            "_source": {
                "recordId": instance.id,
                "ipPort": ip + ":9100",
                "proId": proId,
                "proName": proName,
                "msg": json.dumps(res.json()),
                "recordTrans": instance.tran_info,
                "time": time.time(),
            }
        }
a = helpers.bulk(es, actions=[data])

4.使用search查询(最多一次只能查询10000)

def prometheus_search_new(recordId, ipPort, proName):
    query_body = {
        "query": {
            "bool": {
                "must": [
                    {
                        "term": {
                            "recordId": recordId
                        }
                    },
                    {
                        "term": {
                            "ipPort.keyword": ipPort
                        }
                    },
                    {
                        "term": {
                            "proName.keyword": proName
                        }
                    },
                ]
            },
        },
        "sort": {"time": {"order": "desc"}},
        "from": 0,
        "size": 10000
    }
    try:
        result = es.search(index=settings.ES_PROMETHEUS_SETTINGS['index'],
                           doc_type=settings.ES_PROMETHEUS_SETTINGS['type'],
                           body=query_body)
        hits = result['hits']['hits']
    except Exception as e:
        hits = []
    return hits

5.使用helpers查询（可以实现分页下次写）

es_result = helpers.scan(
        client=es,
        query=query_body,
        scroll='5m',
        index=settings.ES_SETTINGS ['index'],
        doc_type=settings.ES_SETTINGS ['type'],
        timeout='1m',
        preserve_order=True
    )

6.返回结果

{
  "took": 13,
  "timed_out": false,
  "_shards": {
  	"total": 65,
  	"successful": 65,
  	"skipped": 0,
  	"failed": 0
  },
  "hits": {
  	"total": 152883,
  	"max_score": null,
  	"hits": [
  	{
  		"_index": "new_stat",
  		"_type": "stat",
  		"_id": "q1jWAG0BhYz49-yfOJms",
  		"_score": null,
  		"_source": {
  			"_index": "record",
  			"_type": "prometheus",
  			"_id": "7131prometheus",
  			"_version": 1,
  			"_score": 1,
  			"_source": {
  			"ipPort": "10.13.31.212:9100",
  			"recordTrans": "[{'enable': True, 'url': u'http://10.13.31.212:8008/v1/music/process', 'tran_name': u'\u53ee\u5f53\u5f71\u89c6', 'virtualNum': 100, 'method': u'POST', 'tran_id': 22355}]",
  			"proName": "cup_sys",
  			"recordId": 7131,
  			"time": 1551510988.541931,
  			"proId": 11,
  			"msg": "{"status": "success", "data": {"resultType": "matrix", "result": []}}"
  			}
  		}
  		]
  }
}

另外

elasticsearch通过from~size查询分页时，会受到max_result_window影响，可根据实际适用，设置：

curl -XPUT "http://ip:port/index/_settings"  -d '{"index": {"max_result_window": 1000000}}'