Go操作Elasticsearch总结

Go操作Elasticsearch总结

安装

参见:CentOS 7 RPM安装 Elasticsearch 7.14.1和常用插件

官方入门例子

《Elasticsearch:权威指南》“适应新环境” 通过一个例子,介绍了ES的核心概念。

我们受雇于 Megacorp 公司,作为 HR 部门新的 “热爱无人机” (“We love our drones!”)激励项目的一部分,我们的任务是为此创建一个员工目录。该目录应当能培养员工认同感及支持实时、高效、动态协作,因此有一些业务需求:

  • 支持包含多值标签、数值、以及全文本的数据
  • 检索任一员工的完整信息
  • 允许结构化搜索,比如查询 30 岁以上的员工
  • 允许简单的全文搜索以及较复杂的短语搜索
  • 支持在匹配文档内容中高亮显示搜索片段
  • 支持基于数据创建和管理分析仪表盘

ES-Head使用

Structured Query:
Go操作Elasticsearch总结_第1张图片
Any Request页面可以自己写各种查询语句(看官方API examples时直接贴过来调试):
Go操作Elasticsearch总结_第2张图片

Kibana使用

最常用的是:Management / 开发工具
Go操作Elasticsearch总结_第3张图片

Elasticsearch API总结

插入数据(自动创建索引)

  • website:index,索引名称
  • blog:documentType,索引类型,目前统一设置为_doc,后续ES高版本好像放弃了。
  • 123:id,索引ID。一般不指定,通过ES自动生成,插入速度更快。
POST /website/blog/123
{
  "title": "My first blog entry",
  "text":  "Just trying this out...",
  "date":  "2014/01/01"
}

创建映射

如果不显示创建索引,则一些字段聚合查询的时候会报错,因为没有指定字段类型。聚合查询操作只支持keyword。

增删改查映射可以参考:

  • 《ES权威指南2.x》
  • 7.x最新的API

创建一个显式映射的例子:

PUT /my-index-000001
{
  "mappings": {
    "properties": {
      "age":    { "type": "integer" },  
      "email":  { "type": "keyword"  }, 
      "name":   { "type": "text"  }     
    }
  }
}

支持的数据类型可以参考:Field data types

  • Keywords: The keyword family, including keyword, constant_keyword, and wildcard.
  • Numbers: Numeric types, such as long and double, used to express amounts.
  • Dates: Date types, including date and date_nanos.
  • text: Analyzed, unstructured text.
  • alias: Defines an alias for an existing field.

此时,就可以对email聚合统计(为什么?因为text会被分词,keywords不会分词?),或者去重了。

创建索引

当然,可以直接在创建索引的时候指定。那为何还要单独创建映射?因为es不是列式存储的,不像数据库需要提前创建表,指定有多少个字段(列),es是可以动态增加字段(列)的。

具体参考:https://www.elastic.co/guide/cn/elasticsearch/guide/current/_creating_an_index.html

PUT /my_index
{
    "settings": { ... any settings ... },
    "mappings": {
        "type_one": { ... any mappings ... },
        "type_two": { ... any mappings ... },
        ...
    }
}

删除索引

具体参考:https://www.elastic.co/guide/cn/elasticsearch/guide/current/_deleting_an_index.html

DELETE /my_index

查询API

具体参见:https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html

GET /my-index-000001/_search?from=40&size=20
{   
	"query": {    
  		"term": {       
          	"user.id": "kimchy"     
        }  
	} 
}
{
  "took": 5,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 20,
      "relation": "eq"
    },
    "max_score": 1.3862942,
    "hits": [
      {
        "_index": "my-index-000001",
        "_type" : "_doc",
        "_id": "0",
        "_score": 1.3862942,
        "_source": {
          "@timestamp": "2099-11-15T14:12:12",
          "http": {
            "request": {
              "method": "get"
            },
            "response": {
              "status_code": 200,
              "bytes": 1070000
            },
            "version": "1.1"
          },
          "source": {
            "ip": "127.0.0.1"
          },
          "message": "GET /search HTTP/1.1 200 1070000",
          "user": {
            "id": "kimchy"
          }
        }
      },
      ...
    ]
  }
}

聚合查询API

具体API参考:权威指南2.x:聚合、Aggregations

GET /my-index-000001/_search
{
  "aggs": {
    "my-agg-name": {
      "terms": {
        "field": "my-field"
      }
    }
  }
}

GO操作Elasticsearch

先在pkg.go.dev搜索一下:
Go操作Elasticsearch总结_第4张图片

选第一个,进去后点击Expand展开,就有更详细的入门例子。
PS:v7是针对elasticsearch 7.x版本,使用前要确认自己的es版本。7.x和6.x有些许出入,如果用了7.x开发,降级到6.x难度也不算大。

Import
	"github.com/elastic/go-elasticsearch/v6"
	"github.com/elastic/go-elasticsearch/v6/esapi"
	"github.com/elastic/go-elasticsearch/v6/esutil"
Example
cfg := elasticsearch.Config{
  Addresses: []string{
    "http://localhost:9200",
    "http://localhost:9201",
  },
  Username: "foo",
  Password: "bar",
  Transport: &http.Transport{
    MaxIdleConnsPerHost:   10,
    ResponseHeaderTimeout: time.Second,
    DialContext:           (&net.Dialer{Timeout: time.Second}).DialContext,
    TLSClientConfig: &tls.Config{
      MinVersion:         tls.VersionTLS11,
    },
  },
}

es := elasticsearch.NewClient(cfg)

// 测试
_, err = client.Ping()
if err != nil {
  return err
} else {
  logger.Info("success connect to elasticsearch:", conf.Addresses)
}

// 打印es版本信息
res, err := es.Info()
if err != nil {
  log.Fatalf("Error getting response: %s", err)
}
log.Println(res)

查询操作

方式一:使用go函数
// go 构造json便捷写法
type H map[string]interface{}
queryMap := H{
  "query": H{
    "term": H{
      "user.id": "kimchy",
    },
  },
}

data, err := json.Marshal(queryMap)
if err != nil {
  return
}

// 执行搜索操作
res, err := es.client.Search(
		es.client.Search.WithContext(context.Background()),
		es.client.Search.WithIndex(es.toIndex),
		es.client.Search.WithDocumentType(es.toType),
		es.client.Search.WithBody(strings.NewReader(query)),
		es.client.Search.WithPretty(),
)

defer res.Body.Close()
if err != nil {
	return 
}

// 需要再次判断
if res.IsError() {
		var e map[string]interface{}
		if err := json.NewDecoder(res.Body).Decode(&e); err != nil {
			logger.Sugar.Fatalf("Error parsing the response body: %s", err)
			return nil, err

		} else {
			// Print the response status and error information.
			logger.Sugar.Errorf("[%s] %s: %s",
				res.Status(),
				e["error"].(map[string]interface{})["type"],
				e["error"].(map[string]interface{})["reason"],
			)
		}
}

// 读取所有
b, err := ioutil.ReadAll(res.Body)
if err != nil {
  return nil, err
}
方式二:直接使用http请求

这里以创建映射举例,search同理:

// CreateMapping 创建映射
func (e *Elastic) CreateMapping(index, jsonData string) error {
  	url := fmt.Sprintf("http://10.0.56.153:9200/steel-index")
	req, _ := http.NewRequest("PUT", url, strings.NewReader(jsonData))
	req.Header.Add("Content-Type", "application/json")
	res, err := e.httpClient.Do(req)
	if err != nil {
		return err
	}

	defer res.Body.Close()
	body, _ := ioutil.ReadAll(res.Body)
	logger.Info("success create mapping index: ", index, ",res:", string(body))
	return nil
}

插入

func (e *Elastic) Post(jsonData string) error {
	req := esapi.IndexRequest{
		Index:        e.toIndex,
		DocumentType: e.toType,
		Body:         strings.NewReader(jsonData),
		Timeout:      time.Second * 10,
	}
	res, err := req.Do(context.Background(), e.client)
	if err != nil {
		return err
	}

	defer res.Body.Close()

	if !res.IsError() {
		// Deserialize the response into a map.
		var r map[string]interface{}
		if err := json.NewDecoder(res.Body).Decode(&r); err != nil {
			logger.Sugar.Warnf("Error parsing the response body: %s", err)
		} else {
			// Print the response status and indexed document version.
			logger.Sugar.Infof("[%s] %s; version=%d", res.Status(), r["result"], int(r["_version"].(float64)))
		}
	} else {
		logger.Sugar.Warn("Error post elastic,statsCode:", res.StatusCode, ",string:", res.String())
	}

	return nil
}

批量插入(bulk)

具体可以参考官方的这篇博客:The Go client for Elasticsearch: Working with data

// AddBulk 批量添加进ES
// see more: https://www.elastic.co/cn/blog/the-go-client-for-elasticsearch-working-with-data
func (e *Elastic) AddBulk(index, docType string, numWorkers, flushBytes int, jsonData []string) error {
	bulkIndexer, err := esutil.NewBulkIndexer(esutil.BulkIndexerConfig{
		Index:         index, // The default index name
		DocumentType:  docType,
		Client:        e.client,         // The Elasticsearch client
		NumWorkers:    numWorkers,       // The number of worker goroutines
		FlushBytes:    flushBytes,       // The flush threshold in bytes
		FlushInterval: 30 * time.Second, // The periodic flush interval
	})
	if err != nil {
		return err
	}

	var countSuccessful uint64 = 0

	// 组合,具体参考ES Bulk API:https://www.elastic.co/guide/en/elasticsearch/reference/master/docs-bulk.html#bulk-api-request-body
	for _, str := range jsonData {
		err := bulkIndexer.Add(context.Background(), esutil.BulkIndexerItem{
			// Action field configures the operation to perform (index, create, delete, update)
			Action: "index",
			// DocumentID is the (optional) document ID
			// DocumentID: strconv.Itoa(a.ID),
			// Body is an `io.Reader` with the payload
			Body: strings.NewReader(str),
			// OnSuccess is called for each successful operation
			OnSuccess: func(ctx context.Context, item esutil.BulkIndexerItem, res esutil.BulkIndexerResponseItem) {
				atomic.AddUint64(&countSuccessful, 1)
			},
			// OnFailure is called for each failed operation
			OnFailure: func(ctx context.Context, item esutil.BulkIndexerItem, res esutil.BulkIndexerResponseItem, err error) {
				if err != nil {
					logger.Infof("ERROR: %s", err)
				} else {
					logger.Infof("ERROR: %s: %s", res.Error.Type, res.Error.Reason)
				}
			},
		})
		if err != nil {
			return err
		}
	}

	start := time.Now()
	// Close waits until all added items are flushed and closes the indexer.
	if err := bulkIndexer.Close(context.Background()); err != nil {
		return err
	} else {
		// 统计时间
		biStats := bulkIndexer.Stats()
		dur := time.Since(start)

		if biStats.NumFailed > 0 {
			logger.Warnf(
				"Indexed [%s] documents with [%s] errors in %s (%s docs/sec)",
				Comma(int64(biStats.NumFlushed)),
				Comma(int64(biStats.NumFailed)),
				dur.Truncate(time.Millisecond),
				Comma(int64(1000.0/float64(dur/time.Millisecond)*float64(biStats.NumFlushed))),
			)
		} else {
			logger.Infof(
				"Successfully indexed [%s] documents in %s (%s docs/sec)",
				Comma(int64(biStats.NumFlushed)),
				dur.Truncate(time.Millisecond),
				Comma(int64(1000.0/float64(dur/time.Millisecond)*float64(biStats.NumFlushed))),
			)
		}
	}

	return nil
}

删除

// 删除索引
func (e *Elastic) DeleteIndex(index string) error {
	url := fmt.Sprintf("http://%s/%s", config.DefaultConfig.ES.Addr, index)
	req, _ := http.NewRequest("DELETE", url, nil)
	res, err := e.httpClient.Do(req)
	if err != nil {
		return err
	}

	defer res.Body.Close()
	body, _ := ioutil.ReadAll(res.Body)
	logger.Info("success delete index: ", index, ",res:", string(body))
	return nil
}

关于作者

推荐下自己的开源IM,纯Golang编写:

CoffeeChat:https://github.com/xmcy0011/CoffeeChat
opensource im with server(go) and client(flutter+swift)

参考了TeamTalk、瓜子IM等知名项目,包含服务端(go)和客户端(flutter+swift),单聊和机器人(小微、图灵、思知)聊天功能已完成,目前正在研发群聊功能,欢迎对golang感兴趣的小伙伴Star加关注。
Go操作Elasticsearch总结_第5张图片

你可能感兴趣的:(Golang学习和进阶,后端开发,elasticsearch,go操作es,es)