实时数仓-Doris ON ES

原理介绍:原文点击  

   Doris通过创建外部表方式将Doris的分布式查询规划能力和ES(Elasticsearch)的全文检索能力相结合,提供更完善的OLAP分析场景解决方案,支持:

  1. ES中的多index分布式Join查询

  2. Doris和ES中的表联合查询,更复杂的全文检索过滤

实时数仓-Doris ON ES_第1张图片

   创建ES外表后,FE会请求建表指定的主机,获取所有节点的HTTP端口信息以及index的shard分布信息等,如果请求失败会顺序遍历host列表直至成功或完全失败。

   执行查询时,会根据FE得到的一些节点信息和index的元数据信息,生成查询计划并发给对应的BE节点,BE节点会根据就近原则即优先请求本地部署的ES节点,BE通过HTTP Scroll方式流式的从ES index的每个分片中并发的获取数据

计算完结果后,返回给client端。

    ES节点类型分为主节点、数据节点、协调节点,FE通过主节点获取ES信息,BE直接拉取数据节点获取数据。

实验过程  

实验环境:doris版本0.14.0,elasticsearch版本7.11.1

doris环境搭建及启动这里就不在叙述了,elasticsearch参考ES环境搭建及后续文章。

一、单节点查询:

1、创建doris外部表

CREATE EXTERNAL TABLE `es_table` (  `id` bigint(20) COMMENT "",  `k1` bigint(20) COMMENT "",  `k2` datetime COMMENT "",  `k3` varchar(20) COMMENT "",  `k4` varchar(100) COMMENT "",  `k5` float COMMENT "") ENGINE=ELASTICSEARCHPARTITION BY RANGE(`id`)()PROPERTIES ("host" = "http://192.168.244.129:9200","index" = "test”);

2、ES初始化

1、创建test索引

{  "mappings": {    "properties": {      "k1": {        "type": "long",        "index": "true"      },      "k3": {        "type": "text",        "analyzer": "ik_max_word",        "search_analyzer": "ik_max_word"      },      "k4": {        "type": "text",        "analyzer": "ik_max_word",        "search_analyzer": "ik_max_word"      },      "k5": {        "type": "float"      },      "k2": {        "type": "date",        "format": "yyyy-MM-dd"      }    }  }}

实时数仓-Doris ON ES_第2张图片

2、添加数据

{  "k1": 100,  "k2": "2020-01-01",  "k3": "Trying",  "k4": "Trying out Elasticsearch",  "k5": 10}

数据添加成功后,在mysql客户端连接doris查询ES数据,看到如下结果代表doris查询ES成功。

实时数仓-Doris ON ES_第3张图片

3、批量添加数据

POST /_bulk
{"index":{"_index":"test"}}
{ "k1" : 100, "k2": "2020-01-01", "k3": "Trying out Elasticsearch", "k4": "Trying out Elasticsearch", "k5": 10.0}
{"index":{"_index":"test"}}
{ "k1" : 100, "k2": "2020-01-01", "k3": "Trying out Doris", "k4": "Trying out Doris", "k5": 10.0}
{"index":{"_index":"test"}}
{ "k1" : 100, "k2": "2020-01-01", "k3": "Doris On ES", "k4": "Doris On ES", "k5": 10.0}
{"index":{"_index":"test"}}
{ "k1" : 100, "k2": "2020-01-01", "k3": "Doris", "k4": "Doris", "k5": 10.0}
{"index":{"_index":"test"}}
{ "k1" : 100, "k2": "2020-01-01", "k3": "ES", "k4": "ES", "k5": 10.0}

执行模糊匹配查询:

实时数仓-Doris ON ES_第4张图片

二、JOIN查询:

1、创建外部表

实时数仓-Doris ON ES_第5张图片

2、ES创建索引test2

{  "mappings": {    "properties": {      "k1": {        "type": "long",        "index": "true"      },      "k3": {        "type": "text",        "analyzer": "ik_max_word",        "search_analyzer": "ik_max_word"      },      "k4": {        "type": "text",        "analyzer": "ik_max_word",        "search_analyzer": "ik_max_word"      },      "k5": {        "type": "float"      },      "k2": {        "type": "date",        "format": "yyyy-MM-dd"      }    }  }}

3、ES添加数据​​​​​​​

POST /_bulk{"index":{"_index":"test2"}}{ "k1" : 200, "k2": "2020-02-01", "k3": "Doris e ", "k4": "ES", "k5": 20.0}

4、执行JOIN查询

实时数仓-Doris ON ES_第6张图片

5、JOIN模糊查询​​​​​​​

select * from test ,test2 where test.k1=test2.k1 and esquery (test.k3, '{        "match": {           "k3": "ES"        }    }');

实时数仓-Doris ON ES_第7张图片

Doris ON  ES 今天就介绍到这里了,觉得有用关注:蓝天Java大数据

你可能感兴趣的:(doris,大数据,big,data)