前面几个章节对Druid的整体架构做了简单的说明,本文主要描述如何部署Druid的环境
Imply提供了一套完整的部署方式,包括依赖库,Druid,图形化的数据展示页面,SQL查询组件等。本文将基于Imply套件进行说明
目录说明如下:
- bin/ - run scripts for included software.
- conf/ - template configurations for a clustered setup.
- conf-quickstart/* - configurations for the single-machine quickstart.
- dist/ - all included software.
- quickstart/ - files related to the single-machine quickstart.
bin/supervise -c conf/supervise/quickstart.conf
安装包中包含一些测试的数据,可以通过执行预先定义好的数据说明文件进行导入
bin/post-index-task --file quickstart/wikiticker-index.json
命令:curl -L -H’Content-Type: application/json’ -XPOST –data-binary @quickstart/wikiticker-top-pages.json http://localhost:8082/druid/v2?pretty
结果:
[ {
"timestamp" : "2016-06-27T00:00:11.080Z",
"result" : [ {
"edits" : 29,
"page" : "Copa América Centenario"
}, {
"edits" : 16,
"page" : "User:Cyde/List of candidates for speedy deletion/Subpage"
},
..........
{
"edits" : 8,
"page" : "World Deaf Championships"
} ]
} ]
集群配置的规划需要根据需求来定制,下面以一个开发环境机器搭建为例,描述如何搭建一个有HA特性的Druid集群.
集群部署有以下几点需要说明
1. 为了保证HA,主节点部署两台
2. 管理节点与查询节点可以考虑多核大内存的机器
角色 | 机器 | 配置 | 集群角色 |
---|---|---|---|
主节点 | 10.5.24.137 | 8C16G | Coordinator,Overlord |
主节点 | 10.5.24.138 | 8C16G | Coordinator,Overlord |
数据节点,查询节点 | 10.5.24.139 | 8C16G | Historical, MiddleManager, Tranquility,Broker,Pivot Web |
数据节点,查询节点 | 10.5.24.140 | 8C16G | Historical, MiddleManager, Tranquility,(数据节点,查询节点)Broker |
编辑conf/druid/_common/common.runtime.properties 文件内容
1. loadList配置:==此处需要统一在一个位置统一定义,否则会出现extension加载的问题==
druid.extensions.loadList=["mysql-metadata-storage","druid-hdfs-storage"]
#
# Zookeeper
#
druid.zk.service.host=native-lufanfeng-2-5-24-138:2181,native-lufanfeng-3-5-24-139:2181,native-lufanfeng-4-5-24-140:2181
druid.zk.paths.base=/druid
# For MySQL:
druid.metadata.storage.type=mysql
druid.metadata.storage.connector.connectURI=jdbc:mysql://10.5.24.151:3306/druid
druid.metadata.storage.connector.user=root
druid.metadata.storage.connector.password=123456
#druid.storage.type=local
#druid.storage.storageDirectory=var/druid/segments
druid.storage.type=hdfs
druid.storage.storageDirectory=hdfs://10.5.24.137:9000/druid/segments
#druid.indexer.logs.type=file
#druid.indexer.logs.directory=var/druid/indexing-logs
druid.indexer.logs.type=hdfs
druid.indexer.logs.directory=hdfs://10.5.24.137:9000/druid/indexing-logs
:verify bin/verify-java
:verify bin/verify-version-check
coordinator bin/run-druid coordinator conf
!p80 overlord bin/run-druid overlord conf
root@native-lufanfeng-1-5-24-137:~/imply-2.3.8# java -classpath "dist/druid/lib/*" -Ddruid.extensions.directory="dist/druid/extensions" io.druid.cli.Main tools pull-deps -c io.druid.extensions:mysql-metadata-storage:0.10.1 -c io.druid.extensions.contrib:druid-rabbitmq:0.10.1 -h org.apache.hadoop:hadoop-client:2.7.0
==默认mysql-metadata-storage带的mysql驱动是针对Mysql 5.1的,如果使用Mysql的版本是5.5 或是其他版本,可能会出现”Communications link failure”的错误,此时需要更新Mysql的驱动。==
:verify bin/verify-java
:verify bin/verify-node
:verify bin/verify-version-check
broker bin/run-druid broker conf
imply-ui bin/run-imply-ui conf
historical bin/run-druid historical conf
middleManager bin/run-druid middleManager conf
# Uncomment to use Tranquility Server
#!p95 tranquility-server bin/tranquility server -configFile conf/tranquility/server.json
# Uncomment to use Tranquility Kafka
#!p95 tranquility-kafka bin/tranquility kafka -configFile conf/tranquility/kafka.json
settingsLocation:
location: mysql
uri: 'mysql://root:[email protected]:3306/druid'
table: 'pivot_state'
initialSettings:
clusters:
- name: druid
type: druid
host: localhost:8082