DRUID的安装部署

使用的是imply集成的druid,Imply提供了一套完整的部署方式,包括依赖库,Druid,tranquility、图形化的数据展示页面(pivot),SQL查询组件等。

安装版本: imply_3.0.7

部署规划:

角色机器集群角色

主节点master服务server3Coordinator,Overlord

主节点master服务server4Coordinator,Overlord

数据服务、查询服务server5Historical, MiddleManager, Tranquility,Broker,Pivot

数据服务、查询服务server6Historical, MiddleManager, Tranquility,Broker,Pivot

注:1、本druid集群具有HA特性。

2、server7也安装有imply_3.0.7,以便于以后横向拓展。

启动命令:

以后台挂起方式启动druid:

在server3、server4上执行

nohup bin/supervise -c conf/supervise/master.conf > master.log &

在server5、server5上执行

nohup bin/supervise -c conf/supervise/data-with-query.conf > data-with-query.log &

可视化控制台

overlord 控制页面:http://server3:8090/console.html

druid集群页面:http://server3:8081/index.html#/

数据可视化页面:http://server5:9096/datasets

配置文件

conf/druid/_common/common.runtime.properties

#

# Extensions

#

druid.extensions.directory=dist/druid/extensions

druid.extensions.hadoopDependenciesDir=dist/druid/hadoop-dependencies

druid.extensions.loadList=["mysql-metadata-storage","druid-hdfs-storage","druid-kafka-indexing-service","druid-lookups-cached-global","druid-histogram","druid-datasketches"]

#

# Logging

#

# Log all runtime properties on startup. Disable to avoid logging properties on startup:

druid.startup.logging.logProperties=true

#

# Zookeeper

#

druid.zk.service.host=server3:2181,server4:2181,server5:2181

druid.zk.paths.base=/druid

#

# Metadata storage

#

# For Derby server on your Druid Coordinator (only viable in a cluster with a single Coordinator, no fail-over):

#druid.metadata.storage.type=derby

#druid.metadata.storage.connector.connectURI=jdbc:derby://master.example.com:1527/var/druid/metadata.db;create=true

#druid.metadata.storage.connector.host=master.example.com

#druid.metadata.storage.connector.port=1527

# For MySQL:

druid.metadata.storage.type=mysql

druid.metadata.storage.connector.connectURI=jdbc:mysql://server2:3306/druid

druid.metadata.storage.connector.user=root

druid.metadata.storage.connector.password=root

# For PostgreSQL:

#druid.metadata.storage.type=postgresql

#druid.metadata.storage.connector.connectURI=jdbc:postgresql://db.example.com:5432/druid

#druid.metadata.storage.connector.user=...

#druid.metadata.storage.connector.password=...

#

# Deep storage

#

# For local disk (only viable in a cluster if this is a network mount):

#druid.storage.type=local

#druid.storage.storageDirectory=var/druid/segments

# For HDFS:

druid.storage.type=hdfs

druid.storage.storageDirectory=hdfs://server1:9870/druid/segments

#

# Indexing service logs

#

# For local disk (only viable in a cluster if this is a network mount):

#druid.indexer.logs.type=file

#druid.indexer.logs.directory=var/druid/indexing-logs

# For HDFS:

druid.indexer.logs.type=hdfs

druid.indexer.logs.directory=hdfs://server1:9870/druid/indexing-logs

# For S3:

#druid.indexer.logs.type=s3

#druid.indexer.logs.s3Bucket=your-bucket

#druid.indexer.logs.s3Prefix=druid/indexing-logs

#

# Service discovery

#

druid.selectors.indexing.serviceName=druid/overlord

druid.selectors.coordinator.serviceName=druid/coordinator

#

# Monitoring

#

druid.monitoring.monitors=["org.apache.druid.java.util.metrics.JvmMonitor"]

druid.emitter=logging

druid.emitter.logging.logLevel=debug

注:需将core-site.xml、hdfs-site.xml、mapred-site.xml、yarn-site.xml等Hadoop的配置文件移至_common文件夹下

conf/pivot/config.yaml

# The port on which the Pivot server will listen on.

port: 9096

# runtime directory

varDir: var/pivot

servingMode: clustered

# User management mode

# By default Imply will not show a login screen and anyone accessing it will automatically be treated as an 'admin'

# Uncomment the line below to enable user authentication, for more info see: https://docs.imply.io/on-prem/configure/config-api

#userMode: native-users

# The initial settings that will be loaded in, in this case a connection will be created for a Druid cluster that is running locally.

stateStore:

type: mysql

connection: 'mysql://user:password@server2:3306/druid'

table: 'pivot_state'

initialSettings:

connections:

- name: druid

type: druid

title: My Druid

host: localhost:8887

brokerHosts: ["server5:8082"]

coordinatorHosts: ["server3:8081"]

overlordHosts: ["server3:8090"]

#

# Pivot must have a state store in order to function

# The state (data cubes, dashboards, etc) can be stored in two ways.

# Choose just one option and comment out the other.

#

# 1) Stored in a sqlite file, editable at runtime with Settings View. Not suitable for running in a cluster.

# 2) Stored in a database, editable at runtime with Settings View. Works well with a cluster of Imply servers.

#

#

# 1) File-backed (sqlite) state (not suitable for running in a cluster)

#

#stateStore:

# type: sqlite

# connection: var/pivot/pivot-settings.sqlite

#

# 2) Database-backed state 'mysql' (MySQL) or 'pg' (Postgres)

#

# stateStore:

# location: mysql

# connection: 'mysql://user:password@server2:3306/druid'

创建conf/supercise/data-with-query.conf文件

:verify bin/verify-java

:verify bin/verify-version-check

broker bin/run-druid broker conf

router bin/run-druid router conf

pivot bin/run-pivot conf

historical bin/run-druid historical conf

middleManager bin/run-druid middleManager conf

# Uncomment to use Tranquility Server

#!p95 tranquility-server bin/tranquility server -configFile conf/tranquility/server.json

# Uncomment to use Tranquility Kafka

#!p95 tranquility-kafka bin/tranquility kafka -configFile conf/tranquility/kafka.json

# Uncomment to use Tranquility Clarity metrics server

#!p95 tranquility-metrics-server java -Xms4g -Xmx4g -cp "dist/tranquility/lib/*:dist/tranquility/conf" com.metamx.tranquility.distribution.DistributionMain server -configFile conf/tranquility/server-for-metrics.yaml

创建conf/supercise/master.conf文件

:verify bin/verify-java

:verify bin/verify-version-check

coordinator bin/run-druid coordinator conf

!p80 overlord bin/run-druid overlord conf

注:

1、因端口号冲突,在配置文件中修改了以下端口号:

修改router端口为8887

修改middlemanager端口号为8092

修改pivot端口号为9096

因druid启动有自检端口号机制,需修改启动文件bin/verify-default-ports中检查的端口号

你可能感兴趣的:(DRUID的安装部署)