Druid单机的安装和测试
1、环境信息
2、安装zookeeper
3、安装druid
4、启动druid
5、验证druid
6、遇到的吭
1、环境信息
Centos6.5
32GB 8C
Zookeeper 3.4.5
Druid 0.9.2
2、安装zookeeper
tar -xzf zookeeper-3.4.6.tar.gz
cd zookeeper-3.4.6
cp conf/zoo_sample.cfg conf/zoo.cfg
./bin/zkServer.sh start3、安装 druid
3、1解压druid
3、2 执行 bin/init 初始化化环境
4、启动druid
4、1启动5个终端执行下面5个命名，或者启动一个将命令执行丢入后台
java `cat conf-quickstart/druid/historical/jvm.config | xargs` -cp "conf-quickstart/druid/_common:conf-quickstart/druid/historical:lib/*" io.druid.cli.Main server historical
java `cat conf-quickstart/druid/broker/jvm.config | xargs` -cp "conf-quickstart/druid/_common:conf-quickstart/druid/broker:lib/*" io.druid.cli.Main server broker
java `cat conf-quickstart/druid/coordinator/jvm.config | xargs` -cp "conf-quickstart/druid/_common:conf-quickstart/druid/coordinator:lib/*" io.druid.cli.Main server coordinator
java `cat conf-quickstart/druid/overlord/jvm.config | xargs` -cp "conf-quickstart/druid/_common:conf-quickstart/druid/overlord:lib/*" io.druid.cli.Main server overlord
java `cat conf-quickstart/druid/middleManager/jvm.config | xargs` -cp "conf-quickstart/druid/_common:conf-quickstart/druid/middleManager:lib/*" io.druid.cli.Main server middleManager4、2验证是否正常
执行完成以后可以查看服务是否启动
http://10.20.23.82:8081/#/indexing-service
http://10.20.23.82:8090/console.html
5、验证druid
5、1 修改 wikiticker-2015-09-12-sampled.json 文件为本地local模式的
{
"type" : "index",
"spec" : {
"ioConfig" : {
"type" : "index",
"firehose" : {
"type" : "local",
"baseDir" : "/home/hadoop/druid-0.9.2/quickstart",
"filter" : "wikiticker-2015-09-12-sampled.json"
}
},
"dataSchema" : {
"dataSource" : "wikiticker",
"granularitySpec" : {
"type" : "uniform",
"segmentGranularity" : "day",
"queryGranularity" : "none",
"intervals" : ["2015-09-12/2015-09-13"]
},
"parser" : {
"type" : "string",
"parseSpec" : {
"format" : "json",
"dimensionsSpec" : { "dimensions" : [ "channel", "cityName", "comment", "countryIsoCode", "countryName", "isAnonymous", "isMinor", "isNew", "isRobot", "isUnpatrolled", "metroCode", "namespace", "page", "regionIsoCode", "regionName", "user" ] },
"timestampSpec" : { "format" : "auto", "column" : "time" } }
},
"metricsSpec" : [
{
"name" : "count",
"type" : "count"
},
{
"name" : "added",
"type" : "longSum",
"fieldName" : "added"
},
{
"name" : "deleted",
"type" : "longSum",
"fieldName" : "deleted"
},
{
"name" : "delta",
"type" : "longSum",
"fieldName" : "delta"
},
{
"name" : "user_unique",
"type" : "hyperUnique",
"fieldName" : "user"
}
]
},
"tuningConfig" : {
"type" : "index",
"partitionsSpec" : {
"type" : "hashed",
"targetPartitionSize" : 5000000
},
"jobProperties" : {}
}
}
}5、2 执行 curl -X ‘POST’ -H ‘Content-Type:application/json’ -d @quickstart/wikiticker-index.json localhost:8090/druid/indexer/v1/task 命令来overlord数据
[hadoop@SZB-L0038787 druid-0.9.2]$ curl -X 'POST' -H 'Content-Type:application/json' -d @quickstart/wikiticker-index.json localhost:8090/druid/indexer/v1/task
{"task":"index_wikiticker_2017-05-22T02:06:52.225Z"}5、3 查看运行的情况，可以在console上面查看是否运行成功
5、4查看数据
运行如下命令
curl -L -H'Content-Type: application/json' -XPOST --data-binary @quickstart/wikiticker-top-pages.json http://localhost:8082/druid/v2/?pretty[hadoop@SZB-L0038787 druid-0.9.2]$ curl -L -H'Content-Type: application/json' -XPOST --data-binary @quickstart/wikiticker-top-pages.json http://localhost:8082/druid/v2/?pretty
[ {
"timestamp" : "2015-09-12T00:46:58.771Z",
"result" : [ {
"edits" : 33,
"page" : "Wikipedia:Vandalismusmeldung"
}, {
"edits" : 28,
"page" : "User:Cyde/List of candidates for speedy deletion/Subpage"
}, {
"edits" : 27,
"page" : "Jeremy Corbyn"
}, {
"edits" : 21,
"page" : "Wikipedia:Administrators' noticeboard/Incidents"
}, {
"edits" : 20,
"page" : "Flavia Pennetta"
}, {
"edits" : 18,
"page" : "Total Drama Presents: The Ridonculous Race"
}, {
"edits" : 18,
"page" : "User talk:Dudeperson176123"
}, {
"edits" : 18,
"page" : "Wikipédia:Le Bistro/12 septembre 2015"
}, {
"edits" : 17,
"page" : "Wikipedia:In the news/Candidates"
}, {
"edits" : 17,
"page" : "Wikipedia:Requests for page protection"
}, {
"edits" : 16,
"page" : "Utente:Giulio Mainardi/Sandbox"
}, {
"edits" : 16,
"page" : "Wikipedia:Administrator intervention against vandalism"
}, {
"edits" : 15,
"page" : "Anthony Martial"
}, {
"edits" : 13,
"page" : "Template talk:Connected contributor"
}, {
"edits" : 12,
"page" : "Chronologie de la Lorraine"
}, {
"edits" : 12,
"page" : "Wikipedia:Files for deletion/2015 September 12"
}, {
"edits" : 12,
"page" : "Гомосексуальный образ жизни"
}, {
"edits" : 11,
"page" : "Constructive vote of no confidence"
}, {
"edits" : 11,
"page" : "Homo naledi"
}, {
"edits" : 11,
"page" : "Kim Davis (county clerk)"
}, {
"edits" : 11,
"page" : "Vorlage:Revert-Statistik"
}, {
"edits" : 11,
"page" : "Конституция Японской империи"
}, {
"edits" : 10,
"page" : "The Naked Brothers Band (TV series)"
}, {
"edits" : 10,
"page" : "User talk:Buster40004"
}, {
"edits" : 10,
"page" : "User:Valmir144/sandbox"
} ]
} ]6、遇到的坑
6、1 大小写的区别，路径的配置baseDir只能这么写
2017-05-22T01:33:31,841 ERROR [task-runner-0-priority-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running task[IndexTask{id=index_wikiticker_2017-05-22T01:33:26.102Z, type=index, dataSource=wikiticker}]
com.metamx.common.IAE: baseDir is null
at io.druid.segment.realtime.firehose.LocalFirehoseFactory.connect(LocalFirehoseFactory.java:89) ~[druid-server-0.9.2.jar:0.9.2]
at io.druid.segment.realtime.firehose.LocalFirehoseFactory.connect(LocalFirehoseFactory.java:46) ~[druid-server-0.9.2.jar:0.9.2]
at io.druid.indexing.common.task.IndexTask.getDataIntervals(IndexTask.java:255) ~[druid-indexing-service-0.9.2.jar:0.9.2]
at io.druid.indexing.common.task.IndexTask.run(IndexTask.java:201) ~[druid-indexing-service-0.9.2.jar:0.9.2]
at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) [druid-indexing-service-0.9.2.jar:0.9.2]
at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) [druid-indexing-service-0.9.2.jar:0.9.2]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_101]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]
2017-05-22T01:33:31,852 INFO [task-runner-0-priority-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_wikiticker_2017-05-22T01:33:26.102Z] status changed to [FAILED].
2017-05-22T01:33:31,858 INFO [task-runner-0-priority-0] io.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {
"id" : "index_wikiticker_2017-05-22T01:33:26.102Z",
"status" : "FAILED",
"duration" : 326
}6、2 源文件的路径，在druid中配置路径baseDir 这个建议填写绝对路径，filter这个只填写文件名称。如果你需要填写信息的话filter这个参数带的路径是没有实际意义的
2017-05-22T01:34:40,650 INFO [task-runner-0-priority-0] io.druid.segment.realtime.firehose.LocalFirehoseFactory - Searching for all [quickstart/wikiticker-2015-09-12-sampled.json] in and beneath [/home/hadoop/druid-0.9.2/.]
2017-05-22T01:34:40,673 ERROR [task-runner-0-priority-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running task[IndexTask{id=index_wikiticker_2017-05-22T01:34:36.301Z, type=index, dataSource=wikiticker}]
com.metamx.common.ISE: Found no files to ingest! Check your schema.
at io.druid.segment.realtime.firehose.LocalFirehoseFactory.connect(LocalFirehoseFactory.java:100) ~[druid-server-0.9.2.jar:0.9.2]
at io.druid.segment.realtime.firehose.LocalFirehoseFactory.connect(LocalFirehoseFactory.java:46) ~[druid-server-0.9.2.jar:0.9.2]
at io.druid.indexing.common.task.IndexTask.getDataIntervals(IndexTask.java:255) ~[druid-indexing-service-0.9.2.jar:0.9.2]
at io.druid.indexing.common.task.IndexTask.run(IndexTask.java:201) ~[druid-indexing-service-0.9.2.jar:0.9.2]
at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) [druid-indexing-service-0.9.2.jar:0.9.2]
at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) [druid-indexing-service-0.9.2.jar:0.9.2]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_101]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]
2017-05-22T01:34:40,678 INFO [task-runner-0-priority-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_wikiticker_2017-05-22T01:34:36.301Z] status changed to [FAILED].
2017-05-22T01:34:40,680 INFO [task-runner-0-priority-0] io.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {
"id" : "index_wikiticker_2017-05-22T01:34:36.301Z",
"status" : "FAILED",
"duration" : 232
}6、3 路径错误，在druid中有一个环境路径需要提前手工创建
environment:java.io.tmpdir=var/tmp
2017-05-22T02:05:47,698 ERROR [task-runner-0-priority-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running task[IndexTask{id=index_wikiticker_2017-05-22T02:05:40.252Z, type=index, dataSource=wikiticker}]
java.lang.IllegalStateException: Failed to create directory within 10000 attempts (tried 1495418747661-0 to 1495418747661-9999)
at com.google.common.io.Files.createTempDir(Files.java:600) ~[guava-16.0.1.jar:?]
at io.druid.segment.indexing.RealtimeTuningConfig.createNewBasePersistDirectory(RealtimeTuningConfig.java:56) ~[druid-server-0.9.2.jar:0.9.2]
at io.druid.segment.indexing.RealtimeTuningConfig.(RealtimeTuningConfig.java:118) ~[druid-server-0.9.2.jar:0.9.2]
at io.druid.indexing.common.task.IndexTask.convertTuningConfig(IndexTask.java:147) ~[druid-indexing-service-0.9.2.jar:0.9.2]
at io.druid.indexing.common.task.IndexTask.generateSegment(IndexTask.java:397) ~[druid-indexing-service-0.9.2.jar:0.9.2]
at io.druid.indexing.common.task.IndexTask.run(IndexTask.java:233) ~[druid-indexing-service-0.9.2.jar:0.9.2]
at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) [druid-indexing-service-0.9.2.jar:0.9.2]
at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) [druid-indexing-service-0.9.2.jar:0.9.2]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_101]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]
2017-05-22T02:05:47,705 INFO [task-runner-0-priority-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_wikiticker_2017-05-22T02:05:40.252Z] status changed to [FAILED].
2017-05-22T02:05:47,711 INFO [task-runner-0-priority-0] io.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {
"id" : "index_wikiticker_2017-05-22T02:05:40.252Z",
"status" : "FAILED",
"duration" : 2776
}