druid.io 使用indexing service 配置出现 - Received FAILED

多节点部署druid.io，使用indexing service进行批量数据导入，出现问题。

提交任务后，在overlord节点消息中出现如下信息：

2016-03-22T19:25:17,555 INFO [Curator-PathChildrenCache-0] io.druid.indexing.overlord.RemoteTaskRunner - Worker[druid04:8091] wrote RUNNING status for task: index_wikipedia_2016-03-22T19:25:17.191+08:00
2016-03-22T19:25:23,219 INFO [Curator-PathChildrenCache-0] io.druid.indexing.overlord.RemoteTaskRunner - Worker[druid04:8091] wrote FAILED status for task: index_wikipedia_2016-03-22T19:25:17.191+08:00
2016-03-22T19:25:23,219 INFO [Curator-PathChildrenCache-0] io.druid.indexing.overlord.RemoteTaskRunner - Worker[druid04:8091] completed task[index_wikipedia_2016-03-22T19:25:17.191+08:00] with status[FAILED]
2016-03-22T19:25:23,219 INFO [Curator-PathChildrenCache-0] io.druid.indexing.overlord.TaskQueue - Received FAILED status for task: index_wikipedia_2016-03-22T19:25:17.191+08:00
2016-03-22T19:25:23,219 INFO [Curator-PathChildrenCache-0] io.druid.indexing.overlord.RemoteTaskRunner - Cleaning up task[index_wikipedia_2016-03-22T19:25:17.191+08:00] on worker[druid04:8091]
2016-03-22T19:25:23,222 INFO [Curator-PathChildrenCache-0] io.druid.indexing.overlord.TaskLockbox - Removing task[index_wikipedia_2016-03-22T19:25:17.191+08:00] from activeTasks
2016-03-22T19:25:23,223 INFO [Curator-PathChildrenCache-0] io.druid.indexing.overlord.TaskLockbox - Removing task[index_wikipedia_2016-03-22T19:25:17.191+08:00] from TaskLock[index_wikipedia_2016-03-22T19:25:17.191+08:00]

发现是task在work节点上运行失败。这时查找到middleManage节点日志，继续查看原因。

2016-03-22T19:25:18,227 INFO [forking-task-runner-0] io.druid.indexing.overlord.ForkingTaskRunner - Logging task index_wikipedia_2016-03-22T19:25:17.191+08:00 output to: /tmp/persistent/task/index_wikipedia_2016-03-22T19:25:17.191+08:00/log
2016-03-22T19:25:23,783 INFO [forking-task-runner-0] io.druid.indexing.overlord.ForkingTaskRunner - Process exited with status[1] for task: index_wikipedia_2016-03-22T19:25:17.191+08:00
2016-03-22T19:25:23,787 INFO [forking-task-runner-0] io.druid.indexing.common.tasklogs.FileTaskLogs - Wrote task log to: /tmp/druid/indexlog/index_wikipedia_2016-03-22T19:25:17.191+08:00.log
2016-03-22T19:25:23,789 INFO [forking-task-runner-0] io.druid.indexing.overlord.ForkingTaskRunner - Removing task directory: /tmp/persistent/task/index_wikipedia_2016-03-22T19:25:17.191+08:00
2016-03-22T19:25:23,811 INFO [WorkerTaskMonitor-0] io.druid.indexing.worker.WorkerTaskMonitor - Job's finished. Completed [index_wikipedia_2016-03-22T19:25:17.191+08:00] with status [FAILED]

很失望的是只能看出异常退出，却没有显示的具体原因。

这时停掉 middlemanager线程，修改middleManager 的配置config/middleManager/runtime.properties 文件，将log信息配置保存在本地的目录。

druid.indexer.logs.type=local
druid.indexer.logs.directory=/tmp/druid/indexlog

配置完成后，重新运行middleManager线程，并且重新提交任务，错误依旧，但是此时可以找到错误的具体原因。查看刚刚配置过的=/tmp/druid/indexlog文件。

) Not enough direct memory.  Please adjust -XX:MaxDirectMemorySize, druid.processing.buffer.sizeBytes, or druid.processing.numThreads: maxDirectMemory[239,075,328], memoryNeeded[4,294,967,296] = druid.processing.buffer.sizeBytes[1,073,741,824] * ( druid.processing.numThreads[3] + 1 )
  at io.druid.guice.DruidProcessingModule.getIntermediateResultsPool(DruidProcessingModule.java:106)
  at io.druid.guice.DruidProcessingModule.getIntermediateResultsPool(DruidProcessingModule.java:106)
  while locating io.druid.collections.StupidPool<java.nio.ByteBuffer> annotated with @io.druid.guice.annotations.Global()
    for parameter 1 at io.druid.query.groupby.GroupByQueryEngine.<init>(GroupByQueryEngine.java:75)
  at io.druid.guice.QueryRunnerFactoryModule.configure(QueryRunnerFactoryModule.java:83)
  while locating io.druid.query.groupby.GroupByQueryEngine
    for parameter 0 at io.druid.query.groupby.GroupByQueryRunnerFactory.<init>(GroupByQueryRunnerFactory.java:79)
  at io.druid.guice.QueryRunnerFactoryModule.configure(QueryRunnerFactoryModule.java:80)
  while locating io.druid.query.groupby.GroupByQueryRunnerFactory
  while locating io.druid.query.QueryRunnerFactory annotated with @com.google.inject.multibindings.Element(setName=,uniqueId=26, type=MAPBINDER)
  at io.druid.guice.DruidBinders.queryRunnerFactoryBinder(DruidBinders.java:36)
  while locating java.util.Map<java.lang.Class<? extends io.druid.query.Query>, io.druid.query.QueryRunnerFactory>
    for parameter 0 at io.druid.query.DefaultQueryRunnerFactoryConglomerate.<init>(DefaultQueryRunnerFactoryConglomerate.java:34)
  while locating io.druid.query.DefaultQueryRunnerFactoryConglomerate
  at io.druid.guice.StorageNodeModule.configure(StorageNodeModule.java:53)
  while locating io.druid.query.QueryRunnerFactoryConglomerate
    for parameter 9 at io.druid.indexing.common.TaskToolboxFactory.<init>(TaskToolboxFactory.java:83)
  at io.druid.cli.CliPeon$1.configure(CliPeon.java:138)
  while locating io.druid.indexing.common.TaskToolboxFactory
    for parameter 0 at io.druid.indexing.overlord.ThreadPoolTaskRunner.<init>(ThreadPoolTaskRunner.java:76)
  at io.druid.cli.CliPeon$1.configure(CliPeon.java:164)
  while locating io.druid.indexing.overlord.ThreadPoolTaskRunner
  while locating io.druid.query.QuerySegmentWalker
    for parameter 3 at io.druid.server.QueryResource.<init>(QueryResource.java:90)
  while locating io.druid.server.QueryResource

一大堆信息，第一句最重要，那就是内存不够导致问题原因。

思考具体哪里内存分配的不够？ 多年从事c 类的语言，java的水平等于0，根本不知道原因，反复对比各种配置 java运行配置。

最后猜测 可能是middleManager fork poen线程时内存不够问题。

查找手册查看mddileManager中关于middle 配置fork poen的参数。

最后找到config/middleManager/runtime.properties中的两项

 

#druid.indexer.fork.property.druid.processing.buffer.sizeBytes=100000000
#druid.indexer.fork.property.druid.processing.numThreads=1

默认的可能buffer是1g,  numThreads是3个，导致内存移除，修改后，重新运行任务，任务顺利完成。