flume接收来的数据太大怎么处理?

在flume中,将kafka作为flume的输入源,通过memory的方式,将数据写到hdfs上,当kafka的数据较大时,出现如下错误

16/03/24 16:57:31 ERROR network.BoundedByteBufferReceive: OOME with size 722234
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
at kafka.network.BoundedByteBufferReceive.byteBufferAllocate(BoundedByteBufferReceive.scala:80)
at kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:63)
at kafka.network.Receive$class.readCompletely(Transmission.scala:56)
at kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
at kafka.network.BlockingChannel.receive(BlockingChannel.scala:100)
at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:81)
at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:71)
at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:109)
at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:109)
at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:109)
at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(SimpleConsumer.scala:108)
at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:108)
at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:108)
at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:107)
at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:96)
at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:88)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
16/03/24 16:57:32 ERROR consumer.ConsumerFetcherThread: [ConsumerFetcherThread-flume_ibd105-1458809849949-5f4498cc-0-46], Error due to 
java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "ConsumerFetcherThread-flume_ibd105-1458809849949-5f4498cc-0-45" java.lang.OutOfMemoryError: GC overhead limit exceeded
16/03/24 16:57:32 ERROR consumer.ConsumerFetcherThread: [ConsumerFetcherThread-flume_ibd105-1458809849949-5f4498cc-0-45], Error due to 
16/03/24 16:57:32 INFO consumer.ConsumerFetcherThread: [ConsumerFetcherThread-flume_ibd105-1458809849949-5f4498cc-0-46], Stopped 
Exception in thread "ConsumerFetcherThread-flume_ibd105-1458809849949-5f4498cc-0-43" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "metrics-meter-tick-thread-2" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "ConsumerFetcherThread-flume_ibd105-1458809849949-5f4498cc-0-44" java.lang.OutOfMemoryError: GC overhead limit exceeded

处理的办法是在flume conf配置下修改配置的大小。

你可能感兴趣的:(flume,面试部分)