Shuffle's getMapOutput() fails with EofException, followed by IllegalStateExcept

    在查看tasktracker日志时，发现报错

2013-12-18 15:56:29,527 WARN org.mortbay.log: Committed before 410 getMapOutput(attempt_201311271626_585288_m_000024_0,2) failed :
org.mortbay.jetty.EofException
...
2013-12-18 15:56:29,528 ERROR org.mortbay.log: /mapOutput
java.lang.IllegalStateException: Committed
...

 

    于是google错误，找到这个链接https://issues.apache.org/jira/browse/MAPREDUCE-5

    tasktracker getMapOutput报错，主要原因是ReduceTask提前关闭jetty连接。reduce 缓冲区里缓冲的数据已经达到阈值，所以就关闭连接，触发flush()操作，将缓冲区内容写入磁盘，然后再次连接jetty。这个错误的结论是不要改，不会导致task或Job失败，可能和reduce的heapsize设置太小有关。

原文如下：

The Jetty EOFException is observed when the Reduce Task prematurely closes a connection to a jetty server. The RT might decide to do this when the expected map output size is less than its memory cache limit, but cannot fetch it now because it has already fetched several other map outputs to memory. So, it triggers a flush of the existing map outputs from memory to disk and once this flush is complete, it will attempt to fetch the map output again by reopening the connection to the Jetty. In short, it is OK to see such messages.
This should neither cause a task nor a job failure. Are you seeing failures? The reason why these error messages appear for some jobs and not for others is primarily got to do with the size of map outputs and the heap size allocated for the reducer.