java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read

环境：

- Spark 2.11-2.0.2

- Hadoop 2.6.5

使用Spark连接HDFS，在高并发的时候，偶尔会遇到以下异常：

2017-01-26 10:30:29,079 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: xxx:50010:DataXceiver error processing WRITE_BLOCK operation  src: /xxx:52827 dst: /xxx:50010
java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/xxx:50010 remote=/xxx:52827]
    at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
    at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
    at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
    at java.io.DataInputStream.read(DataInputStream.java:132)
    at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:192)
    at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
    at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:171)
    at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
    at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:496)
    at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:891)
    at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:758)
    at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
    at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
    at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:239)
    at java.lang.Thread.run(Thread.java:662)

出现以上错误是因为datanode的服务线程连接数都被占用，导致Spark等待超时。可以通过以下方法解决：

• 根据datanode的资源情况适当增加的服务线程数：在hdfs-default.xml配置文件里面新增或修改dfs.datanode.handler.count，默认是10
• 增加客户端的超时时间dfs.client.socket-timeout，默认是60000ms，注意，旧版本的Hadoop用的是dfs.socket.timeout，可参考http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/DeprecatedProperties.html

例如以下Java代码设置超时时间为300000ms=300s=5分钟：

try (JavaSparkContext context = new JavaSparkContext(new SparkConf().setMaster("local[*]").setAppName("test"))) {
    context.sc().hadoopConfiguration().set("dfs.client.socket-timeout", "300000");
    ......