开启HDFS的 dfs.client.read.shortcircuit 配置

1。在 hadoop的配置文件 hdfs-site.xml 添加如下配置:

        dfs.client.read.shortcircuit
        true


        dfs.domain.socket.path
        /home/yeshang/hdfs_sockets/dn_socket


2。进入本地文件系统:

mkdir -p /home/yeshang/hdfs_sockets/
chmod 0755 /home/yeshang/hdfs_sockets
dn_socket 是系统写入,不用手动创建,否则会出问题

3。如果该配置生效 datanode的log会有如下log:

例如:Hadoop/logs/hadoop-yeshang-datanode-cluster2.serversolution.sh.hxt.log

2019-01-04 14:44:38,092 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Balancing bandwith is 1048576 bytes/s
2019-01-04 14:44:38,092 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Number threads for balancing is 5
2019-01-04 14:44:38,092 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Listening on UNIX domain socket: /home/yeshang/hdfs_sockets/dn_socket
2019-01-04 14:44:38,222 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog

你可能感兴趣的:(Spark)