spark启动时Cannot assign requested address: Service 'sparkDriver'问题分析

问题经过

当使用bin/spark-shell命令或使用spark-submit提交任务时，会遇到如下报错。

Setting default log level to “WARN”.
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
19/12/24 09:58:13 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
19/12/24 09:58:15 WARN util.Utils: Service ‘sparkDriver’ could not bind on port 0. Attempting port 1.
19/12/24 09:58:15 WARN util.Utils: Service ‘sparkDriver’ could not bind on port 0. Attempting port 1.
19/12/24 09:58:15 WARN util.Utils: Service ‘sparkDriver’ could not bind on port 0. Attempting port 1.
19/12/24 09:58:15 WARN util.Utils: Service ‘sparkDriver’ could not bind on port 0. Attempting port 1.
19/12/24 09:58:15 WARN util.Utils: Service ‘sparkDriver’ could not bind on port 0. Attempting port 1.
19/12/24 09:58:15 WARN util.Utils: Service ‘sparkDriver’ could not bind on port 0. Attempting port 1.
19/12/24 09:58:15 WARN util.Utils: Service ‘sparkDriver’ could not bind on port 0. Attempting port 1.
19/12/24 09:58:15 WARN util.Utils: Service ‘sparkDriver’ could not bind on port 0. Attempting port 1.
19/12/24 09:58:15 WARN util.Utils: Service ‘sparkDriver’ could not bind on port 0. Attempting port 1.
19/12/24 09:58:15 WARN util.Utils: Service ‘sparkDriver’ could not bind on port 0. Attempting port 1.
19/12/24 09:58:15 WARN util.Utils: Service ‘sparkDriver’ could not bind on port 0. Attempting port 1.
19/12/24 09:58:15 WARN util.Utils: Service ‘sparkDriver’ could not bind on port 0. Attempting port 1.
19/12/24 09:58:15 WARN util.Utils: Service ‘sparkDriver’ could not bind on port 0. Attempting port 1.
19/12/24 09:58:15 WARN util.Utils: Service ‘sparkDriver’ could not bind on port 0. Attempting port 1.
19/12/24 09:58:15 WARN util.Utils: Service ‘sparkDriver’ could not bind on port 0. Attempting port 1.
19/12/24 09:58:15 WARN util.Utils: Service ‘sparkDriver’ could not bind on port 0. Attempting port 1.
19/12/24 09:58:15 ERROR spark.SparkContext: Error initializing SparkContext.
java.net.BindException: Cannot assign requested address: Service ‘sparkDriver’ failed after 16 retries (starting from 0)! Consider explicitly setting the appropriate port for the service ‘sparkDriver’ (for example spark.ui.port for SparkUI) to an available port or increasing spark.port.maxRetries.
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:127)
at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:501)atio.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1218)
at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:496)
at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:481)
at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:965)
at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:210)
at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:353)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:446)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)atio.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
at java.lang.Thread.run(Thread.java:748)

问题分析

可以从错误日志看出，spark无法连接地址，无法执行spark任务。也就是spark本身并未出现bug，而是网络环境的问题。

我的集群是在阿里云上搭建的，首先想到的是阿里云端口未开放。于是我开放阿里云所有的端口发现问题还存在。

在查询了一些资料之后，我发现问题出现在了hosts上。注意！划重点了！
阿里云服务器会有两个ip地址，一个对应内网，一个对应外网。而我hosts中用户名对应的ip地址为外网。而当spark本地运行时，会默认寻找用户名对应的ip地址，但找到了外网的地址，所以造成无法连接的问题。

问题解决

首先不推荐更改hosts，因为这是系统文件。它的更改不仅应用于spark还用影响整个环境下的所有组件。
在不影响环境的情况下，我们可以向spark-env.sh文件中添加一条变量，让spark启动时默认找到本地ip。
vim spark-env.sh

SPARK_LOCAL_IP=127.0.0.1