解决centos7上初始化spark意外

在Linux(CentOS7)上初始化spark遇到“org.apache.spark.SparkException: Invalid Spark URL: spark://HeartbeatReceiver@VM_0_9_centos:34068”问题,具体如下:

ERROR SparkContext:91 - Error initializing SparkContext.
org.apache.spark.SparkException: Invalid Spark URL: spark://HeartbeatReceiver@VM_0_9_centos:34068
 at org.apache.spark.rpc.RpcEndpointAddress$.apply(RpcEndpointAddress.scala:66)
 at org.apache.spark.rpc.netty.NettyRpcEnv.asyncSetupEndpointRefByURI(NettyRpcEnv.scala:134)
 at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:101)
 at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:109)
 at org.apache.spark.util.RpcUtils$.makeDriverRef(RpcUtils.scala:32)
 at org.apache.spark.executor.Executor.(Executor.scala:178)
 at org.apache.spark.scheduler.local.LocalEndpoint.(LocalSchedulerBackend.scala:59)
 at org.apache.spark.scheduler.local.LocalSchedulerBackend.start(LocalSchedulerBackend.scala:127)
 at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:178)
 at org.apache.spark.SparkContext.(SparkContext.scala:501)
 at org.apache.spark.api.java.JavaSparkContext.(JavaSparkContext.scala:58)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
 at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
 at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
 at py4j.Gateway.invoke(Gateway.java:238)
 at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
 at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
 at py4j.GatewayConnection.run(GatewayConnection.java:238)
 at java.lang.Thread.run(Thread.java:748)

解决方法:在spark-env.shell中设置SPARK_LOCAL_IP=127.0.0.1后运行:
export SPARK_LOCAL_HOSTNAME=localhost

export SPARK_LOCAL_HOSTNAME=localhost

即可解决
解决centos7上初始化spark意外_第1张图片

你可能感兴趣的:(spark学习)