Exception: Randomness of hash of string should be disabled via PYTHONHASHSEED
at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRDD.scala:193)
at org.apache.spark.api.python.PythonRunner$$anon$1.<init>(PythonRDD.scala:234)
at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:152)
at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:63)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.api.python.PairwiseRDD.compute(PythonRDD.scala:390)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
... 1 more
经过排查,发现是python 的 PYTHONHASHSEED 环境变量引起的问题。
新版的python为了安全性,在hash计算时加入随加盐,在单会话下,加的盐是一致的,所以hash出来的结果一致。而在集群多台服务器一起计算时,每台服务器有各自的会话,大家都加随机加盐,结果导致各自的hash结果不一致。从而引起程序报错。
解决方法很简单,关闭随机加盐,或设置相同的盐值。
方法一:
python代码里设转置环境变量
conf = SparkConf()
conf.set("spark.executorEnv.PYTHONHASHSEED","0");
方法二:
修改spark-defaults.conf,增加
spark.executorEnv.PYTHONHASHSEED=0