pyspark 版本适配问题

Exception: Python in worker has different version 2.7 than that in driver 3.8, PySpark cannot run with different minor versions.Please check environment variables PYSPARK_PYTHON and PY


PySpark cannot run with different minor versions.Please check environment variables PYSPARK_PYTHON and PY)

原因是机器装着不同版本的python,pyspark不知道用哪个,所以代码需要配置要使用的版本即可,可通过设置环境变量来实现

import os
# 此处为自己环境中的python路径
os.environ['PYSPARK_PYTHON'] = '/usr/local/bin/python3' 

你可能感兴趣的:(pyspark,pyspark)