pycharm 远程连接运行pyspark

1. pycharm 新建项目，添加如下远程 ssh interpreter 解释器，输入连接要连接的host 和username
   
   
   
   
    

2. 将linux 中spark 下的pyspark 复制到python 中

由于是spark2.0 因此只支持python3.5以下

cp -r  /home/software/spark-2.0.1-bin-hadoop2.7/python/pyspark  /usr/local/python3.5/lib/python3.5/site-packages/

 

3. 在pycharm 中编写code，添加JAVA_HOME  PYSPARK_PYTHON SPARK_HOME变量为linux 中对应的环境变量，

 

 

import os
JAVA_HOME = '/home/software/jdk1.8.0_141'
PYSPARK_PYTHON = "/usr/bin/python3.5"
SPARK_HOME='/home/software/spark-2.0.1-bin-hadoop2.7'
os.environ["SPARK_HOME"] = SPARK_HOME
os.environ["JAVA_HOME"] = JAVA_HOME
os.environ["PYSPARK_PYTHON"] = PYSPARK_PYTHON
os.environ["PYSPARK_DRIVER_PYTHON"] = PYSPARK_PYTHON

from pyspark import SparkContext
if __name__ == '__main__':
    #创建spark context
    sc = SparkContext('local[2]','wordcount')
    #通过spark context 获取rdd
    rdd1 = sc.textFile('file:///root/tmpdata/test.txt')
    rdd2 = rdd1.flatMap(lambda line:line.split())
    rdd3 = rdd2.map(lambda x:(x,1))
    rdd4 = rdd3.reduceByKey(lambda x,y:x+y)
    print(rdd4.collect())

4 运行结果