Spark实战(三)本地连接远程Spark(Python环境)
一、Python环境准备
远程服务器上Python版本要与本地想匹配，这里本地使用Anaconda来进行安装，远程服务器之间安装。
wget --no-check-certificate https://www.python.org/ftp/python/3.6.7/Python-3.6.7.tgz
tar -zxvf Python-3.6.7.tgz
sudo ./configure --prefix=/usr/local/python3
make & make install
建立软连接
mv /usr/bin/python /usr/bin/python.bak
ln -s /usr/local/python3/bin/python3.6 /usr/bin/python
本地之间下载Anaconda安装包，会自动安装相关环境。
二、配置PyCharm
在file->setting->project interpreter中新增一个python环境，
tool->Deployment->Configuration中新建一个SFTP
Mappings中Local path路径正斜杠要修改一下改为反斜杠，否则可能会无法识别导致报错
接下来需要在configuration中更改运行参数
因为是远程环境，Environment Variables中需要指明服务器端python路径，spark路径，java路径。
以下是配置中遇到的一些常见错误：
去上面搜索安装pyspark-stubs
上面错误表示py4j版本不兼容，重新下载想匹配的版本
spark FileNotFoundError: [WinError 2] 系统找不到指定的文件，需要查询edit configuration中python interpreter是否选择正确
python3安装完成后默认没有安装pip，需要自己去安装，否则会出先pip未找到的错误
备份原来的源：输入命令sudo cp /etc/apt/sources.list /etc/apt/sources_init.list
输入命令sudo vi /etc/apt/sources.list
添加如下
deb http://mirrors.aliyun.com/ubuntu/ trusty main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ trusty-security main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ trusty-updates main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ trusty-proposed main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ trusty-backports main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ trusty main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ trusty-security main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ trusty-updates main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ trusty-proposed main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ trusty-backports main restricted universe multiverse
sudo apt-get update
sudo apt-get upgrade
sudo apt-get install python3-pip
sudo apt-get install python-pip