CentOS8.2 详细安装、配置Spark3.0.0教程（Pyspark）亲自测试
sudo passwd：更改root密码
su：可以直接切换root密码
yum install -y update
yum groupinstall -y 'Development Tools'
yum install -y gcc openssl-devel bzip2-devel libffi-devel
yum -y install gcc pcre pcre-devel zlib zlib-devel openssl openssl-devel
----------------------------------------------------------------------------------
安装sqlite3:
sudo yum install sqlite-devel
----------------------------------------------------------------------------------
安装vim
yum -y install vim*
----------------------------------------------------------------------------------
安装python
下载和编译python3.8.3
（1）在官方网站下载最新版本python3.8.3
wget https://www.python.org/ftp/python/3.8.3/Python-3.8.3.tgz
mkdir /usr/local/python3
cd Python-3.8.3
./configure --prefix=/usr/local/python3 --enable-optimizations --with-ssl
make && make install
ln -s /usr/local/python3/bin/python3 /usr/local/bin/python3
ln -s /usr/local/python3/bin/pip3 /usr/local/bin/pip3
ln -s /usr/local/bin/python3 /usr/bin/python
pip3 install --upgrade pip
----------------------------------------------------------------------------------
安装redis6
wget http://download.redis.io/releases/redis-6.0.5.tar.gz
tar -zxvf redis-6.0.5.tar.gz
cd redis-6.0.5
make PREFIX=/usr/local/soft/redis install
mkdir /usr/local/soft/redis/conf
cp redis.conf /usr/local/soft/redis/conf/
mkdir -p /data/redis6
cd /data/redis6/
mkdir logs
mkdir data
cd /usr/local/soft/redis/conf
vi redis.conf修改配置文件
绑定访问的ip
bind 192.168.159.131（本机IP地址）
设置密码
requirepass 123456
使以daemon方式运行
daemonize yes
使用的最大内存数量
maxmemory 128MB
#日志保存目录
logfile "/data/redis6/logs/redis.log"
#数据保存目录
dir /data/redis6/data/
io线程数
io-threads 3
生成供systemd使用的service文件
vi /lib/systemd/system/redis6.service
[Unit]
Description=Redis
After=network.target
[Service]
Type=forking
PIDFile=/var/run/redis_6379.pid
ExecStart=/usr/local/soft/redis/bin/redis-server /usr/local/redis/conf/redis.conf
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s QUIT $MAINPID
PrivateTmp=true
[Install]
WantedBy=multi-user.target
重新加载service文件：systemctl daemon-reload
启动:systemctl start redis6
停止:systemctl stop redis6
打开端口6379：firewall-cmd --add-port=6379/tcp --permanent
重启防火墙：systemctl restart firewalld.service
/usr/local/soft/redis/bin/redis-cli -h 192.168.159.131 -p 6379 -a 123456
----------------------------------------------------------------------------------
yum list installed mysql*
rpm -qa | grep mysql*
2.下载mysql rpm
yum localinstall https://dev.mysql.com/get/mysql57-community-release-el7-9.noarch.rpm
3.安装
yum install -y mysql-server
4.设置开机启动Mysql
systemctl enable mysqld.service
5.检查是否已经安装了开机自动启动
systemctl list-unit-files | grep mysqld
6.设置开启服务
systemctl start mysqld.service
7.查看临时登录密码
grep 'temporary password' /var/log/mysql/mysqld.log
如果没有就不用输入密码
mysql -u root -p
8.修改root密码并生效
ALTER USER 'root'@'localhost' IDENTIFIED BY 'Sloveb55555..';
9.创建远程登录用户
CREATE USER 'root'@'%' IDENTIFIED BY 'Sloveb55555..';
10.更改用户加密规则
ALTER USER 'root'@'%' IDENTIFIED WITH mysql_native_password BY 'Sloveb55555..';
11.授权用户远程登录权限
grant all on *.* to 'root'@'%';
12.刷新配置
flush privileges;
13.添加3306端口
firewall-cmd --add-port=3306/tcp --permanent
14.重启防火墙
systemctl restart firewalld.service
----------------------------------------------------------------------------------
PyCharm Pro安装：
到官网下载
执行：tar -xzvf pycharm-professional-2020.1.1.tar.gz -C /usr/local/
执行：cd /usr/local/pycharm-2020.1.1/bin
执行：sh ./pycharm.sh
设置图标
sudo gedit /usr/share/applications/Pycharm.desktop
复制以下代码
[Desktop Entry]
Type=Application
Name=Pycharm
GenericName=pycharm-2020
Comment=Pycharm2020:The Python IDE
Exec="/home/new2020/下载/pycharm-2020.1.1/bin/pycharm.sh" %f
Icon=/home/new2020/下载/pycharm-2020.1.1/bin/pycharm.png
Terminal=pycharm
Categories=Pycharm;
打开以下目录
/usr/share/applications/
复制一份这个pycharm图标到桌面
---------------------------------------------------------------------------------
java自带1.8.0
---------------------------------------------------------------------------------
spark安装：
spark-3.0.0-bin-hadoop2.7.tgz
tar -zxvf spark-3.0.0-bin-hadoop2.7.tgz -C /usr/local/spark
解压到/usr/local/spark目录
配置环境变量并且添加以下配置代码
vim /etc/profile
export SPARK_HOME=/usr/local/spark
export PATH=${PATH}:$SPARK_HOME/bin
export PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.10.7-src.zip:$PYTHONPATH
保存退出执行生效
source /etc/profile
执行：./bin/pyspark
---------------------------------------------------------------------------------
安装pip
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
---------------------------------------------------------------------------------
安装numpy
pip3 install numpy
---------------------------------------------------------------------------------
安装 py4j
pip3 install py4j
降低log4j级别
进入spark-2.4.0-bin-hadoop2.7下面的conf目录
cp conf/log4j.properties.template conf/log4j.properties
vim conf/log4j.properties
将 log4j.rootCategory=INFO, console
替换为 log4j.rootCategory=ERROR, console
保存退出
-------------------------------------------------------------------------------
pip3 install redis