Spark单机版配置

安装scala：

• 使用root用户配置/etc/profile文件，并生效该配置：

export PATH=/usr/local/scala-2.12.3/bin:$PATH

• 将hadoop的权限修改为mylinux

sudo chown -R mylinux:mylinux hadoop-2.8.0/
sudo chown -R mylinux:mylinux ./scala-2.12.3/

Spark安装后，配置环境变量：

~$ vim .bashrc

export HADOOP_HOME=/usr/local/hadoop-2.8.0
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin
export HADOOP_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
# HADOOP PATH END

# SPARK PATH START
export SPARK_HOME=/usr/local/spark
export PATH=$PATH:$SPARK_HOME/bin
# SPARK PATH END

# Scala Path Start
export SCALA_HOME=/usr/local/scala-2.12.3
# Scala Path End

• 保存后,更新：
  source .bashrc
• 查看环境变量：
  echo $PATH

配置conf/spark-env.sh：

cd /usr/local/spark/conf/
vim spark-env.sh

export SPARK_MASTER_HOST=localhost
export SPARK_MASTER_PORT=7066
export SPARK_LOCAL_IP=127.0.0.10
export SPARK_WORKER_CORES=1
export SPARK_WORKER_INSTANCES=1
export SPARK_WORKER_MEMORY=512M

运行Spark

• 启动HDFS

start-dfs.sh
start-yarn.sh

• jps检查是否齐全

3761 ResourceManager
3383 DataNode
3576 SecondaryNameNode
3883 NodeManager
3260 NameNode
3934 Jps

• 启动start-all.sh

cd /usr/local/spark/
./sbin/start-all.sh

• jps 为

3761 ResourceManager
4417 Jps
3383 DataNode
4344 Worker
3576 SecondaryNameNode
4250 Master
3883 NodeManager
3260 NameNode

• 使用spark-shell连接集群
  spark-shell

通过 netstat -nlt 命令查看节点网络情况

默认地址：http://localhost:8080/