生产环境实战spark （6）分布式集群 5台设备 Scala安装

生产环境实战spark （6）分布式集群 Scala安装

1，scala官网下载scala安装到本地。

这里使用spark 2.1.x版本，而spark 2.1 版本需要scala2.1.1版本以上的兼容版本，因此下载安装 scala-2.11.8

下载地址：http://www.scala-lang.org/download/all.html

Spark runs on Java 7+, Python 2.6+/3.4+ and R 3.1+. For the Scala API, Spark 2.1.0 uses Scala 2.11. You will need to use a compatible Scala version (2.11.x).

2，从本地使用winscp工具将scala 2.11.8上传到master节点。

上传后检查

Last login: Wed Apr 19 09:27:17 2017 from 132.150.75.19
[root@master ~]# cd /usr/local
[root@master local]# ls
bin  etc  games  include  jdk1.8.0_121  lib  lib64  libexec  rhzf_setup_scripts  rhzf_spark_setupTools  sbin  share  src
[root@master local]# cd  rhzf_spark_setupTools
[root@master rhzf_spark_setupTools]# ls
jdk-8u121-linux-x64.tar.gz  scala-2.11.8.zip
[root@master rhzf_spark_setupTools]# 

3，解压scala进行安装。

[root@master rhzf_spark_setupTools]# unzip scala-2.11.8.zip
Archive:  scala-2.11.8.zip
   creating: scala-2.11.8/
   creating: scala-2.11.8/man/
   creating: scala-2.11.8/man/man1/
  inflating: scala-2.11.8/man/man1/scala.1  
  inflating: scala-2.11.8/man/man1/scalap.1  
  inflating: scala-2.11.8/man/man1/fsc.1  
  inflating: scala-2.11.8/man/man1/scaladoc.1  
  inflating: scala-2.11.8/man/man1/scalac.1  
   creating: scala-2.11.8/bin/
  inflating: scala-2.11.8/bin/scalac  
  inflating: scala-2.11.8/bin/fsc    
  inflating: scala-2.11.8/bin/fsc.bat  
  inflating: scala-2.11.8/bin/scala  
  inflating: scala-2.11.8/bin/scalap  
  inflating: scala-2.11.8/bin/scaladoc.bat  
  inflating: scala-2.11.8/bin/scaladoc  
  inflating: scala-2.11.8/bin/scalac.bat  
  inflating: scala-2.11.8/bin/scala.bat  
  inflating: scala-2.11.8/bin/scalap.bat  
   creating: scala-2.11.8/doc/
   creating: scala-2.11.8/doc/tools/
  inflating: scala-2.11.8/doc/tools/index.html  
  inflating: scala-2.11.8/doc/tools/scalap.html  
   creating: scala-2.11.8/doc/tools/images/
  inflating: scala-2.11.8/doc/tools/images/scala_logo.png  
 extracting: scala-2.11.8/doc/tools/images/external.gif  
  inflating: scala-2.11.8/doc/tools/scala.html  
   creating: scala-2.11.8/doc/tools/css/
  inflating: scala-2.11.8/doc/tools/css/style.css  
  inflating: scala-2.11.8/doc/tools/fsc.html  
  inflating: scala-2.11.8/doc/tools/scalac.html  
  inflating: scala-2.11.8/doc/tools/scaladoc.html  
  inflating: scala-2.11.8/doc/README  
  inflating: scala-2.11.8/doc/LICENSE.md  
   creating: scala-2.11.8/doc/licenses/
  inflating: scala-2.11.8/doc/licenses/mit_jquery-ui.txt  
  inflating: scala-2.11.8/doc/licenses/mit_sizzle.txt  
  inflating: scala-2.11.8/doc/licenses/apache_jansi.txt  
  inflating: scala-2.11.8/doc/licenses/bsd_asm.txt  
  inflating: scala-2.11.8/doc/licenses/mit_tools.tooltip.txt  
  inflating: scala-2.11.8/doc/licenses/mit_jquery-layout.txt  
  inflating: scala-2.11.8/doc/licenses/mit_jquery.txt  
  inflating: scala-2.11.8/doc/licenses/bsd_jline.txt  
  inflating: scala-2.11.8/doc/License.rtf  
   creating: scala-2.11.8/lib/
  inflating: scala-2.11.8/lib/scala-parser-combinators_2.11-1.0.4.jar  
  inflating: scala-2.11.8/lib/scala-reflect.jar  
  inflating: scala-2.11.8/lib/scalap-2.11.8.jar  
  inflating: scala-2.11.8/lib/scala-continuations-library_2.11-1.0.2.jar  
  inflating: scala-2.11.8/lib/config-1.2.1.jar  
  inflating: scala-2.11.8/lib/akka-actor_2.11-2.3.10.jar  
  inflating: scala-2.11.8/lib/scala-continuations-plugin_2.11.8-1.0.2.jar  
  inflating: scala-2.11.8/lib/scala-actors-migration_2.11-1.1.0.jar  
  inflating: scala-2.11.8/lib/scala-xml_2.11-1.0.4.jar  
  inflating: scala-2.11.8/lib/jline-2.12.1.jar  
  inflating: scala-2.11.8/lib/scala-library.jar  
  inflating: scala-2.11.8/lib/scala-compiler.jar  
  inflating: scala-2.11.8/lib/scala-swing_2.11-1.0.2.jar  
  inflating: scala-2.11.8/lib/scala-actors-2.11.0.jar  
[root@master rhzf_spark_setupTools]# ls
jdk-8u121-linux-x64.tar.gz  scala-2.11.8  scala-2.11.8.zip

复制到/usr/local目录：
[root@master rhzf_spark_setupTools]# mv scala-2.11.8 /usr/local
[root@master rhzf_spark_setupTools]# cd /usr/local
[root@master local]# ls
bin  etc  games  include  jdk1.8.0_121  lib  lib64  libexec  rhzf_setup_scripts  rhzf_spark_setupTools  sbin  scala-2.11.8  share  src

4，配置linux环境变量。

vi  /etc/profile

export JAVA_HOME=/usr/local/jdk1.8.0_121
export SCALA_HOME=/usr/local/scala-2.11.8

export PATH=.:$PATH:$JAVA_HOME/bin:$SCALA_HOME/bin

在命令行中输入source  /etc/profile，使刚才修改的SCALA_HOME及PATH配置文件生效 。

[root@master local]# source  /etc/profile

5，单机master节点上scala安装完成，版本检查。

[root@master local]# scala -version
Scala code runner version 2.11.8 -- Copyright 2002-2016, LAMP/EPFL
[root@master local]# 

6，编写脚本，在woker的节点也安装scala。

[root@master rhzf_setup_scripts]# ls
rhzf_hosts_scp.sh  rhzf_ssh.sh
[root@master rhzf_setup_scripts]# vi rhzf_scala.sh


#!/bin/sh
for i in  238 239 240 241
do
scp   -rq /usr/local/scala-2.11.8  root@10.*.*.$i:/usr/local/scala-2.11.8
scp   -rq /etc/profile  [email protected].$i:/etc/profile
ssh   root@10.*.*.$i source /etc/profile

done

[root@master rhzf_setup_scripts]# ls
rhzf_hosts_scp.sh  rhzf_scala.sh  rhzf_ssh.sh
[root@master rhzf_setup_scripts]# chmod u+x rhzf_scala.sh
[root@master rhzf_setup_scripts]# ./rhzf_scala.sh
[root@master rhzf_setup_scripts]# 

7， 在woker的节点检查scala安装完成。

[root@worker01 ~]# scala -version
Scala code runner version 2.11.8 -- Copyright 2002-2016, LAMP/EPFL
[root@worker01 ~]# 

Last login: Tue Apr 18 12:52:16 2017 from master
[root@worker02 ~]# scala -version
Scala code runner version 2.11.8 -- Copyright 2002-2016, LAMP/EPFL
[root@worker02 ~]# 

Last login: Tue Apr 18 12:52:01 2017 from worker02
[root@worker03 ~]# scala -version
Scala code runner version 2.11.8 -- Copyright 2002-2016, LAMP/EPFL
[root@worker03 ~]# 

Last login: Tue Apr 18 12:08:22 2017 from 132.150.75.19
[root@worker04 ~]# scala -version
Scala code runner version 2.11.8 -- Copyright 2002-2016, LAMP/EPFL
[root@worker04 ~]#