基础架构系列篇-基于win10安装SPARK3与HADOOP环境

基础架构系列篇-基于win10安装SPARK3与HADOOP环境(这里版本是 spark-3.1.2 hadoop-3.2.0)

1. 下载安装spark (这里版本是 spark-3.1.2-bin-hadoop3.2.tgz)

地址: http://spark.apache.org/downloads.html
https://downloads.apache.org/spark
https://archive.apache.org/dist/spark
解压并设置环境变量SPARK_HOME
增加PATH %SPARK_HOME%\bin %SPARK_HOME%\sbin
打开POWERSHELL测试spark-shell

2.下载安装hadoop (按SPARK版本选择)

地址: https://archive.apache.org/dist/hadoop/common/hadoop-3.2.0/
http://archive.apache.org/dist/hadoop/core/
解压并设置环境变量HADOOP_HOME
增加PATH %HADOOP_HOME%\bin %HADOOP_HOME%\sbin

下载WINUTILS (按HADOOP版本选择)
地址: https://github.com/cdarlint/winutils
拷贝WINUTILS的BIN内容放HADOOP解压的BIN路径
(可不拷贝WINUTILS的BIN的hadoop.dll放c盘windows/sysyem32下)
打开POWERSHELL测试hadoop version

设置hadoop参数 路径etc/hadoop
hadoop-env.cmd (可不改)
@set JAVA_HOME=%JAVA_HOME%
set JAVA_HOME=安装的JDK路径

core-site.xml

 

    fs.default.name
    hdfs://localhost:8900


    hadoop.tmp.dir
    /F:/SW/BIGdata/hadoop/hadoop-3.2.0/data/tmp/${user.name}
    A base for other temporary directories.

hdfs-site.xml

    dfs.replication
    1


    dfs.permissions
    false


    dfs.namenode.name.dir
    /F:/SW/BIGdata/hadoop/hadoop-3.2.0/data/dfs/namenode


    dfs.datanode.data.dir
    /F:/SW/BIGdata/hadoop/hadoop-3.2.0/data/dfs/datanode


    fs.checkpoint.dir
    /F:/SW/BIGdata/hadoop/hadoop-3.2.0/data/dfs/snn


    fs.checkpoint.edits.dir
    /F:/SW/BIGdata/hadoop/hadoop-3.2.0/data/dfs/snn

yarn-site.xml

 
 
    yarn.nodemanager.aux-services
    mapreduce_shuffle


    yarn.nodemanager.aux-services.mapreduce.shuffle.class
    org.apache.hadoop.mapred.ShuffleHandler

mapred-site.xml

 
  
    mapreduce.framework.name
    yarn

格式化NameNode
hdfs namenode -format
(版本hadoop-3.2.1异常ERROR namenode.NameNode: Failed to start namenode.
java.lang.UnsupportedOperationException at java.nio.file.Files.setPosixFilePermissions(Files.java:2044))
修改用hadoop-3.2.0版本

启动Hadoop服务
执行start-all
停止stop-all

3. 测试查看

yarn：http://127.0.0.1:8088/cluster
hdfs：http://localhost:9870