Hadoop伪集群模式搭建MAC版[Linux依旧适用]

[TOC]

Hadoop

1 环境搭建

1.1 ssh免密登录

• 需要生产ssh key[可以忽略，但是启动的时候，每次都需要输入密码]

  首先在用户根目录下ls -a查看包括隐藏的文件夹，应该能发现.ssh，如果没有就安装brew ssh

  有了文件.ssh后， ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa

  cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

ssh: Could not resolve hostname localhost: nodename nor servname provided, or not known

• mac配置ssh远程登录：系统偏好设置-》共享-》远程登录

mac ssh: connect to host localhost port 22: Connection refused 

• 需要配置 修改文件/etc/hosts,在里面增加对127.0.0.1的解析[否则second namenode 启动不了，处理方法就是对bogon做个DNS解析指向127.0.0.1]

  127.0.0.1    bogon    localhost 
  

1.2 hadoop-env.sh

# 1.跳转到hadoop-3.14/etc/hadoop 目录
cd hadoop

# 2.修改hadoo-env.sh，mac中的open在窗口打开文件，可以设置默认打开软件
vim hadoop-env.sh 或者 open hadoop-env.sh

# 3.配置 JAVA_HOME[在文件对应的位置中配置]
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_271.jdk/Contents/Home

1.2 core-site.xml

vim core-site.xml

    
    
        fs.defaultFS
        hdfs://localhost:8020
    
    
    
        hadoop.tmp.dir
        /Users/ayden/Documents/repository/hadoop/tmp
    

1.3 hdfs-site.xml

vim hdfs-site.xml

    
    
        dfs.replication
        1
    
    
    
        dfs.http.address
        0.0.0.0:50070
    
    
    
        dfs.namenode.name.dir
        /Users/ayden/Documents/repository/hadoop/tmp/dfs/name
    
    
    
        dfs.datanode.data.dir
        /Users/ayden/Documents/repository/hadoop/tmp/dfs/data
    

1.4 Hadoop环境变量

vim ~/.bash_profile

# Hadoop
export HADOOP_HOME=/Users/ayden/Documents/soft/hadoop-3.1.4
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin

source ~/.bash_profile

1.5 启动

cd /Users/ayden/Documents/soft/hadoop-3.1.4/sbin

• 启动HDFS

  # 启动
  start-dfs.sh
  
  # 检查
  jps
  
  # 访问ui
  http://localhost:50070
  
  # 停止
  stop.sh
  

• 启动HADOOP

  # 启动
  start-all.sh 
  
  # 检查
  jps
  
  # 访问
  http://localhost:8088
  
  # 停止
  stop-all.sh
  

1.6 YARN 环境配置

• YARN配置

# 配置mapreduce在yarn上运行
vim mapred-site.xml

    
        mapreduce.framework.name
        yarn
    
    
        mapreduce.application.classpath
        $HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*
    

# 配置yarn中的mapreduce
vim yarn-site.xml

    
        yarn.nodemanager.aux-services
        mapreduce_shuffle
    
    
        yarn.nodemanager.env-whitelist
        JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME
    

• 启动YARN

start-yarn.sh

• 停止YARN

stop-yarn.sh

• YARN 界面

http://localhost:8088/cluster/scheduler