Centos7上安装hadoop伪集群

阅读更多

一、环境：

jdk home:        /opt/jdk1.8

user:                devops

hadoop home: /opt/hadoop-3.2.0  

hadoop version： 3.2.0

 

 二、安装配置：

 

$cd /opt
$wget http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-3.2.0/hadoop-3.2.0.tar.gz
$tar -xzf ./hadoop-3.2.0.tar.gz
$cd hadoop-3.2.0
#需要修改成对应的 jdk home
$sed -i 's/#  JAVA_HOME=\/usr\/java\/testing.*/JAVA_HOME=\/opt\/jdk1.8/g'  ./etc/hadoop/hadoop-env.sh

$./bin/hadoop version

$vi etc/hadoop/core-site.xml
#输入如下内容:
 
        fs.defaultFS
        hdfs://localhost:9000
 
 
 
$vi etc/hadoop/hdfs-site.xml
#输入如下内容:
  
        dfs.replication
        1
 
 
$vi etc/hadoop/mapred-site.xml
#输入如下内容:
 
        mapreduce.framework.name
        yarn
 
 
       yarn.app.mapreduce.am.env
       HADOOP_MAPRED_HOME=/opt/hadoop-3.2.0
 
 
       mapreduce.map.env
       HADOOP_MAPRED_HOME=/opt/hadoop-3.2.0
 
 
       mapreduce.reduce.env
       HADOOP_MAPRED_HOME=/opt/hadoop-3.2.0
 
 
$vi etc/hadoop/yarn-site.xml
输入如下内容:
 
        yarn.nodemanager.aux-services
        mapreduce_shuffle
 

 

 三、设置免密码登陆

 

 $ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
 $ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
 $cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
 $chmod 0600 ~/.ssh/authorized_keys
 $chmod 755 ~

    注设置完成后一定测试 ssh localhost 无需输入密码才继续往下可参考 https://zhoupinheng.iteye.com/admin/blogs/2436265   

四、启动测试

    

 $bin/hdfs namenode -format
 $sbin/start-dfs.sh
 
 $sbin/start-yarn.sh
   
 $bin/hdfs dfs -mkdir /user
 $bin/hdfs dfs -mkdir /user/devops
 
 $bin/hdfs dfs -put etc/hadoop /user/devops/input
 
 $bin/hdfs dfs -rm -f -r /user/devops/input/shellprofile.d
 $bin/hdfs dfs -rm -f -r /user/devops/output
 #运行测试例子
 $bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.0.jar wordcount input output
 #查看运行结果
 $bin/hdfs dfs -cat output/*
  
 sbin/stop-yarn.sh
 sbin/stop-dfs.sh