【分布式】hadoop安装篇（7）Hadoop 3的伪分布模式安装

准备

$ sudo apt-get install ssh
$ sudo apt-get install pdsh

我们还需要设置pdsh的rcmd，执行下面内容。或创建/etc/pdsh/rcmd_default，里面填入ssh

echo "ssh" > /etc/pdsh/rcmd_default

如果这步骤不设置，我们在启动hdfs时，可能会报告

gsta@gsta005:~/wei/hadoop/hadoop-3.1.0/sbin$ start-dfs.sh
Starting namenodes on [gsta005]
pdsh@gsta005: gsta005: rcmd: socket: Permission denied
Starting datanodes
pdsh@gsta005: localhost: rcmd: socket: Permission denied
Starting secondary namenodes [gsta005]
pdsh@gsta005: gsta005: rcmd: socket: Permission denied

设置免密码登录的ssh

$ ssh-keygen -t rsa -P  -f ~/.ssh/id_rsa
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
$ chmod 0600 ~/.ssh/authorized_keys
验证
$ ssh localhost

配置和启动hdfs

etc/hadoop/core-site.xml

    
        fs.defaultFS
        hdfs://191.8.2.45:9000
    
    
    
        hadoop.tmp.dir
        /home/gsta/wei/hadoop/hadoop-3.1.0/tmp
    

etc/hadoop/hdfs-site.xml

    
        dfs.replication
        1
    

格式化hdfs

$ hdfs namenode -format

启动hdfs

$ start-hdfs.sh
设置相关的文件夹 /user,/user/，命令同v2版本

开启web监控

http://191.8.2.45:9870

如果启动的过程中报错：

gsta@gsta005:~/wei/hadoop$ start-dfs.sh
Starting namenodes on [gsta005]
gsta005: ERROR: JAVA_HOME is not set and could not be found.
pdsh@gsta005: gsta005: ssh exited with exit code 1
Starting datanodes
localhost: ERROR: JAVA_HOME is not set and could not be found.
pdsh@gsta005: localhost: ssh exited with exit code 1
Starting secondary namenodes [gsta005]
gsta005: ERROR: JAVA_HOME is not set and could not be found.
pdsh@gsta005: gsta005: ssh exited with exit code 1

则在hadoop-env.sh中指定JAVA_HOME

export JAVA_HOME=/home/wei/jdk1.8.0_66

配置和启动yarn

etc/hadoop/mapred-site.xml：

        
                mapreduce.framework.name
                yarn
        

        
                mapreduce.application.classpath
                $HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*
        

etc/hadoop/yarn-site.xml：

  
    yarn.nodemanager.aux-services
    mapreduce_shuffle
  
  
    yarn.nodemanager.env-whitelist
    JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME
  

启动yarn

start-yarn.sh

web监控查看

http://191.8.2.45:8088/