hadoop 3.x 伪分布式搭建

hadoop 伪分布式搭建

环境

• CentOS 7
• jdk 1.8
• hadoop 3.3.6

1. 准备

1. 准备环境所需包
2. 上传所有压缩包到服务器

2. 安装jdk

# 解压jdk到/usr/local目录下
tar -xvf jdk-8u431-linux-x64.tar.gz -C /usr/local

先不着急配置java环境变量，后面和hadoop一起配置

3. 安装hadoop

# 解压hadoop到/usr/local目录下
tar -xvf hadoop-3.3.6.tar.gz -C /usr/local

4. 配置hadoop

进入目录 /usr/local/hadoop-3.3.6/sbin，修改如下文件

• mapred-site.xml

  <configuration>
  	<property>
  		
  		<name>mapreduce.framework.namename>
          <value>yarnvalue>
      property>
  configuration>
  

• core-site.xml

  <configuration>
  	
  	<property>
  		<name>hadoop.tmp.dirname>
  		<value>file:/usr/local/hadoop-3.3.6/tmpvalue>
  		<description>Abase for other temporary directories.description>
  	property>
  	
  	<property>
  		<name>fs.defaultFSname>
  		<value>hdfs://localhost:9000value>
  	property>
  	
  	<property>
  		<name>hadoop.http.staticuser.username>
        <value>rootvalue>
    property>
  configuration>
  

• hdfs-site.xml

  <configuration>
    
    <property>
        <name>dfs.replicationname>
        <value>1value>
    property>
    
    <property>
        <name>dfs.namenode.name.dirname>
        <value>file:/usr/local/hadoop-3.3.6/tmp/dfs/namevalue>
    property>
    
    <property>
        <name>dfs.datanode.data.dirname>
        <value>file:/usr/local/hadoop-3.3.6/tmp/dfs/datavalue>
    property>
    
    <property>
        <name>dfs.http.addressname>
        <value>0.0.0.0:50070value>
    property>
  configuration>
  

• yarn-site.xml

  <configuration>
  
    
    <property>
        
        <name>yarn.nodemanager.aux-servicesname>
        <value>mapreduce_shufflevalue>
    property>
  configuration>
  

修改编辑开始和关闭脚本，进入目录/usr/local/hadoop/hadoop-3.3.6/sbin，修改文件如下

注：所有添加内容，都是在文件最开头的 #/usr/bin/env bash 下

• start-dfs.sh

  HDFS_DATANODE_USER=root 
  HADOOP_SECURE_DN_USER=hdfs
  HDFS_NAMENODE_USER=root
  HDFS_SECONDARYNAMENODE_USER=root
  

• stop-dfs.sh

  HDFS_DATANODE_USER=root 
  HADOOP_SECURE_DN_USER=hdfs
  HDFS_NAMENODE_USER=root
  HDFS_SECONDARYNAMENODE_USER=root
  

• start-yarn.sh

  YARN_RESOURCEMANAGER_USER=root
  HADOOP_SECURE_DN_USER=yarn
  YARN_NODEMANAGER_USER=root
  

• stop-yarn.sh

  YARN_RESOURCEMANAGER_USER=root
  HADOOP_SECURE_DN_USER=yarn
  YARN_NODEMANAGER_USER=root
  

修改hadoop-env环境变量，进入目录/usr/local/hadoop-3.3.6/etc/hadoop，打开 hadoop-env.sh 文件追加内容如下

export JAVA_HOME=/usr/local/jdk1.8.0_431
export HADOOP_CONF_DIR=/usr/local/hadoop-3.3.6/etc/hadoop

5. 编辑环境变量

# 打开环境变量文件
vim /etc/profile

文件末尾追加内容

# Java
export JAVA_HOME=/usr/local/jdk1.8.0_431
export MAVEN_HOME=/usr/local/apache-maven-3.9.0
export CLASSPATH=.:${JAVA_HOME}/jre/lib/rt.jar:${JAVA_HOME}/lib/dt.jar:${JAVA_HOME}/lib/tools.jar

# Hadoop
export HADOOP_HOME=/usr/local/hadoop-3.3.6/
export PATH=$PATH:${JAVA_HOME}/bin:${MAVEN_HOME}/bin:${HADOOP_HOME}/bin

执行命令，使其生效

source /etc/profile

6. SSH免密登录设置

# 本地连接本地
ssh localhost
# 退出
exit
# 进入root目录
cd ~/.ssh
# 执行命令，三次回车
ssh-keygen -t rsa
cat id_rsa.pub >> authorized_keys

7.启动

cd /usr/local/hadoop/hadoop-3.3.6/etc/hadoop
# 初始化
hadoop namenode -format
# 启动hadoop
./start-all.sh
# 查看进程
jps

不出意外，结果如下

12720 ResourceManager
15072 Jps
14002 NameNode
14754 NodeManager
14389 SecondaryNameNode
14173 DataNode

用浏览器访问地址：http://[服务器IP]:50070，http://[服务器IP]:8088