Hadoop3.3.4伪分布式环境搭建

---

前言

hadoop学习——伪分布式环境——普通用户搭建

---

一、准备

1. 下载Hadoop

2. 配置环境变量

vi ~/.bash_profile

# 增加如下配置
export HADOOP_HOME=/home/install/hadoop-3.3.4
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

# 使配置生效
source ~/.bash_profile

3. 配置免密

二、Hadoop配置

配置的目录是$HADOOP_HOME/etc/hadop

1. hadoop-env.sh

修改JAVA_HOME，设置为JDK所在的位置

# Linux命令行输出 JDK 位置
echo $JAVA_HOME

2. hdfs-site.xml

<property>
    <name>dfs.replicationname>
    <value>1value>
property>
<property>
    <name>dfs.http.addressname>
    <value>hadoop01:50070value>
property>

3. core-site.xml

 
<property>
    <name>fs.defaultFSname>
    <value>hdfs://hadoop01:9000value>
property>

<property>
    <name>hadoop.tmp.dirname>
    <value>/home/install/data/hadoop/HADOOP_TMP_DIRvalue>
    <description>Abase for other temporary directories.description>
property>

<property>
    <name>hadoop.proxyuser.hduser.groupsname>
    <value>*value>
property>

<property>
    <name>io.file.buffer.sizename>
    <value>131072value>
property>

<property>
    <name>hadoop.proxyuser.hduser.hostsname>
    <value>*value>
property>

4. mapred-site.xml

<property>
    <name>mapreduce.framework.namename>
    <value>yarnvalue>
property>
<property>
    <name>mapreduce.job.counters.maxname>
    <value>1200000value>
property>
<property>
    <name>mapreduce.map.memory.mbname>
    <value>4096value>
property>
<property>
    <name>mapreduce.reduce.memory.mbname>
    <value>8192value>
property>
<property>
    <name>mapreduce.map.java.optsname>
    <value>-Xmx3072mvalue>
property>
<property>
    <name>mapreduce.reduce.java.optsname>
    <value>-Xmx6144mvalue>
property>

5. yarn-site.xml

<property>
    <name>yarn.nodemanager.aux-servicesname>
    <value>mapreduce_shufflevalue>
property>
<property>
    <name>yarn.resourcemanager.hostnamename>
    <value>hadoop01value>
property>
<property>
    <name>yarn.nodemanager.vmem-pmem-rationame>
    <value>4value>
property>
<property>
    <name>yarn.nodemanager.vmem-check-enabledname>
    <value>falsevalue>
property>
<property>
    <name>yarn.nodemanager.resource.memory-mbname>
    <value>20480value>
property>
<property>
    <name>yarn.scheduler.minimum-allocation-mbname>
    <value>1024value>
property>
<property>
    <name>yarn.scheduler.maximum-allocation-mbname>
    <value>20480value>
property>

三、格式化

hadoop namenode -format

四、启动

# 懒一点的方式
start-all.sh

五、访问web页面

http://localhost:50070
如果没有配置，默认端口是9870

---