hadoop集群搭建

官方文档

hadoop

添加hadoop用户

useradd hadoop

/etc/hadoop/hadoop-env.sh

JAVA_HOME=${JAVA_HOME}
JAVA_HOME=/usr/local/java

配置文件

公共参数

core-site.xml

fs.defaultFS
hdfs://mini1:9000




hadoop.tmp.dir
/home/hadoop/hdpdata

hdfs配置（可直接使用默认值）

hdfs-site.xml

dfs.replication
2

mapreduce配置

mapred-site.xml

mapreduce.framework.name
yarn

yarn-site.xml

yarn.resourcemanager.hostname
mini1



yarn.nodemanager.aux-services
mapreduce_shuffle

hadoop添加环境变量

sudo vim /etc/profile

# HADOOP
export HADOOP_HOME=/home/hadoop/apps/hadoop-2.8.5
export PATH=$PATH:${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin

hdfs格式化

hadoop namenode -format

格式化成功截图.png

启动hadoop

namenode

hadoop-daemon start namenode
hadoop-daemon stop namenode
http://****:50070
(http://10.0.1.101:50070/dfshealth.html#tab-overview)
hadoop-daemon start datanode
hadoop-daemon stop datanode

批量启动

配置节点
vi etc/hadoop/slaves

mini2
mini3
mini4

执行命令
start-all.sh(不推荐)
start-hdfs.sh
start-yarn.sh

问题

[hadoop@mini1 hadoop]$ ssh-copy-id mini1

• /usr/bin/ssh-copy-id: ERROR: No identities found
  ~/.ssh/id_rsa id_rsa.pub文件不存在
  执行ssh-keygen 解决
• WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

web页面

http://ip:50070

image.png