【大数据】黑马hadoop学习笔记 集群搭建

 

搭建 

配置hadoop环境

安装编译相关的依赖 

yum install gcc gcc-c++ make autoconf automake libtool curl lzo-devel zlib-devel openssl openssl-devel ncurses-devel snappy snappy-devel bzip2 bzip2-devel lzo lzo-devel lzop libXtst zlib -y
yum install -y doxygen cyrus-sasl* saslwrapper-devel*

安装cmake

安装snappy

安装jdk

需要指向source /etc/profile才能保存环境变量设置

安装maven 

安装ProtocolBuffer 3.7.1

编译hadoop

搭建集群(Hadoop环境可以直接解压黑马的包)角色规划 

角色规划

环境准备

拷贝(我已经安装了jdk)

修改配置文件

hdfs-site.xml

	
    dfs.namenode.secondary.http-address
    node2:9868

 mapred-site.xml

mapreduce.framework.name
yarn



mapreduce.jobhistory.address
 node1:10020



mapreduce.jobhistory.webapp.address
node1:19888


yarn.app.mapreduce.am.env
HADOOP_MAPRED_HOME=${HADOOP_HOME}


mapreduce.map.env
HADOOP_MAPRED_HOME=${HADOOP_HOME}


mapreduce.reduce.env
HADOOP_MAPRED_HOME=${HADOOP_HOME}

yarn-site.xml

yarn.resourcemanager.hostname
node1


yarn.nodemanager.aux-services
mapreduce_shuffle



yarn.nodemanager.pmem-check-enabled
false



yarn.nodemanager.vmem-check-enabled
false



yarn.log-aggregation-enable
true



yarn.log.server.url
http://node1:19888/jobhistory/logs



yarn.log-aggregation.retain-seconds
604800

● workers

node1.itcast.cn
node2.itcast.cn
node3.itcast.cn

环境变量

格式化

启动

初体验