数据仓库环境准备-Hadoop篇

大数据软件版本说明:

hadoop-3.1.4、zookeeper-3.5.8、kafka_2.12-2.6.0、flume-1.9.0、sqoop-1.4.6、hive-3.1.2、mysql-5.7.31-1.el7、spark-3.0.0

一、JDK安装 

1.移除OpenJDK命令:sudo rpm -qa | grep -i java | xargs -n1 sudo rpm -e --nodeps

2.修改/opt目录权限: sudo chmod -r 777 /opt

3.解压jdk至目录: tar -zxvf jdk-8u171-linux-x64.tar.gz -C /opt/module/

4.配置环境变量: sudo vim /etc/profile.d/my_env.sh

5.my_env.sh: 

#JAVA_HOME 

export JAVA_HOME=/opt/module/jdk1.8.0_171 

export PATH=$PATH:$JAVA_HOME/bin

6.source /etc/profile.d/my_env.sh

二、Hadoop配置:

fs.defaultFS

hdfs://hadoop102:8020

hadoop.tmp.dir

/opt/module/hadoop-3.1.4/data

hadoop.http.staticuser.user

linan

hadoop.proxyuser.linan.groups

*

hadoop.proxyuser.linan.hosts

*

hadoop.proxyuser.linan.users

*

dfs.namenode.http-address

hadoop102:9870

dfs.namenode.secondary.http-address

hadoop104:9868

dfs.replication

1

yarn.nodemanager.aux-services

mapreduce_shuffle

yarn.resourcemanager.hostname

hadoop103

yarn.nodemanager.env-whitelist

JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME

yarn.scheduler.minimum-allocation-mb

512

yarn.scheduler.maximum-allocation-mb

4096

yarn.nodemanager.resource.memory-mb

4096

yarn.nodemanager.pmem-check-enabled

false

yarn.nodemanager.vmem-check-enabled

false

yarn.log-aggregation-enable

true

yarn.log.server.url

http://hadoop102:19888/jobhistory/logs

yarn.log-aggregation.retain-seconds

604800

mapreduce.framework.name

yarn

mapreduce.jobhistory.address

hadoop102:10020

mapreduce.jobhistory.webapp.address

hadoop102:19888

/opt/module/hadoop-3.1.4/etc/hadoop/workers:

hadoop102

hadoop103

hadoop104

/opt/module/hadoop-3.1.4/etc/hadoop/hadoop-env.sh:

export JAVA_HOME=/opt/module/jdk1.8.0_171

export HADOOP_HOME=/opt/module/hadoop-3.1.4

export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop

//格式化namenode

rm -rf logs/ data/

bin/hdfs namenode -format

启动:

sbin/start-dfs.sh

sbin/start-yarn.sh

停止:

sbin/stop-dfs.sh

sbin/stop-yarn.sh

测试:

hadoop jar /opt/module/hadoop-3.1.4/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.4.jar pi 1 1

//批量显示脚本xcall:

#!/bin/bash

params=$@

i=2

for((i=2 ; i <= 4 ; i = $i + 1)) ; do

    echo ==============hadoop10$i $params =============

    ssh hadoop10$i "source /etc/profile;$params"

done

集群数据均衡

1、节点间数据均衡

开启数据均衡命令：start-balancer.sh -threshold 10

停止数据均衡命令：stop-balancer.sh

2、磁盘间数据均衡(hadoop3才有)

1)生成均衡计划

hdfs diskbalancer -plan hadoop103

2)执行均衡计划

hdfs diskbalancer -execute hadoop103.plan.json

3)查看当前均衡任务的执行情况

hdfs diskbalancer -query hadoop103

4)取消均衡计划

hdfs diskbalancer -cancel hadoop103.plan.json

Hadoop支持lzo压缩配置

lzo编译源码地址:

https://github.com/twitter/hadoop-lzo

https://www.oberhumer.com/opensource/lzo/

编译lzo源码生成hadoop-lzo-0.4.21.jar包

将编译好的hadoop-lzo-0.4.21.jar放入/opt/module/hadoop-3.1.4/share/hadoop/common目录下

参考地址:

(1)https://wenku.baidu.com/view/61a42f9f0875f46527d3240c844769eae009a3f4.html

(2)https://blog.csdn.net/s_alics/article/details/108513408

core-site配置支持lzo

io.compression.codecs

org.apache.hadoop.io.compression.SnappyCodec,

com.hadoop.compression.lzo.LzoCodec,

com.hadoop.compression.lzo.LzopCodec,

io.compression.codec.lzo.class

com.hadoop.compression.lzo.LzoCodec

测试案例

1、

hadoop fs -mkdir /input

hadoop fs -put word.txt /input

hadoop jar /opt/module/hadoop-3.1.4/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.4.jar wordcount -Dmapreduce.output.fileoutputformat.compress=true -Dmapreduce.output.fileoutputformat.compress.codec=com.hadoop.compression.lzo.LzopCodec /input /output

使用lzo压缩方式支持切片需先创建lzo文件索引

例：bigtable.lzo文件

hadoop jar /opt/module/hadoop-3.1.4/share/hadoop/common/hadoop-lzo-0.4.21.jar

com.hadoop.compression.lzo.DistributedLzoIndexer /input /bigtable.lzo

HDFS调优

hdfs-site:

dfs.namenode.handler.count

21

公式:dfs.namenode.handler.count = 20 * log小e3 = 21

yarn-site:

dfs.namenode.handler.count

21

基准测试

1）hdfs写性能

hadoop jar /opt/module/hadoop-3.1.4/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.1.4-tests.jar TestDFSIO -write -nrFiles 10 -fileSize 128MB

2）hdfs读性能

hadoop jar /opt/module/hadoop-3.1.4/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.1.4-tests.jar TestDFSIO -read -nrFiles 10 -fileSize 128MB

3）删除测试数据

hadoop jar /opt/module/hadoop-3.1.4/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.1.4-tests.jar TestDFSIO -clean