Centos7.8+CDH6.2.1从零开始安装
前言，为什么我要开始重新装一遍呢，因为，我攒了个台式机，还没有装CDH，学习需要用，顺道记录一下安装记录，当做以后的笔记，以后查漏补缺。
一、CDH是什么？
CDH是Cloudera的100%开源平台发行版，包括Apache Hadoop，专门为满足企业需求而构建。CDH提供了企业使用所需的一切。通过将Hadoop与十几个其他关键的开源项目集成，Cloudera创建了一个功能先进的系统，帮助您执行端到端的大数据工作流。
1.1 CDH的重点项目有哪些？
相信大家对这些大数据组件应该都是很熟悉了，那么我们接下来步入正题，进行安装我们的CDH。
二、提前准备
2.1 安装前需要准备环境以及文件
2.1.1 Linux镜像
这里我使用的linux镜像是CentOS-7-x86_64-DVD-2003.iso，如果大家没有，可以选择以下方式进行下载，或者选用自己的Linux镜像。
- 进入阿里云站点：Linux镜像下载
2.1.2 JDK1.8+ Linux版本tar包
百度云连接：https://pan.baidu.com/s/1Re07mLl0t3X17n74GVyUSQ
提取码：g7ve
2.1.3 CDH安装包
链接：https://pan.baidu.com/s/1XVBJpObR4O3b0FVScyaOiA
提取码：or98
2.1.4 一台能够上网的计算机
这里多一句嘴，我以前用的是win7，我感觉win7确实是很好用，但是后来因为一些原因，改换win10，我尝试过很多的win10版本，发现最好的是Windows 10 Enterprise LTSC 2019 (x64) - DVD (Chinese-Simplified)
第一他会更新，但是并不会像别的win10那样进行强制更新，这个更新只会在一个月一次，或者不更新，而且是特别纯净，无任何自带软件。
- 进入我告诉你 Windows 10 Enterprise LTSC 2019
2.1.5 VMware
https://blog.csdn.net/weixin_43704599/article/details/106441149
三、环境准备
3.1 Linux机器配置
3.1.1 机器规划
| hostname | IP | CPU | 内存 | 硬盘 |
|---|---|---|---|---|
| cdh01 | 192.168.1.101 | 2c2核 | 16G | 100G |
| cdh02 | 192.168.1.102 | 1c2核 | 8G | 100G |
| cdh03 | 192.168.1.103 | 1c2核 | 8G | 100G |
| cdh04 | 192.168.1.104 | 1c1核 | 4G | 100G |
因为我的台式机配置比较高，可根据自己的实际情况自定义搭配
3.1.2 Hosts配置
[root@cdh01 ~]# vi /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.101 cdh01
192.168.1.102 cdh02
192.168.1.103 cdh03
192.168.1.104 cdh04
3.1.3 同步脚本
所有节点安装rsync
[root@cdh01 .ssh]# cd /root/
[root@cdh01 ~]# mkdir bin
[root@cdh01 ~]# cd bin/
[root@cdh01 bin]# touch xsync
[root@cdh01 bin]# vi xsync
#!/bin/bash
#1 获取输入参数个数，如果没有参数，直接退出
pcount=$#
if((pcount==0)); then
echo no args;
exit;
fi
#2 获取文件名称
p1=$1
fname=`basename $p1`
echo fname=$fname
#3 获取上级目录到绝对路径
pdir=`cd -P $(dirname $p1); pwd`
echo pdir=$pdir
#4 获取当前用户名称
user=`whoami`
#5 循环
for host in 01 02 03 04
do
echo ------------------- cdh$host --------------
rsync -av $pdir/$fname $user@cdh$host:$pdir
done
[root@cdh01 bin]# chmod u+x xsync
3.1.4 配置免密
[root@cdh01 ~]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:hWgtOj7f96xCfIvZtFvGX6mvga455kacgD2ePajP3qQ root@cdh01
The key's randomart image is:
+---[RSA 2048]----+
| |
| o . |
| * o . |
| + = . |
| o ..S . |
| . . +o*o.. .|
| o ...B.++. ..|
| +..B+B+ .o. |
| +E=B==oo+. |
+----[SHA256]-----+
[root@cdh01 .ssh]# ssh-copy-id cdh01
[root@cdh01 ~]# xsync .ssh/
3.1.5 修改yum源
cd /etc/yum.repos.d/
yum install -y wget
mv CentOS-Base.repo CentOS-Base.repo.bak
wget http://mirrors.aliyun.com/repo/Centos-7.repo
mv Centos-7.repo CentOS-Base.repo
yum clean all
yum makecache
3.1.6 关闭SELINUX（所有节点）
[root@cdh01 ~]# vi /etc/selinux/config
将这一行修改为 SELINUX=disabled
[root@cdh01 ~]# xsync /etc/selinux/config
3.1.7 配置集群时间同步
- 安装ntp（所有节点）
[root@cdh01 ~]# yum install ntp -y
- 在主节点配置时间同步
[root@cdh01 ~]# vi /etc/ntp.conf
driftfile /var/lib/ntp/drift
# Permit time synchronization with our time source, but do not
# permit the source to query or modify the service on this system.
restrict default nomodify notrap nopeer noquery
# Permit all access over the loopback interface. This could
# be tightened as well, but to do so would effect some of
# the administrative functions.
restrict 127.0.0.1
restrict ::1
# Hosts on local network are less restricted.
restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst
server 127.127.1.0
fudge 127.127.1.0 stratum 10
- 修改系统时间与硬件时间同步
vi /etc/sysconfig/ntpd
# Command line options for ntpd
OPTIONS="-g"
SYNC_HWCLOCK=yes
- 配置其他节点时间同步
crontab -e
*/1 * * * * /usr/sbin/ntpdate cdh01
- 开启自启（所有节点）
systemctl start ntpd
systemctl enable ntpd
3.1.8 优化虚拟内存（所有节点）
[root@cdh01 ~]# echo 'vm.swappiness = 0' > /etc/sysctl.d/swappiness.conf
[root@cdh01 ~]# sysctl -p
3.1.9 禁用大叶内存（所有节点）
[root@cdh01 ~]# echo never > /sys/kernel/mm/transparent_hugepage/defrag
[root@cdh01 ~]# echo never > /sys/kernel/mm/transparent_hugepage/enabled
[root@cdh01 ~]# echo "echo never > /sys/kernel/mm/transparent_hugepage/enabled" >> /etc/rc.local
[root@cdh01 ~]# echo "echo never > /sys/kernel/mm/transparent_hugepage/defrag" >> /etc/rc.local
[root@cdh01 ~]# chmod u+x /etc/rc.local
3.1.11 重启所有节点(所有节点关闭防火墙)
[root@cdh01 ~]# systemctl stop firewalld.service
[root@cdh01 ~]# systemctl disable firewalld.service
3.1.11 重启所有节点
reboot
3.2 MySQL部署
3.2.1 MySQL安装
- 下载MySQL仓库和安装MySQL
wget -i -c http://dev.mysql.com/get/mysql57-community-release-el7-10.noarch.rpm
yum -y install mysql57-community-release-el7-10.noarch.rpm
yum -y install mysql-community-server
- 启动MySQL
systemctl start mysqld.service
systemctl status mysqld.service
- 修改密码
grep "password" /var/log/mysqld.log
mysql -uroot -p
然后输入我们之前获取到的密码g&0WNnwv8V:p
ALTER USER 'root'@'localhost' IDENTIFIED BY 'Admin@123';
- 修改MySQL默认编码
vi /etc/my.cnf
character_set_server=utf8
init_connect='SET NAMES utf8'
3.2.2 配置root用户远程访问
mysql -uroot -pAdmin@123
grant all privileges on *.* to 'root' @'%' identified by 'Admin@123';
flush privileges;
3.2.3 创建CDH需要用到的库
mysql> CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
mysql> CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
mysql> CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
mysql> CREATE DATABASE hive DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
mysql> CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
mysql> CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
mysql> quit
Mysql驱动
[root@cdh01 data]# mkdir -p /usr/share/java
[root@cdh01 data]# mv mysql-connector-java-5.1.27-bin.jar /usr/share/java/mysql-connector-java.jar
[root@cdh01 data]# xsync /usr/share/java/
此时建议大家创建快照哦
3.3 Java安装
[root@cdh01 data]# mkdir -p /usr/java/
[root@cdh01 data]# tar -zxvf jdk-8u144-linux-x64.tar.gz -C /usr/java/
[root@cdh01 data]# vi /etc/profile
#JAVA_HOME
export JAVA_HOME=/usr/java/jdk1.8.0_144
export PATH=$PATH:$JAVA_HOME/bin
[root@cdh01 data]# source /etc/profile
[root@cdh01 data]# xsync /usr/share/java/
[root@cdh02 data]# source /etc/profile
[root@cdh03 data]# source /etc/profile
[root@cdh04 data]# source /etc/profile
四、CDH安装
4.1 jar包上传
- unzip 安装
yum install unzip -y
- cm准备
[root@cdh01 data]# unzip cm6.2.1.zip
[root@cdh01 data]# xsync cm6.2.1/
- parcels准备
[root@cdh01 data]# unzip parcels.zip
[root@cdh01 data]# cd parcels
[root@cdh01 parcels]# mkdir -p /opt/cloudera/parcel-repo/
[root@cdh01 parcels]# cp ./* /opt/cloudera/parcel-repo/
[root@cdh01 parcels]# cd /opt/cloudera/parcel-repo/
[root@cdh01 parcel-repo]# mv CDH-6.2.1-1.cdh6.2.1.p0.1580995-el7.parcel.sha1 CDH-6.2.1-1.cdh6.2.1.p0.1580995-el7.parcel.sha
4.2 CM安装
主节点
[root@cdh01 cloudera]# cd /opt/data/cm6.2.1/RPMS/x86_64
[root@cdh01 x86_64]# yum install cloudera-manager-daemons-6.2.1-1426065.el7.x86_64.rpm -y
[root@cdh01 x86_64]# yum install cloudera-manager-agent-6.2.1-1426065.el7.x86_64.rpm -y
[root@cdh01 x86_64]# yum install cloudera-manager-server-6.2.1-1426065.el7.x86_64.rpm -y
其他节点
[root@cdh02 cloudera]# cd /opt/data/cm6.2.1/RPMS/x86_64
[root@cdh02 x86_64]# yum install cloudera-manager-daemons-6.2.1-1426065.el7.x86_64.rpm -y
[root@cdh02 x86_64]# yum install cloudera-manager-agent-6.2.1-1426065.el7.x86_64.rpm -y
4.3 配置Server节点（所有节点）
[root@cdh01 x86_64]vim /etc/cloudera-scm-agent/config.ini
# Hostname of the CM server.
server_host=cdh01
4.3 配置数据库
[root@cdh01 x86_64]# /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm root Admin@123
4.3 启动CM
[root@cdh01 java]# systemctl start cloudera-scm-agent
[root@cdh02 java]# systemctl start cloudera-scm-agent
[root@cdh03 java]# systemctl start cloudera-scm-agent
[root@cdh04 java]# systemctl start cloudera-scm-agent
[root@cdh01 java]# systemctl start cloudera-scm-server
此时观察我们的日志tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log，看到Started Jetty server.就好了
五、组件安装
浏览器输入http://cdh01:7180/cmf/login
账号：admin
密码：admin
5.1集群配置
