CDH6.2.0系统部署手册----------yum在线安装

2019独角兽企业重金招聘Python工程师标准>>>

参考CDH平台部署参考手册

1. 部署信息

操作系统：CentOS 7.6.1810
版本：CDH6.2.0

2. 服务器环境初始化

2.1 操作系统初始化

见本站CentOS 7.3初始化

2.2 设置swap

echo vm.swappiness = 10 >> /etc/sysctl.conf
sysctl -w vm.swappiness=10

2.3 设置巨页

vi /etc/rc.d/rc.local
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
chmod +x /etc/rc.d/rc.local

2.4 关闭防火墙

sudo systemctl stop firewalld
sudo systemctl disable firewalld

2.5 关闭selinux

sudo setenforce 0
sed -i "s/SELINUX=enforce/SELINUX=disabled/g /etc/selinux/config

2.6 设置主机名和hosts

hostnamectl set-hostname cdh-1.sysit.cn
cat > /etc/hosts <<'EOF'
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.50.101.106 cdh-1.sysit.cn cdh-1
10.50.101.64 cdh-2.sysit.cn cdh-2
10.50.101.115 cdh-3.sysit.cn cdh-3
10.50.101.88 cdh-4.sysit.cn cdh-4
10.50.101.82 cdh-5.sysit.cn cdh-5
EOF

2.7 关闭 NUMA（可选）

grubby --update-kernel=ALL --args="numa=off" --update-kernel="$(grubby --default-kernel)"

3. 配置时区及时间同步

本例以ntp为例，如果选用chronyd参见本站其他文档

# 时区
ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
# ntp server
yum install ntp -y
cat >>/etc/ntp.conf <<'EOF'
# 允许客户端同步时间
restrict 10.50.101.0 mask 255.255.255.0 nomodify notrap
# 本机作为时钟源
server 127.127.1.0
fudge 127.127.1.0 stratum 10
EOF
systemctl enable ntpd
systemctl restart ntpd
ntpq -p
# ntp client
yum install ntp -y
sed -i "s/^server/#server/g" /etc/ntp.conf
echo -e "server cdh-1.sysit.cn prefer" >>/etc/ntp.conf
systemctl stop ntpd
ntpdate cdh-1.sysit.cn
systemctl restart ntpd

4. 安装mysql

#mysql5.7
cat > /etc/yum.repos.d/mysql-community-el7.repo<<'EOF'
[mysql57-community]
name=MySQL 5.7 Community Server
baseurl=http://mirror.tuna.tsinghua.edu.cn/mysql/yum/mysql57-community-el7/
enabled=1
gpgcheck=0
EOF
# 安装
yum install mysql-community-server -y
echo "character-set-server=utf8" >>/etc/my.cnf
systemctl restart mysqld
# mysqld.log中查询密码
grep password /var/log/mysqld.log
2019-04-09T07:28:44.292450Z 1 [Note] A temporary password is generated for root@localhost: wg:yykr4Kt;4
# 测试环境关闭强策略密码，生产环境建议开启。
echo "validate-password=off" >>/etc/my.cnf
# 重启mysql
systemctl restart mysqld
# 重置安全配置
mysql_secure_installation
# 登录mysql
mysql -uroot -p
# 创建用于cloudera运行的mysql数据库cmf
create database cmf default character set utf8;
grant all on cmf.* to 'cmf'@'%' identified by 'cmfpassword';
grant all on cmf.* to 'cmf'@'localhost' identified by 'cmfpassword';
# 创建其他组件的mysql数据库
create database hive default character set utf8;
grant all on hive.* to 'hive'@'%' identified by 'hivepassword';
grant all on hive.* to 'hive'@'localhost' identified by 'hivepassword';
create database oozie default character set utf8;
grant all on oozie.* to 'oozie'@'%' identified by 'cmfpassword';
grant all on oozie.* to 'oozie'@'localhost' identified by 'ooziepassword';
create database hue default character set utf8;
grant all on hue.* to 'hue'@'%' identified by 'huepassword';
grant all on hue.* to 'hue'@'localhost' identified by 'huepassword';

5 安装cloudera manager

5.1 配置源

# 所有节点
wget https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/cloudera-manager.repo
mv cloudera-manager.repo /etc/yum.repos.d/

5.2安装JDK及mysql-connector-java

# 所有节点安装，可以从oracle官方下载rpm包安装，这里采用cloudera官方提供的jdk
yum install java -y
# 安装mysql-connector-java
# CentOS自带的mysql-connector-java版本太旧，可以满足cloudera的安装，但是后续安装Hive的时候会出错，原因是cloudera只支持5.1.x的版本，但自带的版本太低。
wget https://cdn.mysql.com//Downloads/Connector-J/mysql-connector-java-5.1.47.tar.gz
tar -zxvf mysql-connector-java-5.1.47.tar.gz
mkdir /usr/share/java
cp mysql-connector-java-5.1.47/mysql-connector-java-5.1.47-bin.jar /usr/share/java/mysql-connector-java.jar

5.3 安装cloudera manager-server

# 在manager server节点安装
yum install cloudera-manager-server -y
# 初始化数据库
# usage: /opt/cloudera/cm/schema/scm_prepare_database.sh [options] (postgresql|mysql|oracle) database username [password]
# --scm-host ：SCM server's hostname. Omit if SCM is colocated with MySQL.
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql --scm-host cdh-1.sysit.cn cmf cmf cmfpassword
# 启动
systemctl start cloudera-scm-server
systemctl enable cloudera-scm-server
systemctl status cloudera-scm-server
# 查看监听端口
ss -antlp | grep 7180

cloudera第一次启动的时候会比较慢，等待一段时间查看7180端口是否监听，监听了即可通过浏览器访问。

默认账号密码是admin/admin

5.4 安装cloudera manager agent

# 可以通过cloudera manager server的web界面安装agent，我这里手动安装agent
yum install cloudera-manager-agent -y
sed -i "s/server_host=localhost/server_host=cdh-1.sysit.cn/g" /etc/cloudera-scm-agent/config.ini
systemctl restart cloudera-scm-agent
systemctl enable cloudera-scm-agent
systemctl status cloudera-scm-agent

6 通过web界面安装自定义安装组件

登录cloudera server 的web界面http://cdh-1.sysit.cn:7180，输入admin/admin账号登录。
根据步骤安装。

点击“continue”进入下一步。

勾选“Yes,I accept the End User License Terms and Conditions. ”，点击“continue”进入下一步。

选择Cloudera Express版本，点击“Continue”进入下一步。

点击“continue”进入下一步

输入Cluster Name，点击“Continue”进入下一步。Cluster Name只是cloudera界面上识别的名字。

New Hosts：用于没有安装cloudera-manager-agent的主机，在这里可以列出需要安装cloudera-manager-agent的主机名或IP地址，通过web界面引导安装agent。
Currently Managed Hosts：用于实现已经安装好了agent，且agent已经将心跳信息发送给server的场景使用。

在这里，之前已经安装好了agent，勾选需要加入集群的主机，点击“Continue”进入下一步。

这里就需要配置CDH的Repository仓库。
因为Repository中文件比较大，从官方网站下载可能比较慢，我们可以下载到本地之后创建一个apache的访问地址，下载地址https://archive.cloudera.com/cdh6/6.2.0/parcels/，需要根据操作系统版本下载，这里下载4个文件：
* https://archive.cloudera.com/cdh6/6.2.0/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel
* https://archive.cloudera.com/cdh6/6.2.0/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel.sha1
* https://archive.cloudera.com/cdh6/6.2.0/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel.sha256
* https://archive.cloudera.com/cdh6/6.2.0/parcels/manifest.json

配置apache的方法，此处略。

在Install Method中，我们选择Use Parcels，点击“More Options”，进入parcel配置界面。