千云物流 -安装企业CM&CDH6.3.2

基础环境准备

cloudera官网下载需要的包

自 2021 年 1 月 31 日起，所有 Cloudera 软件都需要订阅 https://www.cloudera.com/downloads.html
https://www.cloudera.com/downloads/cdp-private-cloud-trial.html
这里官网不提供免费版本了。

环境准备

ssh配置
服务器准备
mysql数据库准备

#获取mysql rpm包
wget http://repo.mysql.com/mysql57-community-release-el7-8.noarch.rpm
#安装rpm 这个rpm是为了更新yum源地址的
rpm -ivh mysql57-community-release-el7-8.noarch.rpm
#使用yum在线安装
yum install -y mysql-server
#启动mysql
systemctl start mysqld
## 从日志找到密码
grep 'temporary password' /var/log/mysqld.log
## 提示错误 You must reset your password using ALTER USER statement before executing this statement.
ALTER USER "root"@"localhost" IDENTIFIED BY "Janle5xxx!"
## 授权数据库
grant all on *.* to 'janle'@'%' identified by 'qY566@#'  with grant option;
## 创建需要的数据库
create database hive; 
create database hue;
create database ooz;
create database scm;

CM的安装和CDH的装备

机器	系统配置	安装服务
hadoop1	8C 32G 300G	Hue Server, Service Monitor, Activity Monitor, Host Monitor, Event Server, Alert Publisher, Oozie Server Hbase Master, Hbase Thrift Server, HDFS Name Node, Hive Gateway, Hive Metastore Server, Hive Server2, Hue Load Balancer, ResourceManager, Zookeeper
hadoop2	4C 8G 100G	Hbase Master, HBase Thrift Server, HSFS SecondaryNameNode, HDFS DataNode, JobHistory Server, Hive Metastore Server, HiveServer2, Load Balancer, Zookeeper
hadoop3	4C 8G 100G	Hbase RegionServer, HDFS DataNode, YARNNodeManager, Zookeeper
hadoop4	4C 8G 100G	Hbase RegionServer, HDFS DataNode, YARNNodeManager

创建需要的文件夹并授权

//--在所有上创建
mkdir -p /opt/cloudera-manager	
//--在server上创建
mkdir -p /opt/cloudera/parcel-repo	
//--在所有的agent节点上创建
mkdir -p /opt/cloudera/parcels	
// --在所有的agent节点上创建
mkdir -p  /opt/cloudera-manager/cm-5.16.1/run/cloudera-scm-agent	
groupadd cloudera-scm
useradd -g cloudera-scm cloudera-scm
//-在server上
chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo/   
//在agent上
chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcels   
## 授权目录安装
```c
mkdir -p /var/lib/cloudera-host-monitor
mkdir /var/lib/cloudera-service-monitor
chown -R cloudera-scm:cloudera-scm /var/lib/cloudera-host-monitor
chown -R cloudera-scm:cloudera-scm /var/lib/cloudera-service-monitor

HDFS配置项dfs.permissions，改为false，否则会因为权限问题不能在hdfs中创建目录和文件

需要的安装资料

https://download.csdn.net/download/u013642886/19955534

需要准备的服务器

机器	系统配置	安装服务
hadoop1	8C 32G 300G	Hue Server, Service Monitor, Activity Monitor, Host Monitor, Event Server, Alert Publisher, Oozie Server Hbase Master, Hbase Thrift Server, HDFS Name Node, Hive Gateway, Hive Metastore Server, Hive Server2, Hue Load Balancer, ResourceManager, Zookeeper
hadoop2	4C 8G 100G	Hbase Master, HBase Thrift Server, HSFS SecondaryNameNode, HDFS DataNode, JobHistory Server, Hive Metastore Server, HiveServer2, Load Balancer, Zookeeper
hadoop3	4C 8G 100G	Hbase RegionServer, HDFS DataNode, YARNNodeManager, Zookeeper
hadoop4	4C 8G 100G	Hbase RegionServer, HDFS DataNode, YARNNodeManager

服务器不要使用docker的服务器，和yarn有冲突

安装jdk

rpm -ivh oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm
export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
source /etc/profile

安装依赖包

yum install -y bind-utils libxslt cyrus-sasl-plain cyrus-sasl-gssapi portmap fuse-libs fuse
yum install -y /lib/lsb/init-functions deltrpm python-deltrpm 
yum install -y mod_ssl openssl-devel python-psycopg2 Mysql-python 

安装httpd【不需要全部安装】

yum install -y httpd createrepo 

配置host

vim /etc/hosts
ip hadoop1
ip hadoop2
ip hadoop3
ip hadoop4

关闭防火墙

systemctl status firewalld.service  //查看防火墙状态
systemctl stop firewalld.service
systemctl disable firewalld.service //禁止防火墙

关闭selinux

vi /etc/sysconfig/selinux
SELINUX=disabled  //为了使配置生效，需要重新启动系统

reboot 重启机器

安装httpd服务

cd /var/www/html
mkdir -p cmd6/6.3.1/redhat7/yum/RPMS/x86_64/
mkdir -p cdh6/6.3.1/parcels/
systemctl start httpd.service

上传cmd6中的文件到此目录，上传allkeys.asc文件到/var/www/html/cmd6/6.3.1/目录下
访问测试地址http://ip/cmd6/6.3.1/readhat7/yum/RPMS/x86_64/
生成repodata目录

cd /var/www/html/cmd6/6.3.1/readhat7/yum
//上传文件后生成reportdata
createrepo .

配置本地yum源

cd /etc/yum.repos.d/
vim cloudera-manager.repo

[cloudera-manager]
name=Cloudera Manager
baseurl=http://ip/cmd6/6.3.1/redhat7/yum/
gpgcheck=0

yum clean all
yum list | grep cloudera

创建cloudera-scm用户（centos7要求有这个用户，os6中不要求）
授权管理员权限

useradd cloudera-scm
passwd cloudera-scm 密码test123
echo "cloudera-scm ALL=(root)NOPASSWORD:ALL" >> /etc/sudoers
su - cloudera-scm  
exit

获取mysql rpm包

wget http://repo.mysql.com/mysql57-community-release-el7-8.noarch.rpm
#安装rpm 这个rpm是为了更新yum源地址的
rpm -ivh mysql57-community-release-el7-8.noarch.rpm
#使用yum在线安装
yum install -y mysql-server
#启动mysql
systemctl start mysqld.service 或者
systemctl start mysqld

从日志找到密码

grep 'temporary password' /var/log/mysqld.log

提示错误 You must reset your password using ALTER USER statement before executing this statement.

//set globel validate_password_policy=0
//set globel validate_password_length=1
ALTER USER "root"@"localhost" IDENTIFIED BY "Janle5xxx!"

授权数据库

grant all on *.* to 'janle'@'%' identified by 'qY566@#'  with grant option;
flush privileges;

创建需要的数据库

create database hive; 
create database hue;
create database ooz;
create database scm;

卸载repo包

yum会有自动更新，耗费时间
yum -y remove mysql57-community-release-el7-8.noarch

安装CM

yum install -y oracle-j2sdk1.8-1.8.0+update181-1.x86_64
yum install -y enterprise-debuginfo-6.3.1-1466458.el7.x86_64
yum install -y cloudera-manager-server-6.3.1-1466458.el7.x86_64
yum install -y cloudera-manager-server-db-2-6.3.1-1466458.el7.x86_64

在yum源中添加新包时需要注意：
如果重新上传过包后，需要删除之前的repodata文件后，重新生成。
重启httpd服务
清楚yum缓存

设置元数据库为mysql

安装clouderaManager Server服务后，设置元数据库为mysql

/opt/cloudera/cm/schema/scm_prepare_database.sh -h localhost mysql scm **密码**

如果提示JDBC Driver not found 将mysql的驱动包放到/opt/cloudera/cm/lib/目录下。

cat /etc/cloudera-scm-server/db.properties

启动server服务进程（端口7180）

systemctl start cloudera-scm-server
//启动日志
tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log
//查看启动状态
systemctl status clouder-scm-server
netstat -an|grep 7180

配置本地parcel包

在数据库初始话完成后，将parcel文件放置到/opt/cloudera/parcel-repo目录中
上传cdh6的parcel等文件到/opt/cloudera/parcel-repo
重命名密钥文件名

mv CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1 CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha

安装CDH

访问http://hadoop1:7180/cmf/login
admin/admin
添加集群-> Select Edition ->免费版本 ->集群名称qy56601
Parcel更新频率修改为1分钟。可以看见生成的parcel的目录。
自定义库设置为: http://hadoop01/cm6/6.3.1

设置swappiness和透明化

临时生效设置

sysctl -w vm.swappiness=10
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
永久生效：
echo "vm.swappiness=10" >> /etc/sysctl.conf
echo "never > /sys/kernel/mm/transparent_hugepage/defrag" >> /etc/rc.local
echo "never > /sys/kernel/mm/transparent_hugepage/enabled" >> /etc/rc.local

有安装问题可以沟通，一定回复。