千云物流 -安装企业CM&CDH6.3.2
基础环境准备
cloudera官网下载需要的包
自 2021 年 1 月 31 日起，所有 Cloudera 软件都需要订阅 https://www.cloudera.com/downloads.html
https://www.cloudera.com/downloads/cdp-private-cloud-trial.html
这里官网不提供免费版本了。
环境准备
ssh配置
服务器准备
mysql数据库准备
#获取mysql rpm包
wget http://repo.mysql.com/mysql57-community-release-el7-8.noarch.rpm
#安装rpm 这个rpm是为了更新yum源地址的
rpm -ivh mysql57-community-release-el7-8.noarch.rpm
#使用yum在线安装
yum install -y mysql-server
#启动mysql
systemctl start mysqld
## 从日志找到密码
grep 'temporary password' /var/log/mysqld.log
## 提示错误 You must reset your password using ALTER USER statement before executing this statement.
ALTER USER "root"@"localhost" IDENTIFIED BY "Janle5xxx!"
## 授权数据库
grant all on *.* to 'janle'@'%' identified by 'qY566@#' with grant option;
## 创建需要的数据库
create database hive;
create database hue;
create database ooz;
create database scm;
CM的安装和CDH的装备
| 机器 | 系统配置 | 安装服务 |
|---|---|---|
| hadoop1 | 8C 32G 300G | Hue Server, Service Monitor, Activity Monitor, Host Monitor, Event Server, Alert Publisher, Oozie Server Hbase Master, Hbase Thrift Server, HDFS Name Node, Hive Gateway, Hive Metastore Server, Hive Server2, Hue Load Balancer, ResourceManager, Zookeeper |
| hadoop2 | 4C 8G 100G | Hbase Master, HBase Thrift Server, HSFS SecondaryNameNode, HDFS DataNode, JobHistory Server, Hive Metastore Server, HiveServer2, Load Balancer, Zookeeper |
| hadoop3 | 4C 8G 100G | Hbase RegionServer, HDFS DataNode, YARNNodeManager, Zookeeper |
| hadoop4 | 4C 8G 100G | Hbase RegionServer, HDFS DataNode, YARNNodeManager |
创建需要的文件夹并授权
//--在所有上创建
mkdir -p /opt/cloudera-manager
//--在server上创建
mkdir -p /opt/cloudera/parcel-repo
//--在所有的agent节点上创建
mkdir -p /opt/cloudera/parcels
// --在所有的agent节点上创建
mkdir -p /opt/cloudera-manager/cm-5.16.1/run/cloudera-scm-agent
groupadd cloudera-scm
useradd -g cloudera-scm cloudera-scm
//-在server上
chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo/
//在agent上
chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcels
## 授权目录安装
```c
mkdir -p /var/lib/cloudera-host-monitor
mkdir /var/lib/cloudera-service-monitor
chown -R cloudera-scm:cloudera-scm /var/lib/cloudera-host-monitor
chown -R cloudera-scm:cloudera-scm /var/lib/cloudera-service-monitor
HDFS配置项dfs.permissions，改为false，否则会因为权限问题不能在hdfs中创建目录和文件
需要的安装资料
https://download.csdn.net/download/u013642886/19955534
需要准备的服务器
| 机器 | 系统配置 | 安装服务 |
|---|---|---|
| hadoop1 | 8C 32G 300G | Hue Server, Service Monitor, Activity Monitor, Host Monitor, Event Server, Alert Publisher, Oozie Server Hbase Master, Hbase Thrift Server, HDFS Name Node, Hive Gateway, Hive Metastore Server, Hive Server2, Hue Load Balancer, ResourceManager, Zookeeper |
| hadoop2 | 4C 8G 100G | Hbase Master, HBase Thrift Server, HSFS SecondaryNameNode, HDFS DataNode, JobHistory Server, Hive Metastore Server, HiveServer2, Load Balancer, Zookeeper |
| hadoop3 | 4C 8G 100G | Hbase RegionServer, HDFS DataNode, YARNNodeManager, Zookeeper |
| hadoop4 | 4C 8G 100G | Hbase RegionServer, HDFS DataNode, YARNNodeManager |
服务器不要使用docker的服务器，和yarn有冲突
安装jdk
rpm -ivh oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm
export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
source /etc/profile
安装依赖包
yum install -y bind-utils libxslt cyrus-sasl-plain cyrus-sasl-gssapi portmap fuse-libs fuse
yum install -y /lib/lsb/init-functions deltrpm python-deltrpm
yum install -y mod_ssl openssl-devel python-psycopg2 Mysql-python
安装httpd【不需要全部安装】
yum install -y httpd createrepo
配置host
vim /etc/hosts
ip hadoop1
ip hadoop2
ip hadoop3
ip hadoop4
关闭防火墙
systemctl status firewalld.service //查看防火墙状态
systemctl stop firewalld.service
systemctl disable firewalld.service //禁止防火墙
关闭selinux
vi /etc/sysconfig/selinux
SELINUX=disabled //为了使配置生效，需要重新启动系统
reboot 重启机器
安装httpd服务
cd /var/www/html
mkdir -p cmd6/6.3.1/redhat7/yum/RPMS/x86_64/
mkdir -p cdh6/6.3.1/parcels/
systemctl start httpd.service
上传cmd6中的文件到此目录，上传allkeys.asc文件到/var/www/html/cmd6/6.3.1/目录下
访问测试地址http://ip/cmd6/6.3.1/readhat7/yum/RPMS/x86_64/
生成repodata目录
cd /var/www/html/cmd6/6.3.1/readhat7/yum
//上传文件后生成reportdata
createrepo .
配置本地yum源
cd /etc/yum.repos.d/
vim cloudera-manager.repo
[cloudera-manager]
name=Cloudera Manager
baseurl=http://ip/cmd6/6.3.1/redhat7/yum/
gpgcheck=0
yum clean all
yum list | grep cloudera
创建cloudera-scm用户（centos7要求有这个用户，os6中不要求）
授权管理员权限
useradd cloudera-scm
passwd cloudera-scm 密码test123
echo "cloudera-scm ALL=(root)NOPASSWORD:ALL" >> /etc/sudoers
su - cloudera-scm
exit
获取mysql rpm包
wget http://repo.mysql.com/mysql57-community-release-el7-8.noarch.rpm
#安装rpm 这个rpm是为了更新yum源地址的
rpm -ivh mysql57-community-release-el7-8.noarch.rpm
#使用yum在线安装
yum install -y mysql-server
#启动mysql
systemctl start mysqld.service 或者
systemctl start mysqld
从日志找到密码
grep 'temporary password' /var/log/mysqld.log
提示错误 You must reset your password using ALTER USER statement before executing this statement.
//set globel validate_password_policy=0
//set globel validate_password_length=1
ALTER USER "root"@"localhost" IDENTIFIED BY "Janle5xxx!"
授权数据库
grant all on *.* to 'janle'@'%' identified by 'qY566@#' with grant option;
flush privileges;
创建需要的数据库
create database hive;
create database hue;
create database ooz;
create database scm;
卸载repo包
yum会有自动更新，耗费时间
yum -y remove mysql57-community-release-el7-8.noarch
安装CM
yum install -y oracle-j2sdk1.8-1.8.0+update181-1.x86_64
yum install -y enterprise-debuginfo-6.3.1-1466458.el7.x86_64
yum install -y cloudera-manager-server-6.3.1-1466458.el7.x86_64
yum install -y cloudera-manager-server-db-2-6.3.1-1466458.el7.x86_64
在yum源中添加新包时需要注意：
如果重新上传过包后，需要删除之前的repodata文件后，重新生成。
重启httpd服务
清楚yum缓存
设置元数据库为mysql
安装clouderaManager Server服务后，设置元数据库为mysql
/opt/cloudera/cm/schema/scm_prepare_database.sh -h localhost mysql scm **密码**
如果提示JDBC Driver not found 将mysql的驱动包放到/opt/cloudera/cm/lib/目录下。
cat /etc/cloudera-scm-server/db.properties
启动server服务进程（端口7180）
systemctl start cloudera-scm-server
//启动日志
tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log
//查看启动状态
systemctl status clouder-scm-server
netstat -an|grep 7180
配置本地parcel包
在数据库初始话完成后，将parcel文件放置到/opt/cloudera/parcel-repo目录中
上传cdh6的parcel等文件到/opt/cloudera/parcel-repo
重命名密钥文件名
mv CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1 CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha
安装CDH
访问http://hadoop1:7180/cmf/login
admin/admin
添加集群-> Select Edition ->免费版本 ->集群名称qy56601
Parcel更新频率修改为1分钟。可以看见生成的parcel的目录。
自定义库设置为: http://hadoop01/cm6/6.3.1
设置swappiness和透明化
临时生效设置
sysctl -w vm.swappiness=10
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
永久生效：
echo "vm.swappiness=10" >> /etc/sysctl.conf
echo "never > /sys/kernel/mm/transparent_hugepage/defrag" >> /etc/rc.local
echo "never > /sys/kernel/mm/transparent_hugepage/enabled" >> /etc/rc.local
有安装问题可以沟通，一定回复。