离线部署CDH5.16.1及各种坑

 

 

所需安装包及版本说明：由于我们的操作系统为CentOS7，需要下载以下文件： cloudera-manager-centos7-cm5.16.1_x86_64.tar.gz CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha1 manifest.json

Cloudera Manager 下载目录 

http://archive.cloudera.com/cm5/cm/5/

CDH 下载目录 

http://archive.cloudera.com/cdh5/parcels/5.16.1/

manifest.json 下载 

http://archive.cloudera.com/cdh5/parcels/5.16.1/manifest.json

 

mkdir -p /opt/cloudera/parcel-repo/

CHD5 相关的 Parcel 包放到主节点的/opt/cloudera/parcel-repo/目录中

CDH-5.12.1-1.cdh5.12.1.p0.3-el7.parcel.sha1 重命名为 CDH-5.12.1-1.cdh5.12.1.p0.3-el7.parcel.sha

这点必须注意，否则，系统会重新下载 CDH-5.12.1-1.cdh5.12.1.p0.3-el6.parcel 文件

 

本文采用离线安装方式

主机名	ip地址	安装服务
node1 (Master)	192.168.252.121	jdk、cloudera-manager、MySql
node2 (Agents)	192.168.252.122	jdk、cloudera-manager
node3 (Agents)	192.168.252.123	jdk、cloudera-manager

二、系统环境搭建

1、网络配置(所有节点)

修改 hostname

依次修改所有节点 node[1-3]

hostnamectl set-hostname node1

 

修改映射关系

1.在 node1 的 /etc/hosts 文件下添加如下内容

$ vi /etc/hosts

2.查看修改后的/etc/hosts 文件内容

[root@node7 ~]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.252.121 node1 192.168.252.122 node2 192.168.252.123 node3 192.168.252.124 node4 192.168.252.125 node5 192.168.252.126 node6 192.168.252.127 node7

 

for a in {2..3} ; do scp /etc/hosts node$a:/etc/hosts ; done

2、SSH 免密码登录

1.在集群node1的 /etc/ssh/sshd_config  文件去掉以下选项的注释

vi /etc/ssh/sshd_config

RSAAuthentication yes #开启私钥验证 PubkeyAuthentication yes #开启公钥验证

2.将集群node1 修改后的 /etc/ssh/sshd_config  通过 scp 命令复制发送到集群的每一个节点

for a in {2..3} ; do scp /etc/ssh/sshd_config node$a:/etc/ssh/sshd_config ; done

3.生成公钥、私钥

1.在集群的每一个节点节点输入命令 ssh-keygen -t rsa -P ''，生成 key，一律回车

ssh-keygen -t rsa -P ''

4.在集群的node1 节点输入命令

将集群每一个节点的公钥id_rsa.pub放入到自己的认证文件中authorized_keys;

for a in {1..3}; do ssh root@node$a cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys; done

5.在集群的node1 节点输入命令

将自己的认证文件 authorized_keys  通过 scp 命令复制发送到每一个节点上去: /root/.ssh/authorized_keys`

for a in {1..3}; do scp /root/.ssh/authorized_keys root@node$a:/root/.ssh/authorized_keys ; done

6.在集群的每一个节点节点输入命令

接重启ssh服务

sudo systemctl restart sshd.service

7.验证 ssh 无密登录

开一个其他窗口测试下能否免密登陆

例如：在node3

ssh root@node2

exit 退出

 

3、关闭防火墙

systemctl stop firewalld.service

systemctl disable firewalld

4、关闭 SELINUX

查看

[root@node1 ~]# getenforce Enforcing [root@node1 ~]# /usr/sbin/sestatus -v SELinux status:

临时关闭

## 设置SELinux 成为permissive模式 ## setenforce 1 设置SELinux 成为enforcing模式 setenforce 0

永久关闭

vi /etc/selinux/config

将 SELINUX=enforcing 改为 SELINUX=disabled

设置后需要重启才能生效

PS 我是修改node1 的 /etc/selinux/config 后，把配置文件复制到其他节点

for a in {2..3}; do scp /etc/selinux/config root@node$a:/etc/selinux/config ; done

 

5、安装 JDK

rpm包放到/opt

for a in {2..3}; do scp /opt/jdk-8u181-linux-x64.rpm root@node$a:/opt/jdk-8u181-linux-x64.rpm ; done

 

rpm -ivh jdk-8u181-linux-x64.rpm

 

vi /etc/profile

 

#jdk export JAVA_HOME=/usr/java/jdk1.8.0_181-amd64/ export JRE_HOME=${JAVA_HOME}/jre export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib export PATH=${JAVA_HOME}/bin:$PATH

 

 

for a in {2..3}; do scp /etc/profile root@node$a:/etc/profile;source /etc/profile ; done

source /etc/profile

 

6、设置 NTP

所有节点安装 NTP

设置同步

yum -y install ntp

ntpdate -d 182.92.12.11

 

7、安装配置 MySql

yum install -y wget

wget -i -c http://dev.mysql.com/get/mysql57-community-release-el7-10.noarch.rpm

rpm -ivh mysql57-community-release-el7-10.noarch.rpm

yum -y install mysql-community-server

systemctl enable mysqld

systemctl start mysqld.service

systemctl status mysqld.service

grep "password" /var/log/mysqld.log

mysql -uroot -p

set global validate_password_policy=0;

set global validate_password_length=1;

set global validate_password_special_char_count=0;

set global validate_password_mixed_case_count=0;

set global validate_password_number_count=0;

select @@validate_password_number_count,@@validate_password_mixed_case_count,@@validate_password_number_count,@@validate_password_length;

 

ALTER USER 'root'@'localhost' IDENTIFIED BY 'password';

grant all privileges on *.* to 'root'@'%' identified by 'password' with grant option;

flush privileges;

exit

 

yum -y remove mysql57-community-release-el7-10.noarch

 

8、下载依赖包

yum -y install chkconfig yum -y install bind-utils yum -y install psmisc yum -y install libxslt yum -y install zlib yum -y install sqlite yum -y install cyrus-sasl-plain yum -y install cyrus-sasl-gssapi yum -y install fuse yum -y install portmap yum -y install fuse-libs yum -y install redhat-lsb

 

三、cloudera manager Server & Agent 安装

1、安装 CM Server & Agent

在所有节点，创建/opt/cloudera-manager

mkdir /opt/cloudera-manager

把下载好的cloudera-manager-centos7-cm5.12.1_x86_64.tar.gz安装包上传至 node1 节点/opt/目录

在 node1 节点拷贝 cloudera-manager-centos7-cm5.12.1_x86_64.tar.gz 到所有 Server、Agent 节点创建 /opt/cloudera-manager 目录：

for a in {2..3}; do scp /opt/cloudera-manager-*.tar.gz root@node$a:/opt/ ; done

所有 Server、Agent 节点节点解压安装 Cloudera Manager Server & Agent

cd /opt tar xvzf cloudera-manager*.tar.gz -C /opt/cloudera-manager

2、创建用户 cloudera-scm（所有节点）

cloudera-scm 用户说明，摘自官网：

Cloudera Manager Server and managed services are configured to use the user account cloudera-scm by default, creating a user with this name is the simplest approach. This created user, is used automatically after installation is complete.

Cloudera管理器服务器和托管服务被配置为在默认情况下使用用户帐户Cloudera-scm，创建具有这个名称的用户是最简单的方法。创建用户，在安装完成后自动使用。

执行：在所有节点创建cloudera-scm用户

useradd --system --home=/opt/cloudera-manager/cm-5.16.1/run/cloudera-scm-server/ --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

3、配置 CM Agent

修改 node1 节点

/opt/cloudera-manager/cm-5.12.1/etc/cloudera-scm-agent/config.ini中的server_host为主节点的主机名。

cd /opt/cloudera-manager/cm-5.16.1/etc/cloudera-scm-agent/ vi config.ini

在node1 操作将 node1 节点修改后的 (复制到所有节点)

for a in {2..3}; do scp /opt/cloudera-manager/cm-5.16.1/etc/cloudera-scm-agent/config.ini root@node$a:/opt/cloudera-manager/cm-5.16.1/etc/cloudera-scm-agent/config.ini ; done

4、配置 CM Server 的数据库

在主节点 node1 初始化CM5的数据库：

下载 mysql 驱动包

cd /opt/cloudera-manager/cm-5.16.1/share/cmf/lib wget http://maven.aliyun.com/nexus/service/local/repositories/hongkong-nexus/content/Mysql/mysql-connector-java/5.1.38/mysql-connector-java-5.1.38.jar

 

for a in {2..3}; do scp /opt/cloudera-manager/cm-5.16.1/share/cmf/lib/mysql-*.jar root@node$a:/opt/cloudera-manager/cm-5.16.1/share/cmf/lib/ ; done

for a in {1..3}; do scp /opt/cloudera-manager/cm-5.16.1/share/cmf/lib/mysql-*.jar root@node$a:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hive/lib/ ; done

for a in {1..3}; do scp /opt/cloudera-manager/cm-5.16.1/share/cmf/lib/mysql-*.jar root@node$a:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/oozie/lib/ ; done

 

#for a in {1..3}; do mkdir /usr/share/java/ && scp /opt/cloudera-manager/cm-5.16.1/share/cmf/lib/mysql-*.jar root@node$a:/usr/share/java/ && mv mysql-connector-java-5.1.41.jar mysql-connector-java.jar ; done

-----需手动下载

启动MySQL服务

systemctl restart mysqld

mysql -uroot -p

create database cm;

exit

 

cd /opt/cloudera-manager/cm-5.16.1/share/cmf/schema/ ./scm_prepare_database.sh mysql cm root password

 

王洪岐：按照原来的写法

scm_prepare_database.sh mysql cm -h -u -p --scm-host scm scm scm

简单密码此步骤会报错java.sql.SQLException: Your password does not satisfy the current policy requirements

drop database cm;

后面的三个scm是库名、用户名、密码，mysql5.7密码最短4位，所以最后一个scm改成scm1

看到如下信息，恭喜您，配置没毛病

[main] DbCommandExecutor INFO Successfully connected to database. All done, your SCM database is configured correctly!

格式:

scm_prepare_database.sh mysql cm -h -u -p --scm-host scm scm scm 对应于：数据库类型 数据库 服务器 用户名 密码 –scm-host Cloudera_Manager_Server 所在节点……

5、创建 Parcel 目录

Manager 节点创建目录/opt/cloudera/parcel-repo，执行：

将下载好的文件

CDH-5.12.1-1.cdh5.12.1.p0.3-el7.parcel CDH-5.12.1-1.cdh5.12.1.p0.3-el7.parcel.sha manifest.json

拷贝到该目录下。

mkdir -p /opt/cloudera/parcel-repo chown cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo cd /opt/cloudera/parcel-repo

重命名，CDH-5.12.1-1.cdh5.12.1.p0.3-el7.parcel.sha1 否则，系统会重新下载 CDH-5.12.1-1.cdh5.12.1.p0.3-el7.parcel

mv CDH-5.12.1-1.cdh5.12.1.p0.3-el7.parcel.sha1 CDH-5.12.1-1.cdh5.12.1.p0.3-el7.parcel.sha

Agent 节点创建目录/opt/cloudera/parcels，执行：

mkdir -p /opt/cloudera/parcels chown cloudera-scm:cloudera-scm /opt/cloudera/parcels

6、启动 CM Manager&Agent 服务

注意，mysql 服务启动，防火墙关闭

在 node1 (master) 执行：

Server

/opt/cloudera-manager/cm-5.16.1/etc/init.d/cloudera-scm-server start

在 node2-7 (Agents) 执行：

Agents

/opt/cloudera-manager/cm-5.16.1/etc/init.d/cloudera-scm-agent start

访问 http://Master:7180 若可以访问（用户名、密码：admin），则安装成功。

Manager 启动成功需要等待一段时间，过程中会在数据库中创建对应的表需要耗费一些时间。

 

四、CDH5 安装

CM Manager && Agent 成功启动后，登录前端页面进行 CDH 安装配置。

admin/admin登录

同意条款——继续

选择免费许可证——继续

 

各个 Agent 节点正常启动后，可以在当前管理的主机列表中看到对应的节点。

选择要安装的节点，点继续。

 

点击，继续，如果配置本地 Parcel 包无误，已下载，应该是瞬间就完成了，然后就是耐心等待分配过程就行了，大约 10 多分钟吧，取决于内网网速。

（若本地 Parcel 有问题，重新检查步骤三、5 是否配置正确）

 

集群设置页面选择需要安装的服务——继续

数据库配置页面，填写连接node1的mysql，用户名密码如下，执行sql创建库之后即可。

create database hue DEFAULT CHARACTER SET utf8; grant all on hue.* TO 'hue'@'%' IDENTIFIED BY 'hue1';

create database hive DEFAULT CHARACTER SET utf8; grant all on hive.* TO 'hive'@'%' IDENTIFIED BY 'hive';

create database oozie DEFAULT CHARACTER SET utf8; grant all on oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie';

 

 

问题

//////////////////////////////////CDH安装失败或中断怎么处理

server和agent分别停止服务

/opt/cloudera-manager/cm-5.16.1/etc/init.d/cloudera-scm-server stop

/opt/cloudera-manager/cm-5.16.1/etc/init.d/cloudera-scm-agent stop

删除Agent节点的UUID 

rm -rf  /opt/cloudera-manager/cm-5.16.1/lib/cloudera-scm-agent/*

清空主节点CM数据库

      进入主节点的Mysql数据库，然后drop database cm;

重新初始化数据库

./scm_prepare_database.sh mysql cm root password

如果报找不到cm库，则create database cm;

server和agent分别启动服务

/opt/cloudera-manager/cm-5.16.1/etc/init.d/cloudera-scm-server start

/opt/cloudera-manager/cm-5.16.1/etc/init.d/cloudera-scm-agent start

////////////////////////////////CDH安装前核查服务启动情况

agent：

ps -ef|grep cloudera

root 17255 1 0 11:12 ? 00:00:05 /opt/cloudera-manager/cm-5.16.1/lib64/cmf/agent/build/env/bin/python /opt/cloudera-manager/cm-5.16.1/lib64/cmf/agent/build/env/bin/supervisord

root 17257 17255 0 11:12 ? 00:00:00 python2.7 /opt/cloudera-manager/cm-5.16.1/lib64/cmf/agent/build/env/bin/cmf-listener -l /opt/cloudera-manager/cm-5.16.1/log/cloudera-scm-agent/cmf_listener.log /opt/cloudera-manager/cm-5.16.1/run/cloudera-scm-agent/events

root 17460 17255 0 11:16 ? 00:00:34 python2.7 /opt/cloudera-manager/cm-5.16.1/lib64/cmf/agent/build/env/bin/flood

root 31658 1 2 13:06 ? 00:00:05 python2.7 /opt/cloudera-manager/cm-5.16.1/lib64/cmf/agent/build/env/bin/cmf-agent --package_dir /opt/cloudera-manager/cm-5.16.1/lib64/cmf/service --agent_dir /opt/cloudera-manager/cm-5.16.1/run/cloudera-scm-agent --lib_dir /opt/cloudera-manager/cm-5.16.1/lib/cloudera-scm-agent --logfile /opt/cloudera-manager/cm-5.16.1/log/cloudera-scm-agent/cloudera-scm-agent.log --daemon --comm_name cmf-agent --pidfile /opt/cloudera-manager/cm-5.16.1/run/cloudera-scm-agent/cloudera-scm-agent.pid

 

正常是上面这四个，如果cmf-agent未启动，将supervisord kill掉再agent start。

cat /opt/cloudera-manager/cm-5.16.1/log/cloudera-scm-agent/supervisord.out

 

安装建库

 

create database hue DEFAULT CHARACTER SET utf8; grant all on hue.* TO 'hue'@'%' IDENTIFIED BY 'hue1';

create database hive DEFAULT CHARACTER SET utf8; grant all on hive.* TO 'hive'@'%' IDENTIFIED BY 'hive';

create database oozie DEFAULT CHARACTER SET utf8; grant all on oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie';

 

////////////////////////CDH安装中途莫名失败，可以重新再来一次

 

////////////////////////hive安装报错

创建 Hive Metastore 数据库表 步骤报错

org.apache.hadoop.hive.metastore.HiveMetaException: Failed to load driver

拷贝驱动到hive的lib目录

for a in {1..3}; do scp /opt/cloudera-manager/cm-5.16.1/share/cmf/lib/mysql-*.jar root@node$a:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hive/lib/ ; done

///////////////////////////oozie安装包错找不到驱动

注：这个很靠谱！以后一定要先设置好

拷贝驱动到/usr/share/java/目录，并且改名 && mv mysql-connector-java-5.1.41.jar mysql-connector-java.jar

///////////////////////启动后好几个服务报红

点击去逐个操作重启

///////////////////////hue LoadBalancer启动不了，提示Failed to find the Apache HTTPD executable.

LoadBalancer所在机器安装httpd即可

yum -y install mod_ssl httpd

 

//////////////////////添加新主机，添加服务节点

所有主机界面有向集群添加主机按钮

添加节点可以进入集群——HBase等——实例——添加角色实例，选择相应实例到特定节点上。

 

 

 

官方参考文档：https://www.cloudera.com/documentation/enterprise/latest/topics/configure_cm_repo.html

参考文档：https://yq.aliyun.com/articles/341408