vmware14安装CetOS7搭建cdh5.13.3环境（二）

创建时间：2019-4-28 10:38:26。

一、创建数据库

create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database amon DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database monitor DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci;

备份快照【mysqlhive】

二、安装CM

2.1 上传

使用xmanager5将以下包：
cloudera-manager-centos7-cm5.13.3_x86_64.tar.gz
CDH-5.13.3-1.cdh5.13.3.p0.2-el7.parcel
CDH-5.13.3-1.cdh5.13.3.p0.2-el7.parcel.sha1
manifest.json
上传到【/usr/local/ninja59k】

2.1.1 重命名sha1

mv CDH-5.13.3-1.cdh5.13.3.p0.2-el7.parcel.sha1 CDH-5.13.3-1.cdh5.13.3.p0.2-el7.parcel.sha

2.1.2 备份快照【uploadcdh5】

2.2 安装

tar -zxvf /usr/local/ninja59k/cloudera-manager-centos7-cm5.13.3_x86_64.tar.gz -C /opt

需要几分钟时间。解压完后两个文件夹【cloudera】【cm-5.13.3】

新的参考资料20190509
https://blog.csdn.net/ltliyue/article/details/82734814
#创建CM用户（所有结点）
mysql>create database cm DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
–授权
grant all privileges on hive.* to ‘scm’@’%’ identified by ‘cdh’ with grant option;
grant all privileges on amon.* to ‘scm’@’%’ identified by ‘cdh’ with grant option;
grant all privileges on hue.* to ‘scm’@’%’ identified by ‘cdh’ with grant option;
grant all privileges on oozie.* to ‘scm’@’%’ identified by ‘cdh’ with grant option;

useradd --system --home=/opt/cloudera-manager/cm-5.7.0/run/cloudera-scm-server --no-create-home --comment “Cloudera SCM User” cloudera-scm

阿里云有问题？？？useradd --system --home=/opt/cm-5.13.3/run/cloudera-scm-server/ --no-create-home --shell=/bin/false --comment “Cloudera SCM User” cloudera-scm
#查看
id cloudera-scm

2.2.2 为CM添加connector.lib

用XMANAGER5将【mysql-connector-java-5.1.47.tar.gz】上传到【/usr/local/ninja59k】。
解压：
tar -zxvf mysql-connector-java-5.1.47.tar.gz
[root@cdh1 mysql-connector-java-5.1.47]# cd /usr/local/ninja59k/mysql-connector-java-5.1.47
[root@cdh1 mysql-connector-java-5.1.47]# ls
build.xml CHANGES COPYING mysql-connector-java-5.1.47-bin.jar mysql-connector-java-5.1.47.jar README README.txt src
将【mysql-connector-java-5.1.47-bin.jar】拷贝到两个目录：
scp mysql-connector-java-5.1.47-bin.jar root@cdh1:/opt/cm-5.13.3/share/cmf/lib/
#mkdir -p /usr/share/java ???
#scp mysql-connector-java-5.1.47-bin.jar root@cdh1:/usr/share/java/???
scp mysql-connector-java-5.1.47-bin.jar root@cdh1:/usr/java/jdk1.8.0_211 #java安装目录

/opt/cm-5.13.3/share/cmf/schema/scm_prepare_database.sh mysql -uroot -p scm scm

2.2.3 SCM数据库初始化

[root@cdh1 mysql-connector-java-5.1.47]# /opt/cm-5.13.3/share/cmf/schema/scm_prepare_database.sh mysql -uroot -p scm scm #初始化server

Enter database password:
Enter SCM password:

JAVA_HOME=/usr/java/jdk1.8.0_211
Verifying that we can write to /opt/cm-5.13.3/etc/cloudera-scm-server
Sun May 05 01:58:50 EDT 2019 WARN: Establishing SSL connection without server’s identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn’t set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to ‘false’. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Creating SCM configuration file in /opt/cm-5.13.3/etc/cloudera-scm-server
Executing: /usr/java/jdk1.8.0_211/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/opt/cm-5.13.3/share/cmf/schema/…/lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /opt/cm-5.13.3/etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
Sun May 05 01:58:52 EDT 2019 WARN: Establishing SSL connection without server’s identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn’t set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to ‘false’. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2019-05-05 01:58:53,499 [main] INFO com.cloudera.enterprise.dbutil.DbCommandExecutor - Successfully connected to database.
All done, your SCM database is configured correctly!

2.2.4 备份vmware快照【scmdb】

2.3 配置端口

2.3.1 配置agent所有从节点指向主节点

每台节点都要进行如下配置：

[root@cdh1 cloudera-scm-agent]# cd /opt/cm-5.13.3/etc/cloudera-scm-agent/
[root@cdh1 cloudera-scm-agent]# vim config.ini
server_host=cdh1 #阿里云改成外网IP吗？
port=7180

#拷贝到其他节点
#scp -r /opt/cm-5.13.3/ root@cdh2:/opt/ #输入yes，需要等待几分钟
#scp -r /opt/cm-5.13.3/ root@cdh3:/opt/ #输入yes，需要等待几分钟

2.3.2 配置数据仓库

该操作只需在主节点执行，在主节点上创建仓库。

[root@cdh1 cloudera-scm-agent]# cd /usr/local/ninja59k/
[root@cdh1 ninja59k]# ls
CDH-5.13.3-1.cdh5.13.3.p0.2-el7.parcel  CDH-5.13.3-1.cdh5.13.3.p0.2-el7.parcel.sha1  cloudera-manager-centos7-cm5.13.3_x86_64.tar.gz  manifest.json  mysql-connector-java-5.1.47  mysql-connector-java-5.1.47.tar.gz
[root@cdh1 ninja59k]# mv CDH-5.13.3-1.cdh5.13.3.p0.2-el7.parcel CDH-5.13.3-1.cdh5.13.3.p0.2-el7.parcel.sha1 manifest.json /opt/cloudera/parcel-repo/
[root@cdh1 ninja59k]# cd /opt/cloudera/parcel-repo/
[root@cdh1 parcel-repo]# mv CDH-5.13.3-1.cdh5.13.3.p0.2-el7.parcel.sha1 CDH-5.13.3-1.cdh5.13.3.p0.2-el7.parcel.sha
[root@cdh1 parcel-repo]# ls
CDH-5.13.3-1.cdh5.13.3.p0.2-el7.parcel  CDH-5.13.3-1.cdh5.13.3.p0.2-el7.parcel.sha  manifest.json

2.3.3 备份vmware快照【cmrepo】

2.4 启动CM

#主节点
[root@cdh1 parcel-repo]# /opt/cm-5.13.3/etc/init.d/cloudera-scm-server start
/opt/cm-5.13.3/etc/init.d/cloudera-scm-server: line 109: pstree: command not found
Starting cloudera-scm-server: [ OK ]

[root@cdh1 ~]# yum install psmisc -y #pstree依赖

#主节点及其他节点
[root@cdh1 ~]# yum install psmisc -y #pstree依赖
[root@cdh1 parcel-repo]# /opt/cm-5.13.3/etc/init.d/cloudera-scm-agent start
/opt/cm-5.13.3/etc/init.d/cloudera-scm-agent: line 102: pstree: command not found
Starting cloudera-scm-agent: [ OK ]
启动需要时间，查看数据库scm中的hosts表，如果里面出现节点信息表示server启动完成。
或者等待显示【 OK 】算完成。

2.5 备份vmware快照【cmweb】

本章结束。

其他信息：

xxxxxxxx2.3.4 配置从节点目录（所有节点）（原教程不适用？？？）

在所有从节点上执行下列操作：

mkdir -p /opt/cloudera/parcels
chown cloudera-scm:cloudera-scm /opt/cloudera/parcels

2.4 初始化数据库

mysql -uroot -pSa123sa4
mysql> set global validate_password_policy=0;
mysql> set global validate_password_mixed_case_count=0;
mysql>  set global validate_password_number_count=3;
mysql> set global validate_password_special_char_count=0;
mysql>  set global validate_password_length=3;
mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'Sa123sa4' WITH GRANT OPTION;
mysql> FLUSH PRIVILEGES;

原文内容：说明：这个脚本就是用来创建和配置CMS需要的数据库的脚本。各参数是指：
mysql：数据库用的是mysql，如果安装过程中用的oracle，那么该参数就应该改为oracle。
-hcdh1：数据库建立在cdh1主机上面。也就是主节点上面。
-uroot：root身份运行mysql。-proot：mysql的root密码是root。
–scm-host cdh1：CMS的主机，一般是和mysql安装的主机是在同一个主机上。
最后三个参数是：数据库名，数据库用户名，数据库密码。

继续出错的信息：

[root@cdh1 ~]# /opt/cloudera-manager/cm-5.13.3/share/cmf/schema/scm_prepare_database.sh mysql -hcdh1 -uroot -pSa123sa4 --scm-host cdh1 scmdbn scmdbu scmdbp
JAVA_HOME=/usr/java
Verifying that we can write to /opt/cloudera-manager/cm-5.13.3/etc/cloudera-scm-server
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
2019-04-28 01:48:48,282 [main] ERROR com.cloudera.enterprise.dbutil.DbProvisioner  - Exception when creating/dropping database with user 'root' and jdbc url 'jdbc:mysql://cdh1/?useUnicode=true&characterEncoding=UTF-8'
java.sql.SQLException: The server time zone value 'EDT' is unrecognized or represents more than one time zone. You must configure either the server or JDBC driver (via the serverTimezone configuration property) to use a more specifc time zone value if you want to utilize time zone support.
省略中间内容
	at com.mysql.cj.jdbc.ConnectionImpl.initializePropsFromServer(ConnectionImpl.java:1319)
	at com.mysql.cj.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:966)
	at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:825)
	... 8 more
--> Error 1, giving up (use --force if you wish to ignore the error)

出现该问题是因为【时区设置】不一致导致。

date
#显示 Sun Apr 28 02:28:53 EDT 2019
mv /etc/localtime /etc/localtime.bak
cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime #修改时间为上海
timedatectl status #查看时区

再次执行sh脚本报错：

/opt/cloudera-manager/cm-5.13.3/share/cmf/schema/scm_prepare_database.sh mysql -hcdh1 -uroot -pSa123sa4 --scm-host cdh1 scmdbn scmdbu scmdbp
#显示内容：
2019-04-28 04:07:57,209 [main] ERROR com.cloudera.enterprise.dbutil.DbProvisioner  - Stack Trace:
java.sql.SQLException: Can't create database 'scmdbn'; database exists
--> Error 1, giving up (use --force if you wish to ignore the error)

可能是刚才执行次数多了，数据库创建了，这里drop掉

mysql:>drop database if exists scmdbn;
>exit

再次执行sh

[root@cdh1 ~]# /opt/cloudera-manager/cm-5.13.3/share/cmf/schema/scm_prepare_database.sh mysql -hcdh1 -uroot -pSa123sa4 --scm-host cdh1 scmdbn scmdbu scmdbp
JAVA_HOME=/usr/java
Verifying that we can write to /opt/cloudera-manager/cm-5.13.3/etc/cloudera-scm-server
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
2019-04-28 04:14:08,423 [main] ERROR com.cloudera.enterprise.dbutil.DbProvisioner  - Exception when creating/dropping database with user 'root' and jdbc url 'jdbc:mysql://cdh1/?useUnicode=true&characterEncoding=UTF-8'
java.sql.SQLException: Your password does not satisfy the current policy requirements

修改默认密码策略：

:>set global validate_password_policy=0;
:>set global validate_password_length=1;
:>FLUSH PRIVILEGES;

终于成功了：

[root@cdh1 ~]# /opt/cloudera-manager/cm-5.13.3/share/cmf/schema/scm_prepare_database.sh mysql -hcdh1 -uroot -pSa123sa4 --scm-host cdh1 scmdbn scmdbu scmdbp
JAVA_HOME=/usr/java
Verifying that we can write to /opt/cloudera-manager/cm-5.13.3/etc/cloudera-scm-server
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
Creating SCM configuration file in /opt/cloudera-manager/cm-5.13.3/etc/cloudera-scm-server
Executing:  /usr/java/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera-manager/cm-5.13.3/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /opt/cloudera-manager/cm-5.13.3/etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
2019-04-28 04:20:06,986 [main] INFO  com.cloudera.enterprise.dbutil.DbCommandExecutor  - Successfully connected to database.
All done, your SCM database is configured correctly!

2.5备份vmware快照【scmdbn】，时间2019-4-28 16:23:23。

3、启动服务

[root@cdh1 ~]# /opt/cloudera-manager/cm-5.13.3/etc/init.d/cloudera-scm-server start
Starting cloudera-scm-server:                              [  OK  ]
[root@cdh1 ~]# /opt/cloudera-manager/cm-5.13.3/etc/init.d/cloudera-scm-agent start
Starting cloudera-scm-agent:                               [  OK  ]

截稿时间：2019-4-28 16:30:24。