Centos7主机搭建CDH6.X机群
目录
1 操作系统要求
1.1 linux 版本
1.2 Cloudera管理器服务器的存储需求
1.3 我的主机规划
2 准备linux系统配置环境
2.1 修改主机名
2.2 配置主机名ip映射
2.3 关闭防火墙
2.4 关闭selinux
3 软件依赖
3.1 安装jdk
3.1.1 检查系统内原有jdk
3.1.2 rpm 安装jdk
3.1.3 配置JAVA_HOME
3.2 ssh密钥配置
3.2.1 生成公钥/私钥对
3.2.2 分发公钥
3.3 修改时区
3.3.1 现有系统时区
3.3.2 修改成东八区
3.4 配置ntp
3.4.1 安装ntp
3.4.2 本地ntp server端配置
3.4.3 本地ntp 客户端
3.5 python依赖
3.6 Perl 依赖
3.7 python-psycopg2依赖
3.7.1 安装epel-release
3.7.2 安装python-pip
3.7.3 升级pip
3.7.3 pip 安装 psycopg2
3.7.4 总结
3.8 iproute package依赖
4 数据库要求
4.1 安装mysql
4.1.1 检查系统总是否原有mysql如果有删除
4.1.2 解压mysql tar 包
4.1.3 rpm安装mysql服务
4.1.4 依次安装mysql其他组件
4.2 修改mysql配置
4.2.1启动mysql
4.2.2 修改root密码
4.2.3 修改字符集要求
5 数据库部署
5.1 主节点部署JDBC
5.2 创建数据库
6 离线部署CM server及agent
6.1 创建软件目录并解压软件
6.2 主节点master部署cm server
6.2.1 rpm安装cloudera manager 守护进程
6.2.2 rpm安装cloudera manager 服务进程
6.3 所有节点作为cm agent部署
6.3.1 安装cloudera agent 前要先安装的依赖
6.3.2 安装 cloudera agnet
6.4 修改agent配置，指向server节点地址
6.5 修改master节点的server配置
7 主节点部署离线parcel源
7.1 安装httpd
7.2 部署离线parcel源
7.3 启动httpd
7.4 web 访问本地parcel源
8 启动cloudera manager 的server和agent
8.1 master节点启动server
8.2 启动agent
9 web页面操作
9.1 登录
9.2 登录后
10 在Cloudera Manager中添加集群
10.1 集群基础
10.2 指定主机
10.3 选择存储库
10.4 Install Parcels
10.5 Inspect Cluster
10.6 集群服务安装
10.6.1初步设置
10.6.2 为服务指定主机
10.6.3 输入已经设置的数据库信息
10.6.4 指定数据磁盘目录位置
10.6.5 安装完成
官网地址：
https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/configure_network_names.html
https://docs.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_version_packaging_download.html
我的安装包和镜像文件地址
链接：https://pan.baidu.com/s/1iM2nxSCY7aYK77MnHzmvZA
提取码：060n
复制这段内容后打开百度网盘手机App，操作更方便哦1 操作系统要求
1.1 linux 版本
我用的是centos7.3
1.2 Cloudera管理器服务器的存储需求
① /usr 1G 以上
② /var 5G~1T
③ /opt 15G以上
其他没什么需要注意的了，根据自己的情况吧
1.3 我的主机规划
| Ip |
Hostname
|
Memory |
核数 |
Disk |
系统 |
| 192.168.1x2.x1 |
cluster2-master |
16G |
8 |
/:50G;/data:200G |
Centos7.3 |
| 192.168.1x2.x2 |
cluster2-slave1 |
10G |
4 |
/:50G;/data:200G |
|
| 192.168.1x2.x3 |
cluster2-slave2 |
10G |
4 |
/:50G;/data:200G |
|
| 汇总 |
|
36G |
16 |
750G |
|
2 准备linux系统配置环境
（全部主机都要操作，全部用root用户）
2.1 修改主机名
vim /etc/hostname
三台机器分别改成：
cluster2-master
cluster2-slave1
cluster2-slave22.2 配置主机名ip映射
cdh推荐我们使用dns服务进行映射。主机数量比较少所以直接
三台机器都要做
vim /etc/hosts
192.168.1X2.x1 cluster2-master
192.168.1X2.x2 cluster2-slave1
192.168.1X2.x3 cluster2-slave2
2.3 关闭防火墙
查看状态：firewall-cmd --state
关闭防火墙：systemctl stop firewalld
防止开机自启：systemctl disable firewalld
查看状态：firewall-cmd --state
2.4 关闭selinux
cdh官网也说了要关，不关会有一些问题。
- Cloudera Enterprise, with the exception of Cloudera Navigator Encrypt, is supported on platforms with Security-Enhanced Linux (SELinux) enabled and in enforcing mode. Cloudera is not responsible for policy support or policy enforcement. If you experience issues with SELinux, contact your OS provider.
vi /etc/selinux/config
将SELINUX设置为：SELINUX=disabled注意别改错了，是SELINUX不是SELINUXTYPE，如果改错了将无法开机。
我就曾经见到有人把disabled单词写错了导致无法开机
比如：
3 软件依赖
所有机器都做依赖
3.1 安装jdk
仅支持64位JDK。Cloudera Manager 6和CDH 6不支持JDK7。尽管CDH 5的所有版本均支持JDK 7，但由Cloudera Manager 6.x管理的CDH 5.x群集必须在所有群集主机上使用JDK 8。Cloudera Manager 6和CDH 6支持Oracle JDK8。CDH5.3和更高版本也支持JDK 8。
Cloudera Enterprise 6.1.0和更高版本以及Cloudera Enterprise 5.16.1和更高版本支持OpenJDK 8。有关安装和迁移的说明，请参阅升级JDK。
CDH 6不支持使用JDK 7编译的应用程序。在升级到CDH 6之前，必须使用JDK 8重新编译应用程序。
任何Cloudera Manager或CDH版本均不支持Oracle JDK 9。
除非明确排除，否则Cloudera支持从引入该支持的版本开始对主要JDK版本的更高版本的更新。当安全受到威胁时，Cloudera会排除或删除对某些Java更新的支持。
不支持在不同JDK版本上的同一集群内运行CDH节点。所有群集主机必须使用相同的JDK更新级别。
除非明确排除，否则从最低要求的版本开始，Cloudera Enterprise 6支持所有JDK 8更新。高于未列出的最小值的更新受支持，但未经测试。Cloudera Manager 6和CDH 6需要JDK 8。
警告：
- 由于JDK问题影响CDH功能，因此不支持JDK 8u40、8u45、8u60和8u242：
- JDK 8u40和8u45受JDK-8077155的影响，这会影响某些Web UI的HTTP身份验证。
- JDK 8u60与AWS开发工具包不兼容，并导致DistCP出现问题。有关更多信息，请参见知识库文章。
- JDK 8u242导致Kerberos故障。
- Hue中的Oozie工作流图显示不适用于低于8u40的JDK版本。
注意：
他说安装java要安装到/usr/java，
请使用yum安装或者rpm安装，这样jdk以会安装到默认位置上，由于CM自身机制的问题，如果这两个软件未安装到默认的位置上会导致许多额外的配置以及很多不期而遇的问题我之前用的tar包安装jdk总是报错JAVA_HOME找不到错误现在我使用rpm安装之后就没问题了java 安装在/usr/java/目录
3.1.1 检查系统内原有jdk
rpm -qa | grep java
检查系统内原有jdk如果有删除干净
我的是新做的系统不需要删除
3.1.2 rpm 安装jdk
[root@cluster2-master software]# rpm -ivh jdk-8u131-linux-x64.rpm
[root@cluster2-slave1 software]# rpm -ivh jdk-8u131-linux-x64.rpm
[root@cluster2-slave2 software]# rpm -ivh jdk-8u131-linux-x64.rpm
3.1.3 配置JAVA_HOME
[root@cluster2-master jdk1.8.0_131]# vim /etc/profile
[root@cluster2-slave1 jdk1.8.0_131]# vim /etc/profile
[root@cluster2-slave2 jdk1.8.0_131]# vim /etc/profileexport JAVA_HOME=/usr/java/jdk1.8.0_131
export PATH=$PATH:$JAVA_HOME/bin:$JAVA_HOME/sbin
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar安装完成后
source /etc/profile
java -version验证下java是否安装成功
3.2 ssh密钥配置
配置免秘钥登录，3台机器都做
3.2.1 生成公钥/私钥对
一直回车即可
[root@cluster2-master software]# ssh-keygen -t rsa
3.2.2 分发公钥
所有机器都这么做给自己反一份公钥，给另外两台也发一份公钥
[root@cluster2-master software]# ssh-copy-id -i cluster2-master //发给自己
[root@cluster2-master software]# ssh-copy-id -i cluster2-slave1 //发给slave1
[root@cluster2-master software]# ssh-copy-id -i cluster2-slave2 //发给slave2
3.3 修改时区
所有机器都这么做
3.3.1 现有系统时区
[root@cluster2-master etc]# ls -al /etc/localtime 
3.3.2 修改成东八区
[root@cluster2-master ~]# timedatectl set-timezone Asia/Shanghai[root@cluster2-master ~]# timedatectl
Local time: Mon 2020-03-30 13:31:31 CST
Universal time: Mon 2020-03-30 05:31:31 UTC
RTC time: Mon 2020-03-30 05:31:31
Time zone: Asia/Shanghai (CST, +0800)
NTP enabled: no
NTP synchronized: yes
RTC in local TZ: no
DST active: n/a
[root@cluster2-master ~]# ls -al /etc/localtime
lrwxrwxrwx 1 root root 35 Mar 30 13:31 /etc/localtime -> ../usr/share/zoneinfo/Asia/Shanghai
3.4 配置ntp
3.4.1 安装ntp
[root@cluster2-master ~]# yum -y install ntp
[root@cluster2-slave1 ~]# yum -y install ntp
[root@cluster2-slave2 ~]# yum -y install ntp
3.4.2 本地ntp server端配置
我选着cluster2-master 机器作为我的ntp server,因此这一步只在cluster2-master上做
增加
server ntp.aliyun.com
[root@cluster2-master etc]# systemctl restart ntpd.service
[root@cluster2-master etc]# systemctl status ntpd.service
[root@cluster2-master etc]# ntpq -p
设置ntp开机启动
因为centos7 默认同步方式是chrony同步，如果chrony已经是开机启动了（systemctl is-enabled chronyd检查是否开机启动），然后使用systemctl enable ntpd 设置开机启动不生效，需要先关掉chrony的开机启动：
systemctl is-enabled chrony
systemctl disable chronyd
然后再设置ntp 开机启动
systemctl enable ntpdsystemctl is-enabled ntpd
3.4.3 本地ntp 客户端
我的客户端在cluster2-slave1和cluster2-slave2这两台机器上所以只需要在这两台机器上做如下操作
vim /etc/ntp.conf
启动ntp 服务
systemctl restart ntpd.service // 启动服务
systemctl status ntpd.service // 查看状态关闭chrony的开机启动
systemctl disable chronyd启动ntp开机启动
systemctl enable ntpd //ntp开机启动
systemctl is-enabled ntpd //查看开机启动是否设置成功
3.5 python依赖
Python-默认情况下，操作系统随附的Python版本和更高版本均支持Hue除外的Cloudera Enterprise 6和更高版本，但与Python 3.0或更高版本不兼容。
例如，Cloudera Enterprise 6在兼容RHEL 6的操作系统上需要Python 2.6或更高版本，但在兼容RHEL 7的操作系统上需要Python 2.7或更高版本。
CDH 6中的Hue在所有操作系统上都需要Python 2.7或更高版本。对于运行Hue的RHEL 6兼容操作系统，必须手动安装Python 2.7。
Spark 2需要Python 2.7或更高版本。如果默认情况下未选择正确的Python级别，请设置PYSPARK_PYTHON 和 PYSPARK_DRIVER_PYTHON 环境变量，以在运行pyspark命令之前指向正确的Python可执行文件。
不支持Python 3。
我的系统centos7.3 自带python2.7.5
3.6 Perl 依赖
Perl -Cloudera Manager需要perl。
后面安装mysql服务的时候也需要这个依赖
perl在linux下可以理解为增强版本的shell，是一种脚本语言，具有程序结构，很多内建功能，也方便调用其它程序。
我的系统centos7.3 自带 perl 5.16.3
3.7 python-psycopg2依赖
3台机器都做
python-psycopg2 -Cloudera Manager 6对软件包有依赖性python-psycopg2。CDH 6中的色相需要更高版本的psycopg2超过Cloudera Manager依赖项所需的数量。有关更多信息，请参阅安装psycopg2Python软件包。
psycopg2 库是 python 用来操作 postgreSQL 数据库的第三方库。使用时需要先进行安装。pip install psycopg2。
3.7.1 安装epel-release
yum -y install epel-release因为安装yum 安装pip 时使用
如果不安装epel-release直接安装pip 会报错
3.7.2 安装python-pip
[root@cluster2-master ~]# yum -y install python-pip
做了3.7.1后安装成功
3.7.3 升级pip
不升级安装python-psycopg2报错如下：
升级
[root@cluster2-master ~]# pip install --upgrade pip升级后
3.7.3 pip 安装 psycopg2
安装完成python-pip并升级之后 之后就可以安装python-psycopg2了,但是还有一些依赖如下
其他依赖：
yum install python-devel postgresql-devel gcc不安装 python-devel postgresql-devel会报错如下
不安装gcc 会报错
安装psycopg2
[root@cluster2-slave1 ~]# pip install psycopg2
3.7.4 总结
pip 安装psycopg2需要提前安装一堆依赖
yum -y install epel-release // 安装python-pip时依赖
yum -y install python-pip // 安装psycopg2时依赖
pip install --upgrade pip // 我在安装psycopg2时提示pip 版本20.0.2
yum install python-devel postgresql-devel gcc // 安装-psycopg2时依赖依赖或者报错
pip install psycopg2 //安装psycopg2
3.8 iproute package依赖
iproute软件包 -Cloudera Enterprise 6依赖于iproute的包。任何运行Cloudera Manager Agent的主机都需要该软件包。所需版本取决于操作系统：就是上面的表
在Linux中，我们经常会涉及到修改主机的路由列表，以前都是使用route这个命令，但是在CentOS7中默认已经不安装net-tools这个包，所以默认是没有route这个命令的，可以使用ip route 代替
系统中有ip route 不需要重新安装
4 数据库要求
4.1 安装mysql
我选择的版本是mysql5.6.47
4.1.1 检查系统总是否原有mysql如果有删除
[root@cluster2-master ~]# rpm -qa | grep mysql4.1.2 解压mysql tar 包
[root@cluster2-master mysql]# tar -xvf MySQL-5.6.47-1.el7.x86_64.rpm-bundle.tar 
4.1.3 rpm安装mysql服务
[root@cluster2-master mysql]# rpm -ivh MySQL-server-5.6.47-1.el7.x86_64.rpm
出现依赖包错误：
表面看是perl 依赖实际上我的系统里是有perl 的已经安装好的，真正缺少的依赖是autoconf
然后安装autoconf 可以了
[root@cluster2-master mysql]# yum -y install autoconf继续安装mysql server 又会出现如下警告导致无法安装成功
这种情况是mariadb-libs-1:5.5.56-2.el7.x86_64包冲突了。把它缷掉即可。
[root@cluster2-master mysql]# yum -y remove mariadb-libs
最后安装就可以成功了
[root@cluster2-master mysql]# rpm -ivh MySQL-server-5.6.47-1.el7.x86_64.rpm
warning: MySQL-server-5.6.47-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:MySQL-server-5.6.47-1.el7 ################################# [100%]
warning: user mysql does not exist - using root
warning: group mysql does not exist - using root
warning: user mysql does not exist - using root
warning: group mysql does not exist - using root
2020-03-27 22:07:01 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2020-03-27 22:07:01 0 [Note] Ignoring --secure-file-priv value as server is running with --bootstrap.
2020-03-27 22:07:01 0 [Note] /usr/sbin/mysqld (mysqld 5.6.47) starting as process 5021 ...
2020-03-27 22:07:01 5021 [Note] InnoDB: Using atomics to ref count buffer pool pages
2020-03-27 22:07:01 5021 [Note] InnoDB: The InnoDB memory heap is disabled
2020-03-27 22:07:01 5021 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2020-03-27 22:07:01 5021 [Note] InnoDB: Memory barrier is not used
2020-03-27 22:07:01 5021 [Note] InnoDB: Compressed tables use zlib 1.2.11
2020-03-27 22:07:01 5021 [Note] InnoDB: Using Linux native AIO
2020-03-27 22:07:01 5021 [Note] InnoDB: Using CPU crc32 instructions
2020-03-27 22:07:01 5021 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2020-03-27 22:07:01 5021 [Note] InnoDB: Completed initialization of buffer pool
2020-03-27 22:07:01 5021 [Note] InnoDB: The first specified data file ./ibdata1 did not exist: a new database to be created!
2020-03-27 22:07:01 5021 [Note] InnoDB: Setting file ./ibdata1 size to 12 MB
2020-03-27 22:07:01 5021 [Note] InnoDB: Database physically writes the file full: wait...
2020-03-27 22:07:01 5021 [Note] InnoDB: Setting log file ./ib_logfile101 size to 48 MB
2020-03-27 22:07:02 5021 [Note] InnoDB: Setting log file ./ib_logfile1 size to 48 MB
2020-03-27 22:07:02 5021 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
2020-03-27 22:07:02 5021 [Warning] InnoDB: New log files created, LSN=45781
2020-03-27 22:07:02 5021 [Note] InnoDB: Doublewrite buffer not found: creating new
2020-03-27 22:07:02 5021 [Note] InnoDB: Doublewrite buffer created
2020-03-27 22:07:02 5021 [Note] InnoDB: 128 rollback segment(s) are active.
2020-03-27 22:07:02 5021 [Warning] InnoDB: Creating foreign key constraint system tables.
2020-03-27 22:07:02 5021 [Note] InnoDB: Foreign key constraint system tables created
2020-03-27 22:07:02 5021 [Note] InnoDB: Creating tablespace and datafile system tables.
2020-03-27 22:07:02 5021 [Note] InnoDB: Tablespace and datafile system tables created.
2020-03-27 22:07:02 5021 [Note] InnoDB: Waiting for purge to start
2020-03-27 22:07:03 5021 [Note] InnoDB: 5.6.47 started; log sequence number 0
2020-03-27 22:07:03 5021 [Note] RSA private key file not found: /var/lib/mysql//private_key.pem. Some authentication plugins will not work.
2020-03-27 22:07:03 5021 [Note] RSA public key file not found: /var/lib/mysql//public_key.pem. Some authentication plugins will not work.
A random root password has been set. You will find it in '/root/.mysql_secret'.
2020-03-27 22:07:03 5021 [Note] Binlog end
2020-03-27 22:07:03 5021 [Note] InnoDB: FTS optimize thread exiting.
2020-03-27 22:07:03 5021 [Note] InnoDB: Starting shutdown...
2020-03-27 22:07:06 5021 [Note] InnoDB: Shutdown completed; log sequence number 1625977
2020-03-27 22:07:06 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2020-03-27 22:07:06 0 [Note] Ignoring --secure-file-priv value as server is running with --bootstrap.
2020-03-27 22:07:06 0 [Note] /usr/sbin/mysqld (mysqld 5.6.47) starting as process 5044 ...
2020-03-27 22:07:06 5044 [Note] InnoDB: Using atomics to ref count buffer pool pages
2020-03-27 22:07:06 5044 [Note] InnoDB: The InnoDB memory heap is disabled
2020-03-27 22:07:06 5044 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2020-03-27 22:07:06 5044 [Note] InnoDB: Memory barrier is not used
2020-03-27 22:07:06 5044 [Note] InnoDB: Compressed tables use zlib 1.2.11
2020-03-27 22:07:06 5044 [Note] InnoDB: Using Linux native AIO
2020-03-27 22:07:06 5044 [Note] InnoDB: Using CPU crc32 instructions
2020-03-27 22:07:06 5044 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2020-03-27 22:07:06 5044 [Note] InnoDB: Completed initialization of buffer pool
2020-03-27 22:07:06 5044 [Note] InnoDB: Highest supported file format is Barracuda.
2020-03-27 22:07:06 5044 [Note] InnoDB: 128 rollback segment(s) are active.
2020-03-27 22:07:06 5044 [Note] InnoDB: Waiting for purge to start
2020-03-27 22:07:06 5044 [Note] InnoDB: 5.6.47 started; log sequence number 1625977
2020-03-27 22:07:06 5044 [Note] RSA private key file not found: /var/lib/mysql//private_key.pem. Some authentication plugins will not work.
2020-03-27 22:07:06 5044 [Note] RSA public key file not found: /var/lib/mysql//public_key.pem. Some authentication plugins will not work.
2020-03-27 22:07:06 5044 [Note] Binlog end
2020-03-27 22:07:06 5044 [Note] InnoDB: FTS optimize thread exiting.
2020-03-27 22:07:06 5044 [Note] InnoDB: Starting shutdown...
2020-03-27 22:07:08 5044 [Note] InnoDB: Shutdown completed; log sequence number 1625987
A RANDOM PASSWORD HAS BEEN SET FOR THE MySQL root USER !
You will find that password in '/root/.mysql_secret'.
You must change that password on your first connect,
no other statement but 'SET PASSWORD' will be accepted.
See the manual for the semantics of the 'password expired' flag.
Also, the account for the anonymous user has been removed.
In addition, you can run:
/usr/bin/mysql_secure_installation
which will also give you the option of removing the test database.
This is strongly recommended for production servers.
See the manual for more instructions.
Please report any problems at http://bugs.mysql.com/
The latest information about MySQL is available on the web at
http://www.mysql.com
Support MySQL by buying support/licenses at http://shop.mysql.com
New default config file was created as /usr/my.cnf and
will be used by default by the server when you start it.
You may edit this file to change server settings
4.1.4 依次安装mysql其他组件
[root@cluster2-master mysql]# rpm -ivh MySQL-client-5.6.47-1.el7.x86_64.rpm // mysql客户端
[root@cluster2-master mysql]# rpm -ivh MySQL-shared-compat-5.6.47-1.el7.x86_64.rpm //这一步就是上面cdh官网文档说明要必须要安装的。
[root@cluster2-master mysql]# rpm -ivh MySQL-shared-5.6.47-1.el7.x86_64.rpm
4.2 修改mysql配置
4.2.1启动mysql
[root@cluster2-master mysql]# systemctl start mysql4.2.2 修改root密码
获取原始密码
[root@cluster2-master mysql]# cat /root/.mysql_secret
# The random password set for the root user at Fri Mar 27 22:07:03 2020 (local time): 8Aj9y.....B9rcBY
可知原始密码是：8Aj9y.....B9rcBY
[root@cluster2-master mysql]# mysql -uroot -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.6.47
Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql>
修改密码为：1xxxx6
mysql> set password = password('1xxxxx6');
Query OK, 0 rows affected (0.00 sec)
mysql> grant all privileges on *.* to 'root'@'%' identified by '1xxxx6';
Query OK, 0 rows affected (0.00 sec)
mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)
4.2.3 修改字符集要求
对所有自定义数据库使用UTF8编码。MySQL和MariaDB必须使用MySQL utf8编码，而不是utf8mb4。
查看现有字符集
mysql> show variables like 'char%';
+--------------------------+----------------------------+
| Variable_name | Value |
+--------------------------+----------------------------+
| character_set_client | utf8 |
| character_set_connection | utf8 |
| character_set_database | latin1 |
| character_set_filesystem | binary |
| character_set_results | utf8 |
| character_set_server | latin1 |
| character_set_system | utf8 |
| character_sets_dir | /usr/share/mysql/charsets/ |
+--------------------------+----------------------------+
8 rows in set (0.00 sec)
退出mysql，修改下配置
[root@cluster2-master mysql]# vim /usr/my.cnf
重启下mysql
[root@cluster2-master mysql]# systemctl restart mysqlmysql> show variables like 'char%';
+--------------------------+----------------------------+
| Variable_name | Value |
+--------------------------+----------------------------+
| character_set_client | utf8 |
| character_set_connection | utf8 |
| character_set_database | utf8 |
| character_set_filesystem | binary |
| character_set_results | utf8 |
| character_set_server | utf8 |
| character_set_system | utf8 |
| character_sets_dir | /usr/share/mysql/charsets/ |
+--------------------------+----------------------------+
8 rows in set (0.00 sec)
character_set_database | utf8
character_set_server | utf8
可见这两个都是utf8了
到此mysql安装成功，使用navicat登录验证成功
5 数据库部署
5.1 主节点部署JDBC
[root@cluster2-master ~]# mkdir -p /usr/share/java
[root@cluster2-master software]# mv mysql-connector-java-5.1.48.jar /usr/share/java/
[root@cluster2-master software]# cd /usr/share/java/
[root@cluster2-master java]# mv mysql-connector-java-5.1.48.jar mysql-connector-java.jar 注意：mysql-connector-java-****.jar 必须重命名为 ：mysql-connector-java.jar
为什么要这么做因为后面agent的配置文件里我们可以看到如下的默认数据库驱动地址指向
我为了保险起见吧各个salve从节点也创建了/usr/share/java目录并把mysql-connector-java.jar scp 拷贝了过去
5.2 创建数据库
创建CDH源数据库、用户、amon服务的数据库
mysql> create database cmf DEFAULT CHARACTER SET utf8; # 创建CDH源数据库
Query OK, 1 row affected (0.00 sec)
mysql> create database amon DEFAULT CHARACTER SET utf8; #amon服务的数据库
Query OK, 1 row affected (0.00 sec)
mysql> grant all on *.* TO 'cmf'@'%' IDENTIFIED BY '1xxxx6';
Query OK, 0 rows affected (0.00 sec)
mysql> grant all on *.* TO 'amon'@'%' IDENTIFIED BY '1xxxx6';
Query OK, 0 rows affected (0.01 sec)
mysql> create database rman DEFAULT CHARACTER SET utf8;
Query OK, 1 row affected (0.00 sec)
mysql> grant all on rman.* TO 'rman'@'%' IDENTIFIED BY '1xxxx6';
Query OK, 0 rows affected (0.00 sec)
mysql> create database hue DEFAULT CHARACTER SET utf8; #创建hue源数据库
Query OK, 1 row affected (0.00 sec)
mysql> grant all on hue.* TO 'hue'@'%' IDENTIFIED BY '1xxxx6';
Query OK, 0 rows affected (0.00 sec)
mysql> create database hive DEFAULT CHARACTER SET utf8; #创建hive源数据库
Query OK, 1 row affected (0.00 sec)
mysql> grant all on hive.* TO 'hive'@'%' IDENTIFIED BY '1xxxx6';
Query OK, 0 rows affected (0.00 sec)
mysql> create database nav DEFAULT CHARACTER SET utf8;
Query OK, 1 row affected (0.00 sec)
mysql> grant all on nav.* TO 'nav'@'%' IDENTIFIED BY '1xxxx6';
Query OK, 0 rows affected (0.00 sec)
mysql> create database oozie DEFAULT CHARACTER SET utf8; #创建oozie源数据库
Query OK, 1 row affected (0.00 sec)
mysql> grant all on oozie.* TO 'oozie'@'%' IDENTIFIED BY '1xxxx6';
Query OK, 0 rows affected (0.00 sec)
mysql> create database sqoop DEFAULT CHARACTER SET utf8; #创建sqoop源数据库
Query OK, 1 row affected (0.00 sec)
mysql> grant all on sqoop.* TO 'sqoop'@'%' IDENTIFIED BY '1xxxx6';
Query OK, 0 rows affected (0.00 sec)
mysql> create database navms DEFAULT CHARACTER SET utf8;
Query OK, 1 row affected (0.00 sec)
mysql> grant all on navms.* TO 'navms'@'%' IDENTIFIED BY '1xxxx6';
Query OK, 0 rows affected (0.00 sec)
mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)
6 离线部署CM server及agent
需要提前下载好cm6.3.1-redhat7.tar.gz安装压缩包
本文刚开始提供的百度网盘可以下载，或者自己去官网找
6.1 创建软件目录并解压软件
所有节点
[root@cluster2-master ~]# mkdir -p /opt/cloudera-manager
[root@cluster2-slave1 ~]# mkdir -p /opt/cloudera-manager
[root@cluster2-slave2 ~]# mkdir -p /opt/cloudera-manager解压到 /opt/cloudera-manager
[root@cluster2-master software]# tar -zxf cm6.3.1-redhat7.tar.gz -C /opt/cloudera-manager/解压后的目录结构
[root@cluster2-master cloudera-manager]# tree
.
└── cm6.3.1
├── repodata
│ ├── 3662f97de72fd44c017bb0e25cee3bc9398108c8efb745def12130a69df2ecb2-filelists.sqlite.bz2
│ ├── 43f3725f730ee7522712039982aa4befadae4db968c8d780c8eb15ae9872cd4d-primary.xml.gz
│ ├── 49e4d60647407a36819f1d8ed901258a13361749b742e3be9065025ad31feb8e-filelists.xml.gz
│ ├── 8afda99b921fd1538dd06355952719652654fc06b6cd14515437bda28376c03d-other.sqlite.bz2
│ ├── b9300879675bdbc300436c1131a910a535b8b5a5dc6f38e956d51769b6771a96-primary.sqlite.bz2
│ ├── e28836e19e07f71480c4dad0f7a87a804dc93970ec5277ad95614e8ffcff0d58-other.xml.gz
│ ├── repomd.xml
│ ├── repomd.xml.asc
│ └── repomd.xml.key
├── RPM-GPG-KEY-cloudera
├── RPMS
│ ├── noarch
│ └── x86_64
│ ├── cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm
│ ├── cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm
│ ├── cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm
│ ├── cloudera-manager-server-db-2-6.3.1-1466458.el7.x86_64.rpm
│ ├── enterprise-debuginfo-6.3.1-1466458.el7.x86_64.rpm
│ └── oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm
└── SRPMS
6 directories, 16 files
注意要使用tree 命令需要先安装tree :yum install tree
scp 到各个slave节点的/opt/cloudera-manager目录
三台机器属于同一内网，相互之间传递文件比我从自己的win机器传递上去快而且他们之间是相互免秘钥的。所以使用scp传递
[root@cluster2-master cloudera-manager]# scp -r cm6.3.1 root@cluster2-slave1:/opt/cloudera-manager/
[root@cluster2-master cloudera-manager]# scp -r cm6.3.1 root@cluster2-slave2:/opt/cloudera-manager/-p :保留原文件的修改时间，访问时间和访问权限。
-r： 递归复制整个目录。
6.2 主节点master部署cm server
cd /opt/cloudera-manager/cm6.3.1/RPMS/x86_646.2.1 rpm安装cloudera manager 守护进程
[root@cluster2-master x86_64]# rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm //rpm安装cloudera manager 守护进程
warning: cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:cloudera-manager-daemons-6.3.1-14################################# [100%]
6.2.2 rpm安装cloudera manager 服务进程
[root@cluster2-master x86_64]# rpm -ivh cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm
warning: cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:cloudera-manager-server-6.3.1-146################################# [100%]
Created symlink from /etc/systemd/system/multi-user.target.wants/cloudera-scm-server.service to /usr/lib/systemd/system/cloudera-scm-server.service.
6.3 所有节点作为cm agent部署
所有节点当然包括master节点也要部署agent
6.3.1 安装cloudera agent 前要先安装的依赖
yum install psmisc libxslt cyrus-sasl-gssapi lsb openssl-devel python-psycopg2 MySQL-python不然会报错如下
psmisc is needed by cloudera-manager-agent-6.3.1-1466458.el7.x86_64
libxslt is needed by cloudera-manager-agent-6.3.1-1466458.el7.x86_64
cyrus-sasl-gssapi is needed by cloudera-manager-agent-6.3.1-1466458.el7.x86_64
/lib/lsb/init-functions is needed by cloudera-manager-agent-6.3.1-1466458.el7.x86_64
openssl-devel is needed by cloudera-manager-agent-6.3.1-1466458.el7.x86_64
python-psycopg2 is needed by cloudera-manager-agent-6.3.1-1466458.el7.x86_64
MySQL-python is needed by cloudera-manager-agent-6.3.1-1466458.el7.x86_64
其中 yum install lsb 可以解决 ：/lib/lsb/init-functions is needed by cloudera-manager-agent-6.3.1-1466458.el7.x86_64的问题
6.3.2 安装 cloudera agnet
cd /opt/cloudera-manager/cm6.3.1/RPMS/x86_64
rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm // master节点已经执行过了可以跳过，salve节点要需要执行这一步
rpm -ivh cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm // master节点也需要执行依赖添加后可以安装成功了
master 节点：
[root@cluster2-master x86_64]# rpm -ivh cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm
warning: cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:cloudera-manager-agent-6.3.1-1466################################# [100%]
Created symlink from /etc/systemd/system/multi-user.target.wants/cloudera-scm-agent.service to /usr/lib/systemd/system/cloudera-scm-agent.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/supervisord.service to /usr/lib/systemd/system/supervisord.service.
cluster2-slave1
[root@cluster2-slave1 x86_64]# rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm
warning: cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:cloudera-manager-daemons-6.3.1-14################################# [100%]
[root@cluster2-slave1 x86_64]# rpm -ivh cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm
warning: cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:cloudera-manager-agent-6.3.1-1466################################# [100%]
Created symlink from /etc/systemd/system/multi-user.target.wants/cloudera-scm-agent.service to /usr/lib/systemd/system/cloudera-scm-agent.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/supervisord.service to /usr/lib/systemd/system/supervisord.service.cluster2-slave2
[root@cluster2-slave2 x86_64]# rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm
warning: cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:cloudera-manager-daemons-6.3.1-14################################# [100%]
[root@cluster2-slave2 x86_64]# rpm -ivh cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm
warning: cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:cloudera-manager-agent-6.3.1-1466################################# [100%]
Created symlink from /etc/systemd/system/multi-user.target.wants/cloudera-scm-agent.service to /usr/lib/systemd/system/cloudera-scm-agent.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/supervisord.service to /usr/lib/systemd/system/supervisord.service.
6.4 修改agent配置，指向server节点地址
所有节点都做
修改前
修改
[root@cluster2-master x86_64]# sed -i "s/server_host=localhost/server_host=cluster2-master/g" /etc/cloudera-scm-agent/config.ini
[root@cluster2-slave1 x86_64]# sed -i "s/server_host=localhost/server_host=cluster2-master/g" /etc/cloudera-scm-agent/config.ini
[root@cluster2-slave2 x86_64]# sed -i "s/server_host=localhost/server_host=cluster2-master/g" /etc/cloudera-scm-agent/config.ini修改后
6.5 修改master节点的server配置
vim /etc/cloudera-scm-server/db.properties
[root@cluster2-master java]# vim /etc/cloudera-scm-server/db.properties
# Copyright (c) 2012 Cloudera, Inc. All rights reserved.
#
# This file describes the database connection.
#
# The database type
# Currently 'mysql', 'postgresql' and 'oracle' are valid databases.
com.cloudera.cmf.db.type=mysql
# The database host
# If a non standard port is needed, use 'hostname:port'
#com.cloudera.cmf.db.host=localhost
com.cloudera.cmf.db.host=cluster2-master
# The database name
#com.cloudera.cmf.db.name=cmf
com.cloudera.cmf.db.name=cmf
# The database user
#com.cloudera.cmf.db.user=cmf
com.cloudera.cmf.db.user=cmf
# The database user's password
#com.cloudera.cmf.db.password=
com.cloudera.cmf.db.password=1xxxx6
# The db setup type
# After fresh install it is set to INIT
# and will be changed post config.
# If scm-server uses Embedded DB then it is set to EMBEDDED
# If scm-server uses External DB then it is set to EXTERNAL
# com.cloudera.cmf.db.setupType=INIT
# 这里不要设置成External 我刚才是就是这样设置的，导致web无法访问，agent 连接被拒绝，下面这条注释掉就可以了
# com.cloudera.cmf.db.setupType=External
~ 刚开始设置成：com.cloudera.cmf.db.setupType=External
报错如下
RuntimeException: INVALID value is specified for db.setupType
把 com.cloudera.cmf.db.setupType=External 注释掉就行了
7 主节点部署离线parcel源
需要提前下载好相关parcel源
本文刚开始提供的百度网盘地址可以下载，或者官网去下载
CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel
CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel.sha1
manifest.json 7.1 安装httpd
yum install -y httpd
[root@cluster2-master ~]# yum install -y httpd7.2 部署离线parcel源
① 创建目录
[root@cluster2-master html]# mkdir -p /var/www/html/liucf/cdh6_3_1/parcel
② 移动下载好的parcel相关文件到 /var/www/html/liucf/cdh6_3_1/parcel
[root@cluster2-master software]# mv CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel /var/www/html/liucf/cdh6_3_1/parcel/CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel
[root@cluster2-master software]# mv CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel.sha1 /var/www/html/liucf/cdh6_3_1/parcel/CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel.sha
[root@cluster2-master software]# mv manifest.json /var/www/html/liucf/cdh6_3_1/parcel/manifest.json7.3 启动httpd
[root@cluster2-master software]# systemctl start httpd7.4 web 访问本地parcel源
http://cluster2-master/liucf/cdh6_3_1/parcel/
8 启动cloudera manager 的server和agent
8.1 master节点启动server
[root@cluster2-master parcel]# jps // 启动前
7769 Jps
[root@cluster2-master parcel]# systemctl start cloudera-scm-server // 启动
[root@cluster2-master parcel]# jps // 启动后
7957 Jps
7881 Main
查看日志目录
[root@cluster2-master parcel]# ls -al /var/log/cloudera-scm-server/
total 56
drwxr-x--- 2 cloudera-scm cloudera-scm 90 Mar 28 20:51 .
drwxr-xr-x. 13 root root 4096 Mar 28 16:20 ..
-rw-r----- 1 cloudera-scm cloudera-scm 52940 Mar 28 20:51 cloudera-scm-server.log
-rw-r----- 1 cloudera-scm cloudera-scm 0 Mar 28 20:51 cmf-server-nio.log
-rw-r----- 1 cloudera-scm cloudera-scm 0 Mar 28 20:51 cmf-server-perf.log
[root@cluster2-master cloudera-scm-server]# more cloudera-scm-server.log查看日志没报错启动正常
通过进程状态也能看到在运行
[root@cluster2-master cloudera-scm-agent]# systemctl status cloudera-scm-server
8.2 启动agent
所有节点都启动
[root@cluster2-master ~]# systemctl start cloudera-scm-agent
[root@cluster2-slave1 ~]# systemctl start cloudera-scm-agent
[root@cluster2-slave2 ~]# systemctl start cloudera-scm-agent
查看日志目录
[root@cluster2-master cloudera-scm-agent]# ls -al /var/log/cloudera-scm-agent
total 40
drwxr-x--- 2 cloudera-scm cloudera-scm 178 Mar 28 21:09 .
drwxr-xr-x. 13 root root 4096 Mar 28 16:20 ..
-rw-r--r-- 1 root root 13369 Mar 28 21:09 cloudera-scm-agent.log
-rw-r--r-- 1 root root 202 Mar 28 21:09 cmf_listener.log
-rw-r--r-- 1 root root 735 Mar 28 21:09 scm-status.log
-rw-r--r-- 1 root root 820 Mar 28 21:09 status-stderr.log
-rw-r--r-- 1 root root 0 Mar 28 21:09 status-stdout.log
-rw-r--r-- 1 root root 999 Mar 28 21:09 supervisord.log
-rw-r--r-- 1 root root 1408 Mar 28 21:09 supervisord.out通过进程状态查看
[root@cluster2-master cloudera-scm-agent]# systemctl status cloudera-scm-agent
注意安装后一定要检查一下日志是否有报错，这个我安装的过程中使用
systemctl status cloudera-scm-server
systemctl status cloudera-scm-agent
查看进程都是active（running）状态但是实际日志是有报错的（见上面5.5），经过排查才解决掉问题，正常访问
9 web页面操作
http://cluster2-master:7180
9.1 登录
用户名：admin
密码 :admin
9.2 登录后
点击继续->勾选【是的，我接受最终用户许可条款和条件。】，继续->
然后选择【Cloudera EnterpriseCloudera Enterprise试用版】，继续
10 在Cloudera Manager中添加集群
10.1 集群基础
10.2 指定主机
多个主机可以用逗号分割
cluster2-master,cluster2-slave1,cluster2-slave2
也可以使用后面【当前管理的主机（3）】
10.3 选择存储库
可见【选择存储库】这一列在我选着的版本里不支持使用自定义的本地存储库了
更多选项中添加搭建的http镜像
我把我自己的http镜像当做远程parcel存储库路径放到最前面
10.4 Install Parcels
10.5 Inspect Cluster
10.6 集群服务安装
10.6.1初步设置
可见一共有上面截图这些服务
选着从必须品开始安装
对Cloudera发行版的管理和支持，包括Hadoop。
服务：HDFS，YARN（包含MapReduce 2），ZooKeeper，Oozie，Hive和Hue
10.6.2 为服务指定主机
点击【继续】之后
10.6.3 输入已经设置的数据库信息
参考5.2 部分的设置
【测试连接】成功之后点击【继续】
10.6.4 指定数据磁盘目录位置
如上截图，把需要存储在本地磁盘的东西都指定到数据盘挂载的目录/data下
等上图显示的所有服务都初始化完成之后
10.6.5 安装完成
最后安装完成
完成后成上面这样，因为我装的是测试集群，配置各方面有些不能满足cloudera manager的最低要求，之后进行相应的配置修改就可以使服务表现的是健康的了
后面还可以添加服务，比如spark,hbase...等等
到此
Centos7主机搭建CDH6.X机群完成