(一)离线快速部署cdh5.16.2
文章目录
- 阿里云机器购买
- 环境准备
- 域名配置
- 主机名修改
- 关闭防火墙
- 禁用selinux
- 配置ssh免密登录
- 安装jdk
- 制作本地镜像
- 安装httpd软件
- 安装createrepo软件
- 配置ntp服务器
- 关闭透明大页
- 安装mysql
- mysql5.7安装
- Mysql5.5安装
- 配置CM元数据库
- 安装cm
- 安装CDH
- 问题清单
- 安装好mysql未卸载干净,导致重新安装不能启动
阿里云机器购买
购买云ESC,选择按量付费并且为按量付费,对于按量付费,在不使用集群的情况下进行停机操作,收取较少的费用
选择区域靠近自己办公区域,并且可用区为同一区域,购买三台2核16G阿里云服务器
下一步
选择自定义密码
创建实例
环境准备
域名配置
配置
[root@note01 ~]# vim /etc/hosts
分发
scp /etc/hosts root@note02:/etc/
scp /etc/hosts root@note03:/etc/
主机名修改
vim /etc/sysconfig/network
修改其HOSTNAME
修改完成重启
reboot
注意hostname不可含有特殊字符,并且需要都为小写
关闭防火墙
先永久关闭
chkconfig iptables off
后临时关闭
service iptables stop
禁用selinux
查询selinux状态,保证为disable
sestatus
未关闭需要
vim /etc/selinux/config
设置为disable
临时生效（长期生效需要重启reboot）
setenforce 0
配置ssh免密登录
配置三台虚拟机,并且设置ssh免密登陆,然后敲（三个回车），就会生成两个文件id_rsa（私钥）、id_rsa.pub（公钥）将公钥拷贝到要免密登录的目标机器上
[root@note01 ~]# ssh-copy-id note01
[root@note01 ~]# ssh-copy-id note02
[root@note01 ~]# ssh-copy-id note03
重复操作，配置note01,note02,note03三台服务器免密登录。
安装jdk
上传jdk并解压
[root@note01 software]# tar -zxvf jdk-8u181-linux-x64.tar.gz
创建目录
mkdir -p /usr/java
将jdk包移动到 /usr/java 下
配置环境变量
vim /etc/profile
增加
export JAVA_HOME=/usr/java/jdk1.8.0_181
export PATH=$JAVA_HOME/bin:$PATH
分发
[root@note01 java]# scp /etc/profile root@note02:/etc
[root@note01 java]# scp /etc/profile root@note03:/etc
[root@note01 java]# scp -r /usr/java root@note02:/usr
[root@note01 java]# scp -r /usr/java root@note03:/usr
生效
[root@note01 java]# source /etc/profile
输入java查看版本
java -version
制作本地镜像
安装httpd软件
在这里插入代码片
[root@note03 ~]# yum install -y httpd
启动
[root@note01 java]# service httpd start
设置开机启动
[root@note01 java]# chkconfig httpd on
查看运行状态
[root@note01 java]# service httpd status
httpd (pid 1901) is running...
创建文件夹
[root@note01 java]# mkdir -p /var/www/html/cdh5.16.2/
拷贝cm安装包
安装createrepo软件
[root@note01 cdh5.16.2]# yum install -y createrepo
在/var/www/html/cdh5.16.2文件夹下执行命令
[root@note01 cdh5.16.2]# createrepo ./
配置CDH安装包作为yum源
[root@note01 cdh5.16.2]# cd /etc/yum.repos.d
[root@note01 yum.repos.d]# vim cloudera-manager.repo
这里地址需要更改为内网地址
[cloudera-manager]
name = Cloudera Manager, Version 5.16.2
baseurl = http://172.19.121.116/cdh5.16.2/
gpgcheck = 0
重建Yum Cache
[root@note01 yum.repos.d]# yum clean all && yum makecache
测试本地镜像是否可用
[root@note01 yum.repos.d]# yum list | grep cloudera
配置ntp服务器
所有节点执行
[root@note01 yum.repos.d]# yum install -y ntp
时间服务器节点
[root@note01 yum.repos.d]# vim /etc/ntp.conf
注释掉所有的restrict开头的配置和server开头的配置
管理节点
#允许哪些⽹网段的机器器来同步时间
restrict 172.19.121.116 mask 255.0.0.0 nomodify notrap nopeer noquery
server 127.127.1.0
fudge 127.127.1.0 stratum 10
其他节点
# local clock
server note01
fudge 127.127.1.0 stratum 1
主节点启动NTP服务 service ntpd start,其他节点不启动
测试
[root@note02 ~]# service ntpd stop
Shutting down ntpd: [ OK ]
[root@note02 ~]# ntpdate note01
10 Feb 18:30:20 ntpdate[2839]: adjust time server 172.19.121.116 offset -0.029700 sec
所有节点启动ntpd服务,并且设置开机自启动
[root@note03 ~]# chkconfig ntpd on
查看是否配置成功
[root@note03 ~]# ntpdc -np
remote local st poll reach delay offset disp
=======================================================================
=172.19.121.116 172.19.121.118 11 64 1 0.00021 0.011171 2.81735
关闭透明大页
[root@note01 redhat_transparent_hugepage]# echo never >/sys/kernel/mm/redhat_transparent_hugepage/enabled
[root@note01 redhat_transparent_hugepage]# echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag
修改配置文件
[root@note03 redhat_transparent_hugepage]# vim /etc/rc.d/rc.local
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.
touch /var/lock/subsys/local
echo never >/sys/kernel/mm/redhat_transparent_hugepage/enabled
echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag
安装mysql
mysql5.7安装
查询是否安装
[root@note01 ~]# rpm -qa | grep mysql
卸载mysql
[root@note01 ~]# yum remove mysql-libs
mysql-5.7.23-1.el6.x86_64.rpm-bundle.tar下载：
链接: https://pan.baidu.com/s/1dGsP-LvIHnuD4FZ2VmsLAw 提取码: qu66
rpm -ivh mysql-community-common-5.7.23-1.el6.x86_64.rpm
rpm -ivh mysql-community-libs-5.7.23-1.el6.x86_64.rpm
rpm -ivh mysql-community-devel-5.7.23-1.el6.x86_64.rpm
rpm -ivh mysql-community-client-5.7.23-1.el6.x86_64.rpm
rpm -ivh mysql-community-server-5.7.23-1.el6.x86_64.rpm
启动mysql
service mysqld start
获取mysql的初始密码
[root@note01 mysql]# grep 'temporary password' /var/log/mysqld.log
2020-02-10T15:41:33.823500Z 1 [Note] A temporary password is generated for root@localhost: Q,ZqgbMih368
使用初始密码登录mysql
修改密码参照
https://blog.csdn.net/memory6364/article/details/82426052
Mysql5.5安装
mysql下载地址
http://ftp.ntu.edu.tw/MySQL/Downloads/MySQL-5.5/
查看是否安装mysql
rpm -qa | grep mysql
若安装需要卸载
[root@note01 ~]# yum remove mysql-*
解压
[root@note01 mysql]# tar -vf MySQL-5.5.60-1.el6.x86_64.rpm-bundle.tar
安装
[root@note01 mysql]# ll
total 294884
-rw-r--r-- 1 root root 150978560 Feb 11 11:07 MySQL-5.5.60-1.el6.x86_64.rpm-bundle.tar
-rw-r--r-- 1 7155 31415 15247620 Feb 27 2018 MySQL-client-5.5.60-1.el6.x86_64.rpm
-rw-r--r-- 1 7155 31415 4982612 Feb 27 2018 MySQL-devel-5.5.60-1.el6.x86_64.rpm
-rw-r--r-- 1 7155 31415 51879508 Feb 27 2018 MySQL-embedded-5.5.60-1.el6.x86_64.rpm
-rw-r--r-- 1 7155 31415 39975220 Feb 27 2018 MySQL-server-5.5.60-1.el6.x86_64.rpm
-rw-r--r-- 1 7155 31415 1759068 Feb 27 2018 MySQL-shared-5.5.60-1.el6.x86_64.rpm
-rw-r--r-- 1 7155 31415 3969756 Feb 27 2018 MySQL-shared-compat-5.5.60-1.el6.x86_64.rpm
-rw-r--r-- 1 7155 31415 33155516 Feb 27 2018 MySQL-test-5.5.60-1.el6.x86_64.rpm
[root@note01 mysql]# rpm -ivh ./MySQL-*
安装好后启动
[root@note01 mysql]# service mysql start
Starting MySQL.Logging to '/var/lib/mysql/note01.err'.
..... SUCCESS!
加入开机启动
[root@note01 mysql]# chkconfig mysql on
配置安装MySQL
[root@note01 mysql]# /usr/bin/mysql_secure_installation
依次输入yynyy 启动输入密码环节为自己想要的密码
进入mysql
[root@note01 mysql]# mysql -u root -p
在另外一个节点安装mysql从节点
参照之前的先清理,并卸载mysql,然后安装mysql
在主节点上执行命令
[root@note01 software]# cp /usr/share/mysql/my-medium.cnf /etc/my.cnf
[root@note01 software]# vim /etc/my.cnf
配置好重启,这里只错测试用,生产环境中需要对mysql进行调优处理
重启
[root@note01 software]# service mysql restart
配置从节点
[root@note02 mysql]# cp /usr/share/mysql/my-medium.cnf /etc/my.cnf
在主节点上进入mysql
[root@note01 software]# mysql -u root -p
mysql> GRANT REPLICATION SLAVE ON *.* TO 'root'@'note02' IDENTIFIED BY '123456';
mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION;
mysql> FLUSH tables with read lock;
解锁表
mysql> UNLOCK TABLES;
保留user表数据样式,其余数据全部删除.
mysql> select Host,User,Password from user;
+------+------+-------------------------------------------+
| Host | User | Password |
+------+------+-------------------------------------------+
| % | root | *6BB4837EB74329105EE4568DDA7DC67ED2CA2AD9 |
+------+------+-------------------------------------------+
1 row in set (0.00 sec)
重新锁表
mysql> FLUSH tables with read lock;
创建zyd数据库,备份
[root@note01 module]# mysqldump -h localhost -uroot -p123456 zyd > ./zyd.sql
在从库中创建数据库
mysql> create database zyd;
mysql> use zyd
Database changed
mysql> source zyd.sql
mysql> use mysql;
mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION;
保证user表数据
mysql> select Host,User,Password from user;
+------+------+-------------------------------------------+
| Host | User | Password |
+------+------+-------------------------------------------+
| % | root | *6BB4837EB74329105EE4568DDA7DC67ED2CA2AD9 |
+------+------+-------------------------------------------+
主节点中
mysql> show master status;
+------------------+----------+--------------+------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+------------------+----------+--------------+------------------+
| mysql-bin.000003 | 107 | | test |
+------------------+----------+--------------+------------------+
从节点上执行
mysql> change master to master_host='note01' , master_user='root' , master_password='123456', master_log_file='mysql-bin.000003' , master_log_pos=107;
mysql> start slave;
其中 master_password 要根据实际情况做修改，master_log_file 就是主节点中show master status的值，master_log_pos的值就是上一步中Position的值。
查看状态
mysql> show slave status \G;
Slave_IO及Slave_SQL的值是YES状态，则表示配置完成。
配置CM元数据库
mysql -u root --password='123456' -e 'create database metastore default character set utf8;'
mysql -u root --password='123456' -e "CREATE USER 'hive'@'%' IDENTIFIED BY '123456';"
mysql -u root --password='123456' -e "GRANT ALL PRIVILEGES ON metastore.* TO 'hive'@'%';"
mysql -u root --password='123456' -e "create user 'amon'@'%' identified by '123456';"
mysql -u root --password='123456' -e 'create database amon default character set utf8;'
mysql -u root --password='123456' -e "grant all privileges on amon.* to 'amon'@'%'"
mysql -u root --password='123456' -e "create user 'rman'@'%' identified by '123456'"
mysql -u root --password='123456' -e 'create database rman default character set utf8;'
mysql -u root --password='123456' -e "grant all privileges on rman.* to 'rman'@'%'"
mysql -u root --password='123456' -e "create user 'sentry'@'%' identified by '123456'"
mysql -u root --password='123456' -e 'create database sentry default character set utf8;'
mysql -u root --password='123456' -e "grant all privileges on sentry.* to 'sentry'@'%'"
mysql -u root --password='123456' -e "create user 'nav'@'%' identified by '123456'"
mysql -u root --password='123456' -e 'create database nav default character set utf8;'
mysql -u root --password='123456' -e "grant all privileges on nav.* to 'nav'@'%'"
mysql -u root --password='123456' -e "create user 'navms'@'%' identified by '123456'"
mysql -u root --password='123456' -e 'create database navms default character set utf8;'
mysql -u root --password='123456' -e "grant all privileges on navms.* to 'navms'@'%'"
mysql -u root --password='123456' -e "create user 'cm'@'%' identified by '123456'"
mysql -u root --password='123456' -e 'create database cm default character set utf8;'
mysql -u root --password='123456' -e "grant all privileges on cm.* to 'cm'@'%'"
mysql -u root --password='123456' -e "create user 'oozie'@'%' identified by '123456'"
mysql -u root --password='123456' -e 'create database oozie default character set utf8;'
mysql -u root --password='123456' -e "grant all privileges on oozie.* to 'oozie'@'%'"
mysql -u root --password='123456' -e "create user 'hue'@'%' identified by '123456'"
mysql -u root --password='123456' -e 'create database hue default character set utf8;'
mysql -u root --password='123456' -e "grant all privileges on hue.* to 'hue'@'%'"
mysql -u root --password='123456' -e "FLUSH PRIVILEGES;"
安装cm
mkdir -p /usr/share/java
上传mysql-connector-java-5.1.47.jar到/usr/share/java 目录
[root@note01 java]# ln -s /usr/share/java/mysql-connector-java-5.1.47.jar /usr/share/java/mysql-connector-java.jar
在主节点上执行
[root@note01 java]# yum install cloudera-manager-server.x86_64 -y
格式化数据库
/usr/share/cmf/schema/scm_prepare_database.sh -h note01 mysql cm cm 123456
启动cm server
[root@note01 java]# service cloudera-scm-server start
安装CDH
上传cdh包
改名
[root@note01 parcel-repo]# mv CDH-5.16.2-1.cdh5.16.2.p0.8-el6.parcel.sha1 CDH-5.16.2-1.cdh5.16.2.p0.8-el6.parcel.sha
重启cm Server
[root@note01 parcel-repo]# service cloudera-scm-server restart
Stopping cloudera-scm-server: [ OK ]
Starting cloudera-scm-server: [ OK ]
进入阿里云打开主节点7180端口
配置规则
配置完成通过Chrome访问
http://外网ip:7180/cmf/login
填写用户名和密码登录admin/admin, 接受用户许可条款，点击【继续】
填写三台主机内网地址,按换行分开并检索
选择自定义服务
角色分配如下
以上截图有误需要添加
配置数据库
方便后期挂盘
点击继续会弹出输入账户和密码操作
问题清单
安装好mysql未卸载干净,导致重新安装不能启动
参考
https://www.cnblogs.com/nicknailo/articles/8563456.html