至少三台物理机或者虚拟机
安装CentOS7 mini版,要求可以互相通信
我的是4台虚拟机,配置均为2G+20G
192.168.65.101 n1
192.168.65.102 n2
192.168.65.103 d1
192.168.65.104 d2
CDH-5.15.0-1.cdh5.15.0.p0.21-el7.parcel
CDH-5.15.0-1.cdh5.15.0.p0.21-el7.parcel.sha1
manifest.json
cloudera-manager-centos7-cm5.15.0_x86_64.tar.gz
mysql-5.7.24-1.el7.x86_64.rpm-bundle.tar
jdk-8u191-linux-x64.tar.gz
Python-2.7.15.tgz
1)备份
mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup
2)下载新的CentOS-Base.repo 到/etc/yum.repos.d/,如果没有wget命令,可使用**yum install wget
**安装或者使用curl
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
或者
curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
3)运行yum makecache
生成缓存
vi /etc/sysconfig/network
修改hostname:
NETWORKING=yes
HOSTNAME=n1
vi /etc/hostname
修改hostname:
n1
直接写入/etc/host或者 vi /etc/host
cat>>/etc/hosts<
所有节点:ssh-keygen
一路回车
所有节点:ssh-copy-id n1
输入yes和密码
[root@n1 ~]# scp /root/.ssh/authorized_keys root@n2:/root/.ssh/
使用ssh d1
互相访问 消除yes确认
临时关闭:systemctl stop firewalld.service
永久关闭:systemctl mask firewalld.service
vi /etc/selinux/config
修改SELINUX=disabled
重启生效
安装
所有节点: yum install ntp
关闭 与NTP相冲突的工具chrony 开机自启
systemctl disable chronyd
设置开机启动
所有节点: chkconfig ntpd on
修改从节点ntp server为主节点
vi /etc/ntp.conf
将
server 0.centos.pool.ntp.org iburst
server 1.centos.pool.ntp.org iburst
server 2.centos.pool.ntp.org iburst
server 3.centos.pool.ntp.org iburst
改为
server n1 iburst
服务启动
所有节点: service ntpd start
可先在主节点上安装,然后scp到其他节点
[root@n1 ~]# mkdir /usr/java
[root@n1 ~]#tar -zxvf jdk-8u191-linux-x64.tar.gz -C /usr/java
[root@n1 ~]#执行
cat>>/etc/profile<
放入mysql的jdbc
[root@n1 ~]# tar -zxvf mysql-connector-java-5.1.46.tar.gz
[root@n1 ~]#cp mysql-connector-java-5.1.46-bin.jar /usr/java/
复制到其他节点
[root@n1 ~]# scp -r /usr/java root@n2:/usr/
[root@n1 ~]# scp -r /etc/profile root@n2:/etc/
所有节点:source /etc/profile
所有节点:java -version
出现jdk版本则安装成功
注意–home即cloudera-manager解压所在的目录
useradd --system --home=/opt/cm-5.15.0/run/cloudera-scm-server/ --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm
rpm -qa | grep mariadb
#查看安装信息
rpm -e mariadb-libs-5.5.56-2.el7.x86_64 --nodeps
#根据查找到的软件包信息卸载
rpm -qa | grep mariadb
再次查看确保无误
解压
tar -xf mysql-5.7.24-1.el7.x86_64.rpm-bundle.tar
安装
rpm -ivh mysql-community-common-5.7.24-1.el7.x86_64.rpm
rpm -ivh mysql-community-libs-5.7.24-1.el7.x86_64.rpm
rpm -ivh mysql-community-client-5.7.24-1.el7.x86_64.rpm
rpm -ivh mysql-community-server-5.7.24-1.el7.x86_64.rpm
在安装server时可能会有依赖包确实
使用yum install安装即可
yum -y install perl net-tools
上面的安装完毕mysql已经安装完了,最后安装
rpm -ivh mysql-community-libs-compat-5.7.24-1.el7.x86_64.rpm
解决t解决hue数据库连接不上
service mysqld start
#启动服务
cat /var/log/mysqld.log |grep password
#查看初始密码
mysql -uroot -p
#登录
set password = password('1234');
#设置密码
报错:
ERROR 1819 (HY000): Yourpassword does not satisfy the current policy requirements
这是密码安全策略的原因,使用复杂密码则不用修改
首先,修改validate_password_policy参数的值
mysql>set global validate_password_policy=0;
mysql> set global validate_password_length=1;
mysql> set password = password('1234');
#授权用户root使用密码passwd从任意主机连接到mysql服务器,新建一个scm用户管理cdh信息
GRANT ALL PRIVILEGES ON *.* TO 'scm'@'%' IDENTIFIED BY '1234' WITH GRANT OPTION;
flush privileges;
create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
主节点执行:tar -zxvf cloudera-manager-centos7-cm5.15.0_x86_64.tar.gz -C /opt
,解压出来的文件夹为cloudera与cm-5.15.0
将前边下载好的mysql-connector-java-5.1.46.tar.gz解压,找到文件夹中的
复制jdbc驱动mysql-connector-java-5.1.45-bin.jar,前面已经解压过
cp /usr/java/mysql-connector-java-5.1.46-bin.jar /opt/cm-5.15.0/share/cmf/lib/
建立数据库
/opt/cm-5.15.0/share/cmf/schema/scm_prepare_database.sh mysql -hn1 -uscm -p scm scm
密码为之前设置的 1234
vi /opt/cm-5.15.0/etc/cloudera-scm-agent/config.ini
server_host=n1
#拷贝到其他节点
[root@n1 ~]# scp -r /opt/cm-5.15.0/ root@n2:/opt/
[root@n1 ~]# scp -r /opt/cm-5.15.0/ root@d1:/opt/
[root@n1 ~]# scp -r /opt/cm-5.15.0/ root@d2:/opt/
将CDH相关文件拷贝到主节点/opt/cloudera/parcel-repo/
#相关文件
CDH-5.15.0-1.cdh5.15.0.p0.21-el7.parcel
CDH-5.15.0-1.cdh5.15.0.p0.21-el7.parcel.sha1
manifest.json
将CDH-5.15.0-1.cdh5.15.0.p0.21-el7.parcel.sha1改成CDH-5.15.0-1.cdh5.15.0.p0.21-el7.parcel.sha,否则安装时会重新下载,很重要!!
[root@n1 ~]# mv CDH-5.15.0-1.cdh5.15.0.p0.21-el7.parcel.sha1 CDH-5.15.0-1.cdh5.15.0.p0.21-el7.parcel.sha
mini版没有psmisc包,需要安装,其他是后边要用到的包
yum -y install psmisc
yum install -y python-lxml
yum -y install perl perl-devel
#主节点
/opt/cm-5.15.0/etc/init.d/cloudera-scm-server start
#主节点及其他节点
/opt/cm-5.15.0/etc/init.d/cloudera-scm-agent start
启动需要点时间,可以查看数据库scm中的HOSTS表,如果里面出现节点信息表示server启动完成
登录主节点IP的7180端口,默认登录用户名和密码都是admin
cp /opt/cm-5.15.0/share/cmf/lib/mysql-connector-java-5.1.46-bin.jar /opt/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hive/lib/
cp /opt/cm-5.15.0/share/cmf/lib/mysql-connector-java-5.1.46-bin.jar /var/lib/oozie/
#/opt/cm-5.15.0/etc/init.d/cloudera-scm-server start
/opt/cm-5.15.0/etc/init.d/cloudera-scm-server: line 109:pstree: command not found
Starting cloudera-scm-server: [FAILED]
#/opt/cm-5.15.0/etc/init.d/cloudera-scm-agent start
/opt/cm-5.15.0/etc/init.d/cloudera-scm-agent: line 108:pstree: command not found
Starting cloudera-scm-agent: [FAILED]
因为系统是最小化安装,默认没有安装
解决方法
yum-y install psmisc
第一个警告直接在所有节点上执行echo 10 > /proc/sys/vm/swappiness
第二个警告按照上面的说明,在所有节点上执行这两条命令
echo never> /sys/kernel/mm/transparent_hugepage/defrag
echo never> /sys/kernel/mm/transparent_hugepage/enabled
echo never >/sys/kernel/mm/transparent_hugepage/defrag >> /etc/rc.local
echo never > /sys/kernel/mm/transparent_hugepage/enabled>> /etc/rc.local
3.中途安装失败,重新安装
关闭所有服务
#主节点
[root@n1 ~]# /opt/cm-5.15.0/etc/init.d/cloudera-scm-server stop
#主节点及其他节点
/opt/cm-5.15.0/etc/init.d/cloudera-scm-agent stop
所有节点删除agent uuid
rm -rf /opt/cm-5.15.0/lib/cloudera-scm-agent/*
删除数据库scm
重新配置数据库
/opt/cm-5.15.0/share/cmf/schema/scm_prepare_database.sh mysql -hn1 -uscm-p scm scm
启动Cloudera server和agent,按照之前步骤安装即可