CDH6.1.0部署手册 亲测

系统环境 三台以上纯净CentOS 7.x

现有三台 192.168.0.187，192.168.0.139，192.168.0.147

环境配置

1.vi /etc/hosts 三台前置机全部配置hosts

192.168.0.187 node01

192.168.0.139 node02

192.168.0.147 node03

 

2.三台前置机设置时区

timedatectl set-timezone 'Asia/Shanghai'

 

 

3. 三台前置机调整linux参数

echo 0 > /proc/sys/vm/swappiness && echo "vm.swappiness=0" >> /etc/sysctl.conf && echo "echo 0 > /proc/sys/vm/swappiness" >> /etc/rc.d/rc.local && cat /proc/sys/vm/swappiness

 

 

4. 三台前置机禁用hugepage透明大页

echo "echo never > /sys/kernel/mm/transparent_hugepage/enabled" >>/etc/rc.d/rc.local && echo "echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag" >>/etc/rc.d/rc.local

 

 

5.三台前置机修改linux最大句柄数

echo "* soft nofile 128000" >> /etc/security/limits.conf && echo "* hard nofile 128000" >> /etc/security/limits.conf &&

echo "* soft nproc 128000" >> /etc/security/limits.conf &&

echo "* hard nproc 128000" >> /etc/security/limits.conf &&

sed -i 's/1024/unlimited/' /etc/security/limits.d/20-nproc.conf &&

ulimit -SHn 128000 &&

ulimit -SHu 128000

 

 

4.主节点安装http服务

yum install httpd &&

chkconfig httpd on &&

service httpd start

5.主节点关闭selinux

setenforce 0

vi /etc/selinux/config 修改SELINUX=disabled    

 

 

6.三台前置机全部关闭防火墙

systemctl stop firewalld

systemctl disable firewalld

 

 

7.ssh免密登录

参考https://blog.51cto.com/lihuansong/2172326

 

8.ntp 时间同步

参考https://blog.51cto.com/lihuansong/2172270

 

主节点安装包准备

CM6 RPM官网：https://archive.cloudera.com/cm6/6.1.0/redhat7/yum/RPMS/x86_64/需要下载该链接下的所有RPM文件然后放入/upload/cloudera-repos/文件夹下，同时还需要下载一个asc文件，同样保存到cloudera-repos目录下ASC文件：https://archive.cloudera.com/cm6/6.1.0/allkeys.asc

将jdk文件（oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm）单独拿出来随便放一个目录我们后面单独装jdk，我是放入了opt目录下要求使用5.1.26以上版本的jdbc驱动，mysql-connector-java-5.1.47.tar.gz https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.47.tar.gz需要下载CDH6 Parcels：https://archive.cloudera.com/cdh6/6.1.0/parcels CDH-6.1.0-1.cdh6.0.1.p0.590678-el7.parcel，CDH-6.1.0-1.cdh6.0.1.p0.590678-el7.parcel.sha256，manifest.json这三个文件

三个节点安装JDK

 

1.卸载原有的jdk (必须使用oracle-j2sdk1.8)

   2.rpm -ivh oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm

 

3.cd /usr/java 查看会发现有个jdk1.8.0_141-cloudera文件

4.然后配置java环境变量vim /etc/profile（新装的CentOS7系统可能没有vim命令，我习惯了vim所以提前安装了，可以用vi 或者安装vim工具）在文件末尾添加
export JAVA_HOME=/usr/java/jdk1.8.0_141-cloudera
export PATH=$PATH:$JAVA_HOME/bin

5. source /etc/profile

用java –version 查看版本

主节点配置Cloudera Manager yum库

1. 安装createrepo 

   yum –y install createrepo

2. 生成rpm 元数据

   cd /upload/cloudera-repos/ 执行createrepo .

3. 将cloudera-repos目录移动到httpd的html目录下：

   mv cloudera-repos /var/www/html/确保可以通过浏览器查看到这些RPM包：
   

创建 yum 内网库 

cd /etc/yum.repos.d 执行 vim cloudera-manager.repo

添加如下内容：

[cloudera-manager]
name=Cloudera Manager 6.0.1
baseurl=http://192.168.66.61/cloudera-repos/
gpgcheck=0
enabled=1保存，退出,然后执行yum clean all && yum makecache主节点安装Cloudera Manager Server

1.注意主节点执行yum install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server

会在opt目录下生成cloudera文件夹

2.Cloudera Manager Server安装完成后，进入到本地Parcel存储库目录：

cd /opt/cloudera/parcel-repo

将第一部分下载的CDH Parcel文件（CDH-6.1.0-1.cdh6.1.0.p0.590678-el7.parcel，CDH-6.1.0-1.cdh6.1.0.p0.590678-el7.parcel.sha256和manifest.json）上传至该目录下,并重命名sha256 文件结尾为sha

 

3.然后执行下面的命令修改文件所有者：chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo/*最终/opt/cloudera/parcel-repo目录内容如下：

 4.配置mysql jdbc驱动，此步骤所有节点都要执行
在所有节点，从前面下载好的mysql-connector-java-5.1.47.tar.gz包中解压出mysql-connector-java-5.1.47-bin.jar文件，将mysql-connector-java-5.1.47-bin.jar文件上传至CM Server节点上的/usr/share/java/目录下并重命名为mysql-connector-java.jar（如果/usr/share/java/目录不存在，需要手动创建）：
tar -zxvf mysql-connector-java-5.1.47.tar.gz
mkdir -p /usr/share/java/
cp mysql-connector-java-5.1.47-bin.jar /usr/share/java/mysql-connector-java.jar

5.Mysql安装（主节点）
安装新版mysql前，需将系统自带的mariadb-lib卸载
[root@node01 java]# rpm -qa|grep mariadb
mariadb-libs-5.5.44-2.el7.centos.x86_64
[root@node01 java]# rpm -e --nodeps mariadb-libs-5.5.44-2.el7.centos.x86_64
[root@node01 java]# rpm -qa|grep mariadb

主节点自行安装 mysql5.7以上
安装完执行以下sql 创建数据库
CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY '1qaz@WSX';
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY '1qaz@WSX';
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY '1qaz@WSX';
GRANT ALL ON metastore.* TO 'hive'@'%' IDENTIFIED BY '1qaz@WSX';
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY '1qaz@WSX';
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY '1qaz@WSX';
GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY '1qaz@WSX';
GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY '1qaz@WSX';
GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY '1qaz@WSX';
flush privileges;

6.设置Cloudera Manager 数据库
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm root

7.主节点启动服务
systemctl start cloudera-scm-server
监控日志
tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

完全启动后，访问主节点 [IP]：7180初始用户 admin/admin 按以下图片配置