Step by Step 实现基于 Cloudera 5.8.2 的企业级安全大数据平台 - 基础部署

因为Cloudera的使用场景较多是私有云,本文介绍的是如何在离线环境下进行部署的parcles方式。

硬件环境

机器类型 配置 规模
Master机型 24 cores, 192GB, SSD * 4(Non-Raid) 5 192.168.1.[2-6]
Slave机型 24 cores, 128GB, SAS * 6(Non-Raid) 11 192.168.1.[7-17]
前端机机型(VM) 6 cores, 16GB, SSD * 1(Raid 5) 1 192.168.1.1

本文不讨论Hadoop角色分布,只描述如何部署Cloudera本身。

Cloudera角色分布:

角色 部署节点 机器类型
Cloudera Server 192.168.1.1 VM
Cloudera Agent 192.168.1.1, 192.168.1.[3-17] VM、Master、Slave
Cloudera Management Services 192.168.1.3 Master
MySQL 192.168.1.2 Master

软件版本

OS: CentOS 7.2

JDK: 1.8.0_73

Cloudera: 5.8.2

依赖资源

  • CDH-5.8.2-1.cdh5.8.2.p0.3-el7.parcel
  • CDH-5.8.2-1.cdh5.8.2.p0.3-el7.parcel.sha1
  • cloudera-manager-centos7-cm5.8.2_x86_64.tar.gz
  • cm5.8.2-centos7.tar.gz
  • manifest.json
  • mysql-connector-java-5.1.34-bin.jar
  • jdk-8u73-linux-x64.rpm

假设所有依赖包都已经上传至/home/admin/soft下,记得下载完CDH-5.8.2-1.cdh5.8.2.p0.3-el7.parcel.sha1后把文件名改为CDH-5.8.2-1.cdh5.8.2.p0.3-el7.parcel.sha,因为后续进行 parcel 版本比对的时候,只会对 repo 目录下的 *.sha 文件内容进行提取,比对 parcel 的 sha 值,以确认我们的 parcel 文件是正确可用没有损毁的。

部署步骤

Step1. SSH免密打通

假设所有步骤都在admin账户下执行,所有服务器的admin账户密码统一,需要打通Cloudera Server至所有agent的SSH免密登录,假设list_agents已经包含了所有agent机器的列表:

192.168.1.1
192.168.1.3
192.168.1.4
192.168.1.5
192.168.1.6
192.168.1.7
192.168.1.8
192.168.1.9
192.168.1.10
192.168.1.11
192.168.1.12
192.168.1.13
192.168.1.14
192.168.1.15
192.168.1.16
192.168.1.17

在192.168.1.1上进行执行:

ssh 192.168.1.1
ssh-keygen
for agent in `cat list_agents`;do ssh-copy-id -i /home/admin/.ssh/id_rsa.pub admin@${agent};done;

当然,我们也可以用expect来做自动化,这里过程省略。

Step2. JDK安装

cd /home/admin/soft
pscp -h list_agents jdk-8u73-linux-x64.rpm ~/
pssh -h list_agents -P "rpm -ivh /home/admin/soft/jdk-8u73-linux-x64.rpm"

修改环境变量:

# .bash_profile 
# Get the aliases and functions 
if [ -f ~/.bashrc ]; then 
    . ~/.bashrc 
fi 
# User specific environment and startup programs 
JAVA_HOME=/usr/java/jdk1.8.0_73 
PATH=$JAVA_HOME/bin:$PATH 
export PATH JAVA_HOME

分发环境变量配置:

pscp -h list_agents ~/.bash_profile /tmp 
pssh -h list_agents "sudo cp /tmp/.bash_profile ~/"

Step3. MySQL安装并初始化数据库,默认用户root无密码

安装MySQL,我们使用 mariadb:

ssh 192.168.1.2
sudo yum install mariadb-server

修改 /etc/my.cnf 以支持utf-8:

[mysql] 
default-character-set=utf8 
[mysqld] 
character_set_server=utf8 
init_connect='SET NAMES utf8'

启动 mysql:

sudo systemctl start mariadb
sudo systemctl enable mariadb

初始化Cloudera Manager数据库:

mysql -uroot
MariaDB > grant all privileges on *.* to 'root'@'%' identified by 'root'; 
MariaDB > grant all privileges on *.* to 'root'@'localhost' identified by 'root'; 
MariaDB > use mysql; 
MariaDB > update user set password=password('root') where user='root'; 
MariaDB > create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci; 
MariaDB > create database reports DEFAULT CHARSET utf8 COLLATE utf8_general_ci; 
MariaDB > create database navigator DEFAULT CHARSET utf8 COLLATE utf8_general_ci; 
MariaDB > create database navigatormeta DEFAULT CHARSET utf8 COLLATE utf8_general_ci; 
MariaDB > create database oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci; 
MariaDB > create database hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci; 
MariaDB > create database sentry DEFAULT CHARSET utf8 COLLATE utf8_general_ci;

Step4. 离线资源准备

在192.168.1.1上对离线包进行解压,并且分发:

sudo tar zxvf /home/admin/soft/cloudera-manager-centos7-cm5.8.2_x86_64.tar.gz -C /opt/
sudo cp /home/admin/soft/mysql-connector-java-5.1.34.jar /opt/cm-5.8.2/share/cmf/lib/

创建Cloudera Manager的初始化数据,mysql密码在之前初始化了,是root:

sudo /opt/cm-5.8.2/share/cmf/schema/scm_prepare_database.sh mysql cm -h127.0.0.1 -uroot -p --port 3306 --scm-host 127.0.0.1 scm scm scm 
mysql -uroot -p
MariaDB > use cm; 
MariaDB > grant all PRIVILEGES on cm to scm;

修改 192.168.1.1 上的 /opt/cm-5.8.2/etc/cloudera-scm-agent/config.iniserver_host修改为主节点hostname(v001001.idc.domain.com),将离线包中的CDH5相关的parcel包放到主节点192.168.1.1/opt/cloudera/parcel-repo/目录中:

sudo mkdir -p /opt/cloudera/parcel-repo/ 
sudo cp /home/admin/soft/cdh5.8.2/CDH-5.8.2-1.cdh5.8.2.p0.3-el7.parcel /opt/cloudera/parcel-repo/ 
sudo cp /home/admin/soft/cdh5.8.2/CDH-5.8.2-1.cdh5.8.2.p0.3-el7.parcel.sha /opt/cloudera/parcel-repo/ 
sudo cp /home/admin/soft/cdh5.8.2/manifest.json /opt/cloudera/parcel-repo/

在主节点192.168.1.1上创建storage目录:

sudo mkdir -p /var/lib/cloudera-scm-server

Step5. 启动Cloudera Server服务

在192.168.1.1上启动Cloudera Server:

sudo /opt/cm-5.8.2/etc/init.d/cloudera-scm-server start

在agent上把mysql-connector-java-5.1.34.jar拷贝至Cloudera Manager的lib目录下:

pscp -h list_agents /home/admin/soft/mysql-connector-java-5.1.34.jar /tmp 
pssh -h list_agents -P "sudo mkdir -p /usr/share/cmf/lib"
pssh -h list_agents -P "sudo mkdir -p /usr/share/java" 
pssh -h list_agents -P "sudo cp /tmp/mysql-connector-java-5.1.34.jar /usr/share/cmf/lib"
pssh -h list_agents -P "sudo cp /tmp/mysql-connector-java-5.1.34.jar /usr/share/java/mysql-connector-java.jar"

Step6. 搭建临时httpd服务器

搭建httpd服务器的目的是为了实现离线本地Repo安装,需要把之前从cm5.8.2-centos7.tar.gz解压缩至Root Dir下,假设Repo地址为http://192.168.1.1/cm/5.8.2/。

sudo yum install -y httpd
sudo tar zxvf cm5.8.2-centos7.tar.gz -C /var/www/html/
sudo rm -rf /var/run/yum.pid

Step7. 进入图形化界面进行部署

默认的Cloudera控制台界面为 http://192.168.1.1:7180
初次登录会要求输入默认的管理员账户名密码,请切记。进入图形化界面后按照以下步骤进行agent发现和集群初始化:

  • 选择 Yes, I accept the End User License Terms and Conditions.,然后一路 Continue
  • Specify hosts for your CDH cluster installation. 界面输入以下机器,并点击 Search
192.168.1.1
192.168.1.3
192.168.1.4
192.168.1.5
192.168.1.6
192.168.1.7
192.168.1.8
192.168.1.9
192.168.1.10
192.168.1.11
192.168.1.12
192.168.1.13
192.168.1.14
192.168.1.15
192.168.1.16
192.168.1.17
  • 全选机器后,点击 Continue
  • 选择Use Parcels (Recommended)
  • More Options中去掉所有远程Repo配置;
  • Select the version of CDH选择CDH-5.8.2-1.cdh5.8.2.p0.3
  • Select the specific release of the Cloudera Manager Agent you want to install on your hosts.选择Custom Repository,并且使用局域网httpd,地址为:http://192.168.1.1/cm/5.8.2/
  • Install Oracle Java SE Development Kit (JDK)不勾选;
  • Single User Mode不勾选;
  • Login To All Hosts As:选择admin,并录入CentOS的admin密码;

正常情况下会自动进行Cloudera安装包分发,以及agent进程启动。

Q: 安装过程中如果遇到:/opt/cloudera/parcels/CDH-5.8.2-1.cdh5.8.2.p0.3/meta/parcel.json 文件无法找到
A: 请把进程停止后清除 uuid 之后再走一遍流程:

sudo /bin/systemctl stop cloudera-scm-agent
sudo rm /var/lib/cloudera-scm-agent/*
sudo /bin/systemctl start cloudera-scm-agent

如果无效,请在安装向导的 More Options 中重新指定 parcel repo 目录,如果 repo 目录空间不足也会导致 parcel.json 文件安装失败。重新指定的方法 Hosts -> Parcels -> Configuration -> Local Parcel Repository Path 修改为其他地址。并且查看 Remote Parcel Repository URLs 配置是否指向本地 httpd 服务器对应目录。
Q: 遇到 ProtocolError:
A: 请把 supervisor 杀掉后重试:

pid=`ps aux | grep "/usr/lib64/cmf/agent/build/env/bin/supervisord" | grep -v grep | awk '{print$2}'`
sudo kill -9 ${pid}

Step8.进行Hadoop服务部署

这块不在这里进行描述,按照您的需要进行部署。但是有一块要注意,记得在部署服务前创建相应 lib 目录并且把 mysql-connector 拷贝至对应服务的 lib 目录:

pscp -h list_agents /home/admin/soft/mysql-connector-java-5.1.34.jar /tmp
pssh -h list_agents "sudo mkdir -p /opt/cloudera/parcels/CDH-5.8.2-1.cdh5.8.2.p0.3/lib/hive/lib/ /opt/cloudera/parcels/CDH-5.8.2-1.cdh5.8.2.p0.3/lib/oozie/lib/ /var/lib/oozie" 
pssh -h list_agents "sudo cp /tmp/mysql-connector-java-5.1.34.jar /opt/cloudera/parcels/CDH-5.8.2-1.cdh5.8.2.p0.3/lib/hive/lib/" 
pssh -h list_agents "sudo cp /tmp/mysql-connector-java-5.1.34.jar /opt/cloudera/parcels/CDH-5.8.2-1.cdh5.8.2.p0.3/lib/oozie/lib/" 
pssh -h list_agents "sudo mkdir -p  /opt/cloudera/parcels/CDH-5.8.2-1.cdh5.8.2.p0.3/lib/sentry/lib"
pssh -h list_agents "sudo cp /tmp/mysql-connector-java-5.1.34.jar /opt/cloudera/parcels/CDH-5.8.2-1.cdh5.8.2.p0.3/lib/sentry/lib/" 
pssh -h list_agents "sudo cp /tmp/mysql-connector-java-5.1.34.jar /var/lib/oozie"
pssh -h list_agents "sudo rm -f /tmp/mysql-connector-java-5.1.34.jar"

Q: 部署 Oozie 的时候报错:mkdir: cannot create directory '/var/lib/oozie/tomcat-deployment': No such file or directory
A: 在 MySQL 中重建 oozie 这个 database,然后 执行:

sudo chown oozie:oozie /var/lib/oozie

Step9. 修改HUE和Oozie的时区

修改时区是针对国内用户的,如果不进行HUE时区修改,会造成HUE中无法使用sqoop,报Sqoop error: Could not get connectors.;如果不进行Oozie时区修改,那么所看到的Oozie日志时间戳会错误。

HUE修改很简单,只需要在Cloudera控制台HUE Service的configuration中对如下属性进行修改,并重启服务即可:

time_zone = Asia/Shanghai

Oozie修改的话,也是在Cloudera控制台HUE Service的configuration找到oozie-env.sh进行修改,添加如下属性:

 
    oozie.processing.timezone 
    GMT+0800 

然后在每台运行Oozie的服务器上执行,完成后重启Oozie服务:

cd /opt/cloudera/parcels/CDH/lib/oozie/libext 
unzip ext-2.2.zip 
chown oozie:oozie -R ext-2.2

最后在Oozie Web Console中的Settings下修改TimezoneCST(Asia/Shanghai)

附 完全卸载脚本

#!/bin/bash 
RELEASE_VERSION=5.8.2
RPM_CMA_VERSION=5.8.2-1.cm582.p0.17.el7.x86_64
RPM_CMD_VERSION=5.8.2-1.cm582.p0.17.el7.x86_64   
pssh -h list -P "sudo /bin/systemctl stop cloudera-scm-agent"

sudo /opt/cm-${RELEASE_VERSION}/etc/init.d/cloudera-scm-server stop 
pssh -h list -P "sudo umount /run/cloudera-scm-agent/process" 
pssh -h list -P "sudo rm -rf /usr/share/cmf /var/lib/cloudera* /var/cache/yum/x86_64/6/cloudera* /var/log/cloudera* /var/run/cloudera* /etc/cloudera* /usr/lib64/cmf /etc/init.d/cloudera-scm-agent /etc/rc3.d/S90cloudera-scm-agent /etc/cloudera-scm-agent" 
pssh -h list -P "sudo rpm -e --noscripts --nodeps cloudera-manager-agent-${RPM_CMA_VERSION}" 
pssh -h list -P "sudo rpm -e --noscripts --nodeps cloudera-manager-daemons-${RPM_CMD_VERSION}" 
pssh -h list -P "sudo rm -rf /var/lib/hadoop-* /var/lib/impala /var/lib/solr /var/lib/zookeeper /var/lib/hue /var/lib/oozie /var/lib/pgsql /var/lib/sqoop2 /data/dfs/ /data/impala/ /data/yarn/ /dfs/ /impala/ /yarn/ /var/run/hadoop-*/ /var/run/hdfs-*/ /usr/bin/hadoop* /usr/bin/zookeeper* /usr/bin/hbase* /usr/bin/hive* /usr/bin/hdfs /usr/bin/mapred /usr/bin/yarn /usr/bin/sqoop* /usr/bin/oozie /etc/hadoop* /etc/zookeeper* /etc/hive* /etc/hue /etc/impala /etc/sqoop* /etc/oozie /etc/hbase* /etc/hcatalog /var/lib/flume-ng /var/lib/hadoop* /var/lib/hue /var/lib/navigator /var/lib/oozie /var/lib/solr /var/lib/sqoop* /var/lib/zookeeper /var/lib/hbase /var/lib/hive /var/lib/impala /var/lib/spark" 
pssh -h list -P "sudo rm -rf /opt/cloudera" 
pssh -h list -P "sudo userdel -r cloudera-scm" 
pssh -h list -P "sudo rm -rf /usr/share/cmf /var/lib/cloudera* /var/cache/yum/x86_64/6/cloudera* /var/log/cloudera* /var/run/cloudera* /etc/cloudera* /usr/lib64/cmf" 
pssh -h list -P "sudo rm -rf /opt/cloudera /opt/cm-${RELEASE_VERSION}/" 

附 Agent 卸载脚本

/sbin/service cloudera-scm-agent stop
/bin/sed -e s/\(server_host=\).*/\1localhost/ -i /etc/cloudera-scm-agent/config.ini
/bin/yum -y erase cloudera-manager-agent
/bin/rm -rf /var/log/cloudera-scm-agent/
/bin/rm -rf /etc/cloudera-scm-agent/

你可能感兴趣的:(Step by Step 实现基于 Cloudera 5.8.2 的企业级安全大数据平台 - 基础部署)