Hadoop项目之Cloudera 5.6.0.1（CDH）的安装部署

1 前言

JDK: 1.8

Cloudera Manager 5.6.0.1

HBase Version 1.0.0

Hadoop Version 2.6.0, revision=c282dc6c30e7d5d27410cabbb328d60fc24266d9

Zookeeper

Hive,

Hue,

Impala  2.1.0 

Oozie

Spark 1.6.1

Sqoop 2

Zookeeper

Scalar 2.10

RESTful API

2 理论基础

2.1 部署软件架构

Oracle JDK
Cloudera Manager Server and Agent packages
Supporting database software
CDH and managed service software

2.2 Cloudera Manager端的相关文件

rpm -ql cloudera-manager-server

显示如下：

/etc/cloudera-scm-server
/etc/cloudera-scm-server/db.properties
/etc/cloudera-scm-server/log4j.properties
/etc/default/cloudera-scm-server
/etc/rc.d/init.d/cloudera-scm-server
/opt/cloudera/csd
/opt/cloudera/parcel-repo
/usr/sbin/cmf-server
/var/log/cloudera-scm-server
/var/run/cloudera-scm-server

文件与目录功能如下：

1）其中/etc/的2-4行为Cloudera Manager服务端配置文件

2）/opt/cloudera/parcel-repo为下载安装包存放目录

2.3 部署步骤

2.4 所需的数据库

3 实践部分

3.1 环境信息

3.1.1 系统信息

OS = CentOS 6.6 x86_64

3.1.2 主机信息

Cloudera Manager:

ip address=10.168.0.120

hostname=cdm-m.cmdschool.org

Cloudera Host1:

ip address=10.168.0.121

hostname=cdm-h1.cmdschool.org

Cloudera Host2:

ip address=10.168.0.122

hostname=cdm-h2.cmdschool.org

Cloudera Host3:

ip address=10.168.0.123

hostname=cdm-h3.cmdschool.org

Cloudera Host4:

ip address=10.168.0.124

hostname=cdm-h4.cmdschool.org

3.2 运行环境配置

3.2.1 关闭selinux

getenforce

如果显示如下：

Enforcing

则执行：

setenforce 0
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config

3.2.2 配置hosts

vim编辑/etc/hosts

10.168.0.120 cdm-m.cmdschool.org cdm-m
10.168.0.121 cdm-h1.cmdschool.org cdm-h1
10.168.0.122 cdm-h2.cmdschool.org cdm-h2
10.168.0.123 cdm-h3.cmdschool.org cdm-h3
10.168.0.124 cdm-h3.cmdschool.org cdm-h4

3.2.3 配置sudo(单用户模式适用)

visudo

增加如下组

%cloudera-scm ALL=(ALL) NOPASSWD: ALL

确认包含如下行：

Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin

vim编辑/etc/pam.d/su，确保包含如下行

session         required        pam_limits.so

3.2.4 关闭防火墙并设置开机不启动

/etc/init.d/iptables stop
chkconfig iptables off

3.2.5 优化虚拟内存需求率

1)检查虚拟内存需求率

cat /proc/sys/vm/swappiness

显示如下：

60

2)临时降低虚拟内存需求率

sysctl vm.swappiness=0

3)永久降低虚拟内存需求率

vim编辑/etc/sysctl.conf

kernel.shmall = 4294967296
vm.swappiness = 0

并运行如下命令使生效

sysctl -p

3.2.6 解决透明大页面问题

1)检查透明大页面问题

cat /sys/kernel/mm/transparent_hugepage/defrag

如果显示为：

[always] madvise never

2)临时关闭透明大页面问题

echo never > /sys/kernel/mm/transparent_hugepage/defrag

确认配置生效：

cat /sys/kernel/mm/transparent_hugepage/defrag

应该显示为：

always madvise [never]

3)配置开机自动生效

vim编辑/etc/rc.local,加入如下内容

echo never > /sys/kernel/mm/transparent_hugepage/defrag

3.3 yum源的安装配置

3.3.1 公共yum源配置

1)配置yum源

下载默认yum源

wget -P /etc/yum.repos.d/ https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/cloudera-manager.repo

修改为指定版本yum源

vim编辑/etc/yum.repos.d/cloudera-manager.repo修改如下参数：

baseurl=https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.6.0/

2)安装jdk

yum install -y oracle-j2sdk1.7

3)安装python

yum install -y python

4)安装ntpd

yum install -y ntp

3.3.2 Cloudera Manager端yum源配置

1)安装Cloudera Manager包

yum install -y cloudera-manager-daemons cloudera-manager-server

2)安装mysql

yum install -y mysql-server mysql-devel mysql

3.3.3 Cloudera Manager Agent端yum源配置

安装Cloudera Manager Agent包

yum install -y cloudera-manager-agent cloudera-manager-daemons

3.4 依赖于yum源的环境配置

3.4.1 配置jdk变量环境

1)vim编辑/etc/profile，末尾加入如下内容

export JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:$PATH

2)导入java环境变量

source /etc/profile

3)测试jdk的配置

java -version

3.4.2权限检查(单用户模式适用)

检查以下目录cloudera-scm用户具有完全的权限

检查当前目录权限:

ls -ld /opt/cloudera/

显示如下

drwxr-xr-x. 4 cloudera-scm cloudera-scm 4096 May 23 13:51 /opt/cloudera/

检查子目录权限:

ls -lR /opt/cloudera/

显示如下

/opt/cloudera/:
total 8
drwxr-xr-x. 2 cloudera-scm cloudera-scm 4096 Feb 12 11:28 csd
drwxr-xr-x. 2 cloudera-scm cloudera-scm 4096 Feb 12 11:28 parcel-repo

/opt/cloudera/csd:
total 0

/opt/cloudera/parcel-repo:
total 0

同样，检查服务器或客户端目录权限

ls -ld /var/log/cloudera-scm-server/
ls -lR /var/log/cloudera-scm-server/
ls -ld /var/lib/cloudera-scm-agent/
ls -lR /var/lib/cloudera-scm-agent/

3.4.4 检查线程限制配置

cat /etc/security/limits.d/cloudera-scm.conf

显示如下：

#
# (c) Copyright 2014 Cloudera, Inc.
#
cloudera-scm    soft  nofile  32768
cloudera-scm    soft  nproc   65536
cloudera-scm    hard  nofile  1048576
cloudera-scm    hard  nproc   unlimited
cloudera-scm    hard  memlock unlimited

3.4.5 Cloudera Manager端配置

1)临时校对时间

ntpdate 0.centos.pool.ntp.org

2)启动并配置ntpd服务自动启动

/etc/init.d/ntpd start
chkconfig ntpd on

3.4.6 Cloudera Manager Agen端配置

1)临时校对时间

ntpdate 10.168.0.120

2)vim编辑/etc/ntp.conf

注释掉外网时间服务器并增加内网时间服务器地址

#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst
server 10.168.0.120 iburst

3)启动并配置ntpd服务自动启动

/etc/init.d/ntpd start
chkconfig ntpd on

3.4.7 安装MySQL JDBC Driver

wget http://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.39.tar.gz
tar zxvf mysql-connector-java-5.1.39.tar.gz
mkdir /usr/share/java/
cp mysql-connector-java-5.1.39/mysql-connector-java-5.1.39-bin.jar /usr/share/java/mysql-connector-java.jar

3.5 Cloudera Manager安装配置

3.5.1 修改mysql参数

1)关闭数据库

/etc/init.d/mysqld stop

2)备份ib_logfile文件

mkdir /var/lib/backup
cd /var/lib/mysql/
mv ib_logfile* /var/lib/backup/

3)vim编辑/etc/my.cnf

加入如下参数

[mysqld]
transaction-isolation = READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
# symbolic-links = 0

key_buffer = 16M
key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1

max_connections = 550
#expire_logs_days = 10
#max_binlog_size = 100M

#log_bin should be on a disk with enough free space. Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your system
#and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log

# For MySQL version 5.1.8 or later. Comment out binlog_format for older versions.
binlog_format = mixed

read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M

# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit  = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

sql_mode=STRICT_ALL_TABLES

3.5.2 启动并设置开机自动启动

/etc/init.d/mysqld start
chkconfig mysqld on

3.5.3 初始化数据库

mysql_secure_installation

向导如下：

[...]
Enter current password for root (enter for none):
OK, successfully used password, moving on...
[...]
Set root password? [Y/n] y
New password:
Re-enter new password:
Remove anonymous users? [Y/n] Y
[...]
Disallow root login remotely? [Y/n] N
[...]
Remove test database and access to it [Y/n] Y
[...]
Reload privilege tables now? [Y/n] Y
All done!

3.5.4 准备scm库

1)方法一

数据库配置：

mysql -uroot -p
create database scm default character set utf8;
grant all privileges on *.* to scm@'cdm-m.cmdschool.org' identified by 'scm';
flush privileges;

vim编辑/etc/cloudera-scm-server/db.properties修改如下参数：

com.cloudera.cmf.db.type=mysql
com.cloudera.cmf.db.host=cdm-m.cmdschool.org
com.cloudera.cmf.db.name=scm
com.cloudera.cmf.db.user=scm
com.cloudera.cmf.db.password=scm

2)方法二(官方建议)

授权temp权限：

mysql -uroot -p
grant all privileges on *.* to 'temp'@'%' identified by 'temp' with grant option;
flush privileges;

生成配置文件：

/usr/share/cmf/schema/scm_prepare_database.sh mysql -h cdm-m.cmdschool.org -utemp -ptemp --scm-host cdm-m.cmdschool.org scm scm scm

显示如下：

JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera
Verifying that we can write to /etc/cloudera-scm-server
Creating SCM configuration file in /etc/cloudera-scm-server
Executing: /usr/java/jdk1.7.0_67-cloudera/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
[main] DbCommandExecutor INFO Successfully connected to database.
All done, your SCM database is configured correctly!

确认生成的结果：

cat /etc/cloudera-scm-server/db.properties

显示如下：

# Auto-generated by scm_prepare_database.sh on Tue May 24 19:08:19 CST 2016
#
# For information describing how to configure the Cloudera Manager Server
# to connect to databases, see the "Cloudera Manager Installation Guide."
#
com.cloudera.cmf.db.type=mysql
com.cloudera.cmf.db.host=cdm-m.cmdschool.org
com.cloudera.cmf.db.name=scm
com.cloudera.cmf.db.user=scm
com.cloudera.cmf.db.password=scm

确认库访问权限:

mysql -uroot -p
show grants for scm@'cdm-m.cmdschool.org';

显示如下：

+----------------------------------------------------------------------------------------------------------------------+
| Grants for [email protected]                                                                                   |
+----------------------------------------------------------------------------------------------------------------------+
| GRANT USAGE ON *.* TO 'scm'@'cdm-m.cmdschool.org' IDENTIFIED BY PASSWORD '*45E6E3C68BDF1AC7EBB5C5A3BCBD5E9437B293BE' |
| GRANT ALL PRIVILEGES ON `scm`.* TO 'scm'@'cdm-m.cmdschool.org'                                                       |
+----------------------------------------------------------------------------------------------------------------------+
2 rows in set (0.00 sec)

清理数据库用户授权：

drop user 'temp'@'%';
flush privileges;

3.5.5 创建附加数据库(可选)

1)附加数据库列表

Role	Database	User	Password
Activity Monitor	amon	amon	amon_password
Reports Manager	rman	rman	rman_password
Hive Metastore Server	metastore	hive	hive_password
Sentry Server	sentry	sentry	sentry_password
Cloudera Navigator Audit Server	nav	nav	nav_password
Cloudera Navigator Metadata Server	navms	navms	navms_password

2)创建数据库并配置管理账号密码

mysql -uroot -p
create database amon default character set utf8;
grant all privileges on amon.* to 'amon'@'%' identified by 'amon_password';

create database rman default character set utf8;
grant all privileges on rman.* to 'rman'@'%' identified by 'rman_password';

create database metastore default character set utf8;
grant all privileges on metastore.* to 'hive'@'%' identified by 'hive_password';

create database sentry default character set utf8;
grant all privileges on sentry.* to 'sentry'@'%' identified by 'sentry_password';

create database nav default character set utf8;
grant all privileges on nav.* to 'nav'@'%' identified by 'nav_password';

create database navms default character set utf8;
grant all privileges on navms.* to 'navms'@'%' identified by 'navms_password';

flush privileges;

3.5.6 配置Oozie库(可选)

1)数据库权限配置

mysql -uroot -p
create database oozie default character set utf8;
grant all privileges on oozie.* to 'oozie'@'localhost' identified by 'oozie';
grant all privileges on oozie.* to 'oozie'@'%' identified by 'oozie';
flush privileges;

2)配置Oozie库所需软连接

3.5.7 启动服务并配置开机启动

/etc/init.d/cloudera-scm-server start
chkconfig cloudera-scm-server on

3.5.8 故障排除

tail -f /var/log/cloudera-scm-server/cloudera-scm-server.out

3.6 Cloudera Manager Agent安装

3.6.1 创建压缩包存放目录

mkdir -p /opt/cloudera/parcels
chown cloudera-scm:cloudera-scm /opt/cloudera/parcels

3.6.2 指定管理服务器和指定包存放目录

vim编辑/etc/cloudera-scm-agent/config.ini确保参数如下并启用：

server_host=cdm-m.cmdschool.org
server_port=7182
parcel_dir=/opt/cloudera/parcels

3.6.3 指定运行单用户模式的用户名(单用户模式适用)

vim编辑/etc/default/cloudera-scm-agent并取消以下行的注释

USER="cloudera-scm"

3.6.4 启动服务并配置服务器开机启动

/etc/init.d/cloudera-scm-agent start
chkconfig cloudera-scm-agent on

3.6.5 故障排除

如下命令监控启动服务的错误输出

tail -f /var/log/cloudera-scm-agent/cloudera-scm-agent.out

发现目录权限问题按如下方式处理

chown -R cloudera-scm:cloudera-scm /var/log/cloudera-scm-agent/
chown -R cloudera-scm:cloudera-scm /var/run/cloudera-scm-agent/events
chown -R cloudera-scm:cloudera-scm /var/run/cloudera-scm-agent/supervisor
chown -R cloudera-scm:cloudera-scm /usr/lib64/cmf/agent/src/cmf/

3.7 登陆配置

http://10.168.0.120:7180/

---------------------------------------

官方文档

http://www.cloudera.com/downloads/manager/5-6-0.html

部署方法B的目录

http://www.cloudera.com/documentation/enterprise/5-6-x/topics/cm_ig_install_path_b.html

单用户模式配置

http://www.cloudera.com/documentation/enterprise/5-6-x/topics/install_singleuser_reqts.html#xd_583c10bfdbd326ba--69adf108-1492ec0ce48--7ade__section_nhj_dlh_kq

mysql的配置概述

www.cloudera.com/documentation/enterprise/5-6-x/topics/cm_ig_installing_configuring_dbs.html

MySQL&MySQL JDBC Driver的详细安装配置

http://www.cloudera.com/documentation/enterprise/5-6-x/topics/cm_ig_mysql.html#cmig_topic_5_5

MySQL JDBC Driver的下载地址

http://dev.mysql.com/downloads/connector/j/5.1.html

JDK的安装配置

http://www.cloudera.com/documentation/enterprise/5-6-x/topics/cdh_ig_jdk_installation.html

配置SCM库

http://www.cloudera.com/documentation/enterprise/5-6-x/topics/cm_ig_installing_configuring_dbs.html#concept_i2r_m3m_hn__section_qjj_pyp_bm

配置Oozie库

http://www.cloudera.com/documentation/enterprise/5-6-x/topics/install_oozie_ext_db.html#admin_oozie_ext_db

Hue服务的数据库配置

http://www.cloudera.com/documentation/enterprise/5-6-x/topics/cm_mc_hue_service.html#xd_583c10bfdbd326ba--7f25092b-13fba2465e5--7e7c

Agent的配置

http://www.cloudera.com/documentation/enterprise/5-6-x/topics/cm_ag_agent_config.html#cmug_topic_5_16__section_kw3_5rq_wm

CDH服务的安装

http://www.cloudera.com/documentation/enterprise/5-6-x/topics/cm_ig_path_b_manual.html

命令行停止CDH服务

http://www.cloudera.com/documentation/enterprise/5-6-x/topics/cdh_ig_services_stop.html#topic_27_3

各版的yum源（下载）地址

http://www.cloudera.com/documentation/enterprise/release-notes/topics/cm_vd.html

CDH的yum源下载地址

http://archive.cloudera.com/cdh5/one-click-install/redhat/6/x86_64/

CDH的手动安装

http://www.cloudera.com/documentation/enterprise/5-6-x/topics/cm_ig_path_b_manual.html

非官方的文档

http://www.it165.net/database/html/201604/15043.html

http://wenku.baidu.com/link?url=x9yTezHZKK2a707ZLN6I8rPkdQbzBAuyy4HOzvPSJqvNJPTZmHnsvhalsEJxYPtbJD2Xybi9tnI0HhjTlrkds27c_-Se4MWfX6v45lzg9VO

http://www.tuicool.com/articles/QFZV3ij

http://www.tuicool.com/articles/ENjmeaY/

进程限制配置原理

http://www.2cto.com/os/201309/240804.html