CDH 5.7离线安装指南

系统环境

硬件环境：window下VMware 11虚拟机

操作系统：红帽6.5服务器版本

Cloudera Manager：5.7

CDH：CDH-5.7.0-1.cdh5.7.0.p0.45-el6.parcel

数据库：Mysql 5.6

参考说明

http://www.cloudera.com/documentation/enterprise/latest/topics/installation_installation.html

下载

Java：http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html

MySQL：http://dev.mysql.com/downloads/mysql/

CM5：http://archive.cloudera.com/cm5/

CDH5：http://archive.cloudera.com/cdh5/

JDBC：http://dev.mysql.com/downloads/connector/j/

配置环境

1.ssh

2.java

3.mysql

4.ntp

5.network

6.防火墙及SELUNUX

以上略。。

安装

Cloudera Manager 5

1. 首先解压CM5的cm-5.7.0和cloudera到Home目录下。

$ sudo tar xzf cloudera-manager*.tar.gz -C /home/

2. 配置Server地址

$ vi /home/cm-5.7.0/etc/cloudera-scm-agent/config.ini
server_host=hadoop001 server_port=7183

3. MySql Driver

$ tar zxvf mysql-connector-java-5.1.37.tar.gz
$ sudo mkdir -p /usr/share/java/
$ sudo cp mysql-connector-java-5.1.37/mysql-connector-java-5.1.37-bin.jar /usr/share/java/mysql-connector-java.jar

4. 因为CM5里包含了SERVER和Agent所以每个节点上都需要安装它。

$ sudo scp -r /home/* agent_host:/home/

5. Cloudera Manger和服务需要Linux帐户来进行管理。所以必须创建一个用户帐户在每一个节点机器上，因为Cloudera默认使用这个帐户来管理Cloudera集群。

$ sudo useradd --system --home=/home/cm-5.7.0/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

6. 创建Cloudera Manager Server本地文件存储地址

$ sudo mkdir /var/log/cloudera-scm-server
$ sudo chown cloudera-scm:cloudera-scm /var/log/cloudera-scm-server

6-1. （可以略过）默认配置使用的路径

• /var/log/cloudera-scm-headlamp
• /var/log/cloudera-scm-firehose
• /var/log/cloudera-scm-alertpublisher
• /var/log/cloudera-scm-eventserver
• /var/lib/cloudera-scm-headlamp
• /var/lib/cloudera-scm-firehose
• /var/lib/cloudera-scm-alertpublisher
• /var/lib/cloudera-scm-eventserver
• /var/lib/cloudera-scm-server

改变目录的所有权

$ sudo chown -R cloudera-scm:cloudera-scm /var/log/cloudera-scm-headlamp

• Use alternate directories:
  1. If the directories you plan to use do not exist, create them. For example, to create /var/cm_logs/cloudera-scm-headlamp for use by the cloudera-scm user, you can use the following commands:

     mkdir /var/cm_logs/cloudera-scm-headlamp
     chown cloudera-scm /var/cm_logs/cloudera-scm-headlamp

  2. Connect to the Cloudera Manager Admin Console.
  3. Select Clusters > Cloudera Management Service
  4. Select Scope > role name.
  5. Click the Configuration tab.
  6. Enter a term in the Search field to find the settings to be changed. For example, you might enter /var or directory.
  7. Update each value with the new locations for Cloudera Manager to use.Note: The configuration property for the Cloudera Manager Server Local Data Storage Directory (default value is: /var/lib/cloudera-scm-server) is located on a different page:
     1. Select Administration > Settings.
     2. Type directory in the Search box.
     3. Enter the directory path in the Cloudera Manager Server Local Data Storage Directory property.
  8. Click Save Changes to commit the changes.

7. 初始化数据库及用户

/home/cm-5.7.0/share/cmf/schema/scm_prepare_database.sh mysql cm -uroot -p123456 --scm-host hadoop001 root 123456

7-1. （可以略过）初始化数据库设置 

http://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_installing_configuring_dbs.html#cmig_topic_5

vi /usr/my.cnf

[mysqld]
transaction-isolation = READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
# symbolic-links = 0

key_buffer = 16M
key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1

max_connections = 550
#expire_logs_days = 10
#max_binlog_size = 100M

#log_bin should be on a disk with enough free space. Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your system
#and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log

# For MySQL version 5.1.8 or later. Comment out binlog_format for older versions.
binlog_format = mixed

read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M

# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit  = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

sql_mode=STRICT_ALL_TABLES

7-2. （可以略过）创建自定义数据库及关联用户

CREATE USER 'amon'@'*' IDENTIFIED BY 'amon';
CREATE USER 'rman'@'*' IDENTIFIED BY 'rman';
CREATE USER 'hive'@'*' IDENTIFIED BY 'hive';
CREATE USER 'sentry'@'*' IDENTIFIED BY 'sentry';
CREATE USER 'nav'@'*' IDENTIFIED BY 'nav';
CREATE USER 'navms'@'*' IDENTIFIED BY 'navms';
CREATE USER 'hue'@'*' IDENTIFIED BY 'hue';

create database amon DEFAULT CHARACTER SET utf8;
grant all on amon.* TO 'amon'@'%' IDENTIFIED BY 'amon';

create database rman DEFAULT CHARACTER SET utf8;
grant all on rman.* TO 'rman'@'%' IDENTIFIED BY 'rman';

create database hive DEFAULT CHARACTER SET utf8;
grant all on hive.* TO 'hive'@'%' IDENTIFIED BY 'hive';

create database sentry DEFAULT CHARACTER SET utf8;
grant all on sentry.* TO 'sentry'@'%' IDENTIFIED BY 'sentry';

create database nav DEFAULT CHARACTER SET utf8;
grant all on nav.* TO 'nav'@'%' IDENTIFIED BY 'nav';

create database navms DEFAULT CHARACTER SET utf8;
grant all on navms.* TO 'navms'@'%' IDENTIFIED BY 'navms';

create database hue DEFAULT CHARACTER SET utf8;
grant all on hue.* to 'hue'@'%' identified by 'hue';

FLUSH PRIVILEGES;

8. 复制CDH安装包到每个节点

$ sudo chown cloudera-scm:cloudera-scm /home/cloudera/parcel-repo $ cp CDH-5.7.0-1.cdh5.7.0.p0.45-el6.parcel /home/cloudera/parcel-repo

9. 启动主节点Clouder Manager Server

$ /home/cm-5.7.0/etc/init.d/cloudera-scm-server start

10. 启动每个节点的Clouder Manager Agent

$ mkdir /home/cm-5.7.0/run/cloudera-scm-agent $ /home/cm-5.7.0/etc/init.d/cloudera-scm-agent start

设置自动启动Agent

$ cp /home/cm-5.7.0/etc/init.d/cloudera-scm-agent /etc/init.d/cloudera-scm-agent
$ chkconfig cloudera-scm-agent on

Cloudera Distribution Hadoop 5.7.0

1. 登录localhost:7180

2. 有3个版本可以使用就不解释了

3. 查看到可以用到的组件

4. 选择需要安装的节点

5. 安装CDH 5.7.0并根据需求安装相应组件

6. 分配节点任务