centos&离线安装Cloudera Manager6.3.1 和CDH6.3.1
1.环境准备（所有节点，root用户环境下）
1.1.设置唯一主机名
sudo hostnamectl set-hostname foo-1.example.com
1.2.设置/etc/hosts
举例：
1.1.1.1 foo-1.example.com foo-1
2.2.2.2 foo-2.example.com foo-2
3.3.3.3 foo-3.example.com foo-3
4.4.4.4 foo-4.example.com foo-4
1.3设置ssh免密（每台都要执行）
1.执行免密ssh-keygen -t rsa
2.将公钥拷贝到包括自己的所有节点
ssh-copy-id 主机名
1.4.设置/etc/sysconfig/network
举例：
HOSTNAME=foo-1.example.com
1.5.禁用防火墙
1）.保存已有规则集：
sudo iptables-save > ~/firewall.rules
2）.禁用防火墙
sudo systemctl disable firewalld
sudo systemctl stop firewalld
1.6.设置SELinux模式
1）首先查看SELinux状态
getenforce
2）如果输出是Permissive或者Disabled则跳过下面设置，如果是enforcing,则继续设置SELinux
-
vim /etc/selinux/configorvim /etc/sysconfig/selinux少数机器是后者； -
修改
LINUX=enforcing为SELINUX=permissive； -
重启系统或者执行以下命令
setenforce 0 -
安装完之后记得还原
修改/etc/selinux/config (or /etc/sysconfig/selinux）中SELINUX=permissive为SELINUX=enforcing -
执行
setenforce 1
1.6.配置NTP服务
# 安装
yum install ntp
# 设置开机启动
chkconfig ntpd on
# 服务启动
systemctl start ntpd
# 手动同步时钟的方法
ntpdate -u 0.cn.pool.ntp.org
2.数据库安装配置和相关数据库创建
2.1.下载数据库
wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm
sudo rpm -ivh mysql-community-release-el7-5.noarch.rpm
sudo yum update
sudo yum install mysql-server
sudo systemctl start mysqld
2.2.配置数据库
1）.停止数据库服务
sudo systemctl stop mysqld
2）.备份/var/lib/mysql/目录下的ib_logfile0和ib_logfile1
3）.修改配置文件vim /etc/my.cnf
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
transaction-isolation = READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
symbolic-links = 0
key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1
max_connections = 550
#expire_logs_days = 10
#max_binlog_size = 100M
#log_bin should be on a disk with enough free space.
#Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your
#system and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log
#In later versions of MySQL, if you enable the binary log and do not set
#a server_id, MySQL will not start. The server_id must be unique within
#the replicating group.
server_id=1
binlog_format = mixed
read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M
# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
sql_mode=STRICT_ALL_TABLES
4).设置开机启动并启动mysql服务
sudo systemctl enable mysqld
sudo systemctl start mysqld
5）.初始化mysql
sudo /usr/bin/mysql_secure_installation
如下，注意Y/N
[...]
Enter current password for root (enter for none):
OK, successfully used password, moving on...
[...]
Set root password? [Y/n] Y
New password:
Re-enter new password:
Remove anonymous users? [Y/n] Y
[...]
Disallow root login remotely? [Y/n] N
[...]
Remove test database and access to it [Y/n] Y
[...]
Reload privilege tables now? [Y/n] Y
All done!
6）.安装MYSQL JDBC 驱动
点击下载
http://www.mysql.com/downloads/connector/j/5.1.html
解压tar zxvf mysql-connector-java-5.1.46.tar.gz
复制解压后驱动到/usr/share/java/,并重命名
sudo mkdir -p /usr/share/java/
cd mysql-connector-java-5.1.46
sudo cp mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar
2.3 创建数据库和账户
CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'Mysql_scm@hiteam123';
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY 'Mysql_amon@hiteam123';
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY 'Mysql_rman@hiteam123';
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY 'Mysql_hue@hiteam123';
GRANT ALL ON metastore.* TO 'hive'@'%' IDENTIFIED BY 'Mysql_hive@hiteam123';
GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY 'Mysql_sentry@hiteam123';
GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY 'Mysql_nav@hiteam123';
GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY 'Mysql_navms@hiteam123';
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'Mysql_oozie@hiteam123';
3.安装Cloudera Manager和CDH
1.下载所需资源
https://archive.cloudera.com/cm6/6.3.1/redhat7/yum/RPMS/x86_64/cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm
https://archive.cloudera.com/cm6/6.3.1/redhat7/yum/RPMS/x86_64/cloudera-manager-server-db-2-6.3.1-1466458.el7.x86_64.rpm
https://archive.cloudera.com/cm6/6.3.1/redhat7/yum/RPMS/x86_64/cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm
https://archive.cloudera.com/cm6/6.3.1/redhat7/yum/RPMS/x86_64/cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm
下载后将cloudera-manager-agent和cloudera-manager-daemons传输到其余子节点中
2.手动安装JDK
1.下载jdk8
2.解压到/usr/java*(没有这个目录就创建一个，必须是这个目录)*
tar xvfz /path/to/jdk-8u-linux-x64.tar.gz -C /usr/java/
3.配置环境变量vim /etc/profile
export JAVA_HOME=/usr/java/jdk1.8.0_241
export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$PATH:$JAVA_HOME/bin
3.安装CM
可以通过配置本地库进行安装https://docs.cloudera.com/documentation/enterprise/upgrade/topics/cm_ig_create_local_package_repo.html
1).主节点安装Server、daemons和agent
yum -y install cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm
yum -y install cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm
yum -y install cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm
2).slave节点安装安装deamons和agent
yum -y install cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm
yum -y install cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm
3).所有节点配置agent的配置vim /etc/cloudera-scm-agent/config.ini
server_host=[master_ip]
4).启动
#主节点
systemctl start cloudera-scm-server
#主节点及其他节点
systemctl start cloudera-scm-agent
5).安装CDH以及错误解决
a.CDH组件安装
进入http://serverip:7180/
如果主节点能联网直接联网操作下载cdh的parcels即可
b.如果无法联网，搭建本地软件包储存库进行进行CDH组件安装
点击进入配置本地软件包储存库
c.解决 HDFS错误“under replicated blocks”
hadoop fs -setrep -R -w 2 /