CentOS 7 安装 CDH 5.16.2 离线安装

一.前期环境准备

所有机器都要操作的，都要进行的环境准备。

安装sshpass  

主要的节点，或者是主节点，或者是主要的跳转到其他节点的服务器，需要安装
yum install -y sshpass
可以自动带密码登录，其他服务器。

 yum list | grep sshpass

sshpass -p '密码' ssh  -o stricthostkeychecking=no slave-4
sshpass -p '密码' scp -rp software/ root@slave-4:/root/software/

所有节点：

更新系统:
yum update -y;

安装vim：

yum install -y vim;

安装开发者工具组：

yum groupinstall "Development Tools" -y;

安装htop,lsof,netstat

yum install -y htop,lsof,netstat;

1.增加SSH外网端口，默认22端口不能暴露给外网。并且只针对公司的外网IP开通。

vim /etc/ssh/sshd_config

systemctl restart sshd;

2.直接关闭防火墙和selinux
增加端口，需要添加selinux规则和防火墙规则，或者关闭selinux和防火墙。

2.1关闭防火墙

systemctl stop firewalld;

取消开机启动
systemctl disable firewalld;

2.2关闭selinux

查看selinux状态

sestatus
getenforce

临时关闭
setenforce 0

永久关闭

vim /etc/selinux/config

   SELINUX=disabled

查看是否成功:
getenforce

3.修改IP：

虚拟机拷贝的主机需要修改网卡uuid
uuidgen eth0

nmcli connection show

vim /etc/sysconfig/network-scripts/ifcfg-eth0

systemctl restart network

4.修改主机名,编辑hosts文件:
方式一:
vim /etc/hostname
master

方式二:
hostnamectl set-hostname master
hostnamectl set-hostname slave-1
hostnamectl set-hostname slave-2
hostnamectl set-hostname slave-3
hostnamectl set-hostname slave-4

主机名生效:
hostname -F /etc/hostname

主机名查看：
hostname

FQDN查看：
hostname -f

生效：
cat /etc/sysconfig/network

cat /etc/hostname

编辑hosts文件:
vim /etc/hosts

192.168.1.16 master
192.168.1.17 slave-1
192.168.1.18 slave-2
192.168.1.19 slave-3
192.168.1.20 slave-4

scp -o stricthostkeychecking=no -P 22 -p /etc/hosts root@slave-1:/etc/

 
5.配置免密码登录(全部节点):
cd /root/.ssh/;ll
rm -rf /root/.ssh/*;ll

ssh-keygen -t rsa

cd /root/.ssh/;ll

在每个节点(包括主节点)执行远程拷贝公钥:
ssh-copy-id -p 22 -o stricthostkeychecking=no -i /root/.ssh/id_rsa.pub root@master

从主节点上拷贝到其他节点：
scp -P 22 -o stricthostkeychecking=no -p /root/.ssh/authorized_keys /root/.ssh/known_hosts root@slave-1:/root/.ssh/
scp -P 22 -o stricthostkeychecking=no -p /root/.ssh/authorized_keys /root/.ssh/known_hosts root@slave-2:/root/.ssh/
scp -P 22 -o stricthostkeychecking=no -p /root/.ssh/authorized_keys /root/.ssh/known_hosts root@slave-3:/root/.ssh/
scp -P 22 -o stricthostkeychecking=no -p /root/.ssh/authorized_keys /root/.ssh/known_hosts root@slave-4:/root/.ssh/

在主节点上执行
ssh -o stricthostkeychecking=no -p 22 master;
exit;
ssh -o stricthostkeychecking=no -p 22 slave-1;
exit;
ssh -o stricthostkeychecking=no -p 22 slave-2;
exit;
ssh -o stricthostkeychecking=no -p 22 slave-3;
exit;
ssh -o stricthostkeychecking=no -p 22 slave-4;
exit;

集群内，互相之间免密码登录配置完毕。

6.时间同步：

全部节点:
yum install -y ntp

6.1：主节点
配置NTP服务器：
如果内网有NTP服务器，或者机器可以访问外网，则可以不配置。

vim /etc/ntp.conf

#这一行的含义是授权该网段上的所有机器可以从这台机器上查询和同步时间.
restrict 192.168.2.0 mask 255.255.255.0 nomodify notrap

#新增:时间服务器列表.可以不增加（如果可以访问外网）
server 0.cn.pool.ntp.org iburst
server 1.cn.pool.ntp.org iburst
server 2.cn.pool.ntp.org iburst
server 3.cn.pool.ntp.org iburst
#新增:允许上层时间服务器主动修改本机时间.（如果可以访问外网）
restrict 0.cn.pool.ntp.org nomodify notrap noquery
restrict 1.cn.pool.ntp.org nomodify notrap noquery
restrict 2.cn.pool.ntp.org nomodify notrap noquery

#新增:当外部时间不可用时，使用本地时间.
server  127.127.1.0     #local clock
fudge   127.127.1.0 stratum 10

6.2：其他节点:客户端 
vim /etc/ntp.conf

server 192.168.1.16

6.3：配置硬件时间同步, 全部节点

vim /etc/sysconfig/ntpd

SYNC_HWCLOCK=yes
OPTIONS="-g -x"

配置开机启动，生效：

systemctl enable ntpd;
systemctl is-enabled ntpd;
systemctl restart ntpd;
systemctl status ntpd;

验证:

ntpdc -np

ntpq -pn

第二种时间同步的简单方式
已经有可以访问的时间服务器
所有节点和时间服务器同步时间

*/2 * * * * /usr/sbin/ntpdate 192.168.1.16;/sbin/hwclock -w > /dev/null 2>&1

hadoop dfs -cat /app/ecom/fcr/ztc-datas/tmp/liuxiao/test/*A | hadoop dfs -put - /app/ecom/fcr/ztc-datas/tmp/liuxiao/test/merge.txt

上网代理：
export http_proxy=http://192.168.1.18:8080
source /etc/profile

安装jdk

添加环境变量：
vim /etc/profile

echo "" >> /etc/profile;
echo "export JAVA_HOME=/usr/java/default" >> /etc/profile;
echo "export JRE_HOME=\$JAVA_HOME/jre" >> /etc/profile;
echo "CLASS_PATH=.:\$JAVA_HOME/lib/dt.jar:\$JAVA_HOME/lib/tools.jar:\$JRE_HOME/lib" >> /etc/profile;
echo "export PATH=\$PATH:\$JAVA_HOME/bin:\$JRE_HOME/bin" >> /etc/profile;
source /etc/profile;
cat /etc/profile;
生效：
source /etc/profile;
验证：
echo  $JAVA_HOME

yum install chrony -y
systemctl enable chronyd.service; 
systemctl start chronyd.service;
chronyc sources;

mysql安装  
主节点

rpm -qa | grep mariadb-libs
rpm -e --nodeps mariadb-libs-5.5.60-1.el7_5.x86_64

 
rpm -e --nodeps mysql-community-common-5.7.27-1.el7.x86_64;
rpm -e --nodeps mysql-community-libs-compat-5.7.27-1.el7.x86_64;
rpm -e --nodeps mysql-community-libs-5.7.27-1.el7.x86_64;
rpm -e --nodeps mysql-community-client-5.7.27-1.el7.x86_64;
rpm -e --nodeps mysql-community-server-5.7.27-1.el7.x86_64;

centos7使用rpm安装mysql遇到的error: Failed dependencies: mysql-community-common(x86-64) >= 5.7.9 is needed
在指令后面加上:  --nodeps --force
rpm -ivh mysql-community-server-5.7.27-1.el7.x86_64.rpm --nodeps --force

rpm -ivh mysql-community-common-5.7.24-1.el7.x86_64.rpm 
rpm -ivh mysql-community-libs-compat-5.7.24-1.el7.x86_64.rpm 
rpm -ivh mysql-community-libs-5.7.24-1.el7.x86_64.rpm
rpm -ivh mysql-community-client-5.7.25-1.el7.x86_64.rpm
rpm -ivh mysql-community-server-5.7.25-1.el7.x86_64.rpm

如果报错误:
error: Failed dependencies:
    libaio.so.1()(64bit) is needed by mysql-community-server-5.7.26-1.el7.x86_64
    libaio.so.1(LIBAIO_0.1)(64bit) is needed by mysql-community-server-5.7.26-1.el7.x86_64
    libaio.so.1(LIBAIO_0.4)(64bit) is needed by mysql-community-server-5.7.26-1.el7.x86_64
yum install libaio -y

yum localinstall mysql-community-server-5.7.24-1.el7.x86_64

开机启动

systemctl enable mysqld

# 重启mysql
systemctl restart mysqld 

修改配置文件:

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
transaction-isolation = READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
symbolic-links = 0

key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1

max_connections = 550
#expire_logs_days = 10
#max_binlog_size = 100M

#log_bin should be on a disk with enough free space.
#Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your
#system and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log

#In later versions of MySQL, if you enable the binary log and do not set
#a server_id, MySQL will not start. The server_id must be unique within
#the replicating group.
server_id=1

binlog_format = mixed

read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M

# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit  = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

sql_mode=STRICT_ALL_TABLES

[mysqld]
#最大连接数
max_connections=550
#设置utf-8格式
character-set-server=utf8
#设置表命不区分大小写
lower_case_table_names=1

#可以不加，
explicit_defaults_for_timestamp=1

获取第一次登陆密码:
grep "password" /var/log/mysqld.log

2020-05-23T11:19:25.326140Z 1 [Note] A temporary password is generated for root@localhost: oHP=>zqUw2?n

[email protected]

执行mysql初始安装设置:

/usr/bin/mysql_secure_installation

[root@hadoop101 lib]# /usr/bin/mysql_secure_installation

Securing the MySQL server deployment.

Enter password for user root: 

The existing password for the user account root has expired. Please set a new password.

New password: 

Re-enter new password: 

VALIDATE PASSWORD PLUGIN can be used to test passwords
and improve security. It checks the strength of password
and allows the users to set only those passwords which are
secure enough. Would you like to setup VALIDATE PASSWORD plugin?

Press y|Y for Yes, any other key for No: y

There are three levels of password validation policy:

LOW    Length >= 8
MEDIUM Length >= 8, numeric, mixed case, and special characters
STRONG Length >= 8, numeric, mixed case, special characters and dictionary                  file

Please enter 0 = LOW, 1 = MEDIUM and 2 = STRONG: 2
Using existing password for root.

Estimated strength of the password: 100 
Change the password for root ? ((Press y|Y for Yes, any other key for No) : 

 ... skipping.
By default, a MySQL installation has an anonymous user,
allowing anyone to log into MySQL without having to have
a user account created for them. This is intended only for
testing, and to make the installation go a bit smoother.
You should remove them before moving into a production
environment.

Remove anonymous users? (Press y|Y for Yes, any other key for No) : y
Success.

Normally, root should only be allowed to connect from
'localhost'. This ensures that someone cannot guess at
the root password from the network.

Disallow root login remotely? (Press y|Y for Yes, any other key for No) : n

 ... skipping.
By default, MySQL comes with a database named 'test' that
anyone can access. This is also intended only for testing,
and should be removed before moving into a production
environment.

Remove test database and access to it? (Press y|Y for Yes, any other key for No) : y
 - Dropping test database...
Success.

 - Removing privileges on test database...
Success.

Reloading the privilege tables will ensure that all changes
made so far will take effect immediately.

Reload privilege tables now? (Press y|Y for Yes, any other key for No) : y
Success.

All done! 

简化版：

Remove anonymous users? [Y/n] Y
[...]
Disallow root login remotely? [Y/n] N
[...]
Remove test database and access to it [Y/n] Y
[...]
Reload privilege tables now? [Y/n] Y
All done!

登录mysql，创建CDH相关数据库:
mysql -u root -p 

create database scm default character set utf8 default collate utf8_general_ci;
create database amon default character set utf8 default collate utf8_general_ci;
create database rman default character set utf8 default collate utf8_general_ci;
create database hue default character set utf8 default collate utf8_general_ci;
create database metastore default character set utf8 default collate utf8_general_ci;
create database sentry default character set utf8 default collate utf8_general_ci;
create database nav default character set utf8 default collate utf8_general_ci;
create database navms default character set utf8 default collate utf8_general_ci;
create database oozie default character set utf8 default collate utf8_general_ci;

grant all privileges on *.* to 'root'@'%' identified by '[email protected]' with grant option;
grant all privileges on *.* to 'root'@'localhost' identified by '[email protected]' with grant option;

grant all privileges on *.* to 'root'@'%' identified by '1qaz@WSX!' with grant option;
grant all privileges on *.* to 'root'@'localhost' identified by '1qaz@WSX!' with grant option;

flush privileges;
exit;

CDH需要的系统设置
cat /etc/sysctl.conf

vim /etc/sysctl.conf
 vm.swappiness = 0

echo "vm.swappiness=0" >> /etc/sysctl.conf
sysctl -p;

cat /etc/rc.local

vim /etc/rc.local

echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag
chmod -R 777 /dev/null

chmod +x /etc/rc.d/rc.local

第二种方式

echo "echo never > /sys/kernel/mm/transparent_hugepage/enabled" >> /etc/rc.local;
echo "echo never > /sys/kernel/mm/transparent_hugepage/defrag" >> /etc/rc.local;
echo "chmod -R 777 /dev/null" >> /etc/rc.local;
chmod +x /etc/rc.d/rc.local;
chmod +x /etc/rc.local;

最大打开数  可以不加 
cat /etc/security/limits.conf

vim /etc/security/limits.conf
* soft nofile 32728
* hard nofile 1029345
* soft nproc 65536
* hard nproc unlimited
* soft memlock unlimited
* hard memlock unlimited

改为:

* - nofile 65535
* - nproc  65000

echo "* - nofile 65535" >> /etc/security/limits.conf;
echo "* - nproc  65000" >> /etc/security/limits.conf;

ulimit -u

scp -rp /etc/sysctl.conf root@slave-1:/etc/
scp -rp /etc/rc.d/rc.local root@slave-1:/etc/rc.d/
scp -rp /etc/security/limits.conf root@slave-1:/etc/security/

wget https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/cloudera-manager.repo -P /etc/yum.repos.d/

sudo rpm --import https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/RPM-GPG-KEY-cloudera

拷贝 parcels

CM下载地址：http://archive.cloudera.com/cm5/cm/5/
离线库下载地址：http://archive.cloudera.com/cdh5/parcels

我这里直接下载的rpm包，配置没那么麻烦

该所有节点执行
安装CM
rpm -ivh cloudera-manager-daemons-5.16.2-1.cm5162.p0.6.el7.x86_64.rpm
rpm -ivh  cloudera-manager-agent-5.16.2-1.cm5162.p0.6.el7.x86_64.rpm

rpm -ivh cloudera-manager-daemons-5.16.2-1.cm5162.p0.7.el7.x86_64.rpm
yum localinstall cloudera-manager-agent-5.16.2-1.cm5162.p0.7.el7.x86_64.rpm 

如果缺少依赖，使用yum安装
yum localinstall cloudera-manager-daemons-5.16.2-1.cm5162.p0.6.el7.x86_64.rpm
yum localinstall cloudera-manager-agent-5.16.2-1.cm5162.p0.6.el7.x86_64.rpm

主节点多一个

rpm -ivh cloudera-manager-server-5.16.2-1.cm5162.p0.7.el7.x86_64.rpm 

mkdir /usr/share/java/
cp mysql-connector-java.jar /usr/share/java/

scp -P 22 /usr/share/java/mysql-connector-java.jar root@slave-1:/usr/share/java/

在主节点修改配置文件

vim /etc/cloudera-scm-agent/config.ini

拷贝到其他节点:
scp  -P 22 /etc/cloudera-scm-agent/config.ini root@slave-1:/etc/cloudera-scm-agent/
scp  -P 22 /etc/cloudera-scm-agent/config.ini root@slave-2:/etc/cloudera-scm-agent/

主节点上，初始化数据库

/usr/share/cmf/schema/scm_prepare_database.sh mysql scm root

如果集群cloudera-scm-server，各个服务启动不起来。 一般是权限问题：

service cloudera-scm-server status

service cloudera-scm-server start

启动前报错:，

/usr/share/cmf/bin/cmf-server: line 92: /dev/null: Permission denied
+======================================================================+
|      Error: JAVA_HOME is not set and Java could not be found         |
+----------------------------------------------------------------------+
| Please download the latest Oracle JDK from the Oracle Java web site  |
|  > http://www.oracle.com/technetwork/java/javase/index.html <        |
|                                                                      |
| Cloudera Manager requires Java 1.6 or later.                         |
| NOTE: This script will find Oracle Java whether you install using    |
|       the binary or the RPM based installer.                         |
+======================================================================+

每个节点执行下条命令:
chmod -R 777 /dev/null;

wget -P /etc/yum.repos.d/

/sbin/chkconfig cloudera-scm-server on
/sbin/chkconfig cloudera-scm-agent on

systemctl enable cloudera-scm-server
systemctl enable cloudera-scm-agent

systemctl start cloudera-scm-server
systemctl start cloudera-scm-agent

systemctl restart cloudera-scm-agent
systemctl restart cloudera-scm-server

systemctl status cloudera-scm-server;
systemctl status cloudera-scm-agent;

systemctl stop cloudera-scm-agent
systemctl stop cloudera-scm-server

yum下载不安装

yumdownloader --resolve --destdir=/root/rpm --releasever=7 cloudera-manager-agent-5.16.2-1.cm5162.p0.6.el7.x86_64

kafka-preferred-replica-election --zookeeper master:2181,slave-1:2181,slave-2:2181,slave-3:2181,slave-4:2181/kafka

集群升级
先更新系统

yum update -y 

如果内网机器不能访问外网，开启nginx正向代理。

创建

ln -s /data/opt/cloudera /opt/cloudera

覆盖/删除
ln -b /data/opt/cloudera /opt/cloudera

kafka-topics --list --zookeeper master:2181,slave-1:2181,slave-2:2181,slave-3:2181,slave-4:2181

zookeeper-client -timeout 5000 -server master:2181,slave-1:2181,slave-2:2181,slave-3:2181,slave-4:2181

kafka-topics --create --zookeeper master:2181,slave-1:2181,slave-2:2181,slave-3:2181,slave-4:2181 --replication-factor 2 --partitions 4 --topic MSG_PG

rm -rf /data/var/local/kafka/*

rpm -e --nodeps  cloudera-manager-server-5.16.2-1.cm5162.p0.1.el7.x86_64

zookeeper-client -timeout 5000 -server  master:2181,slave-1:2181,slave-2:2181,slave-3:2181,slave-4:2181

zookeeper-client -timeout 5000 -server hadoop100:2181,hadoop101:2181,hadoop102:2181,hadoop103:2181,hadoop104:2181

kafka-topics --list  --zookeeper  hadoop100:2181,hadoop101:2181,hadoop102:2181,hadoop103:2181,hadoop104:2181
ls /brokers/topics

问题

Datanode Volume Failures

mv /opt/dfs/dn /opt/dfs/dn2;
mv /tmcdata/dfs/dn /tmcdata/dfs/dn2;