CDH 6.3.1 rpm+http安装部署

一、准备工作

1.离线部署主要分为三块:

• a.MySQL离线部署
• b.CM离线部署
• c.Parcel⽂件离线源部署

2.规划

节点	MySQL部署组件	Parcel文件离线源	CM进程	大数据组件
cdh001		Parcel	Activity Monitor	NN RM DN NM
cdh002	MySql		Alert Publisher Event Server	DN NM
cdh003			Host Monitor Service Monitor	DN NM

3.下载源

支持的版本：requirements_supported_versions

• CM
  cm6.3.1-redhat7.tar.gz
• Parcel
  CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel
  CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel.sha1
  manifest.json
• JDK
  https://www.oracle.com/technetwork/java/javase/downloads/java-archive-javase8-2177648.html
  下载jdk-8u181-linux-x64.tar.gz
• MySQL https://dev.mysql.com/downloads/mysql/5.7.html#downloads
  下载mysql-5.7.26-el7-x86_64.tar.gz
• MySQL jdbc jar
  下载完成后要重命名去掉版本号:
  mv mysql-connector-java-5.1.47.jar mysql-connector-java.jar

二.集群节点初始化

1.安装三台虚拟机

参考:https://www.jianshu.com/p/482cbff461bf

2.当前笔记本或台式机配置hosts文件

• MAC: /etc/hosts
• Window: C:\windows\system32\drivers\etc\hosts

地址 机器名称:
192.168.174.141 cdh001
192.168.174.142 cdh002
192.168.174.143 cdh003

3.设置所有节点的hosts文件

echo "192.168.174.141 cdh001" >> /etc/hosts
echo "192.168.174.142 cdh002" >> /etc/hosts
echo "192.168.174.143 cdh003" >> /etc/hosts

4.关闭所有节点的防火墙及清空规则

systemctl stop firewalld
systemctl disable firewalld
iptables -F

5.关闭所有节点的selinux

vi /etc/selinux/config
将SELINUX=enforcing改为SELINUX=disabled
设置后需要重启才能生效

6.设置所有节点的时区⼀致及时钟同步

• 设置时区

[root@cdh001 ~]# timedatectl
      Local time: Mon 2019-10-28 15:44:50 CST
  Universal time: Mon 2019-10-28 07:44:50 UTC
        RTC time: Mon 2019-10-28 07:44:44
       Time zone: Asia/Shanghai (CST, +0800)
     NTP enabled: n/a
NTP synchronized: no
 RTC in local TZ: no
      DST active: n/a
#查看哪些时区
[root@cdh001 ~]# timedatectl list-timezones
Africa/Abidjan
Africa/Accra
Africa/Addis_Ababa
#所有节点设置亚洲上海时区
[root@cdh001 ~]# timedatectl set-timezone Asia/Shanghai
[root@cdh002 ~]# timedatectl set-timezone Asia/Shanghai
[root@cdh003 ~]# timedatectl set-timezone Asia/Shanghai

• 设置时间

#所有节点安装ntp
[root@cdh001 ~]# yum install -y ntp
[root@cdh002 ~]# yum install -y ntp
[root@cdh003 ~]# yum install -y ntp

#选取cdh001为ntp的主节点
[root@cdh001 ~]# vi /etc/ntp.conf
#time
server 0.asia.pool.ntp.org
server 1.asia.pool.ntp.org
server 2.asia.pool.ntp.org
server 3.asia.pool.ntp.org
#当外部时间不可用时，可使用本地硬件时间
server 127.127.1.0 iburst local clock
#允许哪些网段的机器来同步时间
restrict 192.168.174.0 mask 255.255.255.0 nomodify notrap

#开启ntpd及查看状态
[root@cdh001 ~]# systemctl start ntpd
[root@cdh001 ~]# systemctl status ntpd
● ntpd.service - Network Time Service
   Loaded: loaded (/usr/lib/systemd/system/ntpd.service; disabled; vendor preset: disabled)
   Active: active (running) since Mon 2019-10-28 16:00:30 CST; 2min 47s ago
  Process: 17074 ExecStart=/usr/sbin/ntpd -u ntp:ntp $OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 17075 (ntpd)
   CGroup: /system.slice/ntpd.service
           └─17075 /usr/sbin/ntpd -u ntp:ntp -g

Oct 28 16:00:30 cdh001 ntpd[17075]: Listen and drop on 1 v6wildcard :: UDP 123
Oct 28 16:00:30 cdh001 ntpd[17075]: Listen normally on 2 lo 127.0.0.1 UDP 123
Oct 28 16:00:30 cdh001 ntpd[17075]: Listen normally on 3 ens33 192.168.174.141 UDP 123
Oct 28 16:00:30 cdh001 ntpd[17075]: Listen normally on 4 lo ::1 UDP 123
Oct 28 16:00:30 cdh001 ntpd[17075]: Listen normally on 5 ens33 fe80::53ee:12c5:2ca5:3356...123
Oct 28 16:00:30 cdh001 ntpd[17075]: Listening on routing socket on fd #22 for interface ...tes
Oct 28 16:00:52 cdh001 ntpd[17075]: 0.0.0.0 c016 06 restart
Oct 28 16:00:52 cdh001 ntpd[17075]: 0.0.0.0 c012 02 freq_set kernel 0.000 PPM
Oct 28 16:00:52 cdh001 ntpd[17075]: 0.0.0.0 c011 01 freq_not_set
Oct 28 16:00:57 cdh001 ntpd[17075]: 0.0.0.0 c514 04 freq_mode
Hint: Some lines were ellipsized, use -l to show in full.

#验证
[root@cdh001 ~]# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 LOCAL(0)        .LOCL.           5 l   94   64   16    0.000    0.000   0.000

#其他从节点停止禁用ntpd服务
[root@cdh002 ~]# systemctl stop ntpd
[root@cdh002 ~]# systemctl disable ntpd
[root@cdh003 ~]# systemctl stop ntpd
[root@cdh003 ~]# systemctl disable ntpd

#每10分钟同步cdh001节点时间
[root@cdh002 ~]# /usr/sbin/ntpdate cdh001
28 Oct 16:07:41 ntpdate[9993]: adjust time server 192.168.174.141 offset 0.147138 sec
[root@cdh003 ~]# /usr/sbin/ntpdate cdh001
28 Oct 16:08:02 ntpdate[9993]: step time server 192.168.174.141 offset 1.997036 sec
[root@cdh002 ~]# crontab -e
no crontab for root - using an empty one
*/10 * * * * /usr/sbin/ntpdate cdh001
[root@cdh003 ~]# crontab -e
no crontab for root - using an empty one
*/10 * * * * /usr/sbin/ntpdate cdh001

7.每个节点部署JDK

mkdir -p /usr/java/
tar -zxvf jre-8u181-linux-x64.tar.gz -C /usr/java/
#切记必须修正所属用户及用户组
chown -R root:root /usr/java/jre1.8.0_181

echo "export JAVA_HOME=/usr/java/jre1.8.0_181" >> /etc/profile
source /etc/profile
echo "export PATH=${JAVA_HOME}/bin:${PATH}" >> /etc/profile
source /etc/profile

which java

8.cdh002节点离线部署MySQL5.7

1.解压及创建目录
[root@cdh002 cdh6.3.1]# tar -zxvf mysql-5.7.11-linux-glibc2.5-x86_64.tar.gz -C /usr/local/
[root@cdh002 local]# mv mysql-5.7.11-linux-glibc2.5-x86_64 mysql
[root@cdh002 local]# mkdir mysql/arch mysql/data mysql/tmp

2.创建my.cnf(见文件)
[root@cdh002 local]# vi /etc/my.cnf
[client]
port            = 3306
socket          = /usr/local/mysql/data/mysql.sock
default-character-set=utf8mb4

[mysqld]
port            = 3306
socket          = /usr/local/mysql/data/mysql.sock

skip-slave-start

skip-external-locking
key_buffer_size = 256M
sort_buffer_size = 2M
read_buffer_size = 2M
read_rnd_buffer_size = 4M
query_cache_size= 32M
max_allowed_packet = 16M
myisam_sort_buffer_size=128M
tmp_table_size=32M

table_open_cache = 512
thread_cache_size = 8
wait_timeout = 86400
interactive_timeout = 86400
max_connections = 600

# Try number of CPU's*2 for thread_concurrency
#thread_concurrency = 32 

#isolation level and default engine 
default-storage-engine = INNODB
transaction-isolation = READ-COMMITTED

server-id  = 1739
basedir     = /usr/local/mysql
datadir     = /usr/local/mysql/data
pid-file     = /usr/local/mysql/data/hostname.pid

#open performance schema
log-warnings
sysdate-is-now

binlog_format = ROW
log_bin_trust_function_creators=1
log-error  = /usr/local/mysql/data/hostname.err
log-bin = /usr/local/mysql/arch/mysql-bin
expire_logs_days = 7

innodb_write_io_threads=16

relay-log  = /usr/local/mysql/relay_log/relay-log
relay-log-index = /usr/local/mysql/relay_log/relay-log.index
relay_log_info_file= /usr/local/mysql/relay_log/relay-log.info

log_slave_updates=1
gtid_mode=OFF
enforce_gtid_consistency=OFF

# slave
slave-parallel-type=LOGICAL_CLOCK
slave-parallel-workers=4
master_info_repository=TABLE
relay_log_info_repository=TABLE
relay_log_recovery=ON

#other logs
#general_log =1
#general_log_file  = /usr/local/mysql/data/general_log.err
#slow_query_log=1
#slow_query_log_file=/usr/local/mysql/data/slow_log.err

#for replication slave
sync_binlog = 500


#for innodb options 
innodb_data_home_dir = /usr/local/mysql/data/
innodb_data_file_path = ibdata1:1G;ibdata2:1G:autoextend

innodb_log_group_home_dir = /usr/local/mysql/arch
innodb_log_files_in_group = 4
innodb_log_file_size = 1G
innodb_log_buffer_size = 200M

#根据生产需要，调整pool size 
innodb_buffer_pool_size = 2G
#innodb_additional_mem_pool_size = 50M #deprecated in 5.6
tmpdir = /usr/local/mysql/tmp

innodb_lock_wait_timeout = 1000
#innodb_thread_concurrency = 0
innodb_flush_log_at_trx_commit = 2

innodb_locks_unsafe_for_binlog=1

#innodb io features: add for mysql5.5.8
performance_schema
innodb_read_io_threads=4
innodb-write-io-threads=4
innodb-io-capacity=200
#purge threads change default(0) to 1 for purge
innodb_purge_threads=1
innodb_use_native_aio=on

#case-sensitive file names and separate tablespace
innodb_file_per_table = 1
lower_case_table_names=1

[mysqldump]
quick
max_allowed_packet = 128M

[mysql]
no-auto-rehash
default-character-set=utf8mb4

[mysqlhotcopy]
interactive-timeout

[myisamchk]
key_buffer_size = 256M
sort_buffer_size = 256M
read_buffer = 2M
write_buffer = 2M

3.创建用户组及用户
[root@cdh002 local]# groupadd -g 101 dba
[root@cdh002 local]# useradd -u 514 -g dba -G root -d /usr/local/mysql mysqladmin
useradd: warning: the home directory already exists.
Not copying any file from skel directory into it.
[root@cdh002 local]# id mysqladmin
uid=514(mysqladmin) gid=101(dba) groups=101(dba),0(root)

4.copy 环境变量配置文件至mysqladmin用户的home目录中,为了以下步骤配置个人环境变量
[root@cdh002 local]# cp /etc/skel/.* /usr/local/mysql

5.配置环境变量
[root@cdh002 local]# vi mysql/.bashrc 
# .bashrc

# Source global definitions
if [ -f /etc/bashrc ]; then
        . /etc/bashrc
fi

# Uncomment the following line if you don't like systemctl's auto-paging feature:
# export SYSTEMD_PAGER=

# User specific aliases and functions

export MYSQL_BASE=/usr/local/mysql
export PATH=${MYSQL_BASE}/bin:$PATH


6.赋权限和用户组，切换用户mysqladmin，安装
[root@cdh002 local]# chown  mysqladmin:dba /etc/my.cnf
[root@cdh002 local]# chmod  640 /etc/my.cnf

[root@cdh002 local]# chown -R mysqladmin:dba /usr/local/mysql
[root@cdh002 local]# chmod -R 755 /usr/local/mysql

7.配置服务及开机自启动
[root@cdh002 local]# cd /usr/local/mysql
#将服务文件拷贝到init.d下，并重命名为mysql
[root@cdh002 mysql]# cp support-files/mysql.server /etc/rc.d/init.d/mysql
#赋予可执行权限
[root@cdh002 mysql]# chmod +x /etc/rc.d/init.d/mysql
#删除服务
[root@cdh002 mysql]# chkconfig --del mysql
#添加服务
[root@cdh002 mysql]# chkconfig --add mysql
[root@cdh002 mysql]# chkconfig --level 345 mysql on

8.安装libaio及安装mysql的初始db
[root@cdh002 mysql]# yum -y install libaio
[root@cdh002 mysql]# su - mysqladmin

[mysqladmin@cdh002 ~]$ bin/mysqld \
> --defaults-file=/etc/my.cnf \
> --user=mysqladmin \
> --basedir=/usr/local/mysql/ \
> --datadir=/usr/local/mysql/data/ \
> --initialize

9.查看临时密码
[mysqladmin@cdh002 ~]$ cat data/hostname.err |grep password
2019-10-28T08:50:20.942526Z 1 [Note] A temporary password is generated for root@localhost: /dt>edx-u0qM

10.启动
[mysqladmin@cdh002 ~]$ /usr/local/mysql/bin/mysqld_safe --defaults-file=/etc/my.cnf &

11.登录及修改用户密码
[mysqladmin@cdh002 ~]$ mysql -uroot -p'/dt>edx-u0qM'
mysql> alter user root@localhost identified by '123456';
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '123456';
Query OK, 0 rows affected, 1 warning (0.01 sec)

mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)

mysql> exit

12.重启
[mysqladmin@cdh002 ~]$ service mysql restart

[mysqladmin@cdh002 ~]$ mysql -uroot -p123456
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.11-log MySQL Community Server (GPL)

Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

9.创建CDH的元数据库和用户、 amon服务的数据库及用户

create database cmf DEFAULT CHARACTER SET utf8;
create database amon DEFAULT CHARACTER SET utf8;
grant all on cmf.* TO 'cmf'@'%' IDENTIFIED BY '123456';
grant all on amon.* TO 'amon'@'%' IDENTIFIED BY '123456';
flush privileges;

10.cdh002节点部署mysql jdbc jar

[root@cdh002 mysql]# mkdir -p /usr/share/java/
重命名不能带版本号
cp mysql-connector-java-5.1.47.jar /usr/share/java/mysql-connector-java.jar

三、CDH部署

1.离线部署cm server及agent

1.1.所有节点创建⽬录及解压
mkdir /opt/cloudera-manager
tar -xzvf cm6.3.1-redhat7.tar.gz -C /opt/cloudera-manager/
tar -zxvf cm6.3.1-redhat7.tar.gz -C /opt/cloudera-manager/

1.2.选择cdh001为cm server，不下载依赖包直接部署
cd /opt/cloudera-manager/cm6.3.1/RPMS/x86_64/
rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm --nodeps --force
rpm -ivh cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm --nodeps --force

1.3.所有节点(包含cdh001)为cm agent，不下载依赖包直接部署
cd /opt/cloudera-manager/cm6.3.1/RPMS/x86_64/
rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm --nodeps --force
rpm -ivh cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm --nodeps --force

1.4.所有节点修改agent的配置，指向server的节点cdh001
sed -i "s/server_host=localhost/server_host=cdh001/g" /etc/cloudera-scm-agent/config.ini

1.5.主节点修改server的配置:
vi /etc/cloudera-scm-server/db.properties
com.cloudera.cmf.db.type=mysql
com.cloudera.cmf.db.host=cdh002
com.cloudera.cmf.db.name=cmf
com.cloudera.cmf.db.user=cmf
com.cloudera.cmf.db.password=123456
com.cloudera.cmf.db.setupType=EXTERNAL

2.cdh001节点部署离线parcel源

2.1.安装httpd服务
yum install -y httpd

2.2.部署离线parcel源
mkdir -p /var/www/html/cdh6_parcel
mv CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel /var/www/html/cdh6_parcel/
mv CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel.sha1 /var/www/html/cdh6_parcel/CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel.sha
mv manifest.json /var/www/html/cdh6_parcel/

window/mac: http://cdh001/cdh6_parcel/ 查看是否可以访问

3.cdh001节点启动Server

3.1.启动server
systemctl start cloudera-scm-server

查看⽇志:
cd /var/log/cloudera-scm-server/
tail -F cloudera-scm-server.log
有错误就根据错误解决，没有错误，等待1min，出现7180端⼝，表明是成功的

3.2.等待1min，打开 http://cdh001:7180 账号密码:admin/admin
3.3.假如打不开，去看server的log，根据错误仔细排查错误

4.所有节点启动Agent

systemctl start cloudera-scm-agent

5.接下来，全部Web界面操作

六、常见问题

• Cloudera 建议将 /proc/sys/vm/swappiness 设置为最大值 10。当前设置为 30。使用 sysctl 命令在运行时更改该设置并编辑 /etc/sysctl.conf，以在重启后保存该设置。您可以继续进行安装，但 Cloudera Manager 可能会报告您的主机由于交换而运行状况不良

cat /proc/sys/vm/swappiness
#临时修改
sysctl -w vm.swappiness=0
永久修改
echo "vm.swappiness=0" >> /etc/sysctl.conf

• 已启用透明大页面压缩，可能会导致重大性能问题。请运行“echo never > /sys/kernel/mm/transparent_hugepage/defrag”和“echo never > /sys/kernel/mm/transparent_hugepage/enabled”以禁用此设置，然后将同一命令添加到 /etc/rc.local 等初始化脚本中，以便在系统重启时予以设置

#临时修改
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled

#永久修改
vi /etc/rc.local
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
chmod +x /etc/rc.d/rc.local

• Starting with CDH 6, PostgreSQL-backed Hue requires Psycopg2 version to be at least 2.5.4(可以忽略)

yum install python-pip

pip install --upgrade psycopg2