CDH6.3.2安装(rhel7.6)

CDH6.3.2

一、基本介绍

​ 基于最新CDH发布版本编写安装文档,CDH6安装过程基本与CDH5一致,主要包括以下几步:

​ 1、安装前置准备:包括安装操作系统、关闭防火墙、同步服务器时钟等;

​ 2、mysql安装部署;

​ 3、安装Cloudera Manager;

​ 4、安装CDH集群;

​ 软件要求:

​ CDH 集群搭建需要的安装包包括 JDK、Cloudera-Manager RPM 包(3 个),CDH parcel⽂件(3 个)、MySQL、MySQL 驱动 jar 包。

组件 版本 类型 备注
JDK 8u202 tar jdk-8u202-linux-x64.tar.gz
CM daemons 6.3.1 rpm cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm
CM agent 6.3.1 rpm cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm
CM server 6.3.1 rpm cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm
CDH 各组件⼆ 进制服务包 6.3.2 parcel CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel、CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1、manifest.json
MySQL 8.0.18 tar mysql-8.0.18-el7-x86_64.tar.gz
MySQL驱动包 8.0.18 jar mysql-connector-java-8.0.18.jar

​ 节点分布:

​ CM Server 和 Agent 分配 Cloudera Manager 由 Server 和 Agent 组成,安装 Cloudera Manager Server 的节点称为 Server 节点,安装 Cloudera Manager Agent 的节点称为 Agent 节点。

​ 3个节点的情形

节点 安装的服务
node1 CM daemon,CM server,CM agent
node2 CM daemon,CM agent
node3 CM daemon,CM agent

二、安装前置准备

​ 没有出现其他节点主机名的话,说明每个节点都是需要执行的。

2.1 hosts及hostname

​ rehl7修改主机名使用hostnamectl命令

#节点一
[root@hadoop-1 ~]# hostnamectl set-hostname hadoop-1
#节点二
[root@hadoop-2 ~]# hostnamectl set-hostname hadoop-2
#节点三
[root@hadoop-3 ~]# hostnamectl set-hostname hadoop-3

​ 修改/etc/hosts(每个节点都执行)

[root@hadoop-1 ~]# vim /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
#集群节点地址 集群主机名
172.15.10.145 hadoop-1
172.15.10.146 hadoop-2
172.15.10.147 hadoop-3

2.2 禁用selinux并关闭防火墙

[root@hadoop-1 ~]# vim /etc/selinux/config
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected. 
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted
-------------------------------------------------------------------------------------------------------------------
[root@hadoop-1 ~]# systemctl stop firewalld.service
[root@hadoop-1 ~]# systemctl disable firewalld.service

2.3 系统参数修改

​ swap空间设置

[root@hadoop-1 ~]# echo "vm.swappiness = 0" >> /etc/sysctl.conf
[root@hadoop-1 ~]# sysctl -p

​ 设置透明大页面

[root@hadoop-1 ~]# echo never > /sys/kernel/mm/transparent_hugepage/defrag
[root@hadoop-1 ~]# echo never > /sys/kernel/mm/transparent_hugepage/enabled

​ 增⼤系统打开⽂件数

[root@hadoop-1 ~]# vim /etc/security/limits.conf
#末尾加上
* - nofile 100000
* - nproc 100000
hadoop soft memlock unlimited
hadoop hard memlock unlimited

2.4 ssh免密认证

[root@hadoop-1 ~]# ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa 
[root@hadoop-1 ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@hadoop-1
[root@hadoop-1 ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@hadoop-2
[root@hadoop-1 ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@hadoop-3
#验证方式,ssh hostname,直接连接上了说明配置成功
[root@hadoop-1 ~]# ssh hadoop-2

2.5 安装并启动Apache http

​ 在主节点执行

#安装Apache http
[root@hadoop-1 ~]# yum install -y httpd
#启动Apache http
[root@hadoop-1 cdh]# systemctl start httpd
#设置开机启动Apache http
[root@hadoop-1 cdh]# systemctl enable httpd
#创建安装文件http根目录
[root@hadoop-1 cdh]# mkdir -p /var/www/html/cm6
[root@hadoop-1 cdh]# mkdir -p /var/www/html/cdh6
[root@hadoop-1 cdh]# cd /usr/local/src/cdh/
#上传安装文件到http根目录
[root@hadoop-1 cdh]# cp allkeys.asc *.rpm /var/www/html/cm6
[root@hadoop-1 cdh]# ll /var/www/html/cm6
total 1199784
-rw-r--r-- 1 root root      14041 Jan 22 00:06 allkeys.asc
-rw-r--r-- 1 root root   10483568 Jan 22 00:06 cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm
-rw-r--r-- 1 root root 1203832464 Jan 22 00:06 cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm
-rw-r--r-- 1 root root      11488 Jan 22 00:06 cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm
-rw-r--r-- 1 root root      10996 Jan 22 00:06 cloudera-manager-server-db-2-6.3.1-1466458.el7.x86_64.rpm
-rw-r--r-- 1 root root   14209868 Jan 22 00:06 enterprise-debuginfo-6.3.1-1466458.el7.x86_64.rpm
#创建yum仓库
[root@hadoop-1 cdh]# cd /var/www/html/cm6
[root@hadoop-1 cm6]# yum install -y createrepo
[root@hadoop-1 cm6]# createrepo .

2.6 yum源设置

​ yum源设置有多种方式,无网络可以采用镜像挂载,有网络可以直接用清华源、阿里源等等,此处不做展开,自行百度。这里介绍下本地挂载的方式。

[root@hadoop-1 ~]# vim /etc/yum.repos.d/rhel7.repo
[base]
name=rhel7
baseurl=file:///media/cdrom
enabled=1
gpgcheck=0
[root@hadoop-1 ~]# mkdir -p /media/cdrom 
[root@hadoop-1 ~]# mount /dev/cdrom /media/cdrom
mount: /media/cdrom: WARNING: device write-protected, mounted read-only.
-------------------------------------------------------------------------------------------------------------------
#挂载自建镜像
[root@hadoop-1 cm6]# vim /etc/yum.repos.d/cloudera-manager.repo
[cloudera-manager]
name=Cloudera Manager 6.3.1
baseurl=http://hadoop-1/cm6
gpgcheck=0
enabled=1
autorefresh=0
type=rpm-md
[root@hadoop-1 cm6]# yum clean all
#查看是否挂载成功
[root@hadoop-1 cm6]# yum repolist

​ 网页测试能否正常访问

CDH6.3.2安装(rhel7.6)_第1张图片

2.7 时钟同步设置

​ 检查是否安装ntp,如果没有则使用yum安装

[root@hadoop-1 ~]# rpm -qa | grep ntp
[root@hadoop-1 ~]# yum install -y ntp

​ 主节点hadoop-1配置(设置为标准时间服务器)

[root@hadoop-1 ~]# vim /etc/ntp.conf
# For more information about this file, see the man pages
# ntp.conf(5), ntp_acc(5), ntp_auth(5), ntp_clock(5), ntp_misc(5), ntp_mon(5).

driftfile /var/lib/ntp/drift
#新增:日志目录.
logfile /var/log/ntpd.log

# Permit time synchronization with our time source, but do not
# permit the source to query or modify the service on this system.
restrict default nomodify notrap nopeer noquery

# Permit all access over the loopback interface.  This could
# be tightened as well, but to do so would effect some of
# the administrative functions.
restrict 127.0.0.1
restrict ::1

# Hosts on local network are less restricted.
#restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
#新增:授权172.15.10.0网段上的所有机器可以从这台机器上查询和同步时间
restrict 172.15.10.0 mask 255.255.255.0 nomodify

# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
#修改:注释外部时钟服务器配置,实际生产部署需要外部时钟服务器,否则会有告警
#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst

#broadcast 192.168.1.255 autokey        # broadcast server
#broadcastclient                        # broadcast client
#broadcast 224.0.1.1 autokey            # multicast server
#multicastclient 224.0.1.1              # multicast client
#manycastserver 239.255.254.254         # manycast server
#manycastclient 239.255.254.254 autokey # manycast client
#新增:当外部时间不可用时,使用本地时间.
server 172.15.10.145 iburst
fudge 127.0.0.1 stratum 10

# Enable public key cryptography.
#crypto
-------------------------------------------------------------------------------------------------------------------
[root@hadoop-1 ~]# systemctl enable ntpd
[root@hadoop-1 ~]# systemctl start ntpd

​ 其他节点配置,同步主节点时间

[root@hadoop-2 ~]# echo "server 172.15.10.145" >> /etc/ntp.conf
[root@hadoop-2 ~]# systemctl enable ntpd
[root@hadoop-2 ~]# systemctl start ntpd
#检查方式
[root@hadoop-2 ~]# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 hadoop-1        .INIT.          16 u   32   64    0    0.000    0.000   0.000

2.8 jdk安装

1、卸载默认安装的jdk

[root@hadoop-1 ~]# rpm -qa|grep jdk 
[root@hadoop-1 ~]# rpm -e --nodeps #上一步查询出来的jdk名字

2、安装jdk

​ 目录必须严格要求文中设置,cdh启动的时候强制找/usr/java下面的jdk

[root@hadoop-1 src]# mkdir -p /usr/java
[root@hadoop-1 ~]# cd /usr/local/src
#个人习惯将包上传到/usr/local/src目录
[root@hadoop-1 src]# tar -xzvf jdk-8u202-linux-x64.tar.gz -C /usr/java
[root@hadoop-1 src]# cd /usr/java
[root@hadoop-1 java]# ln -s jdk1.8.0_202/ default
[root@hadoop-1 java]# chmod -R 755 /usr/java
#设置环境变量
[root@hadoop-1 ~]# vim /etc/profile
##新增JAVA环境变量
export JAVA_HOME=/usr/java/default
export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib
export PATH=$JAVA_HOME/bin:$PATH
-------------------------------------------------------------------------------------------------------------------
[root@hadoop-1 java]# source /etc/profile

三、mysql安装部署

​ mysql只需要在server节点部署,以下操作只针对hadoop-1节点

3.1 卸载原有mysql

#查询mysql相关rpm包
[root@hadoop-1 ~]# rpm -qa | grep -i mysql
qt-mysql-4.8.7-2.el7.x86_64 
akonadi-mysql-1.9.2-4.el7.x86_64 
perl-DBD-MySQL-4.023-6.el7.x86_64
#将查询到的包卸载
[root@hadoop-1 ~]# rpm  --nodeps -ev qt-mysql-4.8.7-2.el7.x86_64 akonadi-mysql-1.9.2-4.el7.x86_64 perl-DBD-MySQL-4.023-6.el7.x86_64
#查询mysql相关文件
[root@hadoop-1 ~]# find / -name mysql
#删除mysql相关文件
[root@hadoop-1 ~]# rm -rf /var/lib/mysql
[root@hadoop-1 ~]# rm -rf /usr/lib64/mysql
[root@hadoop-1 ~]# rm -rf /usr/bin/mysql
[root@hadoop-1 ~]# rm -rf /usr/share/mysql
[root@hadoop-1 ~]# rm -rf /var/lib/pcp/config/pmlogconf/mysql
[root@hadoop-1 ~]# rm -rf /etc/selinux/targeted/active/modules/100/mysql
#删除mysql默认读取配置(使用tar包安装情况)
[root@hadoop-1 ~]# rm -rf /etc/my.cnf /etc/my.cnf.d/

3.2 安装mysql

官网地址:mysql8下载

[root@hadoop-1 ~]# cd /usr/local/src
[root@hadoop-1 src]# wget https://dev.mysql.com/get/Downloads/MySQL-8.0/mysql-8.0.18-el7-x86_64.tar.gz

解压压缩包,将解压出的程序目录改名并移动到/usr/local目录下,对其进行初始化操作后便可使用。:

[root@hadoop-1 src]# tar -zxvf mysql-8.0.18-el7-x86_64.tar.gz 
[root@hadoop-1 src]# cp -r  mysql-8.0.18-el7-x86_64/ /usr/local/mysql
[root@hadoop-1 src]# cd /usr/local/mysql/
[root@hadoop-1 mysql]# mkdir data
#创建mysql用户
[root@hadoop-1 mysql]# useradd mysql -M -s /sbin/nologin

修改my.cnf配置,典型的mysql8 my.cnf配置:

[root@hadoop-1 ~]# vim /etc/my.cnf
[client]
port = 3306
#根据实际情况调整mysql.sock配置
socket = /tmp/mysql.sock

[mysqld]
#Mysql服务的唯一编号 每个mysql服务Id需唯一
server-id = 1
#服务端口号 默认3306
port = 3306
#mysql安装根目录
basedir = /usr/local/mysql
#mysql数据文件所在位置
datadir = /usr/local/mysql/data
#pid
pid-file = /usr/local/mysql/data/mysql.pid
#设置socke文件所在目录
socket = /tmp/mysql.sock
#设置临时目录
tmpdir = /tmp
# 用户
user = mysql
# 允许访问的IP网段
bind-address = 0.0.0.0
# 跳过密码登录
#skip-grant-tables
#主要用于MyISAM存储引擎,如果多台服务器连接一个数据库则建议注释下面内容
skip-external-locking
#只能用IP地址检查客户端的登录,不用主机名
skip_name_resolve = 1
#事务隔离级别,默认为可重复读,mysql默认可重复读级别(此级别下可能参数很多间隙锁,影响性能)
transaction_isolation = READ-COMMITTED
#数据库默认字符集,主流字符集支持一些特殊表情符号(特殊表情符占用4个字节)
character-set-server = utf8mb4
#数据库字符集对应一些排序等规则,注意要和character-set-server对应
collation-server = utf8mb4_general_ci
#设置client连接mysql时的字符集,防止乱码
init_connect='SET NAMES utf8mb4'
#是否对sql语句大小写敏感,1表示不敏感
lower_case_table_names = 1
#最大连接数
max_connections = 400
#最大错误连接数
max_connect_errors = 1000
#TIMESTAMP如果没有显示声明NOT NULL,允许NULL值
explicit_defaults_for_timestamp = true
#SQL数据包发送的大小,如果有BLOB对象建议修改成1G
max_allowed_packet = 128M
#MySQL连接闲置超过一定时间后(单位:秒)将会被强行关闭
#MySQL默认的wait_timeout  值为8个小时, interactive_timeout参数需要同时配置才能生效
interactive_timeout = 1800
wait_timeout = 1800
#内部内存临时表的最大值 ,设置成128M。
#比如大数据量的group by ,order by时可能用到临时表,
#超过了这个值将写入磁盘,系统IO压力增大
tmp_table_size = 134217728
max_heap_table_size = 134217728
#数据库错误日志文件
log_error = error.log
#慢查询sql日志设置
slow_query_log = 1
slow_query_log_file = slow.log
#检查未使用到索引的sql
log_queries_not_using_indexes = 1
#针对log_queries_not_using_indexes开启后,记录慢sql的频次、每分钟记录的条数
log_throttle_queries_not_using_indexes = 5
#作为从库时生效,从库复制中如何有慢sql也将被记录
log_slow_slave_statements = 1
#慢查询执行的秒数,必须达到此值可被记录
long_query_time = 8
#检索的行数必须达到此值才可被记为慢查询
min_examined_row_limit = 100
#mysql binlog日志文件保存的过期时间,过期后自动删除
expire_logs_days = 5                   

3.3 初始化MySQL服务程序

​ 对目录进行授权,保证数据能够被mysql系统用户读取。在初始化阶段,应使用mysqld命令确认管理MySQL数据库服务的用户名称、数据保存目录及编码信息。在信息确认无误后开始进行初始化。在初始化的最后阶段,系统会给用户分配一个初始化的临时密码。mysql8版本只有在初始化的时候设置 lower_case_table_names=1才有效,切记。

[root@hadoop-1 mysql]# chown -R mysql:mysql /usr/local/mysql
[root@hadoop-1 mysql]# cd bin
[root@hadoop-1 bin]# ./mysqld --initialize --user=mysql --basedir=/usr/local/mysql --datadir=/usr/local/mysql/data --lower-case-table-names=1
#如果使用了上面的配置文件,会生成日志在data目录下,里面会有初始化生成的密码
[root@hadoop-1 bin]# vim /usr/local/mysql/data/error.log
2022-01-20T15:45:19.609453Z 0 [Warning] [MY-011068] [Server] The syntax 'expire-logs-days' is deprecated and will be removed in a future release. Please use binlog_expire_logs_seconds instead.
2022-01-20T15:45:19.609551Z 0 [System] [MY-013169] [Server] /usr/local/mysql/bin/mysqld (mysqld 8.0.18) initializing of server in progress as process 3281
2022-01-20T15:45:20.506473Z 0 [Warning] [MY-010161] [Server] You need to use --log-bin to make --expire_logs_days work.
2022-01-20T15:45:21.612476Z 5 [Note] [MY-010454] [Server] A temporary password is generated for root@localhost: ww-=dht_A0uj

​ 添加环境变量,方便调用

[root@hadoop-1 bin]# vim ~/.bash_profile
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin:/usr/local/mysql/bin

export PATH
[root@hadoop-1 bin]# source ~/.bash_profile

3.4 配置自启动并启动MySQL

​ libtinfo.so.5文件是MySQL数据库在8.0版本后新添加的重要的函数库文件,但部分情况不存在,需要将libtinfo.so.6.1文件复制过来或者作为链接文件才能正常启动:

[root@hadoop-1 bin]# cd /usr/local/mysql
[root@hadoop-1 mysql]# cp -a support-files/mysql.server /etc/init.d/
[root@hadoop-1 mysql]# chmod a+x /etc/init.d/mysql.server
[root@hadoop-1 mysql]# ln -s /usr/lib64/libtinfo.so.6.1 /usr/lib64/libtinfo.so.5 
[root@hadoop-1 mysql]# /etc/init.d/mysql.server start

​ 登录MySQL并修改密码

[root@hadoop-1 mysql]# mysql -u root -p
Enter password: 初始化的时候生成的密码
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 8
Server version: 8.0.27

Copyright (c) 2000, 2021, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> alter user 'root'@'localhost' identified by 'onaslowboattochina';
#mysql8之后需要额外执行此步才会将密码修改生效
mysql> use mysql;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY 'onaslowboattochina';
#设置远程访问权限,注意Mysql8.0不支持grant all privileges on *.* to root@“%“ identified by “.“;
mysql> CREATE USER 'root'@'%' IDENTIFIED BY 'onaslowboattochina';
Query OK, 0 rows affected (0.00 sec)

mysql> grant all privileges on *.* to 'root'@'%' ;
Query OK, 0 rows affected (0.00 sec)

mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)

3.5 创建数据库

​ 根据实际需要部署的服务,增加或者减少创建的数据库用户数

create database metastore default character set utf8;
CREATE USER 'hive'@'%' IDENTIFIED BY 'onaslowboattochina';
GRANT ALL PRIVILEGES ON metastore. * TO 'hive'@'%';
FLUSH PRIVILEGES;


create database rm default character set utf8;
CREATE USER 'rm'@'%' IDENTIFIED BY 'onaslowboattochina';
GRANT ALL PRIVILEGES ON rpm.* TO 'rm'@'%';
FLUSH PRIVILEGES;

create database amon default character set utf8;
CREATE USER 'amon'@'%' IDENTIFIED BY 'onaslowboattochina';
GRANT ALL PRIVILEGES ON amon.* TO 'amon'@'%';
FLUSH PRIVILEGES;

四、CDH安装

4.1 Cloudera Manager Server 安装(hadoop-1节点)

[root@hadoop-1 ~]# yum install -y cloudera-manager-server

​ cloudera-manager-server安装完成后会自动生成/opt/cloudera/parcel-repo目录,将准备好的 CDH parcel 包 copy⾄/opt/cloudera/parcel-repo,拷贝后将 CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1 以 mv 的⽅式重命名为CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha

[root@hadoop-1 ~]# cd /usr/local/src/cdh/
[root@hadoop-1 cdh]# cp CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1  manifest.json /opt/cloudera/parcel-repo/
[root@hadoop-1 cdh]# cd /opt/cloudera/parcel-repo/
[root@hadoop-1 parcel-repo]# mv CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1 CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha

安装mysql驱动

[root@hadoop-1 parcel-repo]# cd /usr/local/src/cdh/
[root@hadoop-1 cdh]# mkdir -p /usr/share/java
[root@hadoop-1 cdh]# cp mysql-connector-java-8.0.18.jar /usr/share/java/mysql-connector-java.jar

执行CM初始化脚本

[root@hadoop-1 ~]# /opt/cloudera/cm/schema/scm_prepare_database.sh mysql  scm scm onaslowboattochina
#释义																   库  用户 密码		
JAVA_HOME=/usr/java
Verifying that we can write to /etc/cloudera-scm-server
Creating SCM configuration file in /etc/cloudera-scm-server
Executing:  /usr/java/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
[                          main] DbCommandExecutor              INFO  Successfully connected to database.
All done, your SCM database is configured correctly!

启动CMserver

[root@hadoop-1 java]# systemctl start cloudera-scm-server.service
[root@hadoop-1 java]# systemctl status cloudera-scm-server.service
● cloudera-scm-server.service - Cloudera CM Server Service
   Loaded: loaded (/usr/lib/systemd/system/cloudera-scm-server.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2022-01-21 23:14:05 CST; 4s ago
  Process: 32326 ExecStartPre=/opt/cloudera/cm/bin/cm-server-pre (code=exited, status=0/SUCCESS)
 Main PID: 32332 (java)
    Tasks: 44
   CGroup: /system.slice/cloudera-scm-server.service
           └─32332 /usr/java/jdk1.8.0_202/bin/java -cp .:/usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:lib/* -server -Dlog4...

Jan 21 23:14:05 hadoop-1 systemd[1]: Starting Cloudera CM Server Service...
Jan 21 23:14:05 hadoop-1 systemd[1]: Started Cloudera CM Server Service.
Jan 21 23:14:05 hadoop-1 cm-server[32332]: JAVA_HOME=/usr/java/jdk1.8.0_202
Jan 21 23:14:05 hadoop-1 cm-server[32332]: Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
Jan 21 23:14:07 hadoop-1 cm-server[32332]: ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'org.apac...ation logging.
Jan 21 23:14:09 hadoop-1 cm-server[32332]: Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered...y unnecessary.
Hint: Some lines were ellipsized, use -l to show in full.

4.2 Cloudera Manager agent 安装(所有节点)

[root@hadoop-2 ~]# yum install -y cloudera-manager-agent
[root@hadoop-2 ~]# systemctl start cloudera-scm-agent.service
[root@hadoop-2 ~]# systemctl status cloudera-scm-agent.service
● cloudera-scm-agent.service - Cloudera Manager Agent Service
   Loaded: loaded (/usr/lib/systemd/system/cloudera-scm-agent.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2022-01-21 17:03:11 CST; 12s ago
 Main PID: 17500 (cmagent)
    Tasks: 21
   CGroup: /system.slice/cloudera-scm-agent.service
           └─17500 /usr/bin/python2 /opt/cloudera/cm-agent/bin/cm agent

Jan 21 17:03:11 hadoop-2 cm[17500]: [21/Jan/2022 17:03:11 +0000] 17500 MainThread agent        INFO     Re-using pre-existing directory: /var/run/cloudera-scm-agent/flood
Jan 21 17:03:11 hadoop-2 cm[17500]: [21/Jan/2022 17:03:11 +0000] 17500 MainThread agent        INFO     Re-using pre-existing directory: /var/run/cloudera-scm-agent/supervisor/include
Jan 21 17:03:11 hadoop-2 cm[17500]: [21/Jan/2022 17:03:11 +0000] 17500 MainThread agent        INFO     Re-using pre-existing directory: /var/run/cloudera-scm-agent/cgroups
Jan 21 17:03:11 hadoop-2 cm[17500]: [21/Jan/2022 17:03:11 +0000] 17500 MainThread agent        INFO     Re-using pre-existing directory: /var/run/cloudera-scm-agent/process
Jan 21 17:03:11 hadoop-2 cm[17500]: [21/Jan/2022 17:03:11 +0000] 17500 MainThread tmpfs        INFO     Successfully mounted tmpfs at /var/run/cloudera-scm-agent/process
Jan 21 17:03:11 hadoop-2 cm[17500]: [21/Jan/2022 17:03:11 +0000] 17500 MainThread logging      INFO     Logging to /var/log/cloudera-scm-agent/cloudera-scm-agent.log
Jan 21 17:03:18 hadoop-2 cm[17500]: status_server: added process group
Jan 21 17:03:18 hadoop-2 cm[17500]: flood: added process group
Jan 21 17:03:18 hadoop-2 cm[17500]: /opt/cloudera/cm-agent/lib/python2.7/site-packages/psutil/_pslinux.py:477: RuntimeWarning: dirty, writeback, mapped, commit_limit memory stats couldn't be ...d were set to 0
Jan 21 17:03:18 hadoop-2 cm[17500]: warnings.warn(msg, RuntimeWarning)
Hint: Some lines were ellipsized, use -l to show in full.

4.3 集群界面安装

​ 输入server节点的地址加7180默认端口,进入登录界面,默认账号密码为admin;admin
CDH6.3.2安装(rhel7.6)_第2张图片

​ 登陆完成后进入欢迎界面,点击继续即可

CDH6.3.2安装(rhel7.6)_第3张图片

​ 用户许可条款和条件界面,勾选后,点击继续

CDH6.3.2安装(rhel7.6)_第4张图片

​ 测试环境使用的三个节点,目前没有使用到高级功能,点击免费版本即可,选择后点击继续
CDH6.3.2安装(rhel7.6)_第5张图片

​ 进入集群创建环境界面,点击继续

CDH6.3.2安装(rhel7.6)_第6张图片

​ 输入集群名称,自行定义,完成后点击继续

CDH6.3.2安装(rhel7.6)_第7张图片

​ 输入主机名称,完成后点击搜索,系统将查询出来的主机显示到下方,勾选需要部署的主机后点击继续按钮

CDH6.3.2安装(rhel7.6)_第8张图片

​ 进入选择存储库界面,cloudera manager agent栏选择自定义存储库,库地址为之前创建的保存rpm包的地址,CDH other software栏选择使用parcel,下方会展示你放在server节点/opt/cloudera/parcel-repo/目录下的包。确认都勾选完毕后,点击继续。

CDH6.3.2安装(rhel7.6)_第9张图片

​ 进入JDK 安装选项,如果提前每台机器都部署好了jdk,这里可以不用勾选,勾选的话,/var/www/html目录下面要有java的rpm包。根据实际情况设置,完成后点击继续。(按本文档部署无需勾选)

CDH6.3.2安装(rhel7.6)_第10张图片

​ 进入提供ssh登录凭证界面,输入密码后,修改ssh端口为实际的端口(生产环境可能端口不是22),完成后点击继续

CDH6.3.2安装(rhel7.6)_第11张图片

​ 进入 install agents 界面,此处会自动完成agents的安装,等待完成后,点击继续

CDH6.3.2安装(rhel7.6)_第12张图片

​ 进入install parcels界面,此处也是全自动化安装,等待完成即可。

CDH6.3.2安装(rhel7.6)_第13张图片

​ inspect network performance:检测网络性能

​ inspect hosts:检测主机隐患

​ 按照本文档部署,理论上此处检测都是一次性通过,如果结果如下图所示,则点击继续即可。

CDH6.3.2安装(rhel7.6)_第14张图片

4.4集群界面设置

​ 进入部署服务选择界面,根据需要添加需要部署的服务。

CDH6.3.2安装(rhel7.6)_第15张图片

​ 为勾选的服务分配主机,这个需要提前根据自己部署的服务器的性能分配好,哪些节点部署哪些服务,下图只是参考,选择完毕后点击继续。

CDH6.3.2安装(rhel7.6)_第16张图片

​ 进入数据库设置界面,将之前创建的库,用户名,密码分别配置到对应分服务下面,点击测试连接,提示都是successful后,点击继续。

CDH6.3.2安装(rhel7.6)_第17张图片

​ 进入审核更改界面,这里的datanode配置的地址的硬盘需要越大越好。其他无需变更,点击继续即可。

CDH6.3.2安装(rhel7.6)_第18张图片

​ 进入命令详细信息界面,此处为自动化部署,等待即可,出现下图界面,点击继续即可。

CDH6.3.2安装(rhel7.6)_第19张图片

4.5 HDFS及YARN高可用

4. 5.1 HDFS高可用

​ 点击集群下面的HDFS,进入HDFS管理界面

CDH6.3.2安装(rhel7.6)_第20张图片

​ 点击操作,然后选择下面的启用High Acailability(PS:这边是部署完成后写的,所以展示的是禁用High Acailability)

CDH6.3.2安装(rhel7.6)_第21张图片

​ 进入入门界面,输入需要取的集群名称,点击继续即可。

CDH6.3.2安装(rhel7.6)_第22张图片

​ 根据实际情况选择需要部署的主机,完成后点击继续

CDH6.3.2安装(rhel7.6)_第23张图片

​ 进入审核更改界面,目录配置同之前说的一样,配置在容量足够大的目录下。
CDH6.3.2安装(rhel7.6)_第24张图片

​ 进入命令详细信息界面,此处为全自动化部署,下图报错可以忽略,完成后点击继续。

CDH6.3.2安装(rhel7.6)_第25张图片

​ 根据提示完成操作,然后点击完成。

CDH6.3.2安装(rhel7.6)_第26张图片

4.5.2 YARN 高可用

​ 点击操作,然后选择下面的启用High Acailability

CDH6.3.2安装(rhel7.6)_第27张图片

​ 进入入门界面,选择好备机后,点击继续

CDH6.3.2安装(rhel7.6)_第28张图片

​ 进入命令详细信息界面,此处为全程自动化部署,等待即可,出现下图界面说明部署完成,点击完成即可。

CDH6.3.2安装(rhel7.6)_第29张图片

4.6 部分报错解决

4.6.1 Erasure Coding Policy Verification Test

  • 问题分析
    纠删码策略警告是由于当前的HDFS集群DataNode节点数不满足当前所配置的纠删码策略需要的最小主机数

  • 纠删码策略是HDFS集群提供的数据恢复的安全机制
    其中

    • RS-3-2-1024k策略需要至少5台DataNode节点
    • RS-6-3-1024k策略需要至少9台DataNode节点
    • RS-10-4-1024k策略需要至少14台DataNode节点
  • 问题解决
    这里我们的集群无法满足最低的5台DataNode节点要求,所以可以选择关闭纠删策略

    点击报错Erasure Coding Policy Verification Test

CDH6.3.2安装(rhel7.6)_第30张图片

​ 进入Erasure Coding Policy Verification Test界面,点击为此服务更改Fallback Etasure Coding Policy

CDH6.3.2安装(rhel7.6)_第31张图片

CDH6.3.2安装(rhel7.6)_第32张图片

​ 重启HSDF生效

CDH6.3.2安装(rhel7.6)_第33张图片

你可能感兴趣的:(linux,修复,工作问题,hadoop)