cdh部署文档
一.概述CDH
Cloudera版本（Cloudera’s Distribution Including Apache Hadoop，简称“CDH”），基于Web的用户界面,支持大多数Hadoop组件，包括HDFS、MapReduce、Hive、Pig、 Hbase、Zookeeper、Sqoop,简化了大数据平台的安装、使用难度。
配合下面的教程使用
https://www.bilibili.com/video/BV1YE41117xM?from=search&seid=1286781471417665667
二.安装需要环境
提前准备好cm离线包
https://archive.cloudera.com/cm6/6.3.1/redhat7/yum/RPMS/x86_64/
Parcels离线源
https://archive.cloudera.com/cdh6/6.3.2/parcels/
机器最少需要3台，奇数个
官方推荐前置
1.系统推荐使用centos7.x,因为cm依赖python2.7
2.java环境需要jdk1.8,以及jdbc5.1
3.数据库环境。推荐使用mysql5.7
4.CDH-6.3.2套装
5.3台机器NTP时间同步
6./etc/hosts 添加主机名
7.所有主机都设置免密登录！！！！
# host 如图所示
192.168.122.193 hadoop001 # server + agent
192.168.122.196 hadoop002 # agent
192.168.122.197 hadoop003 # agent
1.yum安装mysql5.7
https://dev.mysql.com/doc/mysql-repo-excerpt/5.6/en/linux-installation-yum-repo.html
server机器才需要安装数据库
官方推荐my.cnf的配置
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
transaction-isolation = READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
symbolic-links = 0
key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1
max_connections = 550
#expire_logs_days = 10
#max_binlog_size = 100M
#log_bin should be on a disk with enough free space.
#Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your
#system and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log
#In later versions of MySQL, if you enable the binary log and do not set
#a server_id, MySQL will not start. The server_id must be unique within
#the replicating group.
server_id=1
binlog_format = mixed
read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M
# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
sql_mode=STRICT_ALL_TABLES
创建数据库以及用户，需要自行修改密码验证等级才能使用弱口令
# 创建数据库
CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
# 添加用户授权
GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY '123456';
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY '123456';
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY '123456';
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY '123456';
GRANT ALL ON metastore.* TO 'hive'@'%' IDENTIFIED BY '123456';
GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY '123456';
GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY '123456';
GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY '123456';
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY '123456';
# 刷新权限
FLUSH PRIVILEGES;
2.安装JDK和JDBC
所有机器都安装
# 一定是在/usr/java/jdk1.8.0_251下否则不能启动！！！！！不要改版本号
# 一定是在/usr/java/jdk1.8.0_251下否则不能启动！！！！！不要改版本号
# 一定是在/usr/java/jdk1.8.0_251下否则不能启动！！！！！不要改版本号
mkdir /usr/java
tar -zxvf jdk-8u251-linux-x64.tar.gz -C /usr/java/
# 赋予root权限
chown -R root:root /usr/java/
vim /etc/profile
#java environment
export JAVA_HOME=/usr/java/jdk1.8.0_251/
export CLASSPATH=.:${JAVA_HOME}/lib
export PATH=$PATH:${JAVA_HOME}/bin
# 让配置生效
source /etc/profile
安装jdbc
# 创建目录
mkdir -p /usr/share/java/
tar -zxf mysql-connector-java-5.1.49.tar.gz
cd mysql-connector-java-5.1.49
# 一定要重命名去掉版本号！！！ 目录不能修改
# 一定要重命名去掉版本号！！！ 目录不能修改
# 一定要重命名去掉版本号！！！ 目录不能修改
cp mysql-connector-java-5.1.49.jar /usr/share/java/mysql-connector-java.jar
3. 安装CM server以及agent 和内网parcel源
安装包如下
[root@localhost x86_64]# ll
total 1380420
-rw-r--r-- 1 2001 2001 10483568 Sep 25 2019 cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm
-rw-r--r-- 1 2001 2001 1203832464 Sep 25 2019 cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm
-rw-r--r-- 1 2001 2001 11488 Sep 25 2019 cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm
-rw-r--r-- 1 2001 2001 10996 Sep 25 2019 cloudera-manager-server-db-2-6.3.1-1466458.el7.x86_64.rpm
-rw-r--r-- 1 2001 2001 14209868 Sep 25 2019 enterprise-debuginfo-6.3.1-1466458.el7.x86_64.rpm
-rw-rw-r-- 1 2001 2001 184988341 Sep 25 2019 oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm
hadoop001(master)安装server daemons agent
cm server [hadoop001]:
yum localinstall -y cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm
yum localinstall -y cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm
yum localinstall -y cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm
hadoop002 (agent) 安装daemons agent
cm agent[hadoop002]:
yum localinstall -y cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm
yum localinstall -y cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm
hadoop003(agent) 安装daemons agent
cm agent[hadoop003]:
yum localinstall -y cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm
yum localinstall -y cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm
在CM server机器安装parcel离线源
# 安装apache
yum install -y httpd
# 创建cdh6_parcel目录
mkdir /var/www/html/cdh6_parcel/
# 重点注意cp的时候顺便把sha1要重命名为sha
# 重点注意cp的时候顺便把sha1要重命名为sha
# 重点注意cp的时候顺便把sha1要重命名为sha
cp CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1 /var/www/html/cdh6_parcel/CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha
cp CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel /var/www/html/cdh6_parcel/
cp manifest.json /var/www/html/cdh6_parcel/
#启动httpd
systemctl start httpd
systemctl enable httpd
# 测试parcel页面
curl http://127.0.0.1/cdh6_parcel/
集群中的所有机器都开启内存大页面
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
4. 修改 server以及agent 的配置文件
server机器修改配置/etc/cloudera-scm-server/db.properties
com.cloudera.cmf.db.type=mysql
com.cloudera.cmf.db.host=hadoop001
com.cloudera.cmf.db.name=scm
com.cloudera.cmf.db.user=scm
com.cloudera.cmf.db.password=123456
# setupTypey一定要修改为EXTERNAL
com.cloudera.cmf.db.setupType=EXTERNAL
启动server服务
systemctl start cloudera-scm-server
agent机器修改配置vim /etc/cloudera-scm-agent/config.ini
[General]
# Hostname of the CM server.
server_host=hadoop001
# Port that the CM server is listening on.
server_port=7182
启动agent服务
systemctl start cloudera-scm-agent
启动出错日志排查位置
# server日志
/var/log/cloudera-scm-server
# agent日志
/var/log/cloudera-scm-agent
4. 启动web进行配置
server机器启动后监听7180端口，登录web即可进行配置，需要注意的是web选择parcel源
选择继续同意各种协议，勾选选择免费版
选择机器界面勾选所有机器
选择存储界库，更多选项-远程Parcel存储库url地址更改为，apache搭建的内网存储库url地址
http://hadoop001 /cdh6_parcel/
cdh自检测试，测试网络host等
安装页面进行最小化安装，只安装一个服务即可
CDH公司常用业务
cdh常用的http端口
- 8082 flink web端口
- 8088 yarn web端口
- 7180 cdh web端口
- 9870 Hadoopweb端口