【CDH部署】全网最简单CDH6.2.0环境部署
下载链接
官方下载链接: https://archive.cloudera.com/cdh6/6.2.0/parcels/
百度网盘下载链接: https://pan.baidu.com/s/1Dm5Elf9uQqn14BUbgU3AFQ 提取码: mws3
主机规划
sparkproject1 manager 8G 50GB
sparkproject2 agent 4G 50GB
sparkproject3 agent 4G 50GB
1. 设置IP, hostname, hosts, ssh免密登录
可参考Hadoop中linux基本配置那篇文章
【Hadoop部署】Hadoop环境部署1-Linux基础设置
2. 关闭防火墙
systemctl stop firewalld 关闭防火墙
systemctl disable firewalld 禁止防火墙开机自启
vim /etc/selinux/config —> SELINUX=disabled
3.安装java, 和mysql-connector-java-5.1.34.jar
========
4.一定要装mysql, 不能用mariadb会有依赖包的问题
删除mysql旧的依赖
rpm -qa|grep mariadb
yum remove mariadb
安装mysql
rpm -ivh mysql-community-common-5.7.26-1.el7.x86_64.rpm
rpm -ivh mysql-community-libs-5.7.26-1.el7.x86_64.rpm
rpm -ivh mysql-community-client-5.7.26-1.el7.x86_64.rpm
rpm -ivh mysql-community-server-5.7.26-1.el7.x86_64.rpm
rpm -ivh mysql-community-libs-compat-5.7.26-1.el7.x86_64.rpm
初始化数据库，会生成随机密码
mysqld --initialize --user=root
cat /var/log/mysqld.log | tail -n5
末尾有密码：abCIja+g1aJ?
修改mysql配置文件
vim /etc/my.cnf
在 [mysqld] 标签下添加
init_connect='SET collation_connection = utf8_unicode_ci'
init_connect='SET NAMES utf8'
character-set-server=utf8
collation-server=utf8_unicode_ci
skip-character-set-client-handshake
chmod -R 777 /var/lib/mysql
systemctl start mysqld.service
systemctl enable mysqld.service
进入CLI环境，修改root默认密码。创建必要的表，更新远程访问权限
mysql -uroot -p
修改用户密码
ALTER USER USER() IDENTIFIED BY 'xiangyang';
创建后面cdh需要的表
create database cmserver default charset utf8 collate utf8_general_ci;
create database metadata default charset utf8 collate utf8_general_ci;
create database oozie default charset utf8 collate utf8_general_ci;
create database hue default charset utf8 collate utf8_general_ci;
允许远程访问mysql
update user set host='%' where host='localhost';
flush privileges;
systemctl restart mysqld
以下是模拟Cloudera的仓库环境，本地安装cdh
安装apache服务器
yum install httpd
systemctl start httpd.service
systemctl enable httpd.service
yum install -y createrepo
mkdir -p /var/www/html/cloudera-repos/cm6
cd /var/www/html/cloudera-repos/cm6/
cm文件考过来
createrepo /var/www/html/cloudera-repos/cm6/
yum clean all
yum makecache
vim /etc/yum.repos.d/cloudera-manager.repo
[cloudera-manager]
name = Cloudera Manager, Version 6.2.0
baseurl = http://sparkproject1/cloudera-repos/cm6/
gpgcheck = 0
yum clean all
yum makecache
rpm --import https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/RPM-GPG-KEY-cloudera
sudo yum install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server
cd /opt/cloudera/parcel-repo
parcel拷过来
注册数据库
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql cmserver root xiangyang
启动Cloudera Manager Server
大概要几分钟。可以观察/var/log/cloudera_server中的日志，看有没有报错
service cloudera-scm-server start
成功的话关机克隆，更新一些必要信息
ifconfig ens33 192.168.71.111
vim /etc/sysconfig/network-scripts/ifcfg-ens33
vim /etc/hostname
hostnamectl set-hostname sparkproject2
子节点可以删的组件
yum remove mysql
yum remove httpd
yum remove cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server
rm -rf /var/www/html/cloudera-repos/cm6/
rm -rf /var/www/html/streamsets
问题答疑
如果发现没有权限操作hdfs的情况，使用如下命令
sudo -u hdfs hadoop fs -setrep -R 2 /
StreamSets的安装
拷贝STREAMSETS-3.13.0.jar 到 /opt/cloudera/csd
mkdir -p /var/www/html/streamsets
streamsets文件拷过来
createrepo /var/www/html/streamsets
vim /etc/yum.repos.d/ss.repo
[ssrepo]
name = ss_repo
baseurl = http://sparkproject1/streamsets
enable = true
gpgcheck = false
扩展磁盘
pvcreate /dev/mapper/centos-root
lvextend -L +10G /dev/mapper/centos-root
sync; echo 3 > /proc/sys/vm/drop_caches ; free -h