从零开始虚拟机安装单节点CDH5.15
从零开始虚拟机安装单节点CDH5.15
- 环境和离线文件
- 安装CentOS7
- 更换yum源
- 配置host
- 配置SSH免密登陆
- 关闭selinux
- 安装jdk1.8
- 安装mysql
- 添加jdbc驱动
- 为CDH创建数据库和用户
- ~~导入repo(可不做)~~
- 解压cm
- 初始化CM5的数据库
- 配置cloudera-manger-agent指向主节点服务器
- 创建cloudera-scm用户
- • 安装问题
- • 问题:hue 数据库连接不上
- • 问题:启动HUE Load Balancer角色失败。
- • 配置HUE 遇到Error in sasl_client_start (-4) SASL(-4): no mechanism available: No worthy mechs found
- 结果
环境和离线文件
| 软件 | VMwave Workstation15 |
| Xshell6 | |
| 文件 | CentOS-7-x86_64-Minimal-1804.iso |
| mysql-connector-java-5.1.47.tar.gz | |
| CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel | |
| CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel.sha1 | |
| manifest.json | |
| cloudera-manager-centos7-cm5.15.1_x86_64.tar.gz | |
| 配置分配 | cpu i5-8300H 8核 |
| 内存 13.4G | |
| 硬盘 40G固态 | |
| 占用情况 | 启动CDH后占9.4G内存 |
| 关闭Cloudera Management Service可以节省2G内存 | |
| 硬盘总占用12G |
安装CentOS7
图形界面选择语言中文,打开网络设置,配置静态ip 192.168.5.140 主机名master.hadoop,配置root密码123456
使用xshell登陆,
配置终端提示符颜色
vi .bashrc
export PS1='[\[\e[1;34;1m\]\u@\h \W\[\e[0m\]]$ '
source .bashrc
更换yum源
yum install -y wget
cd /etc/yum.repos.d
sudo mv CentOS-Base.repo CentOS-Base.repo.bak
wget http://mirrors.163.com/.help/CentOS7-Base-163.repo
mv CentOS7-Base-163.repo CentOS-Base.repo
yum clean all
yum makecache
#更新软件,安装常用软件
yum update -y
yum install -y openssh-server vim gcc gcc-c++ glibc-headers bzip2-devel lzo-devel curl openssh-clients zlib-devel autoconf automake cmake libtool openssl-devel fuse-devel snappy-devel telnet unzip zip net-tools.x86_64 firewalld systemd lrzsz psmisc
配置host
vi /etc/hosts
尾部添加192.168.5.140 master.hadoop
配置SSH免密登陆
主要分为两个步骤：首先在所有的节点生成公钥
ssh-keygen -t rsa
然后将所有的节点执行拷贝公钥
ssh-copy-id [email protected]
来自 https://blog.csdn.net/fffsssfff6/article/details/81215416
关闭selinux
vim /etc/selinux/config
把文件里的SELINUX=disabled
关闭防火墙和查看防火墙状态：
systemctl stop firewalld
systemctl disable firewalld
systemctl status firewalld
来自 https://blog.csdn.net/fffsssfff6/article/details/81215416
安装jdk1.8
使用rz上传jdk-8u191-linux-x64.tar.gz mysql-connector-java-5.1.47.tar.gz
mkdir -p /usr/java
tar -xzf jdk-8u191-linux-x64.tar.gz -C /usr/java/
vim /etc/profile
export JAVA_HOME=/usr/java/jdk1.8.0_191
export JRE_HOME=$JAVA_HOME/jre
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
source /etc/profile
安装mysql
wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm
rpm -ivh mysql-community-release-el7-5.noarch.rpm
yum update
yum install mysql-server
systemctl enable mysqld
systemctl start mysqld
#设置mysql root密码
/usr/bin/mysql_secure_installation
添加jdbc驱动
mkdir -p /usr/share/java/
tar xzf mysql-connector-java-5.1.47.tar.gz
cd mysql-connector-java-5.1.47
mv mysql-connector-java-5.1.47-bin.jar /usr/share/java/mysql-connector-java.jar
来自 https://www.cloudera.com/documentation/enterprise/5-15-x/topics/cm_ig_mysql.html#cmig_topic_5_5
为CDH创建数据库和用户
mysql -u root -p123456
#输入密码登陆mysql ，然后创建多个数据库，并完成授权。
CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'scm';
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY 'amon';
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY 'rman';
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY 'hue';
CREATE DATABASE hive DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON hive.* TO 'hive'@'%' IDENTIFIED BY 'hive';
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY 'sentry';
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie';
来自 https://blog.csdn.net/fffsssfff6/article/details/81215416
导入repo(可不做)
wget https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/cloudera-manager.repo -P /etc/yum.repos.d/
rpm --import https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/RPM-GPG-KEY-cloudera
来自https://www.cloudera.com/documentation/enterprise/5-15-x/topics/configure_cm_repo.html
https://www.cloudera.com/documentation/enterprise/5-15-x/topics/configure_cm_repo.html
解压cm
tar zxf cloudera-manager-centos7-cm5.15.1_x86_64.tar.gz -C /opt/
初始化CM5的数据库
/opt/cm-5.15.1/share/cmf/schema/scm_prepare_database.sh mysql scm scm scm
配置cloudera-manger-agent指向主节点服务器
vim /opt/cm-5.15.1/etc/cloudera-scm-agent/config.ini
server_host=master.hadoop
来自 https://blog.csdn.net/shawnhu007/article/details/52579204
创建cloudera-scm用户
useradd --system --home=/opt/cm-5.15.1/run/cloudera-scm-server/ --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm
#来自
#将cdh安装包放到/opt/cloudera/parcel-repo/下,其中CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel.sha1要把末尾1去掉
cd /opt/cloudera/parcel-repo/
mv ~/CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel ./
mv ~/CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel.sha ./
mv ~/manifest.json ./
/opt/cm-5.15.1/etc/init.d/cloudera-scm-server start启动服务端。
/opt/cm-5.15.1/etc/init.d/cloudera-scm-agent start启动Agent服务。
通过http://192.168.5.140:7180访问
• 安装问题
• 问题:hue 数据库连接不上
1，Unexpected error. Unable to verify database connection
查看日志报错是：ImportError: libxslt.so.1: cannot open shared object file: No such file or directory
原因是centos缺少库文件，执行如下命令即可
解决：
yum install krb5-devel cyrus-sasl-gssapi cyrus-sasl-deve libxml2-devel libxslt-devel mysql mysql-devel openldap-devel python-devel python-simplejson sqlite-devel
作者：陈苏漾
来源：CSDN
原文：https://blog.csdn.net/chshgod1/article/details/80475596
• 问题:启动HUE Load Balancer角色失败。
原因：服务器环境没有预先安装httpd，mod_ssl服务
解决方法：
yum -y install httpd
yum -y install mod_ssl
来自 https://segmentfault.com/q/1010000013552053
• 配置HUE 遇到Error in sasl_client_start (-4) SASL(-4): no mechanism available: No worthy mechs found
解决方法：
yum install cyrus-sasl-plain cyrus-sasl-devel cyrus-sasl-gssapi
来自 https://blog.csdn.net/youngqj/article/details/50954275?locationNum=2
结果
参考:https://blog.csdn.net/gtsina/article/details/78048925