Centos7.3(1台) 搭建伪分布式集群
Cloudera Manager:5.7.0
JDK:
jdk-8u144-linux-x64.tar.gz
Cloudera Manager安装包:
cloudera-manager-5.7.5
CDH-Hadoop相关包:
CDH-5.7.0-1.cdh5.7.0.p0.4-el7.parcel
CDH-5.7.0-1.cdh5.7.0.p0.4-el7.parcel.sha1
manifest.json
mysql相关包:
mysql-connector-java-5.1.27-bin.jar
配置涉及到Linux操作中配置hosts、关闭防火墙、关闭selinux等操作,故使用root账户登录并对root进行ssh免密登录;
1.安装后登录root用户,进行权限认证:
sudo vim /etc/gdm/custom.conf
在[daemon]下方进行添加:
AutomaticLoginEnable=true
AutomaticLogin=root
2.重启并修改主机映射和主机名:
ifconfig #获取本机IP
修改主机映射:
vim /etc/hosts
将内容修改为:
127.0.0.1 localhost
本机IP hadoop000 #本机主机名hadoop000
修改主机名:
vim /etc/hostname
将内容修改为:
hadoop000
3.关闭防火墙和SELinux:
systemctl status firewalld #查看防火墙状态
systemctl stop firewalld #关闭防火墙
systemctl disable firewalld #禁止防火墙开机启动
setenforce 0
修改配置文件让机器重启也生效:
vim /etc/sysconfig/selinux
SELINUX=disabled
4.配置ssh免认证:
ssh master
exit
cd ~/.ssh
ssh-keygen –t rsa
cat id_rsa.pub >> authorized_keys
5.安装JDK:
可通过 rpm -qa | grep Java 命令来查询出系统自带的jdk:
rpm -qa | grep jdk查看jdk的具体信息,通过和java -version里面的jdk版本比较,
用rpm -e --nodeps命令卸载相应的jdk;
先卸载自带的OpenJdk
创建jdk安装目录:
mkdir /usr/jvm
解压jdk-8u144-linux-x64.tar.gz到jdk安装目录:
tar -zxvf jdk-8u144-linux-x64.tar.gz -C /usr/jvm
改名:
mv jdk1.8.0_144 jdk1.8.0
配置环境变量:
vim ~/.bash_profile
在末尾添加:
export JAVA_HOME=/usr/jvm/jdk1.8.0
export JRE_HOME=$JAVA_HOME/jre
export PATH=$JAVA_HOME/bin:$JAVA_HOME:$PATH
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tool.jar
使配置生效:
source ~/.bash_profile
6.安装ntp服务
yum -y install ntp
#配置29 vi /etc/ntp.conf
restrict 本机IP nomodify notrap
server 本机IP
重启ntp服务:
systemctl restart ntpd
chkconfig ntpd on
ntpdate -u master
9.安装mysql:
#卸载自带的MariaDB(mysql社区版本)
rpm -e --nodeps mariadb-libs-5.5.35-3.el7.x86_64
安装mysql:
rpm -ivh MySQL-server-5.6.36-1.el7.x86_64.rpm
rpm -ivh MySQL-devel-5.6.36-1.el7.x86_64.rpm
rpm -ivh MySQL-client-5.6.36-1.el7.x86_64.rpm
cp /usr/share/mysql/my-default.cnf /etc/my.cnf
/usr/bin/mysql_install_db
如果出现:
FATAL ERROR: please install the following Perl modules before executing ./scripts/mysql_install_db:
Data::Dumper
解决方法 :安装autoconf库
命令:yum -y install autoconf
service mysql start
cat /root/.mysql_secret #查看root账号密码
登录并修改密码为:123456
10.安装cloudera-manager:
tar xf cloudera-manager-centos7-cm5.7.0_x86_64.tar.gzip -C /opt
拷贝mysql jar
cp mysql-connector-java-5.1.27-bin.jar /usr/share/java/mysql-connector-java.jar
cp mysql-connector-java-5.1.27-bin.jar /opt/cm-5.7.0/share/cmf/lib/
创建用户:
useradd --system --home=/opt/cm-5.7.0/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "cloudera SCM user" cloudera-scm
配置Agent:
vi /opt/cm-5.7.0/etc/cloudera-scm-agent/config.ini
server_host=master
初始化数据库:
#登录mysql后执行
create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database amon DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database reports DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
grant all privileges on *.* to 'root'@'%' identified by '123456' with grant option;
grant all privileges on *.* to 'root'@'localhost' identified by '123456' with grant option;
grant all privileges on *.* to 'root'@'hd-29' identified by '123456' with grant option; flush privileges;
# 初始化cmf
/opt/cm-5.7.0/share/cmf/schema/scm_prepare_database.sh mysql -uroot -p scm scm
拷贝CDH-parcel包:
cp CDH-5.7.0-1.cdh5.7.0.p0.45-el7.parcel /opt/cloudera/parcel-repo
cp CDH-5.7.0-1.cdh5.7.0.p0.45-el7.parcel.sha1 /opt/cloudera/parcel-repo
cp manifest.json /opt/cloudera/parcel-repo
然后修改CDH-5.7.0-1.cdh5.7.0.p0.45-el7.parcel.sha1的名称,去掉末尾 的1,不然安装的时候会重新下载,非常重要!!!
11.启动Cloudera-Manager:
启动CM-server服务:
/opt/cm-5.7.0/etc/init.d/cloudera-scm-server start
登录:http://master:7180/cmf,默认账户和密码都是:admin。出现下图:
至此,Centos7下安装Cloudera-Manager5.7.0 完成。