hadoop CDH + kylin安装

一：实验环境

三台CentOS 6.7主机，IP地址

• 172.16.10.50 master
• 172.16.10.51 slave1
• 172.16.10.52 slave2

JDK版本：1.8.0
CDH版本：5.8.3
Kylin版本：1.5.4.1

cloudera-manager、parcels下载地址：

http://archive.cloudera.com/cm5/cm/5/
http://archive.cloudera.com/cdh5/parcels/

二：安装CDH

系统环境配置：

• 修改主机名
• 配置hosts
• master ssh免密码登录到slave
• 安装jdk
• 安装mysql

安装Cloudera Manager Server(master)

下载cloudera-manager-el6-cm5.3.2_x86_64.tar.gz到/opt目录

[root@master opt]# wget http://archive.cloudera.com/cm5/cm/5/cloudera-manager-el6-cm5.8.3_x86_64.tar.gz
[root@master opt]# tar zxvf cloudera-manager-el6-cm5.8.3_x86_64.tar.gz

下载parcel文件到parcel-repo中

[root@master parcel-repo]# wget http://archive.cloudera.com/cdh5/parcels/5.8.3/CDH-5.8.3-1.cdh5.8.3.p0.2-el6.parcel
[root@master parcel-repo]# wget http://archive.cloudera.com/cdh5/parcels/5.8.3/CDH-5.8.3-1.cdh5.8.3.p0.2-el6.parcel.sha1
[root@master parcel-repo]# wget http://archive.cloudera.com/cdh5/parcels/5.8.3/manifest.json
[root@master parcel-repo]# mv CDH-5.8.3-1.cdh5.8.3.p0.2-el6.parcel.sha1 CDH-5.8.3-1.cdh5.8.3.p0.2-el6.parcel.sha

创建cloudera-scm系统用户

[root@master opt]# useradd --system --home=/opt/cm-5.8.3/run/cloudera-scm-server/ --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

配置Cloudera Manager Server数据库

配置前先将mysql的jdbc库：mysql-connector-java-5.1.35-bin.jar拷贝到/opt/cm-5.8.3/share/cmf/lib/，再授权mysql账号,mysql-connector-java.jar下载地址

http://dev.mysql.com/downloads/connector/j/

mysql> grant all privileges on *.* to 'scm'@'%' IDENTIFIED BY 'scm';
mysql> grant all privileges on *.* to root@master identified by 'root';
mysql> flush privileges;

[root@master opt]# /opt/cm-5.8.3/share/cmf/schema/scm_prepare_database.sh mysql -h 172.16.11.50 -P 3306 -uroot -proot --scm-host master scm scm scm

安装Cloudera Manager Agent(slave)

1. 在master上修改/opt/ cm-5.8.3/etc/cloudera-scm-agent/config.ini中的server host 为主节点主机名：mater
2. 用scp将目录同步到agent上

[root@master opt]# scp -r /opt/cm-5.8.3 slave1:/opt
[root@master opt]# scp -r /opt/cm-5.8.3 slave2:/opt

1. 在slave中创建cloudera-scm系统用户

[root@slave1 opt]# useradd --system --home=/opt/cm-5.8.3/run/cloudera-scm-server/ --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

在master启动Cloudera server

[root@master opt]# /opt/cm-5.8.3/etc/init.d/cloudera-scm-server start

在slave启动Cloudera agent

[root@slave1 opt]# /opt/cm-5.8.3/etc/init.d/cloudera-scm-agent start

登录web管理界面

http://masterip:7180 默认账号密码:admin/admin

遇到的问题：

1. CDH5.8中hive metastore使用mysql作为元数据存储，但是CDH自动生成配置文件中没有生成mysql连接相关参数，手动添加配置：

[root@master ~]# vim /etc/hive/conf/hive-site.xml 

  javax.jdo.option.ConnectionURL
  jdbc:mysql://master/hive?createDatabaseIfNotExist=true


  javax.jdo.option.ConnectionDriverName
  com.mysql.jdbc.Driver


  javax.jdo.option.ConnectionUserName
  root


  javax.jdo.option.ConnectionPassword
  root

1. CDH5.8在web管理界面无法启动hive metastore，需命令行手动启动

[root@master ~]# hive --service metastore &

---

三：安装kylin

下载kylin：

下载地址：http://kylin.apache.org/cn/download/

本次实验只需下载for CDH版即可

[root@master opt]# wget http://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-1.5.4.1/apache-kylin-1.5.4.1-cdh5.7-bin.tar.gz

安装

解压到/opt目录下

[root@master opt]# ll
drwxr-xr-x  10 hdfs hdfs 4096 Nov 18 10:11 apache-kylin-1.5.4.1-cdh5.7-bin
drwxr-xr-x   6 1106 4001 4096 Nov 16 15:13 cloudera
drwxr-xr-x   9 1106 4001 4096 Oct 20 02:11 cm-5.8.3

添加环境变量

[root@master opt]# vim /etc/profile
export JAVA_HOME=/usr/local/java
export KYLIN_HOME=/opt/apache-kylin-1.5.4.1-cdh5.7-bin
export HBASE_HOME=/opt/cloudera/parcels/CDH-5.8.3-1.cdh5.8.3.p0.2/lib/hbase
export HADOOP_HOME=/opt/cloudera/parcels/CDH-5.8.3-1.cdh5.8.3.p0.2/
export HIVE_HOME=/opt/cloudera/parcels/CDH-5.8.3-1.cdh5.8.3.p0.2/lib/hive
export HADOOP_CMD=/opt/cloudera/parcels/CDH-5.8.3-1.cdh5.8.3.p0.2/lib/hadoop/bin/hadoop
export HCAT_HOME=/opt/cloudera/parcels/CDH/lib/hive-hcatalog
export PATH=$PATH:$JAVA_HOME/bin:/usr/local/python2.7/bin

编辑kylin配置文件：

[root@master ~]# vim /opt/apache-kylin-1.5.4.1-cdh5.7-bin/conf/kylin.properties
kylin.job.jar=/opt/apache-kylin-1.5.4-cdh5.7-bin/lib/kylin-job-1.5.4.jar
kylin.coprocessor.local.jar= /opt/apache-kylin-1.5.4-cdh5.7-bin/lib/kylin-coprocessor-1.5.4.jar
kylin.job.yarn.app.rest.check.status.url=http://slave2:8088/ws/v1/cluster/apps/${job_id}?anonymous=true

导入kylin测试数据：

[root@master ~]# cd /opt/apache-kylin-1.5.4.1-cdh5.7-bin/bin/
[root@master bin]# sh sample.sh

启动kylin：

[root@master bin]# ./kylin.sh start

登录后台：http://xxxxx:7070/kylin
账号密码：admin/KYLIN