基于hdp3.0.1安装kylin2.6.5

环境与说明

• 本次测试的环境已提前装好ambari2.7.3和hdp3.0.1
• 操作系统为Centos7.4-1708
• 提前安装好的依赖组件有hadoop系列、ranger权限管理
• Spark未进行安装
• kerberos安全组件未开启

下载安装包

直接从kylin官网下载二进制包

下载地址：https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.5/apache-kylin-2.6.5-bin-hadoop3.tar.gz

将下载的二进制包上传到集群中的某一个节点，解压

[root@worker kylin]# mkdir -p /opt/kylin/
[root@worker kylin]# cd /opt/kylin/
上传至此外
[root@worker kylin]# ll
total 131056
-rw-r--r--. 1 root hadoop 134197437 May 13 15:20 apache-kylin-2.6.5-bin-hadoop3.tar.gz
# 解压
[root@worker kylin]# tar -zxvf apache-kylin-2.6.5-bin-hadoop3.tar.gz

[root@worker kylin]# ll
total 131056
drwxr-xr-x. 8 root root         150 Feb 11 16:33 apache-kylin-2.6.5-bin-hadoop3
-rw-r--r--. 1 root hadoop 134197437 May 13 15:20 apache-kylin-2.6.5-bin-hadoop3.tar.gz
# 重命名
[root@worker kylin]# mv apache-kylin-2.6.5-bin-hadoop3 apache-kylin
# 重命名后查看
[root@worker kylin]# ll
total 131056
drwxr-xr-x. 8 root root         150 Feb 11 16:33 apache-kylin
-rw-r--r--. 1 root hadoop 134197437 May 13 15:20 apache-kylin-2.6.5-bin-hadoop3.tar.gz

添加kylin的环境变量

编辑/etc/profile文件，将如下两行添加至文件最后

# vim /etc/profile
export KYLIN_HOME=/opt/kylin/apache-kylin
export PATH=$KYLIN_HOME/bin:$PATH

使用source命令使更改生效

[root@worker kylin]# source /etc/profile
[root@worker kylin]# echo $KYLIN_HOME
/opt/kylin/apache-kylin

检查env环境

进入KYLIN_HOME/bin目录下，执行check-env.sh脚本

[root@manager bin]# pwd
/opt/kylin/apache-kylin/bin
[root@manager bin]# sh check-env.sh 
Retrieving hadoop conf dir...
KYLIN_HOME is set to /opt/kylin/apache-kylin
mkdir: Permission denied: user=root, access=WRITE, inode="/":hdfs:hdfs:drwxr-xr-x
Failed to create /kylin. Please make sure the user has right to access /kylin

提示root用户没有访问hdfs的权限，这里的解决方法是在ranger中，将hdfs的all-path策略，增加hadoop用户组，因为root用户是属于hadoop的。

[root@manager bin]# id root
uid=0(root) gid=1003(hadoop) groups=1003(hadoop)

ranger中增加对应的权限

再次执行环境检测

[root@manager bin]# sh check-env.sh 
Retrieving hadoop conf dir...
KYLIN_HOME is set to /opt/kylin/apache-kylin

显示如上说明环境检测已完成了，可以下一步了。

启动kylin

进入KYLIN_HOME/bin目录下，执行kylin.sh脚本

[root@manager bin]# sh kylin.sh start
Retrieving hadoop conf dir...
KYLIN_HOME is set to /opt/kylin/apache-kylin
Retrieving hive dependency...

一直卡在这，不往下走了。。

原因是未修改kylin的配置文件，在使用hive cli的时候，没指定用户名和密码，卡在这里是等待输入用户名的…

一劳永逸的解决办法是修改配置文件和脚本。

修改一：
进入KYLIN_HOME/conf目录下，修改kylin.properties配置文件

## Hive client, valid value [cli, beeline]
kylin.source.hive.client=beeline  #修改为beeline
#
## Absolute path to beeline shell, can be set to spark beeline instead of the default hive beeline on PATH
kylin.source.hive.beeline-shell=beeline  #放开此行
#
## Parameters for beeline client, only necessary if hive client is beeline
# kylin.source.hive.beeline-params=-n root --hiveconf hive.security.authorization.sqlstd.confwhitelist.append='mapreduce.job.*|dfs.*' -u jdbc:hive2://localhost:10000
# hive的节点IP根据自己情况改
kylin.source.hive.beeline-params=-n hive -p hive -u jdbc:hive2://master.node:10000

修改二：
进入KYLIN_HOME/bin目录下，修改find-hive-dependency.sh配置文件
将hive_env行的${hive_conf_properties}参数去掉，不然程序会报运行中不能修改配置的错误。

if [ "${client_mode}" == "beeline" ]
then
    beeline_shell=`$KYLIN_HOME/bin/get-properties.sh kylin.source.hive.beeline-shell`
    beeline_params=`bash ${KYLIN_HOME}/bin/get-properties.sh kylin.source.hive.beeline-params`
    # hive_env=`${beeline_shell} ${hive_conf_properties} ${beeline_params} --outputformat=dsv -e "set;" 2>&1 | grep --text 'env:CLASSPATH' `
    hive_env=`${beeline_shell} ${beeline_params} --outputformat=dsv -e "set;" 2>&1 | grep --text 'env:CLASSPATH' `
else
    source ${dir}/check-hive-usability.sh
    hive_env=`hive ${hive_conf_properties} -e set 2>&1 | grep 'env:CLASSPATH'`
fi

再次启动kylin，hive那里已经解决，说spark未安装，无法继续下去

[root@manager bin]# sh kylin.sh start
Retrieving hadoop conf dir...
KYLIN_HOME is set to /opt/kylin/apache-kylin
Retrieving hive dependency...
Retrieving hbase dependency...
Retrieving hadoop conf dir...
Retrieving kafka dependency...
Retrieving Spark dependency...
spark not found, set SPARK_HOME, or run bin/download-spark.sh

修改脚本，屏蔽检测spark

function retrieveDependency() {
    #retrive $hive_dependency and $hbase_dependency
    source ${dir}/find-hive-dependency.sh
    source ${dir}/find-hbase-dependency.sh
    source ${dir}/find-hadoop-conf-dir.sh
    source ${dir}/find-kafka-dependency.sh
    # 屏蔽掉spark检测
    # source ${dir}/find-spark-dependency.sh
    ......

再次启动，检测都没问题了，又出现root用户的权限问题，这次是hbase的，和hdfs一样，在ranger中解决。

Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.security.AccessDeniedException): org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient permissions for user 'root' (action=create)
	at org.apache.ranger.authorization.hbase.AuthorizationSession.publishResults(AuthorizationSession.java:284)
	at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.authorizeAccess(RangerAuthorizationCoprocessor.java:643)
	at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.requirePermission(RangerAuthorizationCoprocessor.java:712)
	at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.preCreateTable(RangerAuthorizationCoprocessor.java:876)
	at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.preCreateTable(RangerAuthorizationCoprocessor.java:427)

再次执行，说hbase中kylin_metadata表已存在

Exception in thread "main" java.lang.IllegalArgumentException: Failed to find metadata store by url: kylin_metadata@hbase
	at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:99)
	at org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:111)
	at org.apache.kylin.rest.service.AclTableMigrationTool.checkIfNeedMigrate(AclTableMigrationTool.java:99)
	at org.apache.kylin.tool.AclTableMigrationCLI.main(AclTableMigrationCLI.java:43)

进入hbase中，把kylin_metadata删除掉再重新执行

hbase(main):001:0> list_namespace_tables "default"
TABLE                                                                                                                                                                                                                                                                         
kylin_metadata                                                                                                                                                                                                                                                                
1 row(s)
Took 0.3842 seconds                                                                                                                                                                                                                                                           
=> ["kylin_metadata"]
hbase(main):002:0> disable "default:kylin_metadata" #先禁用
Took 0.8867 seconds                                                                                                                                                                                                                                                           
hbase(main):003:0> drop "default:kylin_metadata"  #再删除
Took 0.2539 seconds

再次执行启动

[root@manager bin]# sh kylin.sh start
Retrieving hadoop conf dir...
KYLIN_HOME is set to /opt/kylin/apache-kylin
Retrieving hive dependency...
Retrieving hbase dependency...
Retrieving hadoop conf dir...
Retrieving kafka dependency...
Start to check whether we need to migrate acl tables
Retrieving hive dependency...
Retrieving hbase dependency...
Retrieving hadoop conf dir...
Retrieving kafka dependency...
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/kylin/apache-kylin/tool/kylin-tool-2.6.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
.......
2020-05-14 10:30:43,117 INFO  [ReadOnlyZKClient-worker.node:2181,manager.node:2181,master.node:2181@0x5241cf67] zookeeper.ZooKeeper:692 : Session: 0x2720db2d936001a closed
2020-05-14 10:30:43,117 INFO  [ReadOnlyZKClient-worker.node:2181,manager.node:2181,master.node:2181@0x5241cf67-EventThread] zookeeper.ClientCnxn:517 : EventThread shut down

A new Kylin instance is started by root. To stop it, run 'kylin.sh stop'
Check the log at /opt/kylin/apache-kylin/logs/kylin.log
Web UI is at http://manager.node:7070/kylin

页面访问

根据上面启动的提示，web UI地址是http://manager.node:7070/kylin，但是在浏览器上打不开。
查看日志，说是NoClassDefFoundError，这个时候需要找到对应的jar包，来替换kylin内的jar包。

2020-05-14 10:31:34,170 WARN  [localhost-startStop-1] [localhost].[/kylin]:164 : Exception sending context initialized event to listener instance of class [org.springframework.web.context.ContextLoaderListener]
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping': Invocation of init method failed; nested exception is java.lang.NoClassDefFoundError: org/apache/commons/configuration/ConfigurationException
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1628)
......
Caused by: java.lang.NoClassDefFoundError: org/apache/commons/configuration/ConfigurationException
	at java.lang.Class.getDeclaredMethods0(Native Method)
	at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
	at java.lang.Class.getDeclaredMethods(Class.java:1975)
	at org.springframework.util.ReflectionUtils.getDeclaredMethods(ReflectionUtils.java:613)
	at org.springframework.util.ReflectionUtils.doWithMethods(ReflectionUtils.java:524)
	at org.springframework.core.MethodIntrospector.selectMethods(MethodIntrospector.java:68)
	at org.springframework.web.servlet.handler.AbstractHandlerMethodMapping.detectHandlerMethods(AbstractHandlerMethodMapping.java:230)
	at org.springframework.web.servlet.handler.AbstractHandlerMethodMapping.initHandlerMethods(AbstractHandlerMethodMapping.java:214)
	at org.springframework.web.servlet.handler.AbstractHandlerMethodMapping.afterPropertiesSet(AbstractHandlerMethodMapping.java:184)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping.afterPropertiesSet(RequestMappingHandlerMapping.java:127)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1687)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1624)
	... 25 more
Caused by: java.lang.ClassNotFoundException: org.apache.commons.configuration.ConfigurationException
	at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1309)
	at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1137)
	... 37 more

jar包是commons-configuration-1.6.jar，可在ambari目录下找到
复制到kylin 的内置web项目中

cp /usr/lib/ambari-server/commons-configuration-1.6.jar \
	/opt/kylin/apache-kylin/tomcat/webapps/kylin/WEB-INF/lib

停止kylin服务，再次启动

[root@manager bin]# sh kylin.sh stop
Stopping Kylin: 4190
Kylin with pid 4190 has been stopped.
[root@manager bin]# sh kylin.sh start
Retrieving hadoop conf dir...
KYLIN_HOME is set to /opt/kylin/apache-kylin
Retrieving hive dependency...
Retrieving hbase dependency...
Retrieving hadoop conf dir...
Retrieving kafka dependency...
...

这时页面可正常打开

默认用户名为ADMIN，密码为KYLIN，可以正常登录，至此安装部署完成。