Apache kylin 安装文档

kylin安装文档

hadoop spark安装参看这篇

https://blog.csdn.net/longgelaile/article/details/78424474

安装apache-kylin-2.0.0-bin-hbase098.tar.gz 去官网自行下载安装包

linux版本centos

安装版本kylin2.0.0版本 hbase 0.98.6 hive版本apache-hive-2.1.1-bin.tar mysql版本5.6 zk版本采用3.4 zk装了三台集群模式 之前装了两台有点小毛病

hadoop采用的是2.7版本的，但是这个版本比较新需要将0.98版本的hbase中的hadoop依赖jar包替换成2.7的

tar -xvzf apache-kylin-2.0.0-bin-hbase098.tar.gz解压目录

首先配置环境变量

export KYLIN_HOME=/home/hadoop/apache-kylin-2.0.0-bin

并刷新

配置conf下的

ome/hadoop/apache-kylin-2.0.0-bin/conf/

kylin.properties

配置如下

指定端口

kylin.rest.servers=192.168.0.222:7070

 

# Kylin server mode, valid value [all,query, job]

kyin.server.mode=all 指定模式

 

# Hive database name for putting theintermediate flat tables

kylin.job.hive.database.for.intermediatetable=lin

 

#### METADATA | ENV ###

#

## The metadata store in hbase

kylin.metadata.url=/home/hadoop/tmp/kylin_temp

#

## Working folder in HDFS, better bequalified absolute path, make sure user has the right permission to thisdirectory

kylin.env.hdfs-working-dir=/kylin

#

## DEV|QA|PROD. DEV will turn on some devfeatures, QA and PROD has no difference in terms of functions.

kylin.env=QA

#### SERVER | WEB ###

#

## Kylin server mode, valid value [all,query, job]

kylin.server.mode=all

 

首次启动报的错误

Failed to load keystore type JKS with pathconf/.keystore due to /home/hadoop/apache-kylin-2.0.0-bin/tomcat/conf/.keystore(没有那个文件或目录)

java.io.FileNotFoundException: /home/hadoop/apache-kylin-2.0.0-bin/tomcat/conf/.keystore(没有那个文件或目录)

       at java.io.FileInputStream.open(Native Method)

去掉tomcat下的https

在kylin内置tomcat的server.xml中里边有个对https的支持没启用的话 注释掉

先执行

./check-env.sh 检查环境

如果没问题执行

./kylin.sh start

启动后如果构建cube的时候报找不到kylin.properties 的时候 需要将peoperties的配置文件中没用到的选项全部注释掉

如果启动中报spark assembly lib not found这个错误  

修改find-spark-dependency.sh 中的spark_dependency=`find-L $spark_home -name 'spark*.jar' ! -name '*doc*' ! -name '*test*' ! -name'*sources*' ''-printf '%p:' | sed 's/:$//'`

这行将里边的'spark- assembly *.jar替换成'spark*.jar 因为启动的时候去加载spark的jar包

如果在安装kylin的时候遇到这个错误如下

2017-09-18 15:21:16,628 INFO  [main] zookeeper.ZooKeeper:438 : Initiatingclient connection, connectString=VM-10-112-178-109:2181,VM-10-112-178-110:2181sessionTimeout=120000 watcher=org.apache.curator.ConnectionState@13f2749c

2017-09-18 15:21:16,629 INFO  [main-SendThread(VM-10-112-178-110:2181)]zookeeper.ClientCnxn:975 : Opening socket connection to serverVM-10-112-178-110/10.112.178.110:2181. Will not attempt to authenticate usingSASL (unknown error)

2017-09-18 15:21:16,630 INFO  [main-SendThread(VM-10-112-178-110:2181)] zookeeper.ClientCnxn:852: Socket connection established to VM-10-112-178-110/10.112.178.110:2181,initiating session

2017-09-18 15:21:16,631 DEBUG [main]util.ZookeeperDistributedLock:142 : 2933@VM-10-112-178-108 trying to lock/kylin/kylin_metadata/create_htable/kylin_metadata/lock

2017-09-18 15:21:16,634 WARN  [main-SendThread(VM-10-112-178-110:2181)]zookeeper.ClientCnxnSocket:139 : Connected to an old server; r-o mode will beunavailable

2017-09-18 15:21:16,634 INFO  [main-SendThread(VM-10-112-178-110:2181)]zookeeper.ClientCnxn:1235 : Session establishment complete on serverVM-10-112-178-110/10.112.178.110:2181, sessionid = 0x25e93878ddb0022,negotiated timeout = 40000

2017-09-18 15:21:16,639 INFO  [main-EventThread]state.ConnectionStateManager:228 : State change: CONNECTED

2017-09-18 15:21:16,673 INFO  [main] util.ZookeeperDistributedLock:154 :2933@VM-10-112-178-108 acquired lock at/kylin/kylin_metadata/create_htable/kylin_metadata/lock

2017-09-18 15:21:16,679 DEBUG [main]hbase.HBaseConnection:313 : Creating HTable 'kylin_metadata'

2017-09-18 15:21:16,860 DEBUG [main]util.ZookeeperDistributedLock:222 : 2933@VM-10-112-178-108 trying to unlock/kylin/kylin_metadata/create_htable/kylin_metadata/lock

2017-09-18 15:21:16,868 INFO  [main] util.ZookeeperDistributedLock:233 :2933@VM-10-112-178-108 released lock at/kylin/kylin_metadata/create_htable/kylin_metadata/lock

Exception in thread "main"java.lang.IllegalArgumentException: Failed to find metadata store by url:kylin_metadata@hbase

       at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:89)

       atorg.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:101)

       at org.apache.kylin.rest.service.AclTableMigrationTool.checkIfNeedMigrate(AclTableMigrationTool.java:94)

       atorg.apache.kylin.tool.AclTableMigrationCLI.main(AclTableMigrationCLI.java:41)

Caused by:java.lang.reflect.InvocationTargetException

       at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

       atsun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)

       atsun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

       at java.lang.reflect.Constructor.newInstance(Constructor.java:526)

       atorg.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:83)

       ... 3 more

Caused by: java.lang.NoSuchMethodError:org.apache.hadoop.hbase.HTableDescriptor.addFamily(Lorg/apache/hadoop/hbase/HColumnDescriptor;)V

       atorg.apache.kylin.storage.hbase.HBaseConnection.createHTableIfNeeded(HBaseConnection.java:320)

       atorg.apache.kylin.storage.hbase.HBaseResourceStore.createHTableIfNeeded(HBaseResourceStore.java:110)

       atorg.apache.kylin.storage.hbase.HBaseResourceStore.(HBaseResourceStore.java:91)

       ... 8 more

 

上述错误的原因是由于kylin的版本不兼容hbase造成的例如2.0.0版本的kylin官网给出的hbase版本是0.98.*,所以说安装一些新版本可能会造成此错误Caused by: java.lang.NoSuchMethodError:一些类可能不存在了引起导致

还有就是构建的时候可能会爆出连接zk为本地而不是hbase连接的zk集群，这个时候就得重装zk然后 检查hbase是否正常，并且kylin重启

 最终看到的配置如下：

kylin.hbase.region.cut.large=50

deploy.env=QA

acl.adminRole=ROLE_ADMIN

kylin.rest.timezone=GMT+8

kylin.sandbox=true

kylin.job.jar=/opt/apache-kylin-1.5.1-bin/lib/kylin-job-1.5.1.jar

kylin.hdfs.working.dir=/kylin

ldap.user.searchBase=

kylin.job.concurrent.max.limit=10

kylin.job.remote.cli.password=

saml.metadata.file=classpath:sso_metadata.xml

kylin.job.yarn.app.rest.check.interval.seconds=10

mail.sender=

mail.password=

kylin.job.remote.cli.username=

mail.username=

saml.context.serverPort=443

kylin.web.help.length=4

kylin.job.run.as.remote.cmd=false

ldap.service.searchPattern=

kylin.web.contact_mail=

ldap.user.groupSearchBase=

kylin.hbase.region.cut.small=5

kylin.web.hive.limit=20

kylin.job.mapreduce.default.reduce.input.mb=500

kylin.job.hive.database.for.intermediatetable=default

kylin.metadata.url=kylin_metadata@hbase

ldap.password=

ldap.username=

kylin.storage.url=hbase

ganglia.port=8664

ldap.user.searchPattern=

kylin.job.status.with.kerberos=false

ganglia.group=

kylin.hbase.cluster.fs=

acl.defaultRole=ROLE_ANALYST,ROLE_MODELER

saml.context.contextPath=/kylin

mail.host=

kylin.job.remote.cli.working.dir=/tmp/kylin

kylin.web.diagnostic=

saml.context.scheme=https

kylin.job.cubing.inmem.sampling.percent=100

ldap.service.groupSearchBase=

saml.metadata.entityBaseURL=https://hostname/kylin

kylin.hbase.hfile.size.gb=5

ldap.service.searchBase=

[email protected]

mail.enabled=false

kylin.rest.servers=bigdatacloud:7070

kylin.security.profile=testing

kylin.job.retry=0

saml.context.serverName=hostname

kylin.coprocessor.local.jar=/opt/apache-kylin-1.5.1-bin/lib/kylin-coprocessor-1.5.1.jar

ldap.server=ldap://ldap_server:389

kylin.job.remote.cli.hostname=

kylin.query.security.enabled=true

kylin.server.mode=all

kylin.web.help.3=onboard|Cube Design Tutorial|

kylin.web.help.2=tableau|Tableau Guide|

kylin.web.help.1=odbc|ODBC Driver|

kylin.hbase.region.cut.medium=10

kylin.web.help.0=start|Getting Started|

kylin.web.hadoop=

kylin.web.streaming.guide=http://kylin.apache.org/