Kylin(SQL on HBase)
参考
http://kylin.apache.org/docs24/install/index.html
介绍
Apache Kylin™是一个开源的分布式分析引擎，提供Hadoop/Spark之上的SQL查询接口及多维分析（OLAP）能力以支持超大规模数据，最初由eBay Inc. 开发并贡献至开源社区。它能在亚秒内查询巨大的Hive表。
安装
报错Caused by: java.net.UnknownHostException: hdp6:2181: Name or service not known，这个错看了很久，因为hostname和port都是对的，卡了我很久，后来突然发现2019-10-11 20:38:54,715 INFO [Thread-6] zookeeper.ZooKeeper:438 : Initiating client connection, connectString=hdp6:2181:2181,hdp7:2181:2181,hdp8:2181:2181 sessionTimeout=120000 watcher=org.apache.curator.ConnectionState@4da855dd这句中所有的zookeeper都多了一个端口，这才意识到问题
Caused by: org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197)
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:87)
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:115)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.getZooKeeper(CuratorFrameworkImpl.java:477)
at org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:302)
at org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:291)
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
at org.apache.kylin.storage.hbase.util.ZookeeperDistributedLock.peekLock(ZookeeperDistributedLock.java:204)
... 12 more
2019-10-11 20:38:54,715 INFO [Thread-6] zookeeper.ZooKeeper:438 : Initiating client connection, connectString=hdp6:2181:2181,hdp7:2181:2181,hdp8:2181:2181 sessionTimeout=120000 watcher=org.apache.curator.ConnectionState@4da855dd
2019-10-11 20:38:54,717 INFO [close-hbase-conn] hbase.HBaseConnection:136 : Closing HBase connections...
2019-10-11 20:38:54,718 INFO [close-hbase-conn] client.ConnectionManager$HConnectionImplementation:1710 : Closing zookeeper sessionid=0x16db9f171ad0013
2019-10-11 20:38:54,719 ERROR [Thread-6] curator.CuratorZookeeperClient:206 :
java.io.IOException: java.net.UnknownHostException: hdp6:2181: Name or service not known
at org.apache.curator.ConnectionState.close(ConnectionState.java:117)
at org.apache.curator.CuratorZookeeperClient.close(CuratorZookeeperClient.java:202)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.close(CuratorFrameworkImpl.java:309)
at org.apache.kylin.storage.hbase.util.ZookeeperDistributedLock$Factory$1.run(ZookeeperDistributedLock.java:66)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.UnknownHostException: hdp6:2181: Name or service not known
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
at java.net.InetAddress.getAllByName(InetAddress.java:1192)
at java.net.InetAddress.getAllByName(InetAddress.java:1126)
at org.apache.phoenix.shaded.org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:61)
at org.apache.phoenix.shaded.org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445)
at org.apache.curator.utils.DefaultZookeeperFactory.newZooKeeper(DefaultZookeeperFactory.java:29)
at org.apache.curator.framework.imps.CuratorFrameworkImpl$2.newZooKeeper(CuratorFrameworkImpl.java:146)
at org.apache.curator.HandleHolder$1.getZooKeeper(HandleHolder.java:94)
at org.apache.curator.HandleHolder.internalClose(HandleHolder.java:128)
at org.apache.curator.HandleHolder.closeAndClear(HandleHolder.java:71)
at org.apache.curator.ConnectionState.close(ConnectionState.java:113)
... 4 more
2019-10-11 20:38:54,742 INFO [main-EventThread] zookeeper.ClientCnxn:512 : EventThread shut down
2019-10-11 20:38:54,742 INFO [close-hbase-conn] zookeeper.ZooKeeper:684 : Session: 0x16db9f171ad0013 closed
ERROR: Unknown error. Please check full log. 翻看源码如下，很明显这里会自动拼装port
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.kylin.common.util;
import java.util.Arrays;
import javax.annotation.Nullable;
import org.apache.commons.lang3.StringUtils;
import org.apache.hadoop.conf.Configuration;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import com.google.common.base.Function;
import com.google.common.collect.Iterables;
/**
* Use reflection to get zookeeper connect string from HBase configuration.
*/
public class ZooKeeperUtil {
private static final Logger logger = LoggerFactory.getLogger(ZooKeeperUtil.class);
private ZooKeeperUtil() {
throw new IllegalStateException("Class ZooKeeperUtil is an utility class !");
}
public static String getZKConnectStringFromHBase() {
Configuration hconf = null;
try {
Class hbaseConnClz = ClassUtil.forName("org.apache.kylin.storage.hbase.HBaseConnection", Object.class);
hconf = (Configuration) hbaseConnClz.getMethod("getCurrentHBaseConfiguration").invoke(null);
} catch (Throwable ex) {
logger.warn("Failed to get zookeeper connect string from HBase configuration", ex);
return null;
}
final String serverList = hconf.get("hbase.zookeeper.quorum");
final String port = hconf.get("hbase.zookeeper.property.clientPort");
return StringUtils.join(Iterables.transform(Arrays.asList(serverList.split(",")), new Function() {
@Nullable
@Override
public String apply(String input) {
return input + ":" + port;
}
}), ",");
}
}
然后翻看HBase的配置文件，之前把ip和port配置在了一起就导致重复拼接
hbase.zookeeper.quorum
hdp6:2181,hdp7:2181,hdp8:2181
所以，翻看源码得知需要拆成两个配置，如下：
hbase.zookeeper.quorum
hdp6,hdp7,hdp8
hbase.zookeeper.property.clientPort
2181
然后再次启动kylin
bin/kylin.sh start发现报错，如下：
2019-10-11 22:25:31,451 INFO [main-SendThread(hdp7:2181)] zookeeper.ClientCnxn:852 : Socket connection established to hdp7/192.168.1.107:2181, initiating session
2019-10-11 22:25:31,452 DEBUG [main] util.ZookeeperDistributedLock:147 : 37487@hdp4 trying to lock /kylin/kylin_metadata/create_htable/kylin_metadata/lock
2019-10-11 22:25:31,500 INFO [main-SendThread(hdp7:2181)] zookeeper.ClientCnxn:1235 : Session establishment complete on server hdp7/192.168.1.107:2181, sessionid = 0x26db9f17207000a, negotiated timeout = 40000
2019-10-11 22:25:31,516 INFO [main-EventThread] state.ConnectionStateManager:228 : State change: CONNECTED
Exception in thread "main" java.lang.IllegalArgumentException: Failed to find metadata store by url: kylin_metadata@hbase
at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:99)
at org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:111)
at org.apache.kylin.rest.service.AclTableMigrationTool.checkIfNeedMigrate(AclTableMigrationTool.java:99)
at org.apache.kylin.tool.AclTableMigrationCLI.main(AclTableMigrationCLI.java:43)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:92)
... 3 more
Caused by: java.lang.IllegalStateException: Error while peeking at /kylin/kylin_metadata/create_htable/kylin_metadata/lock
at org.apache.kylin.storage.hbase.util.ZookeeperDistributedLock.peekLock(ZookeeperDistributedLock.java:209)
at org.apache.kylin.storage.hbase.util.ZookeeperDistributedLock.isLockedByMe(ZookeeperDistributedLock.java:220)
at org.apache.kylin.storage.hbase.HBaseConnection.createHTableIfNeeded(HBaseConnection.java:352)
at org.apache.kylin.storage.hbase.HBaseResourceStore.createHTableIfNeeded(HBaseResourceStore.java:114)
at org.apache.kylin.storage.hbase.HBaseResourceStore.(HBaseResourceStore.java:88)
... 8 more
Caused by: org.apache.phoenix.shaded.org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /kylin/kylin_metadata/create_htable/kylin_metadata/lock
at org.apache.phoenix.shaded.org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
at org.apache.phoenix.shaded.org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.phoenix.shaded.org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
at org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:302)
at org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:291)
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
at org.apache.kylin.storage.hbase.util.ZookeeperDistributedLock.peekLock(ZookeeperDistributedLock.java:204)
... 12 more
2019-10-11 22:25:32,556 INFO [close-hbase-conn] hbase.HBaseConnection:136 : Closing HBase connections...
2019-10-11 22:25:32,557 INFO [close-hbase-conn] client.ConnectionManager$HConnectionImplementation:1710 : Closing zookeeper sessionid=0x26db9f172070009
2019-10-11 22:25:32,586 INFO [Thread-6] zookeeper.ZooKeeper:684 : Session: 0x26db9f17207000a closed
2019-10-11 22:25:32,587 INFO [main-EventThread] zookeeper.ClientCnxn:512 : EventThread shut down
2019-10-11 22:25:32,604 INFO [close-hbase-conn] zookeeper.ZooKeeper:684 : Session: 0x26db9f172070009 closed
2019-10-11 22:25:32,604 INFO [main-EventThread] zookeeper.ClientCnxn:512 : EventThread shut down
ERROR: Unknown error. Please check full log.
启动成功
A new Kylin instance is started by hadoop. To stop it, run 'kylin.sh stop'
Check the log at /home/hadoop/kylin/logs/kylin.log
Web UI is at http://hadoop:7070/kylin
界面
用默认的用户名和密码 ADMIN/KYLIN 登陆 Kylin 网站
创建项目
刷新数据源
构建模型
1.Model Info
2.Data Model
3.Dimensions
4.Measures
5.Settings
构建CUBE
1.Cube Info
2.Dimensions
3.Measures
4.Refresh Setting
5.Advanced Setting
6.Configuration Overwrites
7.Overview
Build
Monitor
