kylin2.1.0_CDH5.9.3实现streaming CUBE
cloudera manage官方文档说明了CDH集成kafka的版本
Product Compatibility Matrix for CDK Powered By Apache Kafka
CDK Powered By Apache Kafka is currently distributed as a package and in a parcel that is independent of the CDH parcel. The parcel integrates with Cloudera Manager using a Custom Service Descriptor (CSD).
For the latest documentation, see Kafka Documentation.
| Product | Feature | Lowest Supported Cloudera Manager Version | Supported CDH Versions | Integrated into CDH |
|---|---|---|---|---|
| CDK 3.0.0 Powered By Apache Kafka | Cloudera Manager 5.13.x | CDH 5.13.x and higher | No | |
| CDK 2.2.x Powered By Apache Kafka | Cloudera Manager 5.9.x | CDH 5.9.x and higher if Sentry integration is required, CDH 5.4.x and higher otherwise | No | |
| CDK 2.1.x Powered By Apache Kafka | Sentry authorization | Cloudera Manager 5.9.x | CDH 5.9.x and higher if Sentry integration is required, CDH 5.4.x and higher otherwise | No |
| CDK 2.0.x Powered By Apache Kafka | Enhanced security | Cloudera Manager 5.5.3 | CDH 5.4.x and higher | No |
| CDK 1.4.x Powered By Apache Kafka | Distributed both as package and parcel | Cloudera Manager 5.2.x | CDH 5.4.x, 5.5.x, 5.6.x | No |
| CDK 1.3.x Powered By Apache Kafka | Includes Kafka Monitoring | Cloudera Manager 5.2.x | CDH 5.4.x, 5.5.x, 5.6.x | No |
| CDK 1.2.x Powered By Apache Kafka | Cloudera Manager 5.2.x | CDH 5.4.x, 5.5.x, 5.6.x | No |
在kylin2.1.0_CDH5.9.3的版本上安装集成了
CDK 3.0.0 。
修改kylin的配置文件 kylin_job_conf_inmem.xml
mapreduce.map.memory.mb
512
mapreduce.map.java.opts
-Xmx1000m -XX:OnOutOfMemoryError='kill -9 %p'
mapreduce.task.io.sort.mb
50
如果缺少响应依赖的jar包可以执行kylin安装目录bin下的环境文件即可解决。
按照官方文档一步一步可以实现kylin的流式建模。