创建需要的用户和分组
- useradd -r -m -s /bin/bash nifi && useradd -r -m -s /bin/bash kylo && useradd -r -m -s /bin/bash activemq
- groupadd -f kylo && groupadd -f nifi && groupadd -f activemq
创建kylo安装目录
- mkdir /opt/kylo
将安装包解压到安装目录
- tar xvf /home/wcm/sources/kylo-0.10.0.tar -C /opt/kylo
安装前执行预处理脚本, 这里以root用户安装
- /opt/kylo/setup/install/post-install.sh /opt/kylo root root
PS: 三个参数依次为: kylo安装目录, 安装程序的用户, 安装程序的用户组
执行安装向导脚本
- /opt/kylo/setup/setup-wizard.sh
PS1: 安装kylo需要依赖的组件: mysql, elasticsearch, activemq, nifi, kylo
前两者已经安装好, 故只安装后面
PS2 :dladmin密码为kylo-ui登录密码
[root@dpnode05 opt]# /opt/kylo/setup/setup-wizard.sh
The working directory is /opt/kylo/setup
Welcome to the Kylo setup wizard. Lets get started !!!
Please enter Y/y or N/n to the following questions:
Enter the kylo home folder location, hit Enter for '/opt/kylo':
Enter the kylo linux user, hit Enter for 'kylo':
Enter the kylo linux group, hit Enter for 'users': kylo
Would you like to install the database scripts in a database instance? Please enter y/n: y
Would you like Kylo to manage installing and upgrading the database automatically? Please enter y/n: n
OK. Disabling Liquibase in application.properties. Please see the Kylo documentation to see how to generate the database scripts
Press any key to continue: Which database (Enter the number)?
1) MySQL
2) PostgresSQL
3) SQL Server
> 1
Please enter the database hostname or IP, hit Enter for 'localhost'
>
Please enter the database ADMIN username
> root
Please enter the database ADMIN password
> Creating MySQL database 'kylo'
Warning: Using a password on the command line interface can be insecure.
Please enter the password for the dladmin user
>
Please re-enter the password for the dladmin user
>
Please choose an option to configure Java for Kylo, ActiveMQ, and NiFi
1) I already have Java 8 or higher installed as the system Java and want to use that
2) Install Java 8 in the /opt/java folder for me and use that one
3) I have Java 8 or higher installed in another location already. I will provide the location
4) Java is already setup. No changes necessary
> 1
Would you like me to install a local elasticsearch instance? Please enter y/n: n
Would you like me to install a local activemq instance? Please enter y/n: y
Enter the Activemq home folder location, hit Enter for '/opt/activemq':
Enter the user Activemq should run as, hit Enter for 'activemq':
Enter the linux group Activemq should run as, hit Enter for 'activemq':
Would you like me to install a local nifi instance? Please enter y/n: y
Enter Nifi version you wish to install, hit Enter for '1.6.0':
Enter the NiFi home folder location, hit Enter for '/opt/nifi':
Enter the user NiFi should run as, hit Enter for 'nifi':
Enter the linux group NiFi should run as, hit Enter for 'nifi':
Using system Java 8 and remove the JAVA_HOME variable from kylo-ui and kylo-services
Removing custom JAVA_HOME from kylo-ui and kylo-services
installing ActiveMQ
The active home folder is /opt/activemq
Create the /opt/activemq directory
Download activemq and install
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 55.8M 0 192k 0 0 76476 0 0:12:45 0:00:02 0:12:43 125k
修改配置文件
- vim /opt/kylo/kylo-services/conf/application.properties
spring.datasource.url=jdbc:mysql://dpnode05:3306/kylo
spring.datasource.username=root
spring.datasource.password=123456
spring.datasource.driverClassName=com.mysql.jdbc.Driver
复制Mysql驱动
- cp /home/wcm/mysql-connector-java-8.0.15.jar /opt/kylo/kylo-services/lib/
生成sql脚本
- cd /opt/kylo/setup/sql/mysql
- ./setup-mysql.sh dpnode05 root 123456
- cd /opt/kylo/setup/sql
- ./generate-update-ql.sh
-rw-r--r-- 1 root root 0 Feb 22 15:04 kylo-db-update-script.sql
-rw-r--r-- 1 root root 0 Feb 22 15:04 kylo-db-update-script.sql.bac
修改脚本, 解决时区异常问题
vim /opt/kylo/setup/sql/kylo-db-update-script.sql
将第90行中的 1970-01-01 00:00:01 加8小时, 改为 1970-01-01 08:00:01
CREATE TABLE kylo.FEED (id BINARY(16) NOT NULL, name VARCHAR(100) NOT NULL, description VARCHAR(255) NULL, FEED_TYPE VARCHAR(45) NULL, created_time timestamp DEFAULT NOW() NOT NULL, modified_time timestamp DEFAULT '1970-01-01 08:00:01' NOT NULL);
进入Mysql客户端
- 创建数据库: create database if not exists kylo character set utf8 collate utf8_general_ci;
- 使用数据库: use kylo;
- 设置环境变量: SET GLOBAL log_bin_trust_function_creators = 1;
PS: 该环境变量解决sql异常:ERROR 1418 (HY000): This function has none of DETERMINISTIC, NO SQL, or READS SQL DATA in its declaration and binary logging is enabled
执行sql脚本: souce kylo-db-update-script.sql
安装elasticsearch, 启动并执行kylo索引脚本
/opt/kylo/bin/create-kylo-indexes-es.sh
启动依赖程序(elasticesearch, nifi, activemq)
-
检查是否启动
- ps -ef | grep elasticesearch
- ps -ef | grep activemq
- ps -ef | grep nifi
-
没启动则启动, 启动方式:
- nohup $ELASTICHSEARCH_HOME/elasticsearch &
- service activemq start
- service nifi start
启动kylo
- kylo-service start
启动后个服务地址:
- activemq: http://dpnode05:8161/
- nifi: http://dpnode05:8079/nifi/
- kylo: http://dpnode05:8400
PS: kylo-services启动后仍有报错, 尚未解决, 暂时找到问题原因
java.security.AccessControlException: Not authorized to perform the action: Access Templates
at com.thinkbiganalytics.metadata.modeshape.security.action.JcrAllowedActions.checkPermission(JcrAllowedActions.java:226)
at com.thinkbiganalytics.metadata.modeshape.security.DefaultAccessController.lambda$null$0(DefaultAccessController.java:75)
at java.util.Optional.map(Optional.java:215)
at com.thinkbiganalytics.metadata.modeshape.security.DefaultAccessController.lambda$checkPermission$2(DefaultAccessController.java:74)
at com.thinkbiganalytics.metadata.modeshape.JcrMetadataAccess.read(JcrMetadataAccess.java:422)
at com.thinkbiganalytics.metadata.modeshape.JcrMetadataAccess.read(JcrMetadataAccess.java:369)
at com.thinkbiganalytics.metadata.persistence.AggregateMetadataAccess.read(AggregateMetadataAccess.java:86)
at com.thinkbiganalytics.metadata.modeshape.security.DefaultAccessController.checkPermission(DefaultAccessController.java:72)
at com.thinkbiganalytics.metadata.modeshape.security.DefaultAccessController.checkPermission(DefaultAccessController.java:62)
at com.thinkbiganalytics.repository.filesystem.RepositoryMonitor.lambda$getAllRegisteredTemplatesAsMap$3(RepositoryMonitor.java:138)
at com.thinkbiganalytics.metadata.persistence.AggregateMetadataAccess.lambda$null$3(AggregateMetadataAccess.java:124)
at com.thinkbiganalytics.metadata.config.OperationalMetadataTransactionTemplateMetadataAccess$2.doInTransaction(OperationalMetadataTransactionTemplateMetadataAccess.java:187)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:133)
at com.thinkbiganalytics.metadata.config.OperationalMetadataTransactionTemplateMetadataAccess.read(OperationalMetadataTransactionTemplateMetadataAccess.java:144)
at com.thinkbiganalytics.metadata.config.OperationalMetadataTransactionTemplateMetadataAccess.read(OperationalMetadataTransactionTemplateMetadataAccess.java:118)
at com.thinkbiganalytics.metadata.persistence.AggregateMetadataAccess.lambda$wrap$5(AggregateMetadataAccess.java:124)
at com.thinkbiganalytics.metadata.modeshape.JcrMetadataAccess.execute(JcrMetadataAccess.java:444)
at com.thinkbiganalytics.metadata.modeshape.JcrMetadataAccess.read(JcrMetadataAccess.java:401)
at com.thinkbiganalytics.metadata.modeshape.JcrMetadataAccess.read(JcrMetadataAccess.java:369)
at com.thinkbiganalytics.metadata.persistence.AggregateMetadataAccess.read(AggregateMetadataAccess.java:86)
at com.thinkbiganalytics.repository.filesystem.RepositoryMonitor.getAllRegisteredTemplatesAsMap(RepositoryMonitor.java:137)
at com.thinkbiganalytics.repository.filesystem.RepositoryMonitor.watchRepositories(RepositoryMonitor.java:99)
at com.thinkbiganalytics.repository.filesystem.FilesystemRepositoryService.monitorRepositories(FilesystemRepositoryService.java:141)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2019-02-22 15:47:58 WARN localhost-startStop-2:WebappClassLoaderBase:180 - The web application [ROOT] appears to have started a thread named [modeshape-stats-2-thread-1] but has failed to stop it. This
is very likely to create a memory leak. Stack trace of thread:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)