DataSphereStudio 安装
azkaban编译安装
- 遇到的问题
javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
at sun.security.ssl.InputRecord.handleUnknownRecord(InputRecord.java:710)
at sun.security.ssl.InputRecord.read(InputRecord.java:527)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:708)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
原因：浏览器安全证书限制。添加信任即可。
解决办法:https://ip:8443
原因:没有找到活动的executors，需在MySQL数据库里设置端口为12321的executors表的active为1
解决:在开启exec后执行
curl -G "localhost:$(<./executor.port)/executor?action=activate" && echo
问题:javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
分析:如果用http://ip:端口访问不到web界面 得用https://ip:端口访问,但是因为证书没有经过验证,点击高级-->继续访问,这个错就是证书验证,不妨事
问题:azkaban.executor.ExecutorManagerException: azkaban.executor.ExecutorManagerException: java.io.IOException: executor became inactive before setting up the flow 9
分析:Azkaban Executor 启动后不会自动激活 通过翻看 Azkaban Executor Server 源码,发现 Executor 提供了 rest 接口,来手动激活 Executor.
为什么不在 Executor 启动时自动激活呢….官方 issue110 解释如下:
1. We don't update prior to web-server start.
2. Let me introduce how we do a regular Executor deployment. Both old webserver and old executor(3.a) are up initially, and old executor is running jobs (we can not interrupt it). Executor Deployment starts:
3. install new Executor (3.b) on our box without activating.
Run job tests, which targets 3.b and make sure 3.b is working.
4. Activate 3.b, and deactivate 3.a. Even though 3.a is deactivated, jobs still keep runnig there.
Deployment completes.
We would better run tests against new executor before activating it. Is it a bit clear now?
解决:在虚机任意位置执行
curl http://${executorHost}:${executorPort}/executor?action=activate
参考:https://www.jianshu.com/p/5a3f31698042
https://www.cnblogs.com/remainsu/p/azkaban-3731-shi-yong-jian-jie-multiple-executor.html
- dss在打开工作调度组件时出现"页面已重置",谷歌浏览器F12查看有浏览地址出错
分析: 打开页面时浏览器的地址出错
解决: 1.确认dss的visualis-server已启动,未出错
2.确认dss的conf/config.sh中的azkaban的ip和端口没问题
3.确认dss的默认数据库中表dss_application中的schedulis的浏览地址为你的azkaban能登录的地址,如果不是修改,并重启visualis-server
- visualis可视化服务访问报错
- 确保visualis-server已经启动。
- 检查visualis-server安装目录下的application.yml配置，确保以下配置准确无误
url: http://0.0.0.0:0000/dss/visualis 此处url中的IP和端口必须保持与DSS前端Nginx访问的IP地址和端口一致
access:
address: 0.0.0.0 #frontend address，此处保持与DSS前端Nginx访问IP地址一致
port: 0000#frontend port，此处保持与DSS前端Nginx访问端口一致
3.确保数据库表dss_application中 visualis记录行，访问地址与DSS前端Nginx访问IP地址和端口一致。
4.访问visualis出现404错误，确保Nginx配置文件中关于visualis的访问路径配置正确。
location /dss/visualis {
root /data/DSSFront; 示例visualis前端静态文件目录
autoindex on;
}
location / {
root /data/DSSFront/dist; 示例DSS前端静态文件目录
index index.html index.html;
}
- links出现spark或者hive或者python的ENGINEMANAGER和ENTRANCE未出现在eureka中?
可以通过配置对应服务的conf目录下的linkis.properties和linkis-engine.properties(EM才有) hadoop.config.dir参数
hadoop.config.dir=/appcom/config/hadoop-config
spark.config.dir=/appcom/config/spark-config
hive.config.dir=/appcom/config/hive-config
- dss安装好之后scriptis无法显示hive的元数据库数据
- [外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-NsWyqvEE-1584937070882)(https://s2.ax1x.com/2020/01/03/la8fpD.png)]
分析:出现重启失败,是自己记录的pid和进程不相同导致
解决:重启linkis-metadata,在重启以前先kill掉原来的9008的进程
分析:连接数据库时出现url空串错误,无法解析正确的
解决:${LINKIS_HOME}/linkis-metadata/conf/linkis.properties下补全
hive.meta.url=
hive.meta.user=
hive.meta.password=
安装dss
官网链接
https://github.com/WeBankFinTech/DataSphereStudio/blob/master/docs/zh_CN/ch2/DSS%E5%BF%AB%E9%80%9F%E5%AE%89%E8%A3%85%E4%BD%BF%E7%94%A8%E6%96%87%E6%A1%A3.md#%E4%B8%89%E6%A0%87%E5%87%86%E7%89%88dss%E7%8E%AF%E5%A2%83%E9%85%8D%E7%BD%AE%E5%87%86%E5%A4%87
安装时常见的问题
https://github.com/WeBankFinTech/DataSphereStudio/blob/master/docs/zh_CN/ch1/DSS%E5%AE%89%E8%A3%85%E5%B8%B8%E8%A7%81%E9%97%AE%E9%A2%98%E5%88%97%E8%A1%A8.md
未完待续