Datasophon添加第三方组件--FLINKSTANDALONE
Datasophon添加第三方组件--FLINKSTANDALONE
- Datasophon简介
- 制作Flink Standalone安装包
- 编写配置文件
- 编写properties_value.ftl
- 重启datasophon-manager服务
- 页面安装
- 存在问题
Datasophon简介
DataSophon（点击访问官网）是致力于自动化监控、运维、管理大数据基础组件和节点的，帮助您快速构建起稳定，高效的大数据集群服务，具有极易部署、兼容开源生态、兼容复杂环境、方便运维管控的特点。
DataSophon支持非常方便的集成自己需要的组件，目前官方提供了Flink客户端的安装，但是有些时候没有Hadoop环境，运行Flink作业需要部署Flink Standalone,下面演示如何自定义添加FLINKSTANDALONE组件。DataSophon版本号1.1.1，Flink 1.16.2。
制作Flink Standalone安装包
参考官网提供的服务集成协议，
- 我们需要准备一个Flink的安装包，首先下载一个自己需要的版本的Flink，以flink-1.16.2为例，到官网下载flink-1.16.2-bin-scala_2.12.tgz并解压缩，
- 提供Jobmanager和Taskmanager服务的start、stop、status和restart的脚本，下面是我自己写的一个control-flink.sh脚本，需要将该脚本放入flink-1.16.2/bin/ 目录下,
SCRIPT_PATH="$( cd "$( dirname "$0" )" >/dev/null 2>&1 && pwd )"
cd "$SCRIPT_PATH/../"
usage() {
echo "Usage: bash control-flink.sh [jobmanager|taskmanager] [status|restart]"
exit 1
}
APP_NAME="$1"
case "$APP_NAME" in
"jobmanager")
program="StandaloneSessionClusterEntrypoint"
;;
"taskmanager")
program="TaskManagerRunner"
;;
*)
usage
exit 1
;;
esac
is_exist(){
pid=`jps | grep ${program} | awk '{print $1}'`
#如果不存在返回1，存在返回0
if [ -z "${pid}" ]; then
return 1
else
return 0
fi
}
#输出运行状态
status(){
is_exist
if [ $? -eq "0" ]; then
echo "${APP_NAME} is running. Pid is ${pid}"
else
echo "${APP_NAME} is NOT running."
exit 1
fi
}
#启动方法
start(){
is_exist
if [ $? -eq "0" ]; then
echo "${APP_NAME} is already running. pid=${pid} ."
else
bash bin/${APP_NAME}.sh start
echo "${APP_NAME} start success"
fi
}
#停止方法
stop(){
is_exist
if [ $? -eq "0" ]; then
bash bin/${APP_NAME}.sh stop
else
echo "${APP_NAME} is not running"
fi
}
#重启
restart(){
stop
start
}
#根据输入参数，选择执行对应方法，不输入则执行使用说明
case "$2" in
"start")
start
;;
"stop")
stop
;;
"status")
status
;;
"restart")
restart
;;
*)
usage
;;
esac
- 另外我把我常用的jar包flink-doris-connector-1.16-1.4.0.jar和flink-sql-connector-mysql-cdc-2.4.0.jar也放到了flink-1.16.2/lib/目录下。
- 打包
tar czf flink-1.16.2-with-cdc.tar.gz flink-1.16.2
- 将制作好的包上传到datasophon所在机器的/opt/datasophon/DDP/packages目录下，并将包的md5值写入flink-1.16.2-with-cdc.tar.gz.md5文件：
md5sum flink-1.16.2-with-cdc.tar.gz
echo 'a9e11924a9e0408d025f85f21fb577ea' > flink-1.16.2-with-cdc.tar.gz.md5
编写配置文件
在meta/DDP-1.0.0目录下创建FLINKSTANDALONE目录和service_ddl.json服务集成文件。（注意在创建文件夹的时候中间不能有‘-’，例如如果是FLINK-STANDALONE，worker在安装插件时工作目录会按‘-’进行拆分成Flink/STANDALONE，导致安装失败）。
{
"name": "FLINKSTANDALONE",
"label": "FlinkStandalone",
"description": "实时计算引擎Standalone",
"version": "1.16.2",
"sortNum": 23,
"dependencies":[],
"packageName": "flink-1.16.2-with-cdc.tar.gz",
"decompressPackageName": "flink-1.16.2",
"roles": [
{
"name": "JobManager",
"label": "JobManager",
"roleType": "master",
"cardinality": "1+",
"logFile": "log/flink-root-standalonesession-0-localhost.localdomain.log",
"jmxPort": 12356,
"startRunner": {
"timeout": "60",
"program": "bin/control-flink.sh",
"args": [
"jobmanager",
"start"
]
},
"stopRunner": {
"timeout": "600",
"program": "bin/control-flink.sh",
"args": [
"jobmanager",
"stop"
]
},
"statusRunner": {
"timeout": "60",
"program": "bin/control-flink.sh",
"args": [
"jobmanager",
"status"
]
},
"restartRunner": {
"timeout": "60",
"program": "bin/control-flink.sh",
"args": [
"jobmanager",
"restart"
]
},
"externalLink": {
"name": "flink-standalone",
"label": "flink-standalone",
"url": "http://${host}:8083/"
}
},
{
"name": "TaskManager",
"label": "TaskManager",
"roleType": "worker",
"cardinality": "1+",
"logFile": "log/flink-root-taskexecutor-0-localhost.localdomain.log",
"jmxPort": 12358,
"startRunner": {
"timeout": "60",
"program": "bin/control-flink.sh",
"args": [
"taskmanager",
"start"
]
},
"stopRunner": {
"timeout": "600",
"program": "bin/control-flink.sh",
"args": [
"taskmanager",
"stop"
]
},
"statusRunner": {
"timeout": "60",
"program": "bin/control-flink.sh",
"args": [
"taskmanager",
"status"
]
},
"restartRunner": {
"timeout": "60",
"program": "bin/control-flink.sh",
"args": [
"taskmanager",
"restart"
]
}
}
],
"configWriter": {
"generators": [
{
"filename": "flink-conf.yaml",
"configFormat": "properties3",
"outputDirectory": "conf",
"includeParams": [
"rest.port",
"rest.bind-address",
"jobmanager.bind-host",
"jobmanager.memory.process.size",
"taskmanager.bind-host",
"taskmanager.host",
"taskmanager.memory.process.size",
"taskmanager.numberOfTaskSlots",
"parallelism.default",
"jobmanager.execution.failover-strategy",
"rest.address",
"classloader.resolve-order",
"state.backend",
"state.checkpoints.dir: file",
"state.savepoints.dir: file",
"jobmanager.rpc.port",
"jobmanager.rpc.address",
"custom.flink.conf"
]
},
{
"filename": "masters",
"configFormat": "custom",
"outputDirectory": "conf",
"templateName": "properties_value.ftl",
"includeParams": [
"masters"
]
},
{
"filename": "workers",
"configFormat": "custom",
"outputDirectory": "conf",
"templateName": "properties_value.ftl",
"includeParams": [
"workers"
]
}
]
},
"parameters": [
{
"name": "high-availability",
"label": "高可用选择",
"description": "NONE 或者 zookeeper",
"configType": "input",
"required": true,
"type": "input",
"value": "",
"configurableInWizard": true,
"hidden": false,
"defaultValue": "NONE"
},
{
"name": "high-availability.zookeeper.quorum",
"label": "高可用zk地址",
"description": "",
"configType": "input",
"required": true,
"type": "input",
"value": "",
"configurableInWizard": true,
"hidden": false,
"defaultValue": "${zkUrls}"
},
{
"name": "rest.port",
"label": "flink web端口号",
"description": "",
"configType": "input",
"required": true,
"type": "input",
"value": "",
"configurableInWizard": true,
"hidden": false,
"defaultValue": "8083"
},
{
"name": "rest.bind-address",
"label": "flink web绑定地址",
"description": "",
"required": true,
"type": "input",
"value": "",
"configurableInWizard": true,
"hidden": false,
"defaultValue": "0.0.0.0"
},
{
"name": "jobmanager.bind-host",
"label": "jobmanager.bind-host",
"description": "",
"required": true,
"type": "input",
"value": "",
"configurableInWizard": true,
"hidden": false,
"defaultValue": "0.0.0.0"
},
{
"name": "jobmanager.memory.process.size",
"label": "jobmanager.memory.process.size",
"description": "",
"required": true,
"type": "input",
"value": "",
"configurableInWizard": true,
"hidden": false,
"defaultValue": "1600m"
},
{
"name": "taskmanager.bind-host",
"label": "taskmanager.bind-host",
"description": "",
"required": true,
"type": "input",
"value": "",
"configurableInWizard": true,
"hidden": false,
"defaultValue": "0.0.0.0"
},
{
"name": "taskmanager.host",
"label": "taskmanager.host",
"description": "",
"required": true,
"type": "input",
"value": "",
"configurableInWizard": true,
"hidden": false,
"defaultValue": "localhost"
},
{
"name": "taskmanager.memory.process.size",
"label": "taskmanager.memory.process.size",
"description": "",
"required": true,
"type": "input",
"value": "",
"configurableInWizard": true,
"hidden": false,
"defaultValue": "1728m"
},{
"name": "taskmanager.numberOfTaskSlots",
"label": "taskmanager.numberOfTaskSlots",
"description": "",
"required": true,
"type": "input",
"value": "",
"configurableInWizard": true,
"hidden": false,
"defaultValue": "4"
},{
"name": "parallelism.default",
"label": "parallelism.default",
"description": "",
"required": true,
"type": "input",
"value": "",
"configurableInWizard": true,
"hidden": false,
"defaultValue": "1"
},{
"name": "jobmanager.execution.failover-strategy",
"label": "jobmanager.execution.failover-strategy",
"description": "",
"required": true,
"type": "input",
"value": "",
"configurableInWizard": true,
"hidden": false,
"defaultValue": "region"
},{
"name": "rest.address",
"label": "rest.address",
"description": "",
"required": true,
"type": "input",
"value": "",
"configurableInWizard": true,
"hidden": false,
"defaultValue": "localhost"
},{
"name": "classloader.resolve-order",
"label": "classloader.resolve-order",
"description": "",
"required": true,
"type": "input",
"value": "",
"configurableInWizard": true,
"hidden": false,
"defaultValue": "parent-first"
},{
"name": "state.backend",
"label": "state.backend",
"description": "",
"required": true,
"type": "input",
"value": "",
"configurableInWizard": true,
"hidden": false,
"defaultValue": "rocksdb"
},{
"name": "state.checkpoints.dir",
"label": "state.checkpoints.dir",
"description": "",
"required": true,
"type": "input",
"value": "",
"configurableInWizard": true,
"hidden": false,
"defaultValue": "file:///opt/datasophon/flink-1.16.2/flink-checkpoints"
},{
"name": "state.savepoints.dir",
"label": "state.savepoints.dir",
"description": "",
"required": true,
"type": "input",
"value": "",
"configurableInWizard": true,
"hidden": false,
"defaultValue": "file:///opt/datasophon/flink-1.16.2/flink-savepoints"
},{
"name": "jobmanager.rpc.port",
"label": "jobmanager.rpc.port",
"description": "",
"required": true,
"type": "input",
"value": "",
"configurableInWizard": true,
"hidden": false,
"defaultValue": "6123"
},{
"name": "jobmanager.rpc.address",
"label": "jobmanager.rpc.address",
"description": "",
"required": true,
"type": "input",
"value": "",
"configurableInWizard": true,
"hidden": false,
"defaultValue": "localhost"
},
{
"name": "custom.flink.conf",
"label": "custom.flink.conf",
"description": "自定义配置flink-conf.yaml",
"configType": "custom",
"required": false,
"type": "multipleWithKey",
"value": [],
"configurableInWizard": true,
"hidden": false,
"defaultValue": ""
},
{
"name": "masters",
"label": "masters",
"description": "masters机器的IP:Port",
"configType": "custom",
"required": true,
"type": "multipleWithKey",
"value": [],
"configurableInWizard": true,
"hidden": false,
"defaultValue": ""
},
{
"name": "workers",
"label": "workers",
"description": "workers机器的IP",
"configType": "custom",
"required": true,
"type": "multipleWithKey",
"value": [],
"configurableInWizard": true,
"hidden": false,
"defaultValue": ""
}
]
}
服务器上的目录：
编写properties_value.ftl
由于flink conf文件夹下的masters和slaves，不是key,value格式，没有找到合适的生成方式，
需要编写properties_value.ftl，并放置到每个worker的/opt/datasophon/datasophon-worker/conf/templates/目录下，
<#list itemList as item>
${item.value}
#list>
然后重启datasophon-worker服务。
bin/datasophon-worker.sh restart worker
重启datasophon-manager服务
bin/datasophon-api.sh restart api
重启完成后会在数据库表t_ddh_frame_service和t_ddh_frame_service_role中生成对应的记录
t_ddh_frame_service表：
t_ddh_frame_service_role表：
如果对service.json文件修改的话，要想使其生效，需要将这俩表里对应的数据删除，然后重启
页面安装
到这儿的话不出意外打开页面是可以看到添加过后的组件，然后就可以愉快的安装了（当然大概率是会出一些意外的，哈哈，不过根据日志报错很容易排查）。
安装完成的效果，可以方便地在页面上管理实例：
存在问题
- 获取flink日志可能会获取不到，需要修改下service.json中 "logFile"对应的值，
- 这个目前只完成了部署，没有jmx指标监控。