CDH Parcel包制作流程详解
CDH Parcel包制作流程详解
- 背景介绍
CDH可以很方便的添加一些大数据相关服务，但这仅限于cloudera公司提供。若想将第三方服务（如公司自己开发的组件）添加到CDH集群（托管在CDH上），需要按照一定的规则流程制作相关程序包，最后发布到CDH上。
本文以开源项目dolphinscheduler的master服务打包成parcel包部署为例，详解cdh parcel包的制作详细流程。
- 相关规则介绍
一个完整的集成到CDH的parcel包应包含如下几个包：
● parcel：以“.parcel”结尾的gz格式的压缩文件。它必须为一个固定规则的文件名。
parcel包内包含了你的服务组件，同时包含一个重要的描述性文件parcel.json；这个文件记录了你的服务信息，如版本、所属用户、适用的CDH平台版本等parcel必须包置于/opt/cloudera/parcel-repo/目录下才可以被CDH发布程序时识别到。
● parcel命名规则必须如下：
文件名称格式为三段，第一段是包名，第二段是版本号，第三段是运行平台。
例如：DSMASTER-1.3.3.dsmaster.p0.5-el7.parcel
包名：DSMASTER
版本号：1.3.3.dsmaster.p0.5
运行环境：el7
el6是代表centos6系统，centos7则用el7表示
● DSMASTER-1.3.3.jar文件是一个jar包，它记录了你的服务在CDH上的管理规则，如你的服务在CDH页面上显示的图标、依赖的服务、暴露的端口、启动规则等。csd的jar包必须置于/opt/cloudera/csd/目录才可以在添加集群服务时被识别到。
- 相关下载
制作完成的parcel包和csd jar包需要校验其合法性，cdh提供了jar方便我们对制作完成parcel包和csd jar进行校验。下载地址如下：
https://github.com/cloudera/cm_ext
- 制作步骤
1、制作parcel包
1.1parcel包目录结构
parcel包的目录结构由
parcel包目录结构由你的服务目录(lib/dsmaster)和一个meta目录组成。meta目录组成文件如下：
1.2dsmaster_env.sh
声明你的服务运行时的所需的一些变量环境，可以根据你的服务需要可以自行添加设置。
#!/bin/bash
export DSMASTER_HOME=$PARCELS_ROOT/$PARCEL_DIRNAME
1.3parcel.json
Parcel.json是一个重要性的描述文件，这个文件记录了你的服务信息，如版本、所属用户、适用的CDH平台版本等。
{
"schema_version": 1,
"name": "DSMASTER",
"version": "1.3.3.dsmaster.p0.5",
"setActiveSymlink": true,
"depends": "CDH (>= 5.0), CDH (<< 6.0)",
"provides": ["dsmaster"],
"scripts": {
"defines": "es_env.sh"
},
"packages": [{
"name": "dsmaster",
"version": "1.3.3.dsmaster.p0.5"
}],
"components": [{
"name": "dsmaster",
"version": "1.3.3",
"pkg_version": "1.3.3"
}],
"users": {
"dsmaster": {
"longname" : "DSMASTER",
"home" : "/var/lib/dsmaster",
"shell" : "/bin/bash",
"extra_groups": []
}
},
"groups": ["dsmaster"]
}
1.4打包parcel包
执行打包命令打成parcel包。
tar -zcvf DSMASTER-1.3.3.dsmaster.p0.5-el7.parcel DSMASTER-1.3.3.dsmaster.p0.5
java -jar /root/github/cloudera/cm_ext/validator/target/validator.jar -f DSMASTER-1.3.3.dsmaster.p0.5-el7.parcel生成.sha哈希校验文件。
vi DSMASTER-0.0.5.dsmaster.p0.5-el7.parcel.sha
sha1sum DSMASTER-0.0.5.dsmaster.p0.5-el7.parcel
- 制作csd jar
2.1目录结构
csd包的目录如下：
● descriptor：放置服务的规则描述文件service.sdl。
● images：放置服务的图标文件，png格式。不放图标文件，则CDH页面不显示图标。
● scripts ：放置服务的启停脚本，服务的启停通过该目录下的control.sh脚本来实现。
2.2service.sdl
{
"name": "DSMASTER",
"label": "DS Master",
"description": "master service",
"version": 1,
"runAs": {
"user": "root",
"group": "root"
},
"maxInstances": 1,
"icon": "images/icon.png",
"compatibility": {
"generation": 1,
"cdhVersion": {
"min": 5,
"max": 5
}
},
"parcel": {
"requiredTags": ["cdh"],
"optionalTags": ["dsmaster-plugin"]
},
"commands" : [
{
"name" : "ForceStop",
"label" : "Force stop",
"description" : "Force stop of DSm Nodes",
"roleCommand" : "ForceStopRole",
"roleName" : "DSMASTER_NODE",
"runMode" : "all"
}
],
"roles": [{
"name": "DSMASTER_NODE",
"label": "DSMASTER Node",
"pluralLabel": "DSMASTER Nodes",
"jvmBased": true,
"startRunner": {
"program": "scripts/control.sh",
"args": ["start"]
},
"stopRunner" : {"timeout" : "90000",
"runner" : {
"program" : "scripts/control.sh",
"args" : ["stop"]
}
},
"commands": [
{
"name": "ForceStopRole",
"label": "Force Stop Role",
"description": "Force stop of DSMASTER Node",
"expectedExitCodes": [0],
"requiredRoleState": "stopped",
"commandRunner": {
"program": "scripts/control.sh",
"args": ["stop"]
}
}
],
"configWriter": {
"generators": [{
"filename": "dsmaster-conf.properties",
"configFormat": "properties",
"includeParams": [
"dbhost",
"dhname",
"username",
"password",
"zkQuorum"
]
}]
}
}],
"parameters": [
{
"name": "dbhost",
"label": "dbhost",
"description": "dbhost",
"configName": "dbhost",
"required": "true",
"type": "string",
"configurableInWizard": true,
"default": "localhost"
},
{
"name": "dbname",
"label": "dbname",
"description": "dbname",
"configName": "dbname",
"required": "true",
"type": "string",
"configurableInWizard": true,
"default": "dolphinscheduler"
},
{
"name": "username",
"label": "username",
"description": "username",
"configName": "username",
"required": "true",
"type": "string",
"configurableInWizard": true,
"default": "root"
},
{
"name": "password",
"label": "password",
"description": "password",
"configName": "password",
"required": "true",
"type": "string",
"configurableInWizard": true,
"default": "*******"
},
{
"name": "zkQuorum",
"label": "zkQuorum",
"description": "zkQuorum",
"configName": "zkQuorum",
"required": "true",
"type": "string",
"configurableInWizard": true,
"default": "server12:2181,server13:2181,server14:2181"
}
]
}
2.3control.sh
#!/bin/sh
export ES_HOME=/opt/cloudera/parcels/DSMASTER/lib/dsmaster
PROGNAME=`basename "$0"`
warn() {
echo "${PROGNAME}: $*"
}
die() {
warn "$*"
exit 1
}
locateJava() {
echo
# export JAVA_HOME=/usr/java/latest
echo "Changing Java Home to: $JAVA_HOME"
export JAVA="$JAVA_HOME/bin/java"
echo "Changing Java to: $JAVA"
echo
}
config() {
echo
echo "Creating config"
}
init() {
echo "init"
source $CONF_DIR/dsmaster-conf.properties
echo "dbhost: $dbhost"
echo "dbname: $dbname"
echo "username: $username"
echo "password: $password"
echo "zkQuorum: $zkQuorum"
sed -i "s#spring.datasource.url.*#spring.datasource.url=jdbc:mysql://$dbhost/$dbname?characterEncoding=UTF-8\&allowMultiQueries=true#g" $ES_HOME/conf/datasource.properties
sed -i "s#spring.datasource.username.*#spring.datasource.username=$username#g" $ES_HOME/conf/datasource.properties
sed -i "s#spring.datasource.password.*#spring.datasource.password=$password#g" $ES_HOME/conf/datasource.properties
sed -i "s#zookeeper.quorum.*#zookeeper.quorum=$zkQuorum#g" $ES_HOME/conf/zookeeper.properties
}
start() {
echo "Running DSMASTER"
sh $ES_HOME/bin/dolphinscheduler-daemon.sh stop master-server
exec sh $ES_HOME/bin/dolphinscheduler-daemon-cdh.sh start master-server
}
stop() {
echo "stop DSMASTER"
exec sh $ES_HOME/bin/dolphinscheduler-daemon-cdh.sh stop master-server
}
init
case "$1" in
start)
start
;;
stop)
stop
;;
restart)
stop
start
;;
*)
echo "Usage DSMASTER {start|stop|restart}"
;;
esac2.4打包成csd jar
java -jar /root/github/cloudera/cm_ext/validator/target/validator.jar -s descriptor/service.sdl
jar -cvf DSMASTER-1.3.3.jar *- 发布到CDH
1、分发激活parcel包
将parcel包和parcel.sha拷贝到cm节点的/opt/cloudera/parcel-repo/目录下，将csd jar包拷贝到/opt/cloudera/csd目录下。
点击【主机】【Parcel】【检查新Parcel】可以识别出刚放入的parcel包。
点击【分配】【激活】
成功后执行命令重启cloudera-scm-server。
systemctl restart cloudera-scm-server
2、添加服务
首页添加服务。
选择需要添加的服务。
选择服务部署的主机。
更改服务所需的配置。
最后等待执行成功即可。
最后执行成功后在首页可以看到新添加的服务。
