seatunnel入门案例，集群模式
目录
安装部署
解压
环境变量
安装plugin
添加资源jar包
SEATUNNEL 配置文件
env：环境设置
source：数据源设置
sink：数据去向设置
transform: 数据转换设置
运行方式
seatunnel 引擎(zeta)
本地模式
集群模式
安装部署
解压
tar -zxvf apache-SeaTunnel-incubating-2.3.1-bin.tar.gz环境变量
vi /etc/profile
export SEATUNNEL_HOME=/data/soft/seatunnel-2.3.1
export PATH=$PATH:${SEATUNNEL_HOME}/bin安装plugin
# 备份原有的组件配置
cp ${SEATUNNEL_HOME}/config/plugin_config ${SEATUNNEL_HOME}/config/plugin_config_bak
# 创建一个新的
vi ${SEATUNNEL_HOME}/config/plugin_config
# 只安装这三种插件，因为他的插件太多了，下载很慢，用哪个下哪个
--connectors-v2--
connector-jdbc
connector-kafka
connector-console
--end--
# 运行安装命令
sh ${SEATUNNEL_HOME}/bin/install-plugin.sh 2.3.1添加资源jar包
cd ${SEATUNNEL_HOME}/lib
# 添加以下依赖jar，连接mysql需要使用
mysql-connector-j-8.0.31.jarSEATUNNEL 配置文件
env：环境设置
source：数据源设置
sink：数据去向设置
transform: 数据转换设置
env {
execution.parallelism = 1
}
source {
Kafka {
topic = "seatunnel_topic"
format = text
field_delimiter = "#"
schema = {
fields {
name = "string"
age = "int"
}
}
consumer.group = "yuanqu-group11"
bootstrap.servers = "hdp01:6667"
start_mode = "latest"
kafka.config = {
security.protocol = "SASL_PLAINTEXT"
sasl.mechanism = "PLAIN"
sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=\"XXX\" password=\"XX@XXXX\";"
}
}
}
transform {
}
sink {
jdbc {
url = "jdbc:mysql://172.16.10.143:3306/test_db"
driver = "com.mysql.cj.jdbc.Driver"
user = "root"
password = "XXXX"
query = "insert into seatunnel_test(name,age) values(?,?)"
}
}
运行方式
seatunnel 包括三种执行引擎
seatunnel 引擎(zeta)
本地模式
# -e local 指定本地执行
${SEATUNNEL_HOME}/bin/seatunnel.sh --config test.conf -e local集群模式
启动集群
# 启动集群,我这是一台机器伪集群不需要任何配置，如果配置看https://seatunnel.apache.org/docs/2.3.1/seatunnel-engine/deployment
nohup ${SEATUNNEL_HOME}/bin/seatunnel-cluster.sh & 1>/dev/null 2>&1
启动集群任务
# -e local 指定本地执行
nohup ${SEATUNNEL_HOME}/bin/seatunnel.sh --config test.conf & 1>/dev/null 2>&1停止任务
启动任务之后会有一个jobid，如果直接kill 进程是不管用的，seatunnel任务依然会执行，需要使用 -can jobid 结束任务
#我们需要使用 -can, --cancel-job Cancel job by JobId
${SEATUNNEL_HOME}/bin/seatunnel.sh -can 720924469203304449 Authorized only. All activity will be monitored and reported
Last login: Mon Jun 19 14:27:58 2023 from 10.10.10.14
[manager@hdpcli01 ~]$ ${SEATUNNEL_HOME}/bin/seatunnel.sh --list
JAVA_OPTS: -Dhazelcast.client.config=/data/soft/seatunnel-2.3.1/config/hazelcast-client.yaml -Dseatunnel.config=/data/soft/seatunnel-2.3.1/config/seatunnel.yaml -Dhazelcast.config=/data/soft/seatunnel-2.3.1/config/hazelcast.yaml -Dlog4j2.configurationFile=/data/soft/seatunnel-2.3.1/config/log4j2_client.properties -Dseatunnel.logs.path=/data/soft/seatunnel-2.3.1/logs -Dseatunnel.logs.file_name=seatunnel-starter-client -Xms128m -Xmx512m -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/tmp/seatunnel-1757821675590923600 -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=32 -XX:GCLogFileSize=64m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/data/seatunnel/
Jun 19, 2023 4:07:03 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Loading configuration '/data/soft/seatunnel-2.3.1/config/seatunnel.yaml' from System property 'seatunnel.config'
Jun 19, 2023 4:07:03 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Using configuration file at /data/soft/seatunnel-2.3.1/config/seatunnel.yaml
Jun 19, 2023 4:07:03 PM org.apache.seatunnel.engine.common.config.SeaTunnelConfig
INFO: seatunnel.home is /home/manager
Jun 19, 2023 4:07:03 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Loading configuration '/data/soft/seatunnel-2.3.1/config/hazelcast.yaml' from System property 'hazelcast.config'
Jun 19, 2023 4:07:03 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Using configuration file at /data/soft/seatunnel-2.3.1/config/hazelcast.yaml
Jun 19, 2023 4:07:03 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Loading configuration '/data/soft/seatunnel-2.3.1/config/hazelcast-client.yaml' from System property 'hazelcast.client.config'
Jun 19, 2023 4:07:03 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Using configuration file at /data/soft/seatunnel-2.3.1/config/hazelcast-client.yaml
2023-06-19 16:07:04,474 INFO com.hazelcast.client.impl.spi.ClientInvocationService - hz.client_1 [seatunnel] [5.1] Running with 2 response threads, dynamic=true
2023-06-19 16:07:04,562 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTING
2023-06-19 16:07:04,563 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTED
2023-06-19 16:07:04,597 INFO com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel] [5.1] Trying to connect to cluster: seatunnel
2023-06-19 16:07:04,601 INFO com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel] [5.1] Trying to connect to [localhost]:5801
2023-06-19 16:07:04,640 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_CONNECTED
2023-06-19 16:07:04,640 INFO com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel] [5.1] Authenticated with server [localhost]:5801:da118827-3f65-4fe6-b769-1ca26c71d693, server version: 5.1, local address: /127.0.0.1:57914
2023-06-19 16:07:04,642 INFO com.hazelcast.internal.diagnostics.Diagnostics - hz.client_1 [seatunnel] [5.1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
2023-06-19 16:07:04,662 INFO com.hazelcast.client.impl.spi.ClientClusterService - hz.client_1 [seatunnel] [5.1]
Members [1] {
Member [localhost]:5801 - da118827-3f65-4fe6-b769-1ca26c71d693
}
2023-06-19 16:07:04,694 INFO com.hazelcast.client.impl.statistics.ClientStatisticsService - Client statistics is enabled with period 5 seconds.
Job ID Job Name Job Status Submit Time Finished Time
------------------ --------- ---------- ----------------------- -----------------------
723074781175349249 SeaTunnel CANCELED 2023-06-19 15:21:02.191 2023-06-19 15:21:22.951
723074554087342081 SeaTunnel CANCELED 2023-06-19 15:20:08.102 2023-06-19 15:20:28.864
723073982751834113 SeaTunnel CANCELED 2023-06-19 15:17:51.834 2023-06-19 15:18:03.142
723072313091686401 SeaTunnel CANCELED 2023-06-19 15:11:13.734 2023-06-19 15:11:24.541
723071390659379201 SeaTunnel CANCELED 2023-06-19 15:07:33.821 2023-06-19 15:09:55.188
723068648918351873 SeaTunnel CANCELED 2023-06-19 14:56:40.13 2023-06-19 14:57:11.011
723067564967591937 SeaTunnel CANCELED 2023-06-19 14:52:21.739 2023-06-19 14:52:34.226
2023-06-19 16:07:04,817 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTTING_DOWN
2023-06-19 16:07:04,822 INFO com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel] [5.1] Removed connection to endpoint: [localhost]:5801:da118827-3f65-4fe6-b769-1ca26c71d693, connection: ClientConnection{alive=false, connectionId=1, channel=NioChannel{/127.0.0.1:57914->localhost/127.0.0.1:5801}, remoteAddress=[localhost]:5801, lastReadTime=2023-06-19 16:07:04.741, lastWriteTime=2023-06-19 16:07:04.740, closedTime=2023-06-19 16:07:04.818, connected server version=5.1}
2023-06-19 16:07:04,822 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_DISCONNECTED
2023-06-19 16:07:04,825 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTDOWN
2023-06-19 16:07:04,826 INFO org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed SeaTunnel client......
[manager@hdpcli01 ~]$