flink-cdc 使用
接上一篇文章 用上cdc
基于flink-scala 1.14
当前cdc最新版本 2.2.1
<dependency>
<groupId>com.ververicagroupId>
<artifactId>flink-connector-mysql-cdcartifactId>
<version>2.2.1version>
dependency>
看官方文档是和flink 13.5搭配使用，现在使用flink 1.14会有一些问题 原因是flink 1.14引用的guava版本和cdc包中的guava版本不一致，但是artifactId 相同，所以后一个会覆盖前一个导致flink找不到guava报错。
因此，要做一些改造
显示引入 cdc用的guava版本
<dependency>
<groupId>org.apache.flinkgroupId>
<artifactId>flink-shaded-guavaartifactId>
<version>18.0-13.0version>
dependency>
用maven修改artifactId名，在本地重新安装该版本，然后引入
mvn install:install-file
-Dfile=flink-shaded-guava-18.0-13.0.jar
-DgroupId=org.apache.flink
-DartifactId=flink-shaded-guava18
-Dversion=18.0-13.0
-Dpackaging=jar
<dependency>
<groupId>org.apache.flinkgroupId>
<artifactId>flink-shaded-guava18artifactId>
<version>18.0-13.0version>
dependency>
配置完坐标后，在项目lib中删除cdc引入的guava jar包 并手动引入flink 1.14依赖的guava jar包（包路径名与cdc依赖的不一样 可以引入） flink-shaded-guava-30.1.1-jre-14.0
前置工作完成
完整maven坐标
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0modelVersion>
<parent>
<groupId>org.springframework.bootgroupId>
<artifactId>spring-boot-starter-parentartifactId>
<version>2.7.2version>
<relativePath/>
parent>
<groupId>com.examplegroupId>
<artifactId>demoartifactId>
<version>0.0.1-SNAPSHOTversion>
<name>demoname>
<description>Demo project for Spring Bootdescription>
<properties>
<java.version>1.8java.version>
properties>
<dependencies>
<dependency>
<groupId>org.springframework.bootgroupId>
<artifactId>spring-boot-starter-webartifactId>
dependency>
<dependency>
<groupId>org.springframework.bootgroupId>
<artifactId>spring-boot-starter-testartifactId>
<scope>testscope>
dependency>
<dependency>
<groupId>mysqlgroupId>
<artifactId>mysql-connector-javaartifactId>
<version>8.0.21version>
dependency>
<dependency>
<groupId>org.scala-langgroupId>
<artifactId>scala-libraryartifactId>
<version>2.12.7version>
dependency>
<dependency>
<groupId>org.scala-langgroupId>
<artifactId>scala-compilerartifactId>
<version>2.12.7version>
dependency>
<dependency>
<groupId>org.apache.flinkgroupId>
<artifactId>flink-table-api-scala-bridge_2.12artifactId>
<version>1.14.5version>
dependency>
<dependency>
<groupId>org.apache.flinkgroupId>
<artifactId>flink-table-planner_2.12artifactId>
<version>1.14.5version>
dependency>
<dependency>
<groupId>org.apache.flinkgroupId>
<artifactId>flink-connector-jdbc_2.12artifactId>
<version>1.14.5version>
dependency>
<dependency>
<groupId>org.apache.flinkgroupId>
<artifactId>flink-connector-kafka_2.12artifactId>
<version>1.14.5version>
dependency>
<dependency>
<groupId>org.apache.flinkgroupId>
<artifactId>flink-csvartifactId>
<version>1.14.5version>
dependency>
<dependency>
<groupId>org.apache.flinkgroupId>
<artifactId>flink-table-commonartifactId>
<version>1.14.5version>
dependency>
<dependency>
<groupId>org.apache.flinkgroupId>
<artifactId>flink-clients_2.12artifactId>
<version>1.14.5version>
dependency>
<dependency>
<groupId>org.springframework.kafkagroupId>
<artifactId>spring-kafkaartifactId>
dependency>
<dependency>
<groupId>com.ververicagroupId>
<artifactId>flink-connector-mysql-cdcartifactId>
<version>2.2.1version>
dependency>
<dependency>
<groupId>org.apache.flinkgroupId>
<artifactId>flink-shaded-guava18artifactId>
<version>18.0-13.0version>
dependency>
<dependency>
<groupId>org.apache.flinkgroupId>
<artifactId>flink-runtime-web_2.12artifactId>
<version>1.14.5version>
dependency>
dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.bootgroupId>
<artifactId>spring-boot-maven-pluginartifactId>
plugin>
<plugin>
<groupId>org.scala-toolsgroupId>
<artifactId>maven-scala-pluginartifactId>
<version>2.15.2version>
<executions>
<execution>
<id>scala-compile-firstid>
<goals>
<goal>compilegoal>
goals>
<configuration>
<includes>
<include>**/*.scalainclude>
includes>
configuration>
execution>
executions>
plugin>
plugins>
build>
project>
使用cdc
package demo
import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment
import org.apache.flink.table.api.bridge.scala.StreamTableEnvironment
object Test {
private val host_url = "127.0.0.1"
private val port = 3306
private val db_name = "test"
private val table_input = "salary_table"
private val table_output = "salary_count"
private val user = "root"
private val password = "root"
def flinkcdcTest:Unit = {
val env = StreamExecutionEnvironment.createLocalEnvironmentWithWebUI()
env.setParallelism(1)
val tableEnv = StreamTableEnvironment.create(env)
tableEnv.executeSql(
s"""
|create table mysqlInput (
| name string,
| salary double,
| PRIMARY KEY (`name`) NOT ENFORCED
| ) with (
| 'connector' = 'mysql-cdc',
| 'hostname' = '$host_url',
| 'port' = '$port',
| 'username' = '$user',
| 'password' = '$password',
| 'database-name' = '$db_name',
| 'table-name' = '$table_input'
| )
""".stripMargin)
tableEnv.executeSql(
s"""
|create table mysqlOutput (
| salary double,
| cnt bigint not null,
| PRIMARY KEY (`salary`) NOT ENFORCED
| ) with (
| 'connector' = 'jdbc',
| 'url' = 'jdbc:mysql://$host_url:$port/$db_name',
| 'table-name' = '$table_output',
| 'username' = '$user',
| 'password' = '$password',
| 'sink.buffer-flush.max-rows' = '0'
| )
""".stripMargin)
val rltTable = tableEnv.sqlQuery(
"""
|select salary,count(1) as cnt
|from mysqlInput
|group by salary
""".stripMargin)
rltTable.executeInsert("mysqlOutput")
}
}
通过 localhost:8081 访问flink web ui