OpenShift 4 - 使用Debezium实现MySQL的CDC变化数据捕获
《OpenShift 4.x HOL教程汇总》
说明：本文已经在OpenShift 4.6环境中验证
文章目录
- 部署应用和MySQL数据库
-
- 部署MySQL
- 部署应用
- 部署AMQ Streams（Strimzi）和Dezebium
-
- 部署 AMQ Streams（Strimzi）Operator
- 部署Kafka
- 通过MySQL Connector获取变化数据，并发到AMQ Streams的Kafka
-
- 生成Debezium的MySQL Connector容器镜像
- 配置Kafka Connect
- 验证
- 参考
部署应用和MySQL数据库
部署MySQL
- 创建项目
$ oc new-project debezium-demo
- 创建内容如下的debezium.cnf的文件
!include /etc/my.cnf
[mysqld]
server-id = 223344
server_id = 223344
log_bin = ON
binlog_format = ROW
binlog_row_image = full
binlog_rows_query_log_events = ON
expire_logs_days = 10
gtid_mode = ON
enforce_gtid_consistency = ON
- 执行命令，创建configmap
$ oc create configmap db-config --from-file=debezium.cnf
- 创建内容如下的initdb.sql文件
CREATE USER IF NOT EXISTS 'debezium'@'%' IDENTIFIED WITH mysql_native_password BY 'debezium';
GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'debezium'@'%';
FLUSH PRIVILEGES;
- 执行命令，创建configmap
$ oc create configmap db-init --from-file=initdb.sql
- 执行命令，创建configmap
$ oc create secret generic db-creds --from-literal=database-name=music --from-literal=database-password=music --from-literal=database-user=music --from-literal=database-admin-password=music
- 创建内容如下的mysql.yml文件
kind: DeploymentConfig
apiVersion: apps.openshift.io/v1
metadata:
name: spring-music-db
labels:
application: spring-music
app: spring-music
app.kubernetes.io/part-of: spring-music
app.openshift.io/runtime: mysql-database
spec:
replicas: 1
strategy:
type: Recreate
recreateParams:
post:
failurePolicy: Abort
execNewPod:
command:
- /bin/sh
- '-c'
- sleep 10 && MYSQL_PWD="$MYSQL_ROOT_PASSWORD" $MYSQL_PREFIX/bin/mysql -h $SPRING_MUSIC_DB_SERVICE_HOST -u root < /config/initdb.d/initdb.sql
containerName: spring-music-db
volumes:
- db-init
selector:
name: spring-music-db
template:
metadata:
name: spring-music-db
labels:
name: spring-music-db
spec:
volumes:
- name: db-data
emptyDir: {}
- name: db-init
configMap:
name: db-init
- name: db-config
configMap:
name: db-config
containers:
- env:
- name: MYSQL_DEFAULTS_FILE
value: /config/configdb.d/my-debezium.cnf
- name: MYSQL_USER
valueFrom:
secretKeyRef:
name: db-creds
key: database-user
- name: MYSQL_PASSWORD
valueFrom:
secretKeyRef:
name: db-creds
key: database-password
- name: MYSQL_DATABASE
valueFrom:
secretKeyRef:
name: db-creds
key: database-name
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: db-creds
key: database-admin-password
name: spring-music-db
image: ' '
imagePullPolicy: IfNotPresent
volumeMounts:
- name: db-data
mountPath: /var/lib/mysql/data
- name: db-init
mountPath: /config/initdb.d
- name: db-config
mountPath: /config/configdb.d
ports:
- containerPort: 3306
protocol: TCP
livenessProbe:
failureThreshold: 3
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
tcpSocket:
port: 3306
timeoutSeconds: 1
readinessProbe:
exec:
command:
- /bin/sh
- -i
- -c
- MYSQL_PWD="$MYSQL_PASSWORD" mysql -h 127.0.0.1 -u $MYSQL_USER -D $MYSQL_DATABASE -e 'SELECT 1'
failureThreshold: 3
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
memory: 512Mi
securityContext:
privileged: false
triggers:
- type: ConfigChange
- type: ImageChange
imageChangeParams:
automatic: true
containerNames:
- spring-music-db
from:
kind: ImageStreamTag
name: mysql:8.0
namespace: openshift
---
kind: Service
apiVersion: v1
metadata:
name: spring-music-db
labels:
application: spring-music
app: spring-music
annotations:
template.openshift.io/expose-uri: mysql://{.spec.clusterIP}:{.spec.ports[?(.name=="mysql")].port}
spec:
ports:
- name: mysql
port: 3306
protocol: TCP
targetPort: 3306
selector:
name: spring-music-db
- 执行命令，创建应用。
$ oc create -f mysql.yml
- 切换到Developer视图查看Topology，已经部署好MqSQL数据库。
部署应用
- 在Developer视图的“+Add”中进入“container Image”.。
- 在“Image”区域中填入“quay.io/edeandrea/spring-music:latest”，然后选择“Deployment Config”。
- 添加“app.openshift.io/runtime=spring”标签，然后按照下图为Deployment添加3个“Environment Variables”。
Config Map or Secret
NAME: SPRING_DATASOURCE_USERNAME
VALUE: db-creds 的 database-user
Config Map or Secret
NAME: SPRING_DATASOURCE_PASSWORD
VALUE: Secret db-creds 的database-password
Value
NAME: SPRING_DATASOURCE_URL
VALUE: jdbc:mysql://spring-music-db/music
- 在Topology查看应用部署状态。
- 可以访问“spring-music”应用的route访问应用。
部署AMQ Streams（Strimzi）和Dezebium
部署 AMQ Streams（Strimzi）Operator
AMQ Streams Operator是RedHat官方提供的，Strimzi Operator是社区提供的，使用两者任何一个都可以。
- 使用缺省配置安装AMQ Streams Operator或Strimzi Operator。
部署Kafka
- 在Administrator视图安装好的AMQ Streams Operator中找到Kafka。
- 在“Create Kafka”的界面中，在YAML视图中填入以下内容，然后创建Kafka。
kind: Kafka
apiVersion: kafka.strimzi.io/v1beta1
metadata:
name: db-events
namespace: debezium-demo
labels:
app: spring-music-cdc
template: spring-music-cdc
app.kubernetes.io/part-of: spring-music-cdc
spec:
kafka:
replicas: 3
listeners:
plain: {}
jvmOptions:
gcLoggingEnabled: false
config:
auto.create.topics.enable: "true"
num.partitions: 1
offsets.topic.replication.factor: 3
default.replication.factor: 3
transaction.state.log.replication.factor: 3
transaction.state.log.min.isr: 2
storage:
type: persistent-claim
size: 100Gi
deleteClaim: true
template:
statefulset:
metadata:
labels:
app.kubernetes.io/part-of: spring-music-cdc
app: spring-music-cdc
template: spring-music-cdc
annotations:
app.openshift.io/connects-to: db-events-zookeeper
zookeeper:
replicas: 3
storage:
type: persistent-claim
size: 100Gi
deleteClaim: true
template:
statefulset:
metadata:
labels:
app.kubernetes.io/part-of: spring-music-cdc
app: spring-music-cdc
template: spring-music-cdc
entityOperator:
topicOperator: {}
userOperator: {}
- 在Topology中查看部署好的Kafka。
通过MySQL Connector获取变化数据，并发到AMQ Streams的Kafka
生成Debezium的MySQL Connector容器镜像
可以直接使用“quay.io/edeandrea/kafka-connect-debezium-mysql:amq-streams-1.4.0-dbz-1.1.0.Final”镜像，也可以根据以下步骤生成该容器镜像。
- 访问“https://debezium.io/releases/”，通过下图下载Debezium的MySQL Connector Plugin。
- 创建“debezium-connect-image”，然后在下面创建“plugins”目录，再将上一步下载的文件解压缩的plugins目录中。
- 在debezium-connect-image目录中创建如下内容的Dockerfile 文件
FROM registry.redhat.io/amq7/amq-streams-kafka-24-rhel7:1.4.0
USER root:root
COPY ./plugins/ /opt/kafka/plugins
USER jboss:jboss
- 然后根据Dockerfile 生成容器镜像，然后传到quay.io。
配置Kafka Connect
- 在Administrator视图的AMQ Streams Operator进入“Kafka Connect”
- 在YAML视图中使用以下yaml创建资源。
kind: KafkaConnect
apiVersion: kafka.strimzi.io/v1beta1
metadata:
name: db-events
namespace: debezium-demo
labels:
app: spring-music-cdc
template: spring-music-cdc
annotations:
strimzi.io/use-connector-resources: "true"
spec:
replicas: 1
image: "quay.io/edeandrea/kafka-connect-debezium-mysql:amq-streams-1.4.0-dbz-1.1.0.Final"
bootstrapServers: "db-events-kafka-bootstrap:9092"
jvmOptions:
gcLoggingEnabled: false
config:
group.id: spring-music-db
offset.storage.topic: spring-music-db-offsets
config.storage.topic: spring-music-db-configs
status.storage.topic: spring-music-db-status
config.storage.replication.factor: 1
offset.storage.replication.factor: 1
status.storage.replication.factor: 1
config.providers: file
config.providers.file.class: org.apache.kafka.common.config.provider.FileConfigProvider
externalConfiguration:
volumes:
- name: connector-config
secret:
secretName: db-connector-creds
template:
deployment:
metadata:
labels:
app: spring-music-cdc
app.kubernetes.io/part-of: spring-music-cdc
template: spring-music-cdc
annotations:
app.openshift.io/connects-to: db-events-kafka,spring-music-db
- 在Developer视图的Topology查看Kafka Connect部署情况。
- 在Administrator视图的AMQ Streams Operator进入“Kafka Connector”
- 在YAML视图中使用以下yaml创建资源。
kind: KafkaConnector
apiVersion: kafka.strimzi.io/v1alpha1
metadata:
name: db-events
namespace: debezium-demo
labels:
app: spring-music-cdc
strimzi.io/cluster: db-events
spec:
class: io.debezium.connector.mysql.MySqlConnector
tasksMax: 1
config:
database.hostname: spring-music-db
database.port: 3306
database.user: "${file:/opt/kafka/external-configuration/connector-config/connector.properties:dbUsername}"
database.password: "${file:/opt/kafka/external-configuration/connector-config/connector.properties:dbPassword}"
database.dbname: music
database.server.name: spring-music
database.server.id: 223344
database.whitelist: music
database.allowPublicKeyRetrieval: true
database.history.kafka.bootstrap.servers: db-events-kafka-bootstrap:9092
database.history.kafka.topic: dbhistory.music
table.whitelist: music.outbox_events
tombstones.on.delete : false
transforms: outbox
transforms.outbox.type: io.debezium.transforms.outbox.EventRouter
transforms.outbox.route.topic.replacement: "outbox.${routedByValue}.events"
transforms.outbox.table.field.event.id: event_id
transforms.outbox.table.field.event.key: aggregate_id
transforms.outbox.table.field.event.timestamp: event_timestamp
transforms.outbox.table.field.event.type: event_type
transforms.outbox.table.field.event.payload.id: aggregate_id
transforms.outbox.route.by.field: aggregate_type
transforms.outbox.table.fields.additional.placement: "event_id:envelope:eventId,event_timestamp:envelope:eventTimestamp,aggregate_id:envelope:aggregateId,aggregate_type:envelope:aggregateType"
验证
- 在Topology中选中“db-events-kafka”，然后在右侧进入3个pod的任意一个。
- 在Terminal中输入以下命令，然后能从下图看到返回的所有捕获到的数据。
$ bin/kafka-console-consumer.sh --bootstrap-server db-events-kafka-bootstrap:9092 --topic outbox.Album.events --from-beginning
3. 在新的界面中打开部署的应用，点击“add an album”链接添加一个新的album数据，然后ok提交数据。
4. 界面中提示保存成功
5. 返回Terminal，确认出现一行新的数据。格式化后为以下内容，其中在“payload”部分是我们提交的新数据内容。
{
"schema": {
"type": "struct",
"fields": [{
"type": "string",
"optional": true,
"name": "io.debezium.data.Json",
"version": 1,
"field": "payload"
}, {
"type": "string",
"optional": false,
"field": "eventType"
}, {
"type": "int64",
"optional": false,
"field": "eventId"
}, {
"type": "int64",
"optional": false,
"name": "io.debezium.time.MicroTimestamp",
"version": 1,
"field": "eventTimestamp"
}, {
"type": "string",
"optional": false,
"field": "aggregateId"
}, {
"type": "string",
"optional": false,
"field": "aggregateType"
}],
"optional": false
},
"payload": {
"payload": "{\"album\":{\"id\":\"f8f1ad92-f724-45bc-b0ca-e968c369a9c0\",\"genre\":\"Rock\",\"title\":\"Test\",\"artist\":\"Test\",\"albumId\":null,\"trackCount\":0,\"releaseYear\":\"2020\"},\"eventType\":\"ALBUM_CREATED\"}",
"eventType": "ALBUM_CREATED",
"eventId": 30,
"eventTimestamp": 1617626600716385,
"aggregateId": "f8f1ad92-f724-45bc-b0ca-e968c369a9c0",
"aggregateType": "Album"
}
}
参考
https://dzone.com/articles/change-data-capture-with-debezium-a-simple-how-to