OpenShift 4 - 使用Debezium实现MySQL的CDC变化数据捕获

《OpenShift 4.x HOL教程汇总》
说明:本文已经在OpenShift 4.6环境中验证

文章目录

  • 部署应用和MySQL数据库
    • 部署MySQL
    • 部署应用
  • 部署AMQ Streams(Strimzi)和Dezebium
    • 部署 AMQ Streams(Strimzi)Operator
    • 部署Kafka
    • 通过MySQL Connector获取变化数据,并发到AMQ Streams的Kafka
      • 生成Debezium的MySQL Connector容器镜像
      • 配置Kafka Connect
  • 验证
  • 参考

部署应用和MySQL数据库

部署MySQL

  1. 创建项目
$ oc new-project debezium-demo
  1. 创建内容如下的debezium.cnf的文件
!include /etc/my.cnf
 
[mysqld]
server-id = 223344
server_id = 223344
log_bin = ON
binlog_format = ROW
binlog_row_image = full
binlog_rows_query_log_events = ON
expire_logs_days = 10
gtid_mode = ON
enforce_gtid_consistency = ON
  1. 执行命令,创建configmap
$ oc create configmap db-config --from-file=debezium.cnf
  1. 创建内容如下的initdb.sql文件
CREATE USER IF NOT EXISTS 'debezium'@'%' IDENTIFIED WITH mysql_native_password BY 'debezium';
GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'debezium'@'%';
FLUSH PRIVILEGES;
  1. 执行命令,创建configmap
$ oc create configmap db-init --from-file=initdb.sql
  1. 执行命令,创建configmap
$ oc create secret generic db-creds --from-literal=database-name=music --from-literal=database-password=music --from-literal=database-user=music --from-literal=database-admin-password=music
  1. 创建内容如下的mysql.yml文件
kind: DeploymentConfig
apiVersion: apps.openshift.io/v1
metadata:
  name: spring-music-db
  labels:
    application: spring-music
    app: spring-music
    app.kubernetes.io/part-of: spring-music
    app.openshift.io/runtime: mysql-database
spec:
  replicas: 1
  strategy:
    type: Recreate
    recreateParams:
      post:
        failurePolicy: Abort
        execNewPod:
          command:
            - /bin/sh
            - '-c'
            - sleep 10 && MYSQL_PWD="$MYSQL_ROOT_PASSWORD" $MYSQL_PREFIX/bin/mysql -h $SPRING_MUSIC_DB_SERVICE_HOST -u root < /config/initdb.d/initdb.sql
          containerName: spring-music-db
          volumes:
            - db-init
  selector:
    name: spring-music-db
  template:
    metadata:
      name: spring-music-db
      labels:
        name: spring-music-db
    spec:
      volumes:
        - name: db-data
          emptyDir: {}
        - name: db-init
          configMap:
            name: db-init
        - name: db-config
          configMap:
            name: db-config
      containers:
        - env:
            - name: MYSQL_DEFAULTS_FILE
              value: /config/configdb.d/my-debezium.cnf
            - name: MYSQL_USER
              valueFrom:
                secretKeyRef:
                  name: db-creds
                  key: database-user
            - name: MYSQL_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: db-creds
                  key: database-password
            - name: MYSQL_DATABASE
              valueFrom:
                secretKeyRef:
                  name: db-creds
                  key: database-name
            - name: MYSQL_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: db-creds
                  key: database-admin-password
          name: spring-music-db
          image: ' '
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - name: db-data
              mountPath: /var/lib/mysql/data
            - name: db-init
              mountPath: /config/initdb.d
            - name: db-config
              mountPath: /config/configdb.d
          ports:
            - containerPort: 3306
              protocol: TCP
          livenessProbe:
            failureThreshold: 3
            initialDelaySeconds: 30
            periodSeconds: 10
            successThreshold: 1
            tcpSocket:
              port: 3306
            timeoutSeconds: 1
          readinessProbe:
            exec:
              command:
                - /bin/sh
                - -i
                - -c
                - MYSQL_PWD="$MYSQL_PASSWORD" mysql -h 127.0.0.1 -u $MYSQL_USER -D $MYSQL_DATABASE -e 'SELECT 1'
            failureThreshold: 3
            initialDelaySeconds: 5
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          resources:
            limits:
              memory: 512Mi
          securityContext:
            privileged: false
  triggers:
    - type: ConfigChange
    - type: ImageChange
      imageChangeParams:
        automatic: true
        containerNames:
          - spring-music-db
        from:
          kind: ImageStreamTag
          name: mysql:8.0
          namespace: openshift
---
kind: Service
apiVersion: v1
metadata:
  name: spring-music-db
  labels:
    application: spring-music
    app: spring-music
  annotations:
    template.openshift.io/expose-uri: mysql://{.spec.clusterIP}:{.spec.ports[?(.name=="mysql")].port}
spec:
  ports:
    - name: mysql
      port: 3306
      protocol: TCP
      targetPort: 3306
  selector:
    name: spring-music-db
  1. 执行命令,创建应用。
$ oc create -f mysql.yml
  1. 切换到Developer视图查看Topology,已经部署好MqSQL数据库。
    OpenShift 4 - 使用Debezium实现MySQL的CDC变化数据捕获_第1张图片

部署应用

  1. 在Developer视图的“+Add”中进入“container Image”.。
    OpenShift 4 - 使用Debezium实现MySQL的CDC变化数据捕获_第2张图片
  2. 在“Image”区域中填入“quay.io/edeandrea/spring-music:latest”,然后选择“Deployment Config”。
    OpenShift 4 - 使用Debezium实现MySQL的CDC变化数据捕获_第3张图片
  3. 添加“app.openshift.io/runtime=spring”标签,然后按照下图为Deployment添加3个“Environment Variables”。
    Config Map or Secret
    NAME: SPRING_DATASOURCE_USERNAME
    VALUE: db-creds 的 database-user
    Config Map or Secret
    NAME: SPRING_DATASOURCE_PASSWORD
    VALUE: Secret db-creds 的database-password
    Value
    NAME: SPRING_DATASOURCE_URL
    VALUE: jdbc:mysql://spring-music-db/music
    OpenShift 4 - 使用Debezium实现MySQL的CDC变化数据捕获_第4张图片
  4. 在Topology查看应用部署状态。
    OpenShift 4 - 使用Debezium实现MySQL的CDC变化数据捕获_第5张图片
  5. 可以访问“spring-music”应用的route访问应用。
    OpenShift 4 - 使用Debezium实现MySQL的CDC变化数据捕获_第6张图片

部署AMQ Streams(Strimzi)和Dezebium

部署 AMQ Streams(Strimzi)Operator

AMQ Streams Operator是RedHat官方提供的,Strimzi Operator是社区提供的,使用两者任何一个都可以。

  1. 使用缺省配置安装AMQ Streams Operator或Strimzi Operator。
    OpenShift 4 - 使用Debezium实现MySQL的CDC变化数据捕获_第7张图片
    OpenShift 4 - 使用Debezium实现MySQL的CDC变化数据捕获_第8张图片

部署Kafka

  1. 在Administrator视图安装好的AMQ Streams Operator中找到Kafka。
    OpenShift 4 - 使用Debezium实现MySQL的CDC变化数据捕获_第9张图片
  2. 在“Create Kafka”的界面中,在YAML视图中填入以下内容,然后创建Kafka。
kind: Kafka
apiVersion: kafka.strimzi.io/v1beta1
metadata:
  name: db-events
  namespace: debezium-demo
  labels:
    app: spring-music-cdc
    template: spring-music-cdc
    app.kubernetes.io/part-of: spring-music-cdc
spec:
  kafka:
    replicas: 3
    listeners:
      plain: {}
    jvmOptions:
      gcLoggingEnabled: false
    config:
      auto.create.topics.enable: "true"
      num.partitions: 1
      offsets.topic.replication.factor: 3
      default.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
    storage:
      type: persistent-claim
      size: 100Gi
      deleteClaim: true
    template:
      statefulset:
        metadata:
          labels:
            app.kubernetes.io/part-of: spring-music-cdc
            app: spring-music-cdc
            template: spring-music-cdc
          annotations:
            app.openshift.io/connects-to: db-events-zookeeper
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 100Gi
      deleteClaim: true
    template:
      statefulset:
        metadata:
          labels:
            app.kubernetes.io/part-of: spring-music-cdc
            app: spring-music-cdc
            template: spring-music-cdc
  entityOperator:
    topicOperator: {}
    userOperator: {}
  1. 在Topology中查看部署好的Kafka。
    OpenShift 4 - 使用Debezium实现MySQL的CDC变化数据捕获_第10张图片

通过MySQL Connector获取变化数据,并发到AMQ Streams的Kafka

生成Debezium的MySQL Connector容器镜像

可以直接使用“quay.io/edeandrea/kafka-connect-debezium-mysql:amq-streams-1.4.0-dbz-1.1.0.Final”镜像,也可以根据以下步骤生成该容器镜像。

  1. 访问“https://debezium.io/releases/”,通过下图下载Debezium的MySQL Connector Plugin。
    OpenShift 4 - 使用Debezium实现MySQL的CDC变化数据捕获_第11张图片
  2. 创建“debezium-connect-image”,然后在下面创建“plugins”目录,再将上一步下载的文件解压缩的plugins目录中。
    OpenShift 4 - 使用Debezium实现MySQL的CDC变化数据捕获_第12张图片
  3. 在debezium-connect-image目录中创建如下内容的Dockerfile 文件
FROM registry.redhat.io/amq7/amq-streams-kafka-24-rhel7:1.4.0
USER root:root
COPY ./plugins/ /opt/kafka/plugins
USER jboss:jboss
  1. 然后根据Dockerfile 生成容器镜像,然后传到quay.io。

配置Kafka Connect

  1. 在Administrator视图的AMQ Streams Operator进入“Kafka Connect”
    OpenShift 4 - 使用Debezium实现MySQL的CDC变化数据捕获_第13张图片
  2. 在YAML视图中使用以下yaml创建资源。
kind: KafkaConnect
apiVersion: kafka.strimzi.io/v1beta1
metadata:
  name: db-events
  namespace: debezium-demo
  labels:
    app: spring-music-cdc
    template: spring-music-cdc
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  replicas: 1
  image: "quay.io/edeandrea/kafka-connect-debezium-mysql:amq-streams-1.4.0-dbz-1.1.0.Final"
  bootstrapServers: "db-events-kafka-bootstrap:9092"
  jvmOptions:
    gcLoggingEnabled: false
  config:
    group.id: spring-music-db
    offset.storage.topic: spring-music-db-offsets
    config.storage.topic: spring-music-db-configs
    status.storage.topic: spring-music-db-status
    config.storage.replication.factor: 1
    offset.storage.replication.factor: 1
    status.storage.replication.factor: 1
    config.providers: file
    config.providers.file.class: org.apache.kafka.common.config.provider.FileConfigProvider
  externalConfiguration:
    volumes:
      - name: connector-config
        secret:
          secretName: db-connector-creds
  template:
    deployment:
      metadata:
        labels:
          app: spring-music-cdc
          app.kubernetes.io/part-of: spring-music-cdc
          template: spring-music-cdc
        annotations:
          app.openshift.io/connects-to: db-events-kafka,spring-music-db
  1. 在Developer视图的Topology查看Kafka Connect部署情况。
    OpenShift 4 - 使用Debezium实现MySQL的CDC变化数据捕获_第14张图片
  2. 在Administrator视图的AMQ Streams Operator进入“Kafka Connector”
    OpenShift 4 - 使用Debezium实现MySQL的CDC变化数据捕获_第15张图片
  3. 在YAML视图中使用以下yaml创建资源。
kind: KafkaConnector
apiVersion: kafka.strimzi.io/v1alpha1
metadata:
  name: db-events
  namespace: debezium-demo
  labels:
    app: spring-music-cdc
    strimzi.io/cluster: db-events
spec:
  class: io.debezium.connector.mysql.MySqlConnector
  tasksMax: 1
  config:
    database.hostname: spring-music-db
    database.port: 3306
    database.user: "${file:/opt/kafka/external-configuration/connector-config/connector.properties:dbUsername}"
    database.password: "${file:/opt/kafka/external-configuration/connector-config/connector.properties:dbPassword}"
    database.dbname: music
    database.server.name: spring-music
    database.server.id: 223344
    database.whitelist: music
    database.allowPublicKeyRetrieval: true
    database.history.kafka.bootstrap.servers: db-events-kafka-bootstrap:9092
    database.history.kafka.topic: dbhistory.music
    table.whitelist: music.outbox_events
    tombstones.on.delete : false
    transforms: outbox
    transforms.outbox.type: io.debezium.transforms.outbox.EventRouter
    transforms.outbox.route.topic.replacement: "outbox.${routedByValue}.events"
    transforms.outbox.table.field.event.id: event_id
    transforms.outbox.table.field.event.key: aggregate_id
    transforms.outbox.table.field.event.timestamp: event_timestamp
    transforms.outbox.table.field.event.type: event_type
    transforms.outbox.table.field.event.payload.id: aggregate_id
    transforms.outbox.route.by.field: aggregate_type
    transforms.outbox.table.fields.additional.placement: "event_id:envelope:eventId,event_timestamp:envelope:eventTimestamp,aggregate_id:envelope:aggregateId,aggregate_type:envelope:aggregateType"

验证

  1. 在Topology中选中“db-events-kafka”,然后在右侧进入3个pod的任意一个。
    OpenShift 4 - 使用Debezium实现MySQL的CDC变化数据捕获_第16张图片
  2. 在Terminal中输入以下命令,然后能从下图看到返回的所有捕获到的数据。
$ bin/kafka-console-consumer.sh --bootstrap-server db-events-kafka-bootstrap:9092 --topic outbox.Album.events --from-beginning

OpenShift 4 - 使用Debezium实现MySQL的CDC变化数据捕获_第17张图片
3. 在新的界面中打开部署的应用,点击“add an album”链接添加一个新的album数据,然后ok提交数据。
OpenShift 4 - 使用Debezium实现MySQL的CDC变化数据捕获_第18张图片
4. 界面中提示保存成功
OpenShift 4 - 使用Debezium实现MySQL的CDC变化数据捕获_第19张图片
5. 返回Terminal,确认出现一行新的数据。格式化后为以下内容,其中在“payload”部分是我们提交的新数据内容。
在这里插入图片描述

{
	"schema": {
		"type": "struct",
		"fields": [{
			"type": "string",
			"optional": true,
			"name": "io.debezium.data.Json",
			"version": 1,
			"field": "payload"
		}, {
			"type": "string",
			"optional": false,
			"field": "eventType"
		}, {
			"type": "int64",
			"optional": false,
			"field": "eventId"
		}, {
			"type": "int64",
			"optional": false,
			"name": "io.debezium.time.MicroTimestamp",
			"version": 1,
			"field": "eventTimestamp"
		}, {
			"type": "string",
			"optional": false,
			"field": "aggregateId"
		}, {
			"type": "string",
			"optional": false,
			"field": "aggregateType"
		}],
		"optional": false
	},
	"payload": {
		"payload": "{\"album\":{\"id\":\"f8f1ad92-f724-45bc-b0ca-e968c369a9c0\",\"genre\":\"Rock\",\"title\":\"Test\",\"artist\":\"Test\",\"albumId\":null,\"trackCount\":0,\"releaseYear\":\"2020\"},\"eventType\":\"ALBUM_CREATED\"}",
		"eventType": "ALBUM_CREATED",
		"eventId": 30,
		"eventTimestamp": 1617626600716385,
		"aggregateId": "f8f1ad92-f724-45bc-b0ca-e968c369a9c0",
		"aggregateType": "Album"
	}
}

参考

https://dzone.com/articles/change-data-capture-with-debezium-a-simple-how-to

你可能感兴趣的:(OpenShift,4,CDC,Debezium)