huo_火力全开

Flink之CheckPoint 构建真正的End-to-End Exactly-Once

本博客内容主要讲解: Flink消费Kafka数据写入MySQL模拟Flink端到端精准一次消费

1. 模拟 kafka数据没1s产生一条,产生60条;
2. Flink 消费Kafka,每10一次检查点;
3. 消费到第15条(前面产生一个检查点,不一定是在消费到第10条,因为第一个10s开始时间不一定是从第一条开始算)设置异常数据,数据不会插入
4. 修正异常数据,重启应用程序,Flink重新消费Kafka,数据全部落入MySQL,没有重复,也没有丢失

Exactly-Once 语义

Exactly-Once语义指每个数据输入端的事件只能影响最终结果一次.解释机器出现故障,数据既没有重复,也没有丢失

CheckPoint: 一次CheckPoint进行一次快照;容包含以下内容

应用程序的当前状态
输入流位置

在 Flink 1.4.0 之前，Exactly-Once 语义仅限于 Flink 应用程序内部，并没有扩展到 Flink 数据处理完后发送的大多数外部系统。Flink 应用程序与各种数据输出端进行交互，开发人员需要有能力自己维护组件的上下文来保证 Exactly-Once 语义。

模拟 Kafka->Flink->MySQL 端到端精准一次消费

主要代码

代码1

package com.huonan.twocommit;

import lombok.extern.slf4j.Slf4j;
import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.node.ObjectNode;
import org.apache.flink.streaming.api.CheckpointingMode;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer011;
import org.apache.flink.streaming.util.serialization.JSONKeyValueDeserializationSchema;

import java.util.Properties;

/**
 * 消费kafka消息，sink(自定义)到mysql中，保证kafka to mysql 的Exactly-Once
 * @author ThinkPad
 */

@Slf4j
public class StreamDemoKafka2Mysql {
    private static final String topic_ExactlyOnce = KafkaUtils.topic_ExactlyOnce;

    public static void main(String[] args) throws Exception {
        final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        //设置并行度,为了方便测试，查看消息的顺序，这里设置为1，可以更改为多并行度
        env.setParallelism(1);
        //checkpoint的设置
        //每隔10s进行启动一个检查点【设置checkpoint的周期】
        env.enableCheckpointing(10000);
        //设置模式为：exactly_one，仅一次语义
        env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE);
        //确保检查点之间有1s的时间间隔【checkpoint最小间隔】
        env.getCheckpointConfig().setMinPauseBetweenCheckpoints(1000);
        //检查点必须在10s之内完成，或者被丢弃【checkpoint超时时间】
        env.getCheckpointConfig().setCheckpointTimeout(10000);
        //同一时间只允许进行一次检查点
        env.getCheckpointConfig().setMaxConcurrentCheckpoints(1);
        //表示一旦Flink程序被cancel后，会保留checkpoint数据，以便根据实际需要恢复到指定的checkpoint
        //env.getCheckpointConfig().enableExternalizedCheckpoints(CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
        //设置statebackend,将检查点保存在hdfs上面，默认保存在内存中。这里先保存到本地
//        env.setStateBackend(new FsStateBackend("file:///Users/temp/cp/"));

        //设置kafka消费参数
        Properties properties = new Properties();
        properties.setProperty("bootstrap.servers", "192.168.31.100:9092");
// only required for Kafka 0.8
        properties.setProperty("group.id", topic_ExactlyOnce);
        //   Properties props = new Properties();
        //props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.31.100:9092");
        //   props.put(ConsumerConfig.GROUP_ID_CONFIG, "test-exactly-Once");
        //kafka分区自动发现周期
        //  props.put(FlinkKafkaConsumerBase.KEY_PARTITION_DISCOVERY_INTERVAL_MILLIS, "3000");

        /*SimpleStringSchema可以获取到kafka消息，JSONKeyValueDeserializationSchema可以获取都消息的key,value，metadata:topic,partition，offset等信息*/
        FlinkKafkaConsumer011 kafkaConsumer011 = new FlinkKafkaConsumer011<>(
                topic_ExactlyOnce,
                new JSONKeyValueDeserializationSchema(true),
                properties);

        //加入kafka数据源
        DataStreamSource streamSource = env.addSource(kafkaConsumer011);
//        System.out.println("streamSource:" + streamSource.print());
        streamSource.print();
        //数据传输到下游
        streamSource.addSink(new MySqlTwoPhaseCommitSink()).name("MySqlTwoPhaseCommitSink");
        //触发执行
        env.execute("StreamDemoKafka2Mysql");
    }
}

代码2

package com.huonan.twocommit;

import lombok.extern.slf4j.Slf4j;
import com.alibaba.fastjson.JSON;
import net.sf.json.JSONObject;
import org.apache.flink.api.common.ExecutionConfig;
import org.apache.flink.api.common.typeutils.base.VoidSerializer;
import org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer;
import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.node.ObjectNode;
import org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction;

import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.Timestamp;
import java.text.SimpleDateFormat;
import java.util.Date;

/**
 * 自定义kafka to mysql,继承TwoPhaseCommitSinkFunction,实现两阶段提交
 * @author ThinkPad
 */
@Slf4j
public class MySqlTwoPhaseCommitSink extends TwoPhaseCommitSinkFunction {

   // private static final Logger log = LoggerFactory.getLogger(MySqlTwoPhaseCommitSink.class);

    public MySqlTwoPhaseCommitSink(){
        super(new KryoSerializer<>(Connection.class,new ExecutionConfig()), VoidSerializer.INSTANCE);
    }

    /**
     * 执行数据库入库操作  task初始化的时候调用
     * @param connection
     * @param objectNode
     * @param context
     * @throws Exception
     */
    @Override
    protected void invoke(Connection connection, ObjectNode objectNode, Context context) throws Exception {
        log.info("start invoke...");
        String date = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").format(new Date());
        log.info("===>date:" + date + " " + objectNode);
        log.info("===>date:{} --{}",date,objectNode);
        String value = objectNode.get("value").toString();
        JSONObject valueJson = JSONObject.fromObject(value);
        String value_str = (String) valueJson.get("value");
        log.info("objectNode-value:" + value);
        if (value == null||"".equals(value)) {
            return;
        }
        String sql = "insert into `mysqlExactlyOnce_test` (`value`,`insert_time`) values (?,?)";
        log.info("====执行SQL:{}===",sql);
        PreparedStatement ps = connection.prepareStatement(sql);
        ps.setString(1,value);
        Timestamp value_time = new Timestamp(System.currentTimeMillis());
        ps.setTimestamp(2,value_time);
        log.info("要插入的数据:{}--{}",value,value_time);
        //执行insert语句
        ps.execute();
        //手动制造异常
//        if(Integer.parseInt(value_str) == 15) {
//            System.out.println(1 / 0);
//        }
    }

    /**
     * 获取连接，开启手动提交事物（getConnection方法中）
     * @return
     * @throws Exception
     */
    @Override
    protected Connection beginTransaction() throws Exception {
        log.info("start beginTransaction.......");
        String url = "jdbc:mysql://192.168.31.100:3306/test?useUnicode=true&characterEncoding=UTF-8&zeroDateTimeBehavior=convertToNull&useSSL=false&autoReconnect=true";
        Connection connection = DBConnectUtil.getConnection(url, "root", "123456");
        return connection;
    }

    /**
     *预提交，这里预提交的逻辑在invoke方法中
     * @param connection
     * @throws Exception
     */
    @Override
    protected void preCommit(Connection connection) throws Exception {
        log.info("start preCommit...");
    }

    /**
     * 如果invoke方法执行正常，则提交事务
     * @param connection
     */
    @Override
    protected void commit(Connection connection) {
        log.info("start commit...");
        DBConnectUtil.commit(connection);
//        try {
//            Thread.sleep(5000);
//        } catch (InterruptedException e) {
//            e.printStackTrace();
//            log.error("提交失败！！！");
//        }
    }

    /**
     * 如果invoke执行异常则回滚事物，下一次的checkpoint操作也不会执行
     * @param connection
     */
    @Override
    protected void abort(Connection connection) {
        log.info("start abort rollback...");
        DBConnectUtil.rollback(connection);
    }
}

控制台日志

[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka commitId : 73be1e1168f91ee2
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase - Consumer subtask 0 will start reading the following 1 partitions from the committed group offsets in Kafka: [KafkaTopicPartition{topic='test_TwoPhaseCommit', partition=0}]
[Checkpoint Timer] INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 1 @ 1581164180635 for job 4903f3b0bee84c38dc337b92d6f3361f.
[Kafka 0.10 Fetcher for Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO org.apache.kafka.clients.consumer.ConsumerConfig - ConsumerConfig values: 
	auto.commit.interval.ms = 5000
	auto.offset.reset = latest
	bootstrap.servers = [192.168.31.100:9092]
	check.crcs = true
	client.id = 
	connections.max.idle.ms = 540000
	enable.auto.commit = false
	exclude.internal.topics = true
	fetch.max.bytes = 52428800
	fetch.max.wait.ms = 500
	fetch.min.bytes = 1
	group.id = test_TwoPhaseCommit
	heartbeat.interval.ms = 3000
	interceptor.classes = null
	internal.leave.group.on.close = true
	isolation.level = read_uncommitted
	key.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer
	max.partition.fetch.bytes = 1048576
	max.poll.interval.ms = 300000
	max.poll.records = 500
	metadata.max.age.ms = 300000
	metric.reporters = []
	metrics.num.samples = 2
	metrics.recording.level = INFO
	metrics.sample.window.ms = 30000
	partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
	receive.buffer.bytes = 65536
	reconnect.backoff.max.ms = 1000
	reconnect.backoff.ms = 50
	request.timeout.ms = 305000
	retry.backoff.ms = 100
	sasl.jaas.config = null
	sasl.kerberos.kinit.cmd = /usr/bin/kinit
	sasl.kerberos.min.time.before.relogin = 60000
	sasl.kerberos.service.name = null
	sasl.kerberos.ticket.renew.jitter = 0.05
	sasl.kerberos.ticket.renew.window.factor = 0.8
	sasl.mechanism = GSSAPI
	security.protocol = PLAINTEXT
	send.buffer.bytes = 131072
	session.timeout.ms = 10000
	ssl.cipher.suites = null
	ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
	ssl.endpoint.identification.algorithm = null
	ssl.key.password = null
	ssl.keymanager.algorithm = SunX509
	ssl.keystore.location = null
	ssl.keystore.password = null
	ssl.keystore.type = JKS
	ssl.protocol = TLS
	ssl.provider = null
	ssl.secure.random.implementation = null
	ssl.trustmanager.algorithm = PKIX
	ssl.truststore.location = null
	ssl.truststore.password = null
	ssl.truststore.type = JKS
	value.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer

[Kafka 0.10 Fetcher for Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka version : 0.11.0.2
[Kafka 0.10 Fetcher for Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka commitId : 73be1e1168f91ee2
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start preCommit...
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start beginTransaction.......
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.DBConnectUtil - 获取连接:com.mysql.jdbc.JDBC4Connection@39fe1aea 成功...
[Kafka 0.10 Fetcher for Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Discovered coordinator hadoop000:9092 (id: 2147483647 rack: null) for group test_TwoPhaseCommit.
[flink-akka.actor.default-dispatcher-2] INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 1 for job 4903f3b0bee84c38dc337b92d6f3361f (388201 bytes in 2513 ms).
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction - MySqlTwoPhaseCommitSink 0/1 - checkpoint 1 complete, committing transaction TransactionHolder{handle=com.mysql.jdbc.JDBC4Connection@2f37325d, transactionStartTime=1581164179408} from checkpoint 1
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start commit...
{"value":{"value":"1"},"metadata":{"offset":0,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start invoke...
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:16:26 {"value":{"value":"1"},"metadata":{"offset":0,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:16:26 --{"value":{"value":"1"},"metadata":{"offset":0,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - objectNode-value:{"value":"1"}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ====执行SQL:insert into `test_TwoPhaseCommit` (`value`,`insert_time`) values (?,?)===
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - 要插入的数据:{"value":"1"}--2020-02-08 20:16:26.456
{"value":{"value":"2"},"metadata":{"offset":1,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start invoke...
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:16:26 {"value":{"value":"2"},"metadata":{"offset":1,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:16:26 --{"value":{"value":"2"},"metadata":{"offset":1,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - objectNode-value:{"value":"2"}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ====执行SQL:insert into `test_TwoPhaseCommit` (`value`,`insert_time`) values (?,?)===
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - 要插入的数据:{"value":"2"}--2020-02-08 20:16:26.92
{"value":{"value":"3"},"metadata":{"offset":2,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start invoke...
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:16:27 {"value":{"value":"3"},"metadata":{"offset":2,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:16:27 --{"value":{"value":"3"},"metadata":{"offset":2,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - objectNode-value:{"value":"3"}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ====执行SQL:insert into `test_TwoPhaseCommit` (`value`,`insert_time`) values (?,?)===
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - 要插入的数据:{"value":"3"}--2020-02-08 20:16:27.954
{"value":{"value":"4"},"metadata":{"offset":3,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start invoke...
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:16:28 {"value":{"value":"4"},"metadata":{"offset":3,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:16:28 --{"value":{"value":"4"},"metadata":{"offset":3,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - objectNode-value:{"value":"4"}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ====执行SQL:insert into `test_TwoPhaseCommit` (`value`,`insert_time`) values (?,?)===
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - 要插入的数据:{"value":"4"}--2020-02-08 20:16:28.936
{"value":{"value":"5"},"metadata":{"offset":4,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start invoke...
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:16:29 {"value":{"value":"5"},"metadata":{"offset":4,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:16:29 --{"value":{"value":"5"},"metadata":{"offset":4,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - objectNode-value:{"value":"5"}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ====执行SQL:insert into `test_TwoPhaseCommit` (`value`,`insert_time`) values (?,?)===
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - 要插入的数据:{"value":"5"}--2020-02-08 20:16:29.948
[Checkpoint Timer] INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 2 @ 1581164190527 for job 4903f3b0bee84c38dc337b92d6f3361f.
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start preCommit...
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start beginTransaction.......
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.DBConnectUtil - 获取连接:com.mysql.jdbc.JDBC4Connection@9c53b92 成功...
{"value":{"value":"6"},"metadata":{"offset":5,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start invoke...
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:16:31 {"value":{"value":"6"},"metadata":{"offset":5,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:16:31 --{"value":{"value":"6"},"metadata":{"offset":5,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - objectNode-value:{"value":"6"}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ====执行SQL:insert into `test_TwoPhaseCommit` (`value`,`insert_time`) values (?,?)===
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - 要插入的数据:{"value":"6"}--2020-02-08 20:16:31.771
[flink-akka.actor.default-dispatcher-13] INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 2 for job 4903f3b0bee84c38dc337b92d6f3361f (438148 bytes in 1516 ms).
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction - MySqlTwoPhaseCommitSink 0/1 - checkpoint 2 complete, committing transaction TransactionHolder{handle=com.mysql.jdbc.JDBC4Connection@39fe1aea, transactionStartTime=1581164180804} from checkpoint 2
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start commit...
{"value":{"value":"7"},"metadata":{"offset":6,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start invoke...
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:16:32 {"value":{"value":"7"},"metadata":{"offset":6,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:16:32 --{"value":{"value":"7"},"metadata":{"offset":6,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - objectNode-value:{"value":"7"}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ====执行SQL:insert into `test_TwoPhaseCommit` (`value`,`insert_time`) values (?,?)===
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - 要插入的数据:{"value":"7"}--2020-02-08 20:16:32.152
{"value":{"value":"8"},"metadata":{"offset":7,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start invoke...
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:16:33 {"value":{"value":"8"},"metadata":{"offset":7,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:16:33 --{"value":{"value":"8"},"metadata":{"offset":7,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - objectNode-value:{"value":"8"}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ====执行SQL:insert into `test_TwoPhaseCommit` (`value`,`insert_time`) values (?,?)===
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - 要插入的数据:{"value":"8"}--2020-02-08 20:16:33.057
{"value":{"value":"9"},"metadata":{"offset":8,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start invoke...
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:16:34 {"value":{"value":"9"},"metadata":{"offset":8,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:16:34 --{"value":{"value":"9"},"metadata":{"offset":8,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - objectNode-value:{"value":"9"}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ====执行SQL:insert into `test_TwoPhaseCommit` (`value`,`insert_time`) values (?,?)===
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - 要插入的数据:{"value":"9"}--2020-02-08 20:16:34.062
{"value":{"value":"10"},"metadata":{"offset":9,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start invoke...
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:16:35 {"value":{"value":"10"},"metadata":{"offset":9,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:16:35 --{"value":{"value":"10"},"metadata":{"offset":9,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - objectNode-value:{"value":"10"}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ====执行SQL:insert into `test_TwoPhaseCommit` (`value`,`insert_time`) values (?,?)===
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - 要插入的数据:{"value":"10"}--2020-02-08 20:16:35.075
{"value":{"value":"11"},"metadata":{"offset":10,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start invoke...
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:16:36 {"value":{"value":"11"},"metadata":{"offset":10,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:16:36 --{"value":{"value":"11"},"metadata":{"offset":10,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - objectNode-value:{"value":"11"}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ====执行SQL:insert into `test_TwoPhaseCommit` (`value`,`insert_time`) values (?,?)===
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - 要插入的数据:{"value":"11"}--2020-02-08 20:16:36.069
{"value":{"value":"12"},"metadata":{"offset":11,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start invoke...
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:16:37 {"value":{"value":"12"},"metadata":{"offset":11,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:16:37 --{"value":{"value":"12"},"metadata":{"offset":11,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - objectNode-value:{"value":"12"}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ====执行SQL:insert into `test_TwoPhaseCommit` (`value`,`insert_time`) values (?,?)===
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - 要插入的数据:{"value":"12"}--2020-02-08 20:16:37.074
{"value":{"value":"13"},"metadata":{"offset":12,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start invoke...
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:16:38 {"value":{"value":"13"},"metadata":{"offset":12,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:16:38 --{"value":{"value":"13"},"metadata":{"offset":12,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - objectNode-value:{"value":"13"}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ====执行SQL:insert into `test_TwoPhaseCommit` (`value`,`insert_time`) values (?,?)===
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - 要插入的数据:{"value":"13"}--2020-02-08 20:16:38.071
{"value":{"value":"14"},"metadata":{"offset":13,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start invoke...
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:16:39 {"value":{"value":"14"},"metadata":{"offset":13,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:16:39 --{"value":{"value":"14"},"metadata":{"offset":13,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - objectNode-value:{"value":"14"}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ====执行SQL:insert into `test_TwoPhaseCommit` (`value`,`insert_time`) values (?,?)===
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - 要插入的数据:{"value":"14"}--2020-02-08 20:16:39.103
{"value":{"value":"15"},"metadata":{"offset":14,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start invoke...
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:16:40 {"value":{"value":"15"},"metadata":{"offset":14,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:16:40 --{"value":{"value":"15"},"metadata":{"offset":14,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - objectNode-value:{"value":"15"}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ====执行SQL:insert into `test_TwoPhaseCommit` (`value`,`insert_time`) values (?,?)===
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - 要插入的数据:{"value":"15"}--2020-02-08 20:16:40.102
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start abort rollback...
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO org.apache.flink.runtime.taskmanager.Task - Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1) (d8d24f87d0606d90cf2c4149e19773e2) switched from RUNNING to FAILED.
java.lang.ArithmeticException: / by zero

MySQL插入数据展示

重启Flink

日志

[Kafka 0.10 Fetcher for Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka version : 0.11.0.2
[Kafka 0.10 Fetcher for Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka commitId : 73be1e1168f91ee2
[Kafka 0.10 Fetcher for Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Discovered coordinator hadoop000:9092 (id: 2147483647 rack: null) for group test_TwoPhaseCommit.
{"value":{"value":"6"},"metadata":{"offset":5,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start invoke...
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:41 {"value":{"value":"6"},"metadata":{"offset":5,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:41 --{"value":{"value":"6"},"metadata":{"offset":5,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - objectNode-value:{"value":"6"}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ====执行SQL:insert into `test_TwoPhaseCommit` (`value`,`insert_time`) values (?,?)===
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - 要插入的数据:{"value":"6"}--2020-02-08 20:23:41.937
{"value":{"value":"7"},"metadata":{"offset":6,"topic":"test_TwoPhaseCommit","partition":0}}
{"value":{"value":"8"},"metadata":{"offset":7,"topic":"test_TwoPhaseCommit","partition":0}}
{"value":{"value":"9"},"metadata":{"offset":8,"topic":"test_TwoPhaseCommit","partition":0}}
{"value":{"value":"10"},"metadata":{"offset":9,"topic":"test_TwoPhaseCommit","partition":0}}
{"value":{"value":"11"},"metadata":{"offset":10,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start invoke...
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 {"value":{"value":"7"},"metadata":{"offset":6,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 --{"value":{"value":"7"},"metadata":{"offset":6,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - objectNode-value:{"value":"7"}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ====执行SQL:insert into `test_TwoPhaseCommit` (`value`,`insert_time`) values (?,?)===
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - 要插入的数据:{"value":"7"}--2020-02-08 20:23:42.053
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start invoke...
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 {"value":{"value":"8"},"metadata":{"offset":7,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 --{"value":{"value":"8"},"metadata":{"offset":7,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - objectNode-value:{"value":"8"}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ====执行SQL:insert into `test_TwoPhaseCommit` (`value`,`insert_time`) values (?,?)===
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - 要插入的数据:{"value":"8"}--2020-02-08 20:23:42.058
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start invoke...
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 {"value":{"value":"9"},"metadata":{"offset":8,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 --{"value":{"value":"9"},"metadata":{"offset":8,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - objectNode-value:{"value":"9"}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ====执行SQL:insert into `test_TwoPhaseCommit` (`value`,`insert_time`) values (?,?)===
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - 要插入的数据:{"value":"9"}--2020-02-08 20:23:42.063
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start invoke...
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 {"value":{"value":"10"},"metadata":{"offset":9,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 --{"value":{"value":"10"},"metadata":{"offset":9,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - objectNode-value:{"value":"10"}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ====执行SQL:insert into `test_TwoPhaseCommit` (`value`,`insert_time`) values (?,?)===
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - 要插入的数据:{"value":"10"}--2020-02-08 20:23:42.067
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start invoke...
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 {"value":{"value":"11"},"metadata":{"offset":10,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 --{"value":{"value":"11"},"metadata":{"offset":10,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - objectNode-value:{"value":"11"}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ====执行SQL:insert into `test_TwoPhaseCommit` (`value`,`insert_time`) values (?,?)===
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - 要插入的数据:{"value":"11"}--2020-02-08 20:23:42.073
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start invoke...
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 {"value":{"value":"12"},"metadata":{"offset":11,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 --{"value":{"value":"12"},"metadata":{"offset":11,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - objectNode-value:{"value":"12"}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ====执行SQL:insert into `test_TwoPhaseCommit` (`value`,`insert_time`) values (?,?)===
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - 要插入的数据:{"value":"12"}--2020-02-08 20:23:42.077
{"value":{"value":"12"},"metadata":{"offset":11,"topic":"test_TwoPhaseCommit","partition":0}}
{"value":{"value":"13"},"metadata":{"offset":12,"topic":"test_TwoPhaseCommit","partition":0}}
{"value":{"value":"14"},"metadata":{"offset":13,"topic":"test_TwoPhaseCommit","partition":0}}
{"value":{"value":"15"},"metadata":{"offset":14,"topic":"test_TwoPhaseCommit","partition":0}}
{"value":{"value":"16"},"metadata":{"offset":15,"topic":"test_TwoPhaseCommit","partition":0}}
{"value":{"value":"17"},"metadata":{"offset":16,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start invoke...
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 {"value":{"value":"13"},"metadata":{"offset":12,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 --{"value":{"value":"13"},"metadata":{"offset":12,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - objectNode-value:{"value":"13"}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ====执行SQL:insert into `test_TwoPhaseCommit` (`value`,`insert_time`) values (?,?)===
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - 要插入的数据:{"value":"13"}--2020-02-08 20:23:42.092
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start invoke...
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 {"value":{"value":"14"},"metadata":{"offset":13,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 --{"value":{"value":"14"},"metadata":{"offset":13,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - objectNode-value:{"value":"14"}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ====执行SQL:insert into `test_TwoPhaseCommit` (`value`,`insert_time`) values (?,?)===
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - 要插入的数据:{"value":"14"}--2020-02-08 20:23:42.097
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start invoke...
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 {"value":{"value":"15"},"metadata":{"offset":14,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 --{"value":{"value":"15"},"metadata":{"offset":14,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - objectNode-value:{"value":"15"}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ====执行SQL:insert into `test_TwoPhaseCommit` (`value`,`insert_time`) values (?,?)===
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - 要插入的数据:{"value":"15"}--2020-02-08 20:23:42.103
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start invoke...
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 {"value":{"value":"16"},"metadata":{"offset":15,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 --{"value":{"value":"16"},"metadata":{"offset":15,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - objectNode-value:{"value":"16"}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ====执行SQL:insert into `test_TwoPhaseCommit` (`value`,`insert_time`) values (?,?)===
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - 要插入的数据:{"value":"16"}--2020-02-08 20:23:42.107
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start invoke...
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 {"value":{"value":"17"},"metadata":{"offset":16,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 --{"value":{"value":"17"},"metadata":{"offset":16,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - objectNode-value:{"value":"17"}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ====执行SQL:insert into `test_TwoPhaseCommit` (`value`,`insert_time`) values (?,?)===
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - 要插入的数据:{"value":"17"}--2020-02-08 20:23:42.113
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start invoke...
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 {"value":{"value":"18"},"metadata":{"offset":17,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 --{"value":{"value":"18"},"metadata":{"offset":17,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - objectNode-value:{"value":"18"}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ====执行SQL:insert into `test_TwoPhaseCommit` (`value`,`insert_time`) values (?,?)===
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - 要插入的数据:{"value":"18"}--2020-02-08 20:23:42.118
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start invoke...
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 {"value":{"value":"19"},"metadata":{"offset":18,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 --{"value":{"value":"19"},"metadata":{"offset":18,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - objectNode-value:{"value":"19"}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ====执行SQL:insert into `test_TwoPhaseCommit` (`value`,`insert_time`) values (?,?)===
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - 要插入的数据:{"value":"19"}--2020-02-08 20:23:42.123
{"value":{"value":"18"},"metadata":{"offset":17,"topic":"test_TwoPhaseCommit","partition":0}}
{"value":{"value":"19"},"metadata":{"offset":18,"topic":"test_TwoPhaseCommit","partition":0}}
{"value":{"value":"20"},"metadata":{"offset":19,"topic":"test_TwoPhaseCommit","partition":0}}
{"value":{"value":"21"},"metadata":{"offset":20,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start invoke...
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 {"value":{"value":"20"},"metadata":{"offset":19,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 --{"value":{"value":"20"},"metadata":{"offset":19,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - objectNode-value:{"value":"20"}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ====执行SQL:insert into `test_TwoPhaseCommit` (`value`,`insert_time`) values (?,?)===
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - 要插入的数据:{"value":"20"}--2020-02-08 20:23:42.129
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start invoke...
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 {"value":{"value":"21"},"metadata":{"offset":20,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 --{"value":{"value":"21"},"metadata":{"offset":20,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - objectNode-value:{"value":"21"}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ====执行SQL:insert into `test_TwoPhaseCommit` (`value`,`insert_time`) values (?,?)===
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - 要插入的数据:{"value":"21"}--2020-02-08 20:23:42.146
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start invoke...
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 {"value":{"value":"22"},"metadata":{"offset":21,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 --{"value":{"value":"22"},"metadata":{"offset":21,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - objectNode-value:{"value":"22"}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ====执行SQL:insert into `test_TwoPhaseCommit` (`value`,`insert_time`) values (?,?)===
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - 要插入的数据:{"value":"22"}--2020-02-08 20:23:42.155
{"value":{"value":"22"},"metadata":{"offset":21,"topic":"test_TwoPhaseCommit","partition":0}}
{"value":{"value":"23"},"metadata":{"offset":22,"topic":"test_TwoPhaseCommit","partition":0}}
{"value":{"value":"24"},"metadata":{"offset":23,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start invoke...
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 {"value":{"value":"23"},"metadata":{"offset":22,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 --{"value":{"value":"23"},"metadata":{"offset":22,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - objectNode-value:{"value":"23"}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ====执行SQL:insert into `test_TwoPhaseCommit` (`value`,`insert_time`) values (?,?)===
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - 要插入的数据:{"value":"23"}--2020-02-08 20:23:42.164
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start invoke...
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 {"value":{"value":"24"},"metadata":{"offset":23,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 --{"value":{"value":"24"},"metadata":{"offset":23,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - objectNode-value:{"value":"24"}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ====执行SQL:insert into `test_TwoPhaseCommit` (`value`,`insert_time`) values (?,?)===
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - 要插入的数据:{"value":"24"}--2020-02-08 20:23:42.174
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start invoke...
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 {"value":{"value":"25"},"metadata":{"offset":24,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 --{"value":{"value":"25"},"metadata":{"offset":24,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - objectNode-value:{"value":"25"}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ====执行SQL:insert into `test_TwoPhaseCommit` (`value`,`insert_time`) values (?,?)===
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - 要插入的数据:{"value":"25"}--2020-02-08 20:23:42.184
{"value":{"value":"25"},"metadata":{"offset":24,"topic":"test_TwoPhaseCommit","partition":0}}
{"value":{"value":"26"},"metadata":{"offset":25,"topic":"test_TwoPhaseCommit","partition":0}}
{"value":{"value":"27"},"metadata":{"offset":26,"topic":"test_TwoPhaseCommit","partition":0}}
{"value":{"value":"28"},"metadata":{"offset":27,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start invoke...
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 {"value":{"value":"26"},"metadata":{"offset":25,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 --{"value":{"value":"26"},"metadata":{"offset":25,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - objectNode-value:{"value":"26"}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ====执行SQL:insert into `test_TwoPhaseCommit` (`value`,`insert_time`) values (?,?)===
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - 要插入的数据:{"value":"26"}--2020-02-08 20:23:42.198
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start invoke...
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 {"value":{"value":"27"},"metadata":{"offset":26,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 --{"value":{"value":"27"},"metadata":{"offset":26,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - objectNode-value:{"value":"27"}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ====执行SQL:insert into `test_TwoPhaseCommit` (`value`,`insert_time`) values (?,?)===
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - 要插入的数据:{"value":"27"}--2020-02-08 20:23:42.21
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start invoke...
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 {"value":{"value":"28"},"metadata":{"offset":27,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 --{"value":{"value":"28"},"metadata":{"offset":27,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - objectNode-value:{"value":"28"}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ====执行SQL:insert into `test_TwoPhaseCommit` (`value`,`insert_time`) values (?,?)===
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - 要插入的数据:{"value":"28"}--2020-02-08 20:23:42.217
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start invoke...
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 {"value":{"value":"29"},"metadata":{"offset":28,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 --{"value":{"value":"29"},"metadata":{"offset":28,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - objectNode-value:{"value":"29"}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ====执行SQL:insert into `test_TwoPhaseCommit` (`value`,`insert_time`) values (?,?)===
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - 要插入的数据:{"value":"29"}--2020-02-08 20:23:42.222
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start invoke...
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 {"value":{"value":"30"},"metadata":{"offset":29,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ===>date:2020-02-08 20:23:42 --{"value":{"value":"30"},"metadata":{"offset":29,"topic":"test_TwoPhaseCommit","partition":0}}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - objectNode-value:{"value":"30"}
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - ====执行SQL:insert into `test_TwoPhaseCommit` (`value`,`insert_time`) values (?,?)===
[Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - 要插入的数据:{"value":"30"}--2020-02-08 20:23:42.227
{"value":{"value":"29"},"metadata":{"offset":28,"topic":"test_TwoPhaseCommit","partition":0}}
{"value":{"value":"30"},"metadata":{"offset":29,"topic":"test_TwoPhaseCommit","partition":0}}
[Checkpoint Timer] INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 1 @ 1581164623870 for job 53a5dcc154f25208468215e44cd4e629.
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start preCommit...
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start beginTransaction.......
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.DBConnectUtil - 获取连接:com.mysql.jdbc.JDBC4Connection@278447fe 成功...
[flink-akka.actor.default-dispatcher-3] INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 1 for job 53a5dcc154f25208468215e44cd4e629 (461479 bytes in 2955 ms).
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction - MySqlTwoPhaseCommitSink 0/1 - checkpoint 1 complete, committing transaction TransactionHolder{handle=com.mysql.jdbc.JDBC4Connection@74a7a112, transactionStartTime=1581164619994} from checkpoint 1
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start commit...
[Checkpoint Timer] INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 2 @ 1581164633869 for job 53a5dcc154f25208468215e44cd4e629.
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start preCommit...
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start beginTransaction.......
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.DBConnectUtil - 获取连接:com.mysql.jdbc.JDBC4Connection@95b1062 成功...
[flink-akka.actor.default-dispatcher-9] INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 2 for job 53a5dcc154f25208468215e44cd4e629 (388201 bytes in 320 ms).
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction - MySqlTwoPhaseCommitSink 0/1 - checkpoint 2 complete, committing transaction TransactionHolder{handle=com.mysql.jdbc.JDBC4Connection@278447fe, transactionStartTime=1581164624085} from checkpoint 2
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start commit...
[Checkpoint Timer] INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 3 @ 1581164643868 for job 53a5dcc154f25208468215e44cd4e629.
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start preCommit...
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start beginTransaction.......
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.DBConnectUtil - 获取连接:com.mysql.jdbc.JDBC4Connection@17174574 成功...
[flink-akka.actor.default-dispatcher-9] INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 3 for job 53a5dcc154f25208468215e44cd4e629 (388201 bytes in 137 ms).
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction - MySqlTwoPhaseCommitSink 0/1 - checkpoint 3 complete, committing transaction TransactionHolder{handle=com.mysql.jdbc.JDBC4Connection@95b1062, transactionStartTime=1581164633925} from checkpoint 3
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start commit...
[Checkpoint Timer] INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 4 @ 1581164653868 for job 53a5dcc154f25208468215e44cd4e629.
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start preCommit...
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start beginTransaction.......
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.DBConnectUtil - 获取连接:com.mysql.jdbc.JDBC4Connection@2330d50 成功...
[flink-akka.actor.default-dispatcher-12] INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 4 for job 53a5dcc154f25208468215e44cd4e629 (388201 bytes in 77 ms).
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction - MySqlTwoPhaseCommitSink 0/1 - checkpoint 4 complete, committing transaction TransactionHolder{handle=com.mysql.jdbc.JDBC4Connection@17174574, transactionStartTime=1581164643900} from checkpoint 4
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start commit...
[Checkpoint Timer] INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 5 @ 1581164663875 for job 53a5dcc154f25208468215e44cd4e629.
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start preCommit...
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start beginTransaction.......
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.DBConnectUtil - 获取连接:com.mysql.jdbc.JDBC4Connection@1c39cf1d 成功...
[flink-akka.actor.default-dispatcher-17] INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 5 for job 53a5dcc154f25208468215e44cd4e629 (388201 bytes in 675 ms).
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction - MySqlTwoPhaseCommitSink 0/1 - checkpoint 5 complete, committing transaction TransactionHolder{handle=com.mysql.jdbc.JDBC4Connection@2330d50, transactionStartTime=1581164653900} from checkpoint 5
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start commit...
[Checkpoint Timer] INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 6 @ 1581164673869 for job 53a5dcc154f25208468215e44cd4e629.
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start preCommit...
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start beginTransaction.......
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.DBConnectUtil - 获取连接:com.mysql.jdbc.JDBC4Connection@66ddf181 成功...
[flink-akka.actor.default-dispatcher-17] INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 6 for job 53a5dcc154f25208468215e44cd4e629 (388201 bytes in 228 ms).
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction - MySqlTwoPhaseCommitSink 0/1 - checkpoint 6 complete, committing transaction TransactionHolder{handle=com.mysql.jdbc.JDBC4Connection@1c39cf1d, transactionStartTime=1581164664254} from checkpoint 6
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start commit...
[Checkpoint Timer] INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 7 @ 1581164683876 for job 53a5dcc154f25208468215e44cd4e629.
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start preCommit...
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start beginTransaction.......
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.DBConnectUtil - 获取连接:com.mysql.jdbc.JDBC4Connection@13ed850d 成功...
[flink-akka.actor.default-dispatcher-9] INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 7 for job 53a5dcc154f25208468215e44cd4e629 (388201 bytes in 144 ms).
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction - MySqlTwoPhaseCommitSink 0/1 - checkpoint 7 complete, committing transaction TransactionHolder{handle=com.mysql.jdbc.JDBC4Connection@66ddf181, transactionStartTime=1581164673976} from checkpoint 7
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start commit...
[Checkpoint Timer] INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 8 @ 1581164693868 for job 53a5dcc154f25208468215e44cd4e629.
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start preCommit...
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start beginTransaction.......
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.DBConnectUtil - 获取连接:com.mysql.jdbc.JDBC4Connection@709699a1 成功...
[flink-akka.actor.default-dispatcher-17] INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 8 for job 53a5dcc154f25208468215e44cd4e629 (388201 bytes in 206 ms).
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction - MySqlTwoPhaseCommitSink 0/1 - checkpoint 8 complete, committing transaction TransactionHolder{handle=com.mysql.jdbc.JDBC4Connection@13ed850d, transactionStartTime=1581164683918} from checkpoint 8
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start commit...
[Checkpoint Timer] INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 9 @ 1581164703868 for job 53a5dcc154f25208468215e44cd4e629.
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start preCommit...
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start beginTransaction.......
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.DBConnectUtil - 获取连接:com.mysql.jdbc.JDBC4Connection@4cd6b1ac 成功...
[flink-akka.actor.default-dispatcher-24] INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 9 for job 53a5dcc154f25208468215e44cd4e629 (388201 bytes in 157 ms).
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction - MySqlTwoPhaseCommitSink 0/1 - checkpoint 9 complete, committing transaction TransactionHolder{handle=com.mysql.jdbc.JDBC4Connection@709699a1, transactionStartTime=1581164693907} from checkpoint 9
[Async calls on Source: Custom Source -> (Sink: Print to Std. Out, Sink: MySqlTwoPhaseCommitSink) (1/1)] INFO com.huonan.twocommit.MySqlTwoPhaseCommitSink - start commit...

MySQL数据,Kafka没在写数据,MySQL数据不多不少

在测试用遇到问题,启动Flink应用程序报错

[flink-akka.actor.default-dispatcher-45] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Sink: MySqlTwoPhaseCommitSink (1/1) (22a08fab62aca0907f7554fa480f639c) switched from RUNNING to FAILED.
java.lang.Exception: Error while triggering checkpoint 32 for Source: Custom Source -> Sink: MySqlTwoPhaseCommitSink (1/1)
	at org.apache.flink.runtime.taskmanager.Task$1.run(Task.java:1174)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
	at java.util.concurrent.FutureTask.run(FutureTask.java)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.Exception: Could not perform checkpoint 32 for operator Source: Custom Source -> Sink: MySqlTwoPhaseCommitSink (1/1).
	at org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpoint(StreamTask.java:570)
	at org.apache.flink.streaming.runtime.tasks.SourceStreamTask.triggerCheckpoint(SourceStreamTask.java:116)
	at org.apache.flink.runtime.taskmanager.Task$1.run(Task.java:1163)
	... 6 more
Caused by: java.lang.Exception: Could not complete snapshot 32 for operator Source: Custom Source -> Sink: MySqlTwoPhaseCommitSink (1/1).
	at org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:422)
	at org.apache.flink.streaming.runtime.tasks.StreamTask$CheckpointingOperation.checkpointStreamOperator(StreamTask.java:1113)
	at org.apache.flink.streaming.runtime.tasks.StreamTask$CheckpointingOperation.executeCheckpointing(StreamTask.java:1055)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.checkpointState(StreamTask.java:729)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:641)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpoint(StreamTask.java:564)
	... 8 more
Caused by: com.esotericsoftware.kryo.KryoException: java.lang.UnsupportedOperationException
Serialization trace:
indexToMysqlCharset (com.mysql.jdbc.JDBC4Connection)

各种百度问题查找,最后发现是MySQL版本的问题
把5.1.34改为5.1.47就好了

 
            mysql
            mysql-connector-java

            5.1.47

详细代码

Flink-Kafka端到端精准一次处理:详细解读

你可能感兴趣的:(Flink)

nosql数据库技术与应用知识点皆过客，揽星河 NoSQL nosql 数据库大数据数据分析数据结构非关系型数据库
Nosql知识回顾大数据处理流程数据采集(flume、爬虫、传感器)数据存储(本门课程NoSQL所处的阶段)Hdfs、MongoDB、HBase等数据清洗(入仓)Hive等数据处理、分析(Spark、Flink等)数据可视化数据挖掘、机器学习应用(Python、SparkMLlib等)大数据时代存储的挑战(三高)高并发(同一时间很多人访问)高扩展(要求随时根据需求扩展存储)高效率(要求读写速度快)
全面指南：用户行为从前端数据采集到实时处理的最佳实践数字沉思营销流量运营系统架构前端内容运营大数据
引言在当今的数据驱动世界，实时数据采集和处理已经成为企业做出及时决策的重要手段。本文将详细介绍如何通过前端JavaScript代码采集用户行为数据、利用API和Kafka进行数据传输、通过Flink实时处理数据的完整流程。无论你是想提升产品体验还是做用户行为分析，这篇文章都将为你提供全面的解决方案。设计一个通用的ClickHouse表来存储用户事件时，需要考虑多种因素，包括事件类型、时间戳、用户信
详解 Flink 的常见部署方式文刀小桂 Flink flink 大数据
一、常见部署模式分类1.按是否依赖外部资源调度1.1Standalone模式独立模式(Standalone)是独立运行的，不依赖任何外部的资源管理平台，只需要运行所有Flink组件服务1.2Yarn模式Yarn模式是指客户端把Flink应用提交给Yarn的ResourceManager,Yarn的ResourceManager会在Yarn的NodeManager上创建容器。在这些容器上，Flink
大数据之flink与hive 星辰_mya 大数据 flink hive
其实吧我不太想写flink，因为线上经验确实不多，这也是我需要补的地方，没有条件创造条件，先来一篇吧flink：高性能低延迟流批一体的分布式计算框架基于事件时间对实时数据精准处理快速响应支持批处理，高效离线分析和数据挖掘数据仓库的引擎丰富数据源/接收器，集成多种数据存储格式和源，比较常见就是咱们今天的主题hive了checkpoint恢复机制，故障恢复快速恢复计算任务分布式弹性扩展，据业务灵活增加
Java中的大数据处理框架对比分析省赚客app开发者 java 开发语言
Java中的大数据处理框架对比分析大家好，我是微赚淘客系统3.0的小编，是个冬天不穿秋裤，天冷也要风度的程序猿！今天，我们将深入探讨Java中常用的大数据处理框架，并对它们进行对比分析。大数据处理框架是现代数据驱动应用的核心，它们帮助企业处理和分析海量数据，以提取有价值的信息。本文将重点介绍ApacheHadoop、ApacheSpark、ApacheFlink和ApacheStorm这四种流行的
一文搞懂 Flink Task 数据交互之数据写源码 mn_kw flink 交互 java
一文搞懂FlinkTask数据交互之数据写源码1.RecordWriterOutput2.RecordWriter3.数据分区器ChannelSelector4.数据输出模型ResultPartition5.子模型ResultSubpartition6.本地buffer池LocalBufferPool7.获取buffer8.将buffer添加到ResultSubpartitionFlink重要源码
概率图模型（PGM）综述医学影像处理概率图模型概率图模型综述
RefLink:http://www.sigvc.org/bbs/thread-728-1-1.htmlGraphicalModel的基本类型基本的GraphicalModel可以大致分为两个类别：贝叶斯网络(BayesianNetwork)和马尔可夫随机场(MarkovRandomField)。它们的主要区别在于采用不同类型的图来表达变量之间的关系：贝叶斯网络采用有向无环图(DirectedAc
Python基础知识进阶之正则表达式_头歌python正则表达式进阶前端陈萨龙程序员 python 学习面试
最后硬核资料：关注即可领取PPT模板、简历模板、行业经典书籍PDF。技术互助：技术群大佬指点迷津，你的问题可能不是问题，求资源在群里喊一声。面试题库：由技术群里的小伙伴们共同投稿，热乎的大厂面试真题，持续更新中。知识体系：含编程语言、算法、大数据生态圈组件（Mysql、Hive、Spark、Flink）、数据仓库、Python、前端等等。网上学习资料一大堆，但如果学到的知识不成体系，遇到问题时只是
大数据新视界 --大数据大厂之Flink强势崛起：大数据新视界的璀璨明珠青云交大数据新视界 Flink 大数据数据类型实时处理流处理框架对比应用场景数据处理大数据新视界数据库
亲爱的朋友们，热烈欢迎你们来到青云交的博客！能与你们在此邂逅，我满心欢喜，深感无比荣幸。在这个瞬息万变的时代，我们每个人都在苦苦追寻一处能让心灵安然栖息的港湾。而我的博客，正是这样一个温暖美好的所在。在这里，你们不仅能够收获既富有趣味又极为实用的内容知识，还可以毫无拘束地畅所欲言，尽情分享自己独特的见解。我真诚地期待着你们的到来，愿我们能在这片小小的天地里共同成长，共同进步。本博客的精华专栏：Ja
flink增量检查点降低状态依赖实现的详细步骤 goTsHgo Flink 大数据分布式 flink 大数据
增量检查点启动恢复的时间是很久的，业务上不能接受，所以可以通过降低状态依赖来减少恢复的时间。降低状态依赖尽可能减少状态的复杂性和依赖关系，通过拆分状态或将状态外部化到其他服务中，从而降低恢复的开销。实施措施：将状态分割为更小的单元，减少每次恢复的状态量。使用外部状态存储服务，减少Flink状态后端的负担。拆分状态和将状态外部化到其他服务可以帮助减少作业的状态依赖，从而降低恢复时间和复杂度。以下是详
flink table factory基础知识 loukey_j
一、概述在flink中很多组件都是TableFactory的子类。比如序列化，反序列化，tableSinkFactory,tableSourceFactory.TableFactory是用来创建序列化，反序列器，tableSource和tableSink的工厂。二、TableFactory源码在flink框架中，TableFactory的子类并不是程序员自己随心new出来的。flink的提供给程序
2024年最全使用Python求解方程_python解方程(1)，字节面试官迟到 2401_84569545 程序员 python 学习面试
最后硬核资料：关注即可领取PPT模板、简历模板、行业经典书籍PDF。技术互助：技术群大佬指点迷津，你的问题可能不是问题，求资源在群里喊一声。面试题库：由技术群里的小伙伴们共同投稿，热乎的大厂面试真题，持续更新中。知识体系：含编程语言、算法、大数据生态圈组件（Mysql、Hive、Spark、Flink）、数据仓库、Python、前端等等。网上学习资料一大堆，但如果学到的知识不成体系，遇到问题时只是
01-Flink安装部署及入门案例（仅供学习），音视频时代你还不会NDK开发小猪佩琪962 2024年程序员学习 flink 学习大数据
先自我介绍一下，小编浙江大学毕业，去过华为、字节跳动等大厂，目前阿里P7深知大多数程序员，想要提升技能，往往是自己摸索成长，但自己不成体系的自学效果低效又漫长，而且极易碰到天花板技术停滞不前！因此收集整理了一份《2024年最新大数据全套学习资料》，初衷也很简单，就是希望能够帮助到想自学提升又不知道该从何学起的朋友。既有适合小白学习的零基础资料，也有适合3年以上经验的小伙伴深入学习提升的进阶课程，涵
比较Spark与Flink 傲雪凌霜，松柏长青大数据后端 spark flink 大数据
ApacheSpark和ApacheFlink都是目前非常流行的大数据处理引擎，但它们在架构、处理模式、应用场景等方面有一些显著的区别。下面是二者的对比：1.处理模式Spark:主要支持批处理（BatchProcessing），也能通过SparkStreaming处理流式数据，但SparkStreaming本质上是通过微批（micro-batching）的方式处理流数据，延迟相对较高。SparkS
Apache Flink：实时流处理与批处理的统一框架小码快撩 flink 大数据
导语在大数据处理领域，流处理和批处理是两种主要的处理方式。然而，传统的系统通常将这两者视为独立的任务，需要不同的工具和框架来处理。ApacheFlink是一个开源的流处理框架，它打破了这种界限，提供了一个统一的平台来处理实时流数据和批处理数据。一、基本概念与架构ApacheFlink的基本概念与架构主要包括以下几个核心组成部分：基本概念1.流处理模型：无界流(UnboundedStreams):数
flink独立集群部署嘎子吱吱吱吱 flink hadoop linux
#flink独立集群部署说明安装环境三台服务器47.106.23.1（master）47.112.173.2（worker1）47.115.162.3（worker1）提前装好jdk和ssh,以下操作最好不要用root账号提前下载好flink的包并解压设置三台服务器之间ssh免密登录生成本机秘钥以47.106.23.1为例（其他两台参考本服务器）#生成本机秘钥cd;ssh-keygen-trsa-
Flink的时间与watermarks详解大数据技术与数仓
当我们在使用Flink的时候，避免不了要和时间(time)、水位线(watermarks)打交道，理解这些概念是开发分布式流处理应用的基础。那么Flink支持哪些时间语义？Flink是如何处理乱序事件的？什么是水位线？水位线是如何生成的？水位线的传播方式是什么？让我们带着这些问题来开始本文的内容。时间语义基本概念时间是Flink等流处理中最重要的概念之一，在Flink中Time可以分为三种：Eve
实时数仓之实时数仓架构(Hudi)(1) 2401_84164527 程序员架构
目前比较流行的实时数仓架构有两类，其中一类是以Flink+Doris为核心的实时数仓架构方案；另一类是以湖仓一体架构为核心的实时数仓架构方案。本文针对Flink+Hudi湖仓一体架构进行介绍，这套架构的特点是可以基于一套数据完全实现Lambda架构。实时数仓架构图如下：技术框架Kafka：用于接入数据源；FlinkCDC：如果直接接入业务数据源可以考虑CDC方式，如果通过Kafka缓冲接入业务数据
2024年大数据最新实时数仓之实时数仓架构(Hudi) 2401_84185556 程序员大数据架构
技术框架Kafka：用于接入数据源；FlinkCDC：如果直接接入业务数据源可以考虑CDC方式，如果通过Kafka缓冲接入业务数据可以忽略;Flink：用于数据ETL，包括接入数据、处理数据及输出数据全链路数据计算任务；Spark：用于数据ETL，包括处理数据及输出数据全链路数据计算任务；Hudi：湖仓一体数据管理框架，用来管理模型数据，包括ODS/DWD/DWS/DIM/ADS等；Doris：O
实时数仓之实时数仓架构(Hudi)(1)，2024年最新熬夜整理华为最新大数据开发笔试题 2401_84181221 程序员架构大数据
+Hudi：湖仓一体数据管理框架，用来管理模型数据，包括ODS/DWD/DWS/DIM/ADS等；+Doris：OLAP引擎，同步数仓结果模型，对外提供数据服务支持；+Hbase：用来存储维表信息，维表数据来源一部分有Flink加工实时写入，另一部分是从Spark任务生产，其主要作用用来支持FlinkETL处理过程中的LookupJoin功能。这里选用Hbase原因主要因为Table的HbaseC
Flink - CEP kikiki1
Hadoop3.2集群新版本的搭建详细讲解过程，从下面第一张官方的图来看，最新版是3.2，所以大猪将使用3.2的版本来演示，过程中遇到的坑留给自己，把路留给你们，IT之路还有大猪。大猪为了把文章压缩极简方便小伙伴阅读，将使用root帐号进行所有操作。准备两台主机10.211.55.11、10.211.55.12对应的hostname为m1.example.com、m2.example.com具体命
chapter01 Java语言概述知识点Note 月下绯烟 Java java 开发语言
JavaSEJavaEEJavaME大数据Java基础常用技术栈mysqlJDBCSSMspring+springmvc+mybatisLinuxnacosHadoopFlinkJAVAEE消息队列rabbitMQdocker数据库redisspringbootspringcloudsshstruts+spring+hibernate过时技术栈很少用JAVA虚拟机jvm分布式微服务高并发常见dos
【无标题】大数据之批处理，流处理，批流一体概念数字天下大数据
批处理批处理是将一定量的数据集合在一起，形成一个数据批次，然后对这个批次中的数据进行处理。Spark和Flink都支持批处理，其中Spark使用的是批处理模型，即将一批数据一次性读入内存，然后对其进行处理，处理完成后再将结果写入磁盘。Flink也支持批处理，但使用的是基于流处理的批处理模式，即将一批数据分成多个数据流进行处理，可以实现更高效的内存管理和更低的延迟。流处理流式处理是一种将数据流式地处
python flink_《Flink官方文档》Python 编程指南测试版 weixin_39846361 python flink
原文链接译者：hjjxd校对：清英Flink中的分析程序实现了对数据集的某些操作(例如，数据过滤，映射，合并，分组)。这些数据最初来源于特定的数据源(例如来自于读文件或数据集合)。操作执行的结果通过数据池以写入数据到(分布式)文件系统或标准输出(例如命令行终端)的形式返回。Flink程序可以运行在不同的环境中，既能够独立运行，也可以嵌入到其他程序中运行。程序可以运行在本地的JVM上，也可以运行在服
flink---window 搞数据的小杰 flink 大数据
Window介绍DataStream:https://nightlies.apache.org/flink/flink-docs-release-1.17/zh/docs/dev/datastream/operators/windows/SQL:https://nightlies.apache.org/flink/flink-docs-release-1.17/zh/docs/dev/table/
Flink(1.13) 的window机制(一) 万事万物
窗口概述在流处理应用中，数据是连续不断的，因此我们不可能等到所有数据都到了才开始处理。当然我们可以每来一个消息就处理一次，但是有时我们需要做一些聚合类的处理，例如：在过去的1分钟内有多少用户点击了我们的网页。在这种情况下，我们必须定义一个窗口，用来收集最近一分钟内的数据，并对这个窗口内的数据进行计算。流式计算是一种被设计用于处理无限数据集的数据处理引擎，而无限数据集是指一种不断增长的本质上无限的数
pyflink 自定义函数 scan724 Flink实时计算 python 开发语言
frompyflink.datastreamimportStreamExecutionEnvironmentfrompyflink.common.typeinfoimportTypesfrompyflink.datastreamimportStreamExecutionEnvironmentfrompyflink.tableimportStreamTableEnvironmentfrompyfli
flink 问题记录 Jhon_yh flink flink hadoop 大数据
文章目录1.Causedby:java.lang.UnsatisfiedLinkError:org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSums(IILjava/nio/ByteBuffer;ILjava/nio/ByteBuffer;IILjava/lang/String;JZ)V原因java.util.concurrent.Ex
Pyflink教程(三)：自定义函数 yuxj记录学习学习笔记学习 pyflink
该文章例子pyflink环境是apache-flink==1.13.6Python自定义函数是PyFlinkTableAPI中最重要的功能之一，其允许用户在PyFlinkTableAPI中使用Python语言开发的自定义函数，极大地拓宽了PythonTableAPI的使用范围。简单来说就是有的业务逻辑和需求是sql语句满足不了或太麻烦的，需要用过函数来实现。PythonUDFPythonUDF，即
pyflink 滚动窗口实例菜鸟社长菜鸟的大数据进阶之路大数据进阶之路 kafka big data python flink
写在前头：更多大数据相关精彩内容请进我的知识星球，每周定期更新正篇技术路线：模拟kafka生产者发送数据——>flink对kafka数据实时计算处理——>处理后的数据发送到kafka1、模拟客流数据的生产者，参考https://blog.csdn.net/qq_22611181/article/details/1199002502、flink聚合操作原理介绍，参考https://blog.csdn
ASM系列四利用Method 组件动态注入方法逻辑 lijingyao8206 字节码技术 jvm AOP 动态代理 ASM
这篇继续结合例子来深入了解下Method组件动态变更方法字节码的实现。通过前面一篇，知道ClassVisitor 的visitMethod()方法可以返回一个MethodVisitor的实例。那么我们也基本可以知道，同ClassVisitor改变类成员一样，MethodVIsistor如果需要改变方法成员，注入逻辑，也可以
java编程思想 --内部类百合不是茶 java 内部类匿名内部类
内部类;了解外部类并能与之通信内部类写出来的代码更加整洁与优雅 1,内部类的创建内部类是创建在类中的 package com.wj.InsideClass; /* * 内部类的创建 */ public class CreateInsideClass { public CreateInsideClass(
web.xml报错 crabdave web.xml
web.xml报错 The content of element type "web-app" must match "(icon?,display- name?,description?,distributable?,context-param*,filter*,filter-mapping*,listener*,servlet*,s
泛型类的自定义麦田的设计者 java android 泛型
为什么要定义泛型类，当类中要操作的引用数据类型不确定的时候。采用泛型类，完成扩展。例如有一个学生类 Student{ Student(){ System.out.println("I'm a student....."); } } 有一个老师类
CSS清除浮动的4中方法 IT独行者 JavaScript UI css
清除浮动这个问题，做前端的应该再熟悉不过了，咱是个新人，所以还是记个笔记，做个积累，努力学习向大神靠近。CSS清除浮动的方法网上一搜，大概有N多种，用过几种，说下个人感受。 1、结尾处加空div标签 clear:both 1 2 3 4 .div 1 { background : #000080 ; border : 1px s
Cygwin使用windows的jdk 配置方法 _wy_ jdk windows cygwin
1.[vim /etc/profile] JAVA_HOME="/cgydrive/d/Java/jdk1.6.0_43" (windows下jdk路径为D:\Java\jdk1.6.0_43) PATH="$JAVA_HOME/bin:${PATH}" CLAS
linux下安装maven 无量 maven linux 安装
Linux下安装maven(转) 1.首先到Maven官网下载安装文件，目前最新版本为3.0.3，下载文件为 apache-maven-3.0.3-bin.tar.gz，下载可以使用wget命令； 2.进入下载文件夹，找到下载的文件，运行如下命令解压 tar -xvf apache-maven-2.2.1-bin.tar.gz 解压后的文件夹
tomcat的https 配置,syslog-ng配置 aichenglong tomcat http跳转到https syslong-ng配置 syslog配置
1) tomcat配置https,以及http自动跳转到https的配置 1)TOMCAT_HOME目录下生成密钥(keytool是jdk中的命令) keytool -genkey -alias tomcat -keyalg RSA -keypass changeit -storepass changeit
关于领号活动总结 alafqq 活动
关于某彩票活动的总结具体需求，每个用户进活动页面，领取一个号码，1000中的一个；活动要求 1，随机性，一定要有随机性； 2，最少中奖概率，如果注数为3200注，则最多中4注 3，效率问题，（不能每个人来都产生一个随机数，这样效率不高）； 4，支持断电（仍然从下一个开始），重启服务；（存数据库有点大材小用，因此不能存放在数据库）解决方案 1，事先产生随机数1000个，并打
java数据结构冒泡排序的遍历与排序百合不是茶 java
java的冒泡排序是一种简单的排序规则冒泡排序的原理：比较两个相邻的数，首先将最大的排在第一个，第二次比较第二个，此后一样；针对所有的元素重复以上的步骤，除了最后一个例题；将int array[]
JS检查输入框输入的是否是数字的一种校验方法 bijian1013 js
如下是JS检查输入框输入的是否是数字的一种校验方法： <form method=post target="_blank"> 数字：<input type="text" name=num onkeypress="checkNum(this.form)"><br> </form>
Test注解的两个属性：expected和timeout bijian1013 java JUnit expected timeout
JUnit4：Test文档中的解释：　　The Test annotation supports two optional parameters. 　　The first, expected, declares that a test method should throw an exception. 　　If it doesn't throw an exception or if it
[Gson二]继承关系的POJO的反序列化 bit1129 POJO
父类 package inheritance.test2; import java.util.Map; public class Model { private String field1; private String field2; private Map<String, String> infoMap
【Spark八十四】Spark零碎知识点记录 bit1129 spark
1. ShuffleMapTask的shuffle数据在什么地方记录到MapOutputTracker中的 ShuffleMapTask的runTask方法负责写数据到shuffle map文件中。当任务执行完成成功，DAGScheduler会收到通知，在DAGScheduler的handleTaskCompletion方法中完成记录到MapOutputTracker中
WAS各种脚本作用大全 ronin47 WAS 脚本
　　　http://www.ibm.com/developerworks/cn/websphere/library/samples/SampleScripts.html 　　　无意中，在WAS官网上发现的各种脚本作用，感觉很有作用，先与各位分享一下　　　获取下载这些示例 jacl 和 Jython 脚本可用于在 WebSphere Application Server 的不同版本中自
java-12.求 1+2+3+..n不能使用乘除法、 for 、 while 、 if 、 else 、 switch 、 case 等关键字以及条件判断语句 bylijinnan switch
借鉴网上的思路，用java实现： public class NoIfWhile { /** * @param args * * find x=1+2+3+....n */ public static void main(String[] args) { int n=10; int re=find(n); System.o
Netty源码学习-ObjectEncoder和ObjectDecoder bylijinnan java netty
Netty中传递对象的思路很直观： Netty中数据的传递是基于ChannelBuffer（也就是byte[]）；那把对象序列化为字节流，就可以在Netty中传递对象了相应的从ChannelBuffer恢复对象，就是反序列化的过程 Netty已经封装好ObjectEncoder和ObjectDecoder 先看ObjectEncoder ObjectEncoder是往外发送
spring 定时任务中cronExpression表达式含义 chicony cronExpression
一个cron表达式有6个必选的元素和一个可选的元素，各个元素之间是以空格分隔的，从左至右，这些元素的含义如下表所示：代表含义是否必须允许的取值范围 &nb
Nutz配置Jndi ctrain JNDI
1、使用JNDI获取指定资源： var ioc = { dao : { type :"org.nutz.dao.impl.NutDao", args : [ {jndi :"jdbc/dataSource"} ] } } 以上方法,仅需要在容器中配置好数据源,注入到NutDao即可.
解决 /bin/sh^M: bad interpreter: No such file or directory daizj shell
在Linux中执行.sh脚本，异常/bin/sh^M: bad interpreter: No such file or directory。分析：这是不同系统编码格式引起的：在windows系统中编辑的.sh文件可能有不可见字符，所以在Linux系统下执行会报以上异常信息。解决： 1）在windows下转换：利用一些编辑器如UltraEdit或EditPlus等工具
[转]for 循环为何可恨？ dcj3sjt126com 程序员读书
Java的闭包(Closure)特征最近成为了一个热门话题。一些精英正在起草一份议案，要在Java将来的版本中加入闭包特征。然而，提议中的闭包语法以及语言上的这种扩充受到了众多Java程序员的猛烈抨击。不久前，出版过数十本编程书籍的大作家Elliotte Rusty Harold发表了对Java中闭包的价值的质疑。尤其是他问道“for 循环为何可恨？”[http://ju
Android实用小技巧 dcj3sjt126com android
1、去掉所有Activity界面的标题栏　　修改AndroidManifest.xml 　　在application 标签中添加android:theme="@android:style/Theme.NoTitleBar" 2、去掉所有Activity界面的TitleBar 和StatusBar 　　修改AndroidManifes
Oracle 复习笔记之序列 eksliang Oracle 序列 sequence Oracle sequence
转载请出自出处：http://eksliang.iteye.com/blog/2098859 1.序列的作用序列是用于生成唯一、连续序号的对象一般用序列来充当数据库表的主键值 2.创建序列语法如下： create sequence s_emp start with 1 --开始值 increment by 1 --増长值 maxval
有“品”的程序员 gongmeitao 工作
完美程序员的10种品质　　完美程序员的每种品质都有一个范围，这个范围取决于具体的问题和背景。没有能解决所有问题的完美程序员（至少在我们这个星球上），并且对于特定问题，完美程序员应该具有以下品质：　　1. 才智非凡- 能够理解问题、能够用清晰可读的代码翻译并表达想法、善于分析并且逻辑思维能力强（范围：用简单方式解决复杂问题）　　
使用KeleyiSQLHelper类进行分页查询 hvt sql .net C#asp.net hovertree
本文适用于sql server单主键表或者视图进行分页查询，支持多字段排序。KeleyiSQLHelper类的最新代码请到http://hovertree.codeplex.com/SourceControl/latest下载整个解决方案源代码查看。或者直接在线查看类的代码：http://hovertree.codeplex.com/SourceControl/latest#HoverTree.D
SVG 教程（三）圆形，椭圆，直线天梯梦 svg
SVG <circle> SVG 圆形 - <circle> <circle> 标签可用来创建一个圆：下面是SVG代码： <svg xmlns="http://www.w3.org/2000/svg" version="1.1"> <circle cx="100" c
链表栈 luyulong java 数据结构
public class Node { private Object object; private Node next; public Node() { this.next = null; this.object = null; } public Object getObject() { return object; } public
基础数据结构和算法十：2-3 search tree sunwinner Algorithm 2-3 search tree
Binary search tree works well for a wide variety of applications, but they have poor worst-case performance. Now we introduce a type of binary search tree where costs are guaranteed to be loga
spring配置定时任务 stunizhengjia spring timer
最近因工作的需要，用到了spring的定时任务的功能,觉得spring还是很智能化的,只需要配置一下配置文件就可以了,在此记录一下，以便以后用到： //------------------------定时任务调用的方法------------------------------ /** * 存储过程定时器 */ publi
ITeye 8月技术图书有奖试读获奖名单公布 ITeye管理员活动
ITeye携手博文视点举办的8月技术图书有奖试读活动已圆满结束，非常感谢广大用户对本次活动的关注与参与。 8月试读活动回顾： http://webmaster.iteye.com/blog/2102830 本次技术图书试读活动的优秀奖获奖名单及相应作品如下（优秀文章有很多，但名额有限，没获奖并不代表不优秀）：《跨终端Web》 gleams：http

Flink之CheckPoint 构建 真正的End-to-End Exactly-Once