Apache Flink
称为终极流式框架
,不仅仅提供高吞吐、低延迟和Exactly-Once语义的实时计算能力,还提供了基于流式引擎处理批量数据的计算能力,真正意义上实现了批流统一
,无疑是继Spark和Storm的后起之秀。
但是刚刚入门Flink,会接触到水印
或水位线
陌生的技术词汇,但到底什么是水位线,又给Apache Flink蒙上了一层神秘的面纱。这里就为大家揭开Watermark的神秘面纱。
在流式数据处理中,Flink根据时间产生的位置不同,将时间区分为三种时间语义,分别为事件时间、事件接入时间和事件处理时间。
事件时间
,即事件行为发生的时间,如系统终端用户注册时间、订单下单时间以及订单支付时间等,决定了事件真实产生时间。
事件接入时间
,或摄入时间
,即数据接入Flink系统,在DataSource接入时生成的接入时间。
处理时间
,数据通过各个算子实例执行转换操作过程,算子实例所在系统的时间即为数据处理时间。
在Flink中,默认情况下使用的是Process Time时间语义,如果用户选择使用Event Time或者Ingestion Time语义, 则需要在创建的StreamExecutionEnvironment
中调用setStreamTimeCharacteristic()
方法设定系统的时间概念,
// 使用EventTime
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)
// 使用IngestionTime
env.setStreamTimeCharacteristic(TimeCharacteristic.IngestionTime)
使用EventTime
时间语义处理流式数据时,数据从Event
产生,流经Source
,再到Operator
,这中间需要一定的时间。理论情况下,数据是按照EventTime先后顺序传输到Operator
进行处理;但是也不排除由于网络延迟
、消息积压
和背压
等原因而导致乱序
情况;特别是使用Kafka的时候,无法保证多分区间数据的顺序。因此,在进行Window计算的时候,不能无限期地等下去,必须有机制保证在特定的时间后, 触发Window进行计算,即这个机制就是Watermark(水位线)。
直译为水印
,本质是时间戳
,能一定程度上解决数据乱序或者延迟到达问题。
Watermark = 当前窗口最大的事件时间 - 最大允许数据延迟的时间/乱序时间
最大允许数据延迟的时间设置
// 设置时间语义
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
// 指定水印分配策略
ds.assignTimestampsAndWatermarks(
// 注意:WatermarkStrategy为Flink 1.11版本提供
// 参数为最大延迟时间,或者最大无序度,或者最大乱序时间。
// 值maxOutOfOrderness=2s
WatermarkStrategy.<Event>forBoundedOutOfOrderness(Duration.ofSeconds(2))
// 指定事件时间数据
.withTimestamAssigner(e, timestamp) -> e.getEventTime())
);
BoundedOutOfOrdernessWatermarks
源码分析
@Public
public class BoundedOutOfOrdernessWatermarks<T> implements WatermarkGenerator<T> {
// 当前窗口最大的事件时间
private long maxTimestamp;
// 窗口最大允许的延误时间
private final long outOfOrdernessMillis;
public BoundedOutOfOrdernessWatermarks(Duration maxOutOfOrderness) {
// 延误时间, 以毫秒表示
this.outOfOrdernessMillis = maxOutOfOrderness.toMillis();
// 刚开始保证时间为最小
this.maxTimestamp = Long.MIN_VALUE + outOfOrdernessMillis + 1;
}
/**
* 所有事件均会调用该方法,不限于该窗口数据
* @param event 时间
* @param eventTimestamp 事件时间
* @param out Watermark输出器
*/
@Override
public void onEvent(T event, long eventTimestamp, WatermarkOutput out){
// 计算窗口最大的事件时间,保证窗口水印单调递增
maxTimestamp = Math.max(maxTimestamp, eventTimestamp);
}
@Override
public void onPeriodicEmit(WatermarkOutput output) {
// 水印 = 当前窗口最大事件时间 - 最大延误时间 - 1
// 水印需要减一的原因为:窗口为左闭右开的问题
output.emitWatermark(new Watermark(maxTimestamp - outOfOrdernessMillis - 1));
}
}
Watermark >= 窗口结束时间
Watermark = 当前窗口最大的事件时间 - 最大允许数据延迟的时间/乱序时间
Watermark = 当前窗口最大的事件时间 - 最大允许数据延迟的时间/乱序时间 >= 窗口结束时间
当前窗口最大的事件时间 >= 窗口结束时间 + 最大允许数据延迟的时间/乱序时间
在Apache Flink的窗口处理过程中,如果时间超过窗口最大结束时间,将触发数据的计算操作(如汇总、分组等)。但是,对于乱序数据来说,很容易错过窗口计算时间,导致数据丢失。而应用水位线(Watermark)机制,能一定程度上解决数据乱序或者延迟到达问题。
图示,当事件流数据C到达时,事件流数据C的时间超过窗口X的结束时间,因此窗口X将触发计算,并且新建窗口U接收事件流数据C。当事件流数据D和E接入时,由于窗口X已被触发计算,所以事件流数据D和E将丢失。
图示窗口加入Watermark计算,当事件流数据C到达时,Watermark为10:09:00但是小于窗口X的结束时间,没达到窗口X的计算条件,不触发窗口X计算。与此同时,新建窗口U接收事件流数据C。当事件流数据D/E到达时,窗口X还未触发计算,于是事件流数据D/E加入到窗口X,一定程度解决了2秒内数据乱序问题。当事件流数据F到达的时候,Watermark值为10:10:00且大于等于窗口X的结束时间,达到窗口X的计算条件,触发窗口X计算。
标点水位线
,通过数据流中某些特殊标记事件时间触发生成新的水位线。这种方式下,窗口的触发与时间无关,而是决定于何时收到标记事件。
在实际的生产中Punctuated方式在TPS很高的场景下会产生大量的Watermark在一定程度上对下游算子造成压力,所以只有在实时性要求非常高的场景才会选择Punctuated的方式进行Watermark的生成。
周期性水位线
,系统会周期性的(一定时间间隔)产生一个Watermark。水位线提升的时间间隔是由用户设置的,在两次水位线提升时隔内会有一部分消息流入,用户可以根据这部分数据来计算出新的水位线。
在实际的生产中Periodic的方式必须结合时间和积累条数两个维度继续周期性产生Watermark,否则在极端情况下会有很大的延时。
举个例子,最简单的水位线算法就是取目前为止最大的事件时间,然而这种方式比较暴力,对乱序事件的容忍程度比较低,容易出现大量迟到事件。
public class WatermarkMain {
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
// 基于Watermark事件时间的窗口计算
env.getConfig().setAutoWatermarkInterval(5000);
// 获取事件流数据
DataStreamSource<OrderEvent> eventDataStream = env.addSource(new SourceFunction<OrderEvent>() {
private static final long serialVersionUID = 5652749729728486680L;
// 开关
private boolean switchFlag = true;
@Override
public void run(SourceContext<OrderEvent> ctx) throws Exception {
Random random = new Random();
while (switchFlag) {
String orderId = UUID.randomUUID().toString().replaceAll("-", "");
int userId = random.nextInt(2);
int money = random.nextInt(100);
long eventTime = System.currentTimeMillis() - random.nextInt(5) * 1000;
OrderEvent orderEvent = new OrderEvent(orderId, userId, money, eventTime);
System.out.println("data: " + orderEvent);
// 发送元素
ctx.collect(orderEvent);
// 休眠1s
TimeUnit.SECONDS.sleep(1);
}
}
@Override
public void cancel() {
switchFlag = false;
}
});
// 为事件流数据添加Watermark和指定事件时间
// SingleOutputStreamOperator eventWatermarkDataStream
// = eventDataStream.assignTimestampsAndWatermarks(
// // 设置最大允许延误时间为3s
// WatermarkStrategy.forBoundedOutOfOrderness(Duration.ofSeconds(3))
// // 设置时间戳数据
// .withTimestampAssigner((e, timestamp) -> e.getEventTime())
// );
SingleOutputStreamOperator<OrderEvent> eventWatermarkDataStream = eventDataStream
.assignTimestampsAndWatermarks(
new WatermarkStrategy<OrderEvent>() {
@Override
public WatermarkGenerator<OrderEvent> createWatermarkGenerator(
WatermarkGeneratorSupplier.Context ctx
) {
return new WatermarkGenerator<OrderEvent>() {
/** 最大允许延迟时间 */
private final int outOfOrdernessMills = 3000;
/** 用户ID **/
private Integer userId;
/** 事件时间 **/
private Long eventTime;
/** 事件最大时间戳 */
private Long maxTimestamp = Long.MIN_VALUE + outOfOrdernessMills + 1;
// 时间格式化
private FastDateFormat df = FastDateFormat.getInstance("HH:mm:ss");
@Override
public void onEvent(OrderEvent event, long eventTimestamp, WatermarkOutput output) {
this.userId = event.userId;
this.eventTime = event.eventTime;
maxTimestamp = Math.max(maxTimestamp, eventTimestamp);
System.out.println("watermark on event: " + event);
}
@Override
public void onPeriodicEmit(WatermarkOutput out) {
Watermark watermark = new Watermark(maxTimestamp - outOfOrdernessMills - 1);
String note = String.format("watermark emit key:%s current time:%s " +
"event time:%s watermark:%s",
userId, df.format(System.currentTimeMillis()), df.format(eventTime),
df.format(maxTimestamp - outOfOrdernessMills - 1));
System.out.println(note);
out.emitWatermark(watermark);
}
};
}
}.withTimestampAssigner((e, timestamp) -> e.getEventTime()));
// 添加窗口计算
SingleOutputStreamOperator<String> outDataStream = eventWatermarkDataStream.keyBy(OrderEvent::getUserId)
// 设定滚动窗口
.window(TumblingEventTimeWindows.of(Time.seconds(5)))
// 指定窗口应用函数
.apply(new WindowFunction<OrderEvent, String, Integer, TimeWindow>() {
private static final long serialVersionUID = 7034105248794615763L;
// 时间格式化
private FastDateFormat df = FastDateFormat.getInstance("HH:mm:ss");
@Override
public void apply(Integer key, TimeWindow window, Iterable<OrderEvent> events,
Collector<String> out) throws Exception {
List<String> eventTimeList = Lists.newLinkedList();
for (OrderEvent event : events) {
String time = df.format(event.getEventTime());
eventTimeList.add(time);
}
String windowStartTime = df.format(window.getStart());
String windowEndTime = df.format(window.getEnd());
String rs = String.format("key:%s window:[%s,%s) window event times:%s",
key, windowStartTime, windowEndTime, eventTimeList.toString());
out.collect(rs);
}
});
outDataStream.print("WaterMarkResult::");
env.execute("WatermarkMain");
}
@Data
@NoArgsConstructor
@AllArgsConstructor
public static class OrderEvent implements Serializable {
private static final long serialVersionUID = 2082940433103599734L;
/** 订单ID */
private String orderId;
/** 用户ID */
private Integer userId;
/** 金额 */
private Integer money;
/** 事件时间 */
private Long eventTime;
}
}
打印结果
data: WatermarkMain.OrderEvent(orderId=9836bbbbc7e044b5b189f18e88bf9b07, userId=1, money=23, eventTime=1653442790579)
watermark on event: WatermarkMain.OrderEvent(orderId=9836bbbbc7e044b5b189f18e88bf9b07, userId=1, money=23, eventTime=1653442790579)
data: WatermarkMain.OrderEvent(orderId=6a8bbb3fc921410b92a06db59dcd2259, userId=0, money=86, eventTime=1653442787588)
watermark on event: WatermarkMain.OrderEvent(orderId=6a8bbb3fc921410b92a06db59dcd2259, userId=0, money=86, eventTime=1653442787588)
data: WatermarkMain.OrderEvent(orderId=b97f05f6ab054ca285f700f25ad0e9e5, userId=0, money=78, eventTime=1653442790601)
watermark on event: WatermarkMain.OrderEvent(orderId=b97f05f6ab054ca285f700f25ad0e9e5, userId=0, money=78, eventTime=1653442790601)
data: WatermarkMain.OrderEvent(orderId=8fcf42f3c4dd4933b58f4d2ecd893f71, userId=0, money=99, eventTime=1653442790612)
watermark on event: WatermarkMain.OrderEvent(orderId=8fcf42f3c4dd4933b58f4d2ecd893f71, userId=0, money=99, eventTime=1653442790612)
data: WatermarkMain.OrderEvent(orderId=7634bc064aff46249a0da49256d4525b, userId=0, money=2, eventTime=1653442792622)
watermark on event: WatermarkMain.OrderEvent(orderId=7634bc064aff46249a0da49256d4525b, userId=0, money=2, eventTime=1653442792622)
watermark emit key:0 current time:09:39:55 event time:09:39:52 watermark:09:39:49
data: WatermarkMain.OrderEvent(orderId=b406777faac5492fbf95410c27d4b06c, userId=0, money=47, eventTime=1653442792631)
watermark on event: WatermarkMain.OrderEvent(orderId=b406777faac5492fbf95410c27d4b06c, userId=0, money=47, eventTime=1653442792631)
data: WatermarkMain.OrderEvent(orderId=b8a66d2723bc4900a27cc151f7c3f8aa, userId=0, money=58, eventTime=1653442792643)
watermark on event: WatermarkMain.OrderEvent(orderId=b8a66d2723bc4900a27cc151f7c3f8aa, userId=0, money=58, eventTime=1653442792643)
data: WatermarkMain.OrderEvent(orderId=2ffa652b9a1b4b4e8a4dcf340627bf14, userId=1, money=97, eventTime=1653442796643)
watermark on event: WatermarkMain.OrderEvent(orderId=2ffa652b9a1b4b4e8a4dcf340627bf14, userId=1, money=97, eventTime=1653442796643)
data: WatermarkMain.OrderEvent(orderId=f476012749194a789c3b4ad3cf18af3a, userId=0, money=42, eventTime=1653442797652)
watermark on event: WatermarkMain.OrderEvent(orderId=f476012749194a789c3b4ad3cf18af3a, userId=0, money=42, eventTime=1653442797652)
data: WatermarkMain.OrderEvent(orderId=dec3ab891c734f50a97c0eb2d48ec880, userId=1, money=30, eventTime=1653442799664)
watermark on event: WatermarkMain.OrderEvent(orderId=dec3ab891c734f50a97c0eb2d48ec880, userId=1, money=30, eventTime=1653442799664)
watermark emit key:1 current time:09:40:00 event time:09:39:59 watermark:09:39:56
WaterMarkResult:::6> key:0 window:[09:39:45,09:39:50) window event times:[09:39:47]
WaterMarkResult:::6> key:1 window:[09:39:50,09:39:55) window event times:[09:39:50]
WaterMarkResult:::6> key:0 window:[09:39:50,09:39:55) window event times:[09:39:50, 09:39:50, 09:39:52, 09:39:52, 09:39:52]
data: WatermarkMain.OrderEvent(orderId=398a0e808d264a9b940041b7e4989cc1, userId=0, money=61, eventTime=1653442797669)
watermark on event: WatermarkMain.OrderEvent(orderId=398a0e808d264a9b940041b7e4989cc1, userId=0, money=61, eventTime=1653442797669)
data: WatermarkMain.OrderEvent(orderId=56c03f14c5d544cc9465456f8909b756, userId=0, money=77, eventTime=1653442798673)
watermark on event: WatermarkMain.OrderEvent(orderId=56c03f14c5d544cc9465456f8909b756, userId=0, money=77, eventTime=1653442798673)
data: WatermarkMain.OrderEvent(orderId=e820f9c4b24349c9898cd9d160cc6df8, userId=1, money=66, eventTime=1653442800675)
watermark on event: WatermarkMain.OrderEvent(orderId=e820f9c4b24349c9898cd9d160cc6df8, userId=1, money=66, eventTime=1653442800675)
data: WatermarkMain.OrderEvent(orderId=eab266cfff4348efaef06cf3c796b52b, userId=1, money=33, eventTime=1653442800681)
watermark on event: WatermarkMain.OrderEvent(orderId=eab266cfff4348efaef06cf3c796b52b, userId=1, money=33, eventTime=1653442800681)
data: WatermarkMain.OrderEvent(orderId=7d15b0698b514e65b5216878c242c8f6, userId=0, money=18, eventTime=1653442801693)
watermark on event: WatermarkMain.OrderEvent(orderId=7d15b0698b514e65b5216878c242c8f6, userId=0, money=18, eventTime=1653442801693)
watermark emit key:0 current time:09:40:05 event time:09:40:01 watermark:09:39:58
data: WatermarkMain.OrderEvent(orderId=0dd465031b6849adbacd269080591424, userId=0, money=75, eventTime=1653442803705)
watermark on event: WatermarkMain.OrderEvent(orderId=0dd465031b6849adbacd269080591424, userId=0, money=75, eventTime=1653442803705)
data: WatermarkMain.OrderEvent(orderId=56f691902fdf4ab79e149e2329b417c1, userId=0, money=34, eventTime=1653442806705)
watermark on event: WatermarkMain.OrderEvent(orderId=56f691902fdf4ab79e149e2329b417c1, userId=0, money=34, eventTime=1653442806705)
data: WatermarkMain.OrderEvent(orderId=9c0b07f89dc446cba93e30dde2520420, userId=1, money=46, eventTime=1653442805715)
watermark on event: WatermarkMain.OrderEvent(orderId=9c0b07f89dc446cba93e30dde2520420, userId=1, money=46, eventTime=1653442805715)
data: WatermarkMain.OrderEvent(orderId=bd6fcb15f48d4a75b9029119a868d75f, userId=0, money=92, eventTime=1653442804726)
watermark on event: WatermarkMain.OrderEvent(orderId=bd6fcb15f48d4a75b9029119a868d75f, userId=0, money=92, eventTime=1653442804726)
data: WatermarkMain.OrderEvent(orderId=32d1062edd0d4af39c4fa18f42034cb8, userId=1, money=56, eventTime=1653442806738)
watermark on event: WatermarkMain.OrderEvent(orderId=32d1062edd0d4af39c4fa18f42034cb8, userId=1, money=56, eventTime=1653442806738)
watermark emit key:1 current time:09:40:10 event time:09:40:06 watermark:09:40:03
WaterMarkResult:::6> key:1 window:[09:39:55,09:40:00) window event times:[09:39:56, 09:39:59]
WaterMarkResult:::6> key:0 window:[09:39:55,09:40:00) window event times:[09:39:57, 09:39:57, 09:39:58]
data: WatermarkMain.OrderEvent(orderId=ae8870f9afbc4bff954f6f9c9d663145, userId=1, money=16, eventTime=1653442809751)
watermark on event: WatermarkMain.OrderEvent(orderId=ae8870f9afbc4bff954f6f9c9d663145, userId=1, money=16, eventTime=1653442809751)
data: WatermarkMain.OrderEvent(orderId=41dfe734ac0f498bb51bb8f8e7d4ffb3, userId=1, money=17, eventTime=1653442810763)
watermark on event: WatermarkMain.OrderEvent(orderId=41dfe734ac0f498bb51bb8f8e7d4ffb3, userId=1, money=17, eventTime=1653442810763)
data: WatermarkMain.OrderEvent(orderId=2b63d9f2d1d8406f9cb5baf4e12ea879, userId=0, money=51, eventTime=1653442810777)
watermark on event: WatermarkMain.OrderEvent(orderId=2b63d9f2d1d8406f9cb5baf4e12ea879, userId=0, money=51, eventTime=1653442810777)
data: WatermarkMain.OrderEvent(orderId=7ef1ab91f3b44c47b63e81513363e8b1, userId=1, money=23, eventTime=1653442813787)
watermark on event: WatermarkMain.OrderEvent(orderId=7ef1ab91f3b44c47b63e81513363e8b1, userId=1, money=23, eventTime=1653442813787)
data: WatermarkMain.OrderEvent(orderId=f4a3e8dba5a742b791c01a87c9b26ff1, userId=0, money=83, eventTime=1653442814797)
watermark on event: WatermarkMain.OrderEvent(orderId=f4a3e8dba5a742b791c01a87c9b26ff1, userId=0, money=83, eventTime=1653442814797)
watermark emit key:0 current time:09:40:15 event time:09:40:14 watermark:09:40:11
WaterMarkResult:::6> key:0 window:[09:40:00,09:40:05) window event times:[09:40:01, 09:40:03, 09:40:04]
WaterMarkResult:::6> key:1 window:[09:40:00,09:40:05) window event times:[09:40:00, 09:40:00]
WaterMarkResult:::6> key:0 window:[09:40:05,09:40:10) window event times:[09:40:06]
WaterMarkResult:::6> key:1 window:[09:40:05,09:40:10) window event times:[09:40:05, 09:40:06, 09:40:09]
data: WatermarkMain.OrderEvent(orderId=69d4cfd851f24e32a133cbffd0adec7c, userId=1, money=40, eventTime=1653442812805)
watermark on event: WatermarkMain.OrderEvent(orderId=69d4cfd851f24e32a133cbffd0adec7c, userId=1, money=40, eventTime=1653442812805)
data: WatermarkMain.OrderEvent(orderId=bd023967a45a4242b91860ad45ff369c, userId=1, money=72, eventTime=1653442813812)
watermark on event: WatermarkMain.OrderEvent(orderId=bd023967a45a4242b91860ad45ff369c, userId=1, money=72, eventTime=1653442813812)
data: WatermarkMain.OrderEvent(orderId=edd1e26508334b97802e6a10316f4c7a, userId=0, money=46, eventTime=1653442814823)
watermark on event: WatermarkMain.OrderEvent(orderId=edd1e26508334b97802e6a10316f4c7a, userId=0, money=46, eventTime=1653442814823)
data: WatermarkMain.OrderEvent(orderId=50985530b3964324802d51efbd2e458c, userId=0, money=41, eventTime=1653442818829)
watermark on event: WatermarkMain.OrderEvent(orderId=50985530b3964324802d51efbd2e458c, userId=0, money=41, eventTime=1653442818829)
data: WatermarkMain.OrderEvent(orderId=5d0cc8a6bda745f49630698fb332a18c, userId=0, money=0, eventTime=1653442819843)
watermark on event: WatermarkMain.OrderEvent(orderId=5d0cc8a6bda745f49630698fb332a18c, userId=0, money=0, eventTime=1653442819843)
watermark emit key:0 current time:09:40:20 event time:09:40:19 watermark:09:40:16
WaterMarkResult:::6> key:0 window:[09:40:10,09:40:15) window event times:[09:40:10, 09:40:14, 09:40:14]
WaterMarkResult:::6> key:1 window:[09:40:10,09:40:15) window event times:[09:40:10, 09:40:13, 09:40:12, 09:40:13]
data: WatermarkMain.OrderEvent(orderId=90d99b5a5bdd472a961c2a95b364f3db, userId=0, money=82, eventTime=1653442816852)
watermark on event: WatermarkMain.OrderEvent(orderId=90d99b5a5bdd472a961c2a95b364f3db, userId=0, money=82, eventTime=1653442816852)
data: WatermarkMain.OrderEvent(orderId=28e3362d279d404bb9921bd15f5cb690, userId=1, money=52, eventTime=1653442820866)
watermark on event: WatermarkMain.OrderEvent(orderId=28e3362d279d404bb9921bd15f5cb690, userId=1, money=52, eventTime=1653442820866)
data: WatermarkMain.OrderEvent(orderId=161a48b9be6a4be3bb216a21d1717cba, userId=0, money=63, eventTime=1653442819880)
watermark on event: WatermarkMain.OrderEvent(orderId=161a48b9be6a4be3bb216a21d1717cba, userId=0, money=63, eventTime=1653442819880)
data: WatermarkMain.OrderEvent(orderId=f0103748ea2c443c89be1c4b25df1619, userId=1, money=54, eventTime=1653442819892)
watermark on event: WatermarkMain.OrderEvent(orderId=f0103748ea2c443c89be1c4b25df1619, userId=1, money=54, eventTime=1653442819892)
data: WatermarkMain.OrderEvent(orderId=42a3c931fd9e40419cca4877de352006, userId=0, money=23, eventTime=1653442822908)
watermark on event: WatermarkMain.OrderEvent(orderId=42a3c931fd9e40419cca4877de352006, userId=0, money=23, eventTime=1653442822908)
watermark emit key:0 current time:09:40:25 event time:09:40:22 watermark:09:40:19
data: WatermarkMain.OrderEvent(orderId=48a769391850415c98e49ef6370dca7c, userId=1, money=15, eventTime=1653442822920)
watermark on event: WatermarkMain.OrderEvent(orderId=48a769391850415c98e49ef6370dca7c, userId=1, money=15, eventTime=1653442822920)
data: WatermarkMain.OrderEvent(orderId=67421f48773e49488f87701fc06a5a35, userId=0, money=49, eventTime=1653442826930)
watermark on event: WatermarkMain.OrderEvent(orderId=67421f48773e49488f87701fc06a5a35, userId=0, money=49, eventTime=1653442826930)
data: WatermarkMain.OrderEvent(orderId=44bdedbc9a5b4a75bf95255f86f68498, userId=0, money=92, eventTime=1653442825937)
watermark on event: WatermarkMain.OrderEvent(orderId=44bdedbc9a5b4a75bf95255f86f68498, userId=0, money=92, eventTime=1653442825937)
data: WatermarkMain.OrderEvent(orderId=f3695e874db4498c9c7dff5c0679792b, userId=0, money=28, eventTime=1653442825949)
watermark on event: WatermarkMain.OrderEvent(orderId=f3695e874db4498c9c7dff5c0679792b, userId=0, money=28, eventTime=1653442825949)
data: WatermarkMain.OrderEvent(orderId=4ec4e6e404ad47cf8b28a3a27a5b0db5, userId=0, money=66, eventTime=1653442826960)
watermark on event: WatermarkMain.OrderEvent(orderId=4ec4e6e404ad47cf8b28a3a27a5b0db5, userId=0, money=66, eventTime=1653442826960)
watermark emit key:0 current time:09:40:30 event time:09:40:26 watermark:09:40:23
WaterMarkResult:::6> key:0 window:[09:40:15,09:40:20) window event times:[09:40:18, 09:40:19, 09:40:16, 09:40:19]
WaterMarkResult:::6> key:1 window:[09:40:15,09:40:20) window event times:[09:40:19]
data: WatermarkMain.OrderEvent(orderId=958b72cf8b2b489ca4628f29b9a4c74c, userId=0, money=12, eventTime=1653442829975)
watermark on event: WatermarkMain.OrderEvent(orderId=958b72cf8b2b489ca4628f29b9a4c74c, userId=0, money=12, eventTime=1653442829975)
data: WatermarkMain.OrderEvent(orderId=495753d94291434b9559b78147dd34a3, userId=0, money=82, eventTime=1653442828986)
watermark on event: WatermarkMain.OrderEvent(orderId=495753d94291434b9559b78147dd34a3, userId=0, money=82, eventTime=1653442828986)
data: WatermarkMain.OrderEvent(orderId=fd3562acd2fe458395b1b213fc9afa56, userId=1, money=27, eventTime=1653442832002)
watermark on event: WatermarkMain.OrderEvent(orderId=fd3562acd2fe458395b1b213fc9afa56, userId=1, money=27, eventTime=1653442832002)
data: WatermarkMain.OrderEvent(orderId=9050092238d445f199a5beed4e7ac359, userId=0, money=48, eventTime=1653442832006)
watermark on event: WatermarkMain.OrderEvent(orderId=9050092238d445f199a5beed4e7ac359, userId=0, money=48, eventTime=1653442832006)
data: WatermarkMain.OrderEvent(orderId=9b74e005628b469d9c34a13710a44a99, userId=0, money=76, eventTime=1653442833006)
watermark on event: WatermarkMain.OrderEvent(orderId=9b74e005628b469d9c34a13710a44a99, userId=0, money=76, eventTime=1653442833006)