结论:窗口的开始时间,并不一定是最早的记录时间,而是一个计算出来的时间
Flink的窗口函数起始时间计算公式:
timestamp - (timestamp - offset + windowSize) % windowSize
# timestamp为最早时间的记录时间戳
# offset 为参数偏移时间
# windowSize为窗口大小
# 以上计算单位全部统一到毫秒再进行计算
业务代码: .window(TumblingEventTimeWindows.of(Time.seconds(3L)))
其中滚动窗口api:TumblingEventTimeWindows of(Time size, Time offset)
源码:
类:org.apache.flink.streaming.api.windowing.windows.TimeWindow
public static long getWindowStartWithOffset(long timestamp, long offset, long windowSize) {
return timestamp - (timestamp - offset + windowSize) % windowSize;
}
如果offset不输入,默认为0,那么起始时间就是timestamp - timestamp+windowsize和windowsize的取模,假定以秒为小单位。
输入时间是个偶数秒T,windowsize长度为2s,offset为0的情况下,那么经过取模部分值为0,起始时间就是T。
写个小程序计算下启动时间:
import java.time.Instant;
import java.time.LocalDateTime;
import java.time.ZoneId;
import java.time.ZoneOffset;
import java.time.format.DateTimeFormatter;
public class WindowStartTimeCalculate {
public static void main(String[] args) throws Exception {
String[] ss = {"2020-04-20 10:12:44", "2020-04-20 8:50:44", "2020-04-20 14:12:44", "2020-04-20 18:12:44", "2020-04-20 20:12:44"};
for (String date : ss) {
cac(Time.days(1L), date, Time.hours(16L));
}
}
/**
* @param w 窗口大小
* @param d 事件时间
* @param o 偏移时间
* @throws Exception
*/
public static void cac(Time w, String d, Time o) throws Exception {
LocalDateTime parse = LocalDateTime.parse(d, DateTimeFormatter.ofPattern("yyyy-MM-dd H:mm:ss"));
long timestamp = parse.toInstant(ZoneOffset.ofHours(8)).toEpochMilli();
long offset = o.toMilliseconds();
long windowSize = w.toMilliseconds();
long startTime = timestamp - (timestamp - offset + windowSize) % windowSize;
System.out.println("offset:" + offset + ", windowSize: " + windowSize + ", " + startTime + "--" + LocalDateTime.ofInstant(Instant.ofEpochMilli(startTime), ZoneId.systemDefault()));
}
}
窗口1天:
偏移时间不填写,那么起始时间是:2020-04-20T08:00
偏移时间1小时Time.hours(1L),那么起始时间是:2020-04-20T09:00
依次类推...
偏移时间16小时,则起始时间为:2020-04-20T00:00
偏移时间23小时,则起始时间为:2020-04-20T07:00
窗口3s,规律不明显,大部分都是-1s:
offset:0, windowSize: 3000, record:2020-04-20 10:12:01 ,startTime:2020-04-20T10:12:00
offset:0, windowSize: 3000, record:2020-04-20 08:50:14 ,startTime:2020-04-20T08:50:12
offset:0, windowSize: 3000, record:2020-04-20 14:12:37 ,startTime:2020-04-20T14:12:36
offset:0, windowSize: 3000, record:2020-04-20 18:12:49 ,startTime:2020-04-20T18:12:48
offset:0, windowSize: 3000, record:2020-04-20 20:12:50 ,startTime:2020-04-20T20:12:48
窗口10s,则起始时间以事件时间个位数为0的时间开始:
offset:0, windowSize: 10000, record:2020-04-20 10:12:01 ,startTime:2020-04-20T10:12:00
offset:0, windowSize: 10000, record:2020-04-20 8:50:14 ,startTime:2020-04-20T08:50:10
offset:0, windowSize: 10000, record:2020-04-20 14:12:37 ,startTime:2020-04-20T14:12:30
offset:0, windowSize: 10000, record:2020-04-20 18:12:49 ,startTime:2020-04-20T18:12:40
offset:0, windowSize: 10000, record:2020-04-20 20:12:50 ,startTime:2020-04-20T20:12:50
窗口5秒:
offset:0, windowSize: 5000, record:2020-04-20 10:12:01 ,startTime:2020-04-20T10:12:00
offset:0, windowSize: 5000, record:2020-04-20 8:50:14 ,startTime:2020-04-20T08:50:10
offset:0, windowSize: 5000, record:2020-04-20 14:12:37 ,startTime:2020-04-20T14:12:35
offset:0, windowSize: 5000, record:2020-04-20 18:12:49 ,startTime:2020-04-20T18:12:45
offset:0, windowSize: 5000, record:2020-04-20 20:12:50 ,startTime:2020-04-20T20:12:50
窗口为500毫秒,偏移为空值,则起始时间就是最早记录:
offset:0, windowSize: 500, record:2020-04-20 10:12:01 ,startTime:2020-04-20T10:12:01
offset:0, windowSize: 500, record:2020-04-20 8:50:14 ,startTime:2020-04-20T08:50:14
offset:0, windowSize: 500, record:2020-04-20 14:12:37 ,startTime:2020-04-20T14:12:37
offset:0, windowSize: 500, record:2020-04-20 18:12:49 ,startTime:2020-04-20T18:12:49
offset:0, windowSize: 500, record:2020-04-20 20:12:50 ,startTime:2020-04-20T20:12:50