这个模块做的是实时统计用户每10分钟内的搜索次数，也就是10分钟级别的搜索频率。用户搜索时，服务端会把搜索数据发送的Kafka中。
直接看Flink的部分吧，这部分做的事情就是消费Kafka中的数据然后基于Event Time（事件时间）的10分钟级别的滚动窗口统计搜索次数。然后将结果集sink到mysql中。这个功能实现起来还是比较简单的，再加上用FlinkSQL来做就更容易了。先看代码吧
2 具体实现
2.1核心代码部分
创建源表
CREATE TABLE KAFKA_SOURCE_SEARCH_DATA (
data VARCHAR,
ts timestamp(3),
WATERMARK FOR ts as ts - INTERVAL '5' SECOND
) WITH (
'connector.type' = 'kafka',
'connector.version' = 'universal',
'connector.properties.group.id' = 'group-flink',
'connector.topic' = 'search_data',
'connector.startup-mode' = 'earliest-offset',
'connector.properties.zookeeper.connect' = 'localhost:2181',
'connector.properties.bootstrap.servers' = 'kafka1:9094',
'format.type' = 'json'
)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
创建目标表
CREATE TABLE MYSQL_SINK_SEARCH_FREQUENCY (
cnt BIGINT,
cnt_time timestamp(3)
) WITH (
'connector.type' = 'jdbc',
'connector.url' = 'jdbc:mysql://123.207.11.229:3306/search_lite?useUnicode=true&characterEncoding=UTF-8&serverTimezone=UTC&autoReconnect=true&failOverReadOnly=false',
'connector.table' = 'user_search_frequency',
'connector.username' = 'root',
'connector.password' = 'yxyj6900@',
'connector.write.flush.max-rows' = '1'
)
1
2
3
4
5
6
7
8
9
10
11
计算逻辑
INSERT INTO MYSQL_SINK_SEARCH_FREQUENCY(cnt,cnt_time)
SELECT
COUNT(*) as cnt,
TUMBLE_START(ts, INTERVAL '10' MINUTE) as cnt_time
FROM KAFKA_SOURCE_SEARCH_DATA
GROUP BY TUMBLE(ts, INTERVAL '10' MINUTE)
1
2
3
4
5
6
发送到Kafka的JSON样本
{"data": "8-1受集度为q的均布载荷作用的矩形杠杆截面简支梁","ts": "2020-06-06T15:01:39.780Z"}
1
其实思路还是蛮清晰，先设计好源表和目标表，然后在设计计算逻辑。关于源表和目标表的创建以及FlinkSQL的使用，官网提供了详细的文档，这里就不再赘述了。
这里需要提的是关于时间的格式，时间格式要遵循Flink支持的格式，一开始我用的其他时间格式导致报了莫名其妙的错误。
private static final SimpleDateFormat df=new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'");
1
嗯，其实写到这里，基本上就算完成了，可以直接使用Flink提供的sql-client提交就行了注意需要手动安装需要的依赖。但我这里使用的Java开发打包Jar提交的（后面再来写通过sql-client提交任务）。
2.2 Java代码部分
使用官方提供的模板来创建项目，先看需要的依赖（部分代码）
org.apache.flink
flink-connector-kafka_2.11
${flink.version}
org.apache.flink
flink-json
${flink.version}
mysql
mysql-connector-java
8.0.19
org.apache.flink
flink-jdbc_2.11
${flink.version}
https://www.thetoptens.com/m/wqaue848/348872/
https://www.thetoptens.com/m/siuwui22868/348874/
https://www.thetoptens.com/m/mgqyome4022/348875/
https://www.thetoptens.com/m/siuwui22868/348887/
https://www.thetoptens.com/m/wqaue848/348891/
https://www.thetoptens.com/m/mgqyome4022/348893/
https://www.thetoptens.com/m/siuwui22868/348902/
https://www.thetoptens.com/m/wqaue848/348904/
https://www.thetoptens.com/m/mgqyome4022/348910/
https://www.thetoptens.com/m/siuwui22868/348919/
https://www.thetoptens.com/m/mgqyome4022/348923/
https://www.thetoptens.com/m/wqaue848/348925/
https://www.thetoptens.com/m/siuwui22868/348933/
https://www.thetoptens.com/m/mgqyome4022/348937/
https://www.thetoptens.com/m/wqaue848/348944/
https://www.thetoptens.com/m/siuwui22868/348951/
https://www.thetoptens.com/m/mgqyome4022/348956/
https://www.thetoptens.com/m/wqaue848/348961/
https://www.thetoptens.com/m/siuwui22868/348967/
https://www.thetoptens.com/m/mgqyome4022/348971/
https://www.thetoptens.com/m/wqaue848/348974/
https://www.thetoptens.com/m/siuwui22868/348983/
https://www.thetoptens.com/m/mgqyome4022/348988/
https://www.thetoptens.com/m/wqaue848/348991/
https://www.thetoptens.com/m/siuwui22868/349003/
https://www.thetoptens.com/m/mgqyome4022/349008/
https://www.thetoptens.com/m/wqaue848/349013/
https://www.thetoptens.com/m/siuwui22868/349020/
https://www.thetoptens.com/m/mgqyome4022/349025/
https://www.thetoptens.com/m/wqaue848/349031/
https://www.thetoptens.com/m/siuwui22868/349036/
https://www.thetoptens.com/m/mgqyome4022/349041/
https://www.thetoptens.com/m/wqaue848/349049/
https://www.thetoptens.com/m/siuwui22868/349056/
https://www.thetoptens.com/m/mgqyome4022/349061/
https://www.thetoptens.com/m/wqaue848/349067/
https://www.thetoptens.com/m/siuwui22868/349071/
https://www.thetoptens.com/m/mgqyome4022/349076/
https://www.thetoptens.com/m/wqaue848/349082/
https://www.thetoptens.com/m/siuwui22868/349089/
https://www.thetoptens.com/m/mgqyome4022/349094/
https://www.thetoptens.com/m/wqaue848/349105/
https://www.thetoptens.com/m/siuwui22868/349106/
https://www.thetoptens.com/m/mgqyome4022/349109/
https://www.thetoptens.com/m/siuwui22868/349123/
https://www.thetoptens.com/m/wqaue848/349124/
https://www.thetoptens.com/m/mgqyome4022/349126/
https://www.thetoptens.com/m/wqaue848/349140/
https://www.thetoptens.com/m/siuwui22868/349142/
https://www.thetoptens.com/m/mgqyome4022/349147/
https://www.thetoptens.com/m/siuwui22868/349159/
https://www.thetoptens.com/m/wqaue848/349160/
https://www.thetoptens.com/m/mgqyome4022/349162/
https://www.thetoptens.com/m/siuwui22868/349173/
https://www.thetoptens.com/m/wqaue848/349177/
https://www.thetoptens.com/m/mgqyome4022/349181/
https://www.thetoptens.com/m/wqaue848/349193/
https://www.thetoptens.com/m/siuwui22868/349195/
https://www.thetoptens.com/m/mgqyome4022/349198/
https://www.thetoptens.com/m/wqaue848/349207/
https://www.thetoptens.com/m/siuwui22868/349210/
https://www.thetoptens.com/m/mgqyome4022/349212/
https://www.thetoptens.com/m/siuwui22868/349224/
https://www.thetoptens.com/m/wqaue848/349225/
https://www.thetoptens.com/m/mgqyome4022/349228/
https://www.thetoptens.com/m/siuwui22868/349240/
https://www.thetoptens.com/m/wqaue848/349244/
https://www.thetoptens.com/m/mgqyome4022/349247/
https://www.thetoptens.com/m/siuwui22868/349258/
https://www.thetoptens.com/m/wqaue848/349261/
https://www.thetoptens.com/m/mgqyome4022/349264/
https://www.thetoptens.com/m/wqaue848/349278/
https://www.thetoptens.com/m/siuwui22868/349280/
https://www.thetoptens.com/m/mgqyome4022/349284/
https://www.thetoptens.com/m/wqaue848/349296/
https://www.thetoptens.com/m/siuwui22868/349298/
https://www.thetoptens.com/m/mgqyome4022/349303/
https://www.thetoptens.com/m/wqaue848/349312/
https://www.thetoptens.com/m/siuwui22868/349318/
https://www.thetoptens.com/m/mgqyome4022/349325/
https://www.thetoptens.com/m/wqaue848/349328/
https://www.thetoptens.com/m/siuwui22868/349336/
https://www.thetoptens.com/m/mgqyome4022/349344/
https://www.thetoptens.com/m/wqaue848/349348/
https://www.thetoptens.com/m/siuwui22868/349352/
https://www.thetoptens.com/m/wqaue848/349362/
https://www.thetoptens.com/m/mgqyome4022/349364/
https://www.thetoptens.com/m/siuwui22868/349373/
https://www.thetoptens.com/m/mgqyome4022/349382/
https://www.thetoptens.com/m/wqaue848/349386/
https://www.thetoptens.com/m/siuwui22868/349391/
https://www.thetoptens.com/m/mgqyome4022/349400/
https://www.thetoptens.com/m/wqaue848/349407/
https://www.thetoptens.com/m/siuwui22868/349412/
https://www.thetoptens.com/m/mgqyome4022/349417/
https://www.thetoptens.com/m/wqaue848/349429/
https://www.thetoptens.com/m/mgqyome4022/349432/
https://www.thetoptens.com/m/siuwui22868/349434/
https://www.thetoptens.com/m/mgqyome4022/349448/
https://www.thetoptens.com/m/wqaue848/349449/
https://www.thetoptens.com/m/siuwui22868/349451/
https://www.thetoptens.com/m/mgqyome4022/349465/
https://www.thetoptens.com/m/wqaue848/349468/
https://www.thetoptens.com/m/siuwui22868/349470/
https://www.thetoptens.com/m/mgqyome4022/349485/
https://www.thetoptens.com/m/siuwui22868/349486/
https://www.thetoptens.com/m/wqaue848/349487/
https://www.thetoptens.com/m/siuwui22868/349506/
https://www.thetoptens.com/m/mgqyome4022/349507/
https://www.thetoptens.com/m/wqaue848/349508/
https://www.thetoptens.com/m/siuwui22868/349521/
https://www.thetoptens.com/m/mgqyome4022/349525/
https://www.thetoptens.com/m/wqaue848/349526/
https://www.thetoptens.com/m/siuwui22868/349539/
https://www.thetoptens.com/m/wqaue848/349546/
https://www.thetoptens.com/m/mgqyome4022/349545/
https://www.thetoptens.com/m/siuwui22868/349559/
https://www.thetoptens.com/m/wqaue848/349563/
https://www.thetoptens.com/m/mgqyome4022/349567/
https://www.thetoptens.com/m/siuwui22868/349575/
https://www.thetoptens.com/m/wqaue848/349582/
https://www.thetoptens.com/m/mgqyome4022/349593/
https://www.thetoptens.com/m/siuwui22868/349594/
https://www.thetoptens.com/m/wqaue848/349606/
https://www.thetoptens.com/m/mgqyome4022/349610/
https://www.thetoptens.com/m/siuwui22868/349613/
https://www.thetoptens.com/m/wqaue848/349627/
https://www.thetoptens.com/m/mgqyome4022/349632/
https://www.thetoptens.com/m/siuwui22868/349635/
https://www.thetoptens.com/m/wqaue848/349643/
https://www.thetoptens.com/m/mgqyome4022/349653/
https://www.thetoptens.com/m/siuwui22868/349656/
https://www.thetoptens.com/m/wqaue848/349666/
https://www.thetoptens.com/m/mgqyome4022/349671/
https://www.thetoptens.com/m/siuwui22868/349682/
https://www.thetoptens.com/m/wqaue848/349684/
https://www.thetoptens.com/m/mgqyome4022/349688/
https://www.thetoptens.com/m/siuwui22868/349704/
https://www.thetoptens.com/m/wqaue848/349705/
https://www.thetoptens.com/m/mgqyome4022/349710/
https://www.thetoptens.com/m/siuwui22868/349726/
https://www.thetoptens.com/m/wqaue848/349727/
https://www.thetoptens.com/m/mgqyome4022/349730/
https://www.thetoptens.com/m/wqaue848/349743/
https://www.thetoptens.com/m/mgqyome4022/349746/
https://www.thetoptens.com/m/siuwui22868/349748/
https://www.thetoptens.com/m/mgqyome4022/349761/
https://www.thetoptens.com/m/wqaue848/349762/
https://www.thetoptens.com/m/siuwui22868/349767/
https://www.thetoptens.com/m/wqaue848/349783/
https://www.thetoptens.com/m/mgqyome4022/349784/
https://www.thetoptens.com/m/siuwui22868/349792/
https://www.thetoptens.com/m/wqaue848/349803/
https://www.thetoptens.com/m/mgqyome4022/349807/
https://www.thetoptens.com/m/siuwui22868/349815/
https://www.thetoptens.com/m/wqaue848/349823/
https://www.thetoptens.com/m/mgqyome4022/349828/
https://www.thetoptens.com/m/siuwui22868/349833/
https://www.thetoptens.com/m/wqaue848/349843/
https://www.thetoptens.com/m/mgqyome4022/349849/
https://www.thetoptens.com/m/siuwui22868/349858/
https://www.thetoptens.com/m/wqaue848/349866/
https://www.thetoptens.com/m/mgqyome4022/349870/
https://www.thetoptens.com/m/siuwui22868/349876/
https://www.thetoptens.com/m/wqaue848/349886/
https://www.thetoptens.com/m/mgqyome4022/349895/
https://www.thetoptens.com/m/siuwui22868/349896/
https://www.thetoptens.com/m/wqaue848/349913/
https://www.thetoptens.com/m/siuwui22868/349916/
https://www.thetoptens.com/m/mgqyome4022/349917/
https://www.thetoptens.com/m/wqaue848/349931/
https://www.thetoptens.com/m/siuwui22868/349934/
https://www.thetoptens.com/m/mgqyome4022/349941/
https://www.thetoptens.com/m/wqaue848/349951/
https://www.thetoptens.com/m/siuwui22868/349958/
https://www.thetoptens.com/m/mgqyome4022/349965/
https://www.thetoptens.com/m/wqaue848/349974/
https://www.thetoptens.com/m/siuwui22868/349982/
https://www.thetoptens.com/m/mgqyome4022/349986/
https://www.thetoptens.com/m/wqaue848/350000/
https://www.thetoptens.com/m/siuwui22868/350008/
https://www.thetoptens.com/m/mgqyome4022/350012/
https://www.thetoptens.com/m/wqaue848/350027/
https://www.thetoptens.com/m/siuwui22868/350029/
https://www.thetoptens.com/m/mgqyome4022/350033/
https://www.thetoptens.com/m/wqaue848/350050/
https://www.thetoptens.com/m/siuwui22868/350056/
https://www.thetoptens.com/m/mgqyome4022/350059/
https://www.thetoptens.com/m/wqaue848/350074/
https://www.thetoptens.com/m/siuwui22868/350079/
https://www.thetoptens.com/m/mgqyome4022/350083/
https://www.thetoptens.com/m/wqaue848/350100/
https://www.thetoptens.com/m/siuwui22868/350101/
https://www.thetoptens.com/m/mgqyome4022/350105/
https://www.thetoptens.com/m/wqaue848/350126/
https://www.thetoptens.com/m/siuwui22868/350129/
https://www.thetoptens.com/m/mgqyome4022/350128/
https://www.thetoptens.com/m/wqaue848/350149/
https://www.thetoptens.com/m/siuwui22868/350152/
https://www.thetoptens.com/m/mgqyome4022/350155/
https://www.thetoptens.com/m/siuwui22868/350174/
https://www.thetoptens.com/m/wqaue848/350175/
https://www.thetoptens.com/m/mgqyome4022/350185/
https://www.thetoptens.com/m/siuwui22868/350199/
https://www.thetoptens.com/m/wqaue848/350202/
https://www.thetoptens.com/m/mgqyome4022/350211/
https://www.thetoptens.com/m/siuwui22868/350224/
https://www.thetoptens.com/m/wqaue848/350230/
https://www.thetoptens.com/m/mgqyome4022/350242/
https://www.thetoptens.com/m/siuwui22868/350253/
https://www.thetoptens.com/m/wqaue848/350259/
https://www.thetoptens.com/m/mgqyome4022/350266/
https://www.thetoptens.com/m/siuwui22868/350278/
https://www.thetoptens.com/m/wqaue848/350287/
https://www.thetoptens.com/m/mgqyome4022/350293/
https://www.thetoptens.com/m/siuwui22868/350307/
https://www.thetoptens.com/m/wqaue848/350312/
https://www.thetoptens.com/m/mgqyome4022/350316/
https://www.thetoptens.com/m/siuwui22868/350331/
https://www.thetoptens.com/m/wqaue848/350335/
https://www.thetoptens.com/m/mgqyome4022/350340/
https://www.thetoptens.com/m/siuwui22868/350356/
https://www.thetoptens.com/m/wqaue848/350362/
https://www.thetoptens.com/m/mgqyome4022/350369/
https://www.thetoptens.com/m/siuwui22868/350383/
https://www.thetoptens.com/m/wqaue848/350393/
https://www.thetoptens.com/m/mgqyome4022/350399/
https://www.thetoptens.com/m/siuwui22868/350412/
https://www.thetoptens.com/m/wqaue848/350422/
https://www.thetoptens.com/m/mgqyome4022/350425/
https://www.thetoptens.com/m/siuwui22868/350437/
https://www.thetoptens.com/m/wqaue848/350444/
https://www.thetoptens.com/m/mgqyome4022/350451/
https://www.thetoptens.com/m/siuwui22868/350466/
https://www.thetoptens.com/m/wqaue848/350477/
https://www.thetoptens.com/m/mgqyome4022/350481/
https://www.thetoptens.com/m/siuwui22868/350492/
https://www.thetoptens.com/m/wqaue848/350503/
https://www.thetoptens.com/m/mgqyome4022/350509/
https://www.thetoptens.com/m/siuwui22868/350522/
https://www.thetoptens.com/m/wqaue848/350529/
https://www.thetoptens.com/m/mgqyome4022/350538/
https://www.thetoptens.com/m/siuwui22868/350552/
https://www.thetoptens.com/m/wqaue848/350556/
https://www.thetoptens.com/m/mgqyome4022/350566/
https://www.thetoptens.com/m/siuwui22868/350586/
https://www.thetoptens.com/m/wqaue848/350587/
https://www.thetoptens.com/m/mgqyome4022/350595/
https://www.thetoptens.com/m/siuwui22868/350616/
https://www.thetoptens.com/m/wqaue848/350618/
https://www.thetoptens.com/m/mgqyome4022/350620/
https://www.thetoptens.com/m/siuwui22868/350652/
https://www.thetoptens.com/m/mgqyome4022/350653/
https://www.thetoptens.com/m/wqaue848/350655/
https://www.thetoptens.com/m/mgqyome4022/350682/
https://www.thetoptens.com/m/siuwui22868/350684/
https://www.thetoptens.com/m/wqaue848/350692/
https://www.thetoptens.com/m/mgqyome4022/350717/
https://www.thetoptens.com/m/siuwui22868/350721/
https://www.thetoptens.com/m/wqaue848/350729/
https://www.thetoptens.com/m/mgqyome4022/350747/
https://www.thetoptens.com/m/siuwui22868/350754/
https://www.thetoptens.com/m/wqaue848/350765/
https://www.thetoptens.com/m/mgqyome4022/350781/
https://www.thetoptens.com/m/siuwui22868/350796/
https://www.thetoptens.com/m/wqaue848/350798/
https://www.thetoptens.com/m/mgqyome4022/350816/
https://www.thetoptens.com/m/siuwui22868/350831/
https://www.thetoptens.com/m/wqaue848/350832/
https://www.thetoptens.com/m/mgqyome4022/350853/
https://www.thetoptens.com/m/siuwui22868/350859/
https://www.thetoptens.com/m/wqaue848/350862/
https://www.thetoptens.com/m/mgqyome4022/350886/
https://www.thetoptens.com/m/siuwui22868/350891/
https://www.thetoptens.com/m/wqaue848/350896/
https://www.thetoptens.com/m/mgqyome4022/350915/
https://www.thetoptens.com/m/siuwui22868/350920/
https://www.thetoptens.com/m/wqaue848/350928/
https://www.thetoptens.com/m/siuwui22868/350951/
https://www.thetoptens.com/m/mgqyome4022/350956/
https://www.thetoptens.com/m/wqaue848/350962/
https://www.thetoptens.com/m/siuwui22868/350983/
https://www.thetoptens.com/m/mgqyome4022/350984/
https://www.thetoptens.com/m/wqaue848/350992/
https://www.thetoptens.com/m/siuwui22868/351019/
https://www.thetoptens.com/m/mgqyome4022/351021/
https://www.thetoptens.com/m/wqaue848/351031/
https://www.thetoptens.com/m/mgqyome4022/351052/
https://www.thetoptens.com/m/siuwui22868/351058/
https://www.thetoptens.com/m/wqaue848/351065/
https://www.thetoptens.com/m/mgqyome4022/351086/
https://www.thetoptens.com/m/siuwui22868/351093/
https://www.thetoptens.com/m/wqaue848/
"http://www.w3.org/TR/html4/loose.dtd">;
;
Internal Server Error
Something went wrong when trying to load this page, and we're very sorry for that. Our coders, server admins, and database nerds have been notified so they know to put down their game controllers and get to work.
URL: /ajax/ajaxlist.asp
Error: 500 Internal Server Error
Server Time: 7/10/2020 1:43:52 AM
Status: Logged
/
https://www.thetoptens.com/m/mgqyome4022/
"http://www.w3.org/TR/html4/loose.dtd">;
;
Internal Server Error
Something went wrong when trying to load this page, and we're very sorry for that. Our coders, server admins, and database nerds have been notified so they know to put down their game controllers and get to work.
URL: /ajax/ajaxlist.asp
Error: 500 Internal Server Error
Server Time: 7/10/2020 1:43:53 AM
Status: Logged
/
https://www.thetoptens.com/m/siuwui22868/
"http://www.w3.org/TR/html4/loose.dtd">;
;
Internal Server Error
Something went wrong when trying to load this page, and we're very sorry for that. Our coders, server admins, and database nerds have been notified so they know to put down their game controllers and get to work.
URL: /ajax/ajaxlist.asp
Error: 500 Internal Server Error
Server Time: 7/10/2020 1:43:57 AM
Status: Logged
/
https://www.thetoptens.com/m/wqaue848/
"http://www.w3.org/TR/html4/loose.dtd">;
;
Internal Server Error
Something went wrong when trying to load this page, and we're very sorry for that. Our coders, server admins, and database nerds have been notified so they know to put down their game controllers and get to work.
URL: /ajax/ajaxlist.asp
Error: 500 Internal Server Error
Server Time: 7/10/2020 1:44:45 AM
Status: Logged
/
https://www.thetoptens.com/m/mgqyome4022/
"http://www.w3.org/TR/html4/loose.dtd">;
;
Internal Server Error
Something went wrong when trying to load this page, and we're very sorry for that. Our coders, server admins, and database nerds have been notified so they know to put down their game controllers and get to work.
URL: /ajax/ajaxlist.asp
Error: 500 Internal Server Error
Server Time: 7/10/2020 1:45:44 AM
Status: Logged
/
https://www.thetoptens.com/m/wqaue848/
"http://www.w3.org/TR/html4/loose.dtd">;
;
Internal Server Error
Something went wrong when trying to load this page, and we're very sorry for that. Our coders, server admins, and database nerds have been notified so they know to put down their game controllers and get to work.
URL: /ajax/ajaxlist.asp
Error: 500 Internal Server Error
Server Time: 7/10/2020 1:46:31 AM
Status: Logged
/
https://www.thetoptens.com/m/mgqyome4022/
"http://www.w3.org/TR/html4/loose.dtd">;
;
Internal Server Error
Something went wrong when trying to load this page, and we're very sorry for that. Our coders, server admins, and database nerds have been notified so they know to put down their game controllers and get to work.
URL: /ajax/ajaxlist.asp
Error: 500 Internal Server Error
Server Time: 7/10/2020 1:50:07 AM
Status: Logged
/
https://www.thetoptens.com/m/wqaue848/
"http://www.w3.org/TR/html4/loose.dtd">;
;
Internal Server Error
Something went wrong when trying to load this page, and we're very sorry for that. Our coders, server admins, and database nerds have been notified so they know to put down their game controllers and get to work.
URL: /ajax/ajaxlist.asp
Error: 500 Internal Server Error
Server Time: 7/10/2020 1:56:13 AM
Status: Logged
/