Flink博客记录

Flink

配置参数说明

https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/deployment/config/

一、Flink 记录

1.1 内存管理

1.1.1 Flink内存管理

https://www.cnblogs.com/lighten/p/13053828.html

1.1.2 Flink_Flink TaskManager 内存模型 ，以及相关内存配置参数

https://blog.csdn.net/u010003835/article/details/106294342

1.2 Flink 记录

–>> 一文搞懂 Flink 的 Exactly Once 和 At Least Once

https://mp.weixin.qq.com/s/QcL33ghaM-riceOr218oXg

1.2.1 Kafka偏移量提交

https://stackoverflow.com/questions/54585740/commit-kafka-offsets-manually-in-flink

1.2.2 Flink savepoint

https://zhuanlan.zhihu.com/p/94594551

https://zhuanlan.zhihu.com/p/79526638

1.2.3 checkpoint

1.2.3.1 checkpoint 并行度改变后，状态重分配

https://blog.csdn.net/lvwenyuan_1/article/details/98511963

1.2.3.2 checkpoint 恢复

https://mp.weixin.qq.com/s/rncSvPKdFrT2tFMs6TZpcA

1.2.4 反压

http://wuchong.me/blog/2016/04/26/flink-internals-how-to-handle-backpressure/

1.2.5 序列化

http://wuchong.me/blog/2016/04/29/flink-internals-memory-manage/

https://developer.aliyun.com/article/721997

1.2.6 CPU L1 L2 L3 是什么，作用是什么？

https://blog.csdn.net/u010632165/article/details/106795478

1.2.7 Flink on yarn 指定节点

https://blog.csdn.net/weixin_47235286/article/details/121883864

yarn-session.sh -n 2 -jm 1024 -tm 1024 -d -s 2

-n 2 表示指定两个容器 
-jm 1024 表示jobmanager 1024M内存 
-tm 1024表示taskmanager 1024M内存 
-d --detached  任务后台运行 
-s  指定每一个taskmanager分配多少个slots(处理进程)。建议设置为每个机器的CPU核数。一般情况下，vcore的数量等于处理的slot（-s）的数量
-nm,--name YARN上为一个自定义的应用设置一个名字
-q,--query 显示yarn中可用的资源 (内存, cpu核数)
-qu,--queue <arg> 指定YARN队列.
-z,--zookeeperNamespace <arg> 针对HA模式在zookeeper上创建NameSpace

Flink JobListener

Flink 维表更新处理办法

https://zhuanlan.zhihu.com/p/490835750

Flink trigger

https://blog.csdn.net/qq_31866793/article/details/102842570
https://www.jianshu.com/p/e32e04a1d6c4
—>>> https://blog.csdn.net/weixin_43495317/article/details/124241897

Flink 多个source watermark

https://my.oschina.net/qiangzigege/blog/2993330?p={{currentPage+1}}

异步 Asyc

https://blog.csdn.net/qq_46045632/article/details/122588486?utm_medium=distribute.pc_aggpage_search_result.none-task-blog-2_{aggregatepage}first_rank_ecpm_v1~rank_v31_ecpm-1-122588486.pc_agg_new_rank&utm_term=flink+%E5%AE%9E%E6%97%B6%E6%9F%A5%E8%AF%A2+%E7%BB%B4%E8%A1%A8&spm=1000.2123.3001.4430

Flink 流批一体的实践与探索

https://blog.csdn.net/zhaodedong/article/details/105424236

监控

Flink 任务实时监控最佳实践

https://mp.weixin.qq.com/s?__biz=Mzg3MDE0MjUzMA==&mid=2247484144&idx=1&sn=12462bf23cbb4e318857dd53375666b5&chksm=ce9305ddf9e48ccb64e5712ee399b2a10fe1b1fcb7081211109af90da961d6c8a59135860bc0&mpshare=1&scene=23&srcid=0808f76eX4ygFWId3wSbDzbT&sharer_sharetime=1628421639852&sharer_shareid=d73403798b935b929a6ed550a277d0ee#rd

二、FlinkSQL

CDH FlinkSQL 集成 hive

https://www.jianshu.com/p/e964ba3fb8f8

FlinkSQL平台化

Flink SQL on Zeppelin - 打造自己的可视化Flink SQL开发平台

https://mp.weixin.qq.com/s/QbBCO4ul6-hofl-AErld9g

大数据开发之Flink SQL建设实时数仓实践

https://blog.csdn.net/m0_58371965/article/details/122577670

基于 Flink + Hive 构建流批一体准实时数仓

https://mp.weixin.qq.com/s?__biz=MzU3Mzg4OTMyNQ==&mid=2247489376&idx=1&sn=000fda2c695c88a6542f5cebb3207b64&chksm=fd3b9922ca4c1034b8bc0bff6d091c31c13b41443c3a71f5420007d91bc24b60bb102d6a9a94&xtrack=1&scene=90&subscene=93&sessionid=1601205791&clicktime=1601205953&enterid=1601205953&ascene=56&devicetype=android-26&version=3.0.31.2998&nettype=cmnet&abtest_cookie=AAACAA%3D%3D&lang=zh_CN&exportkey=AeaDMVfzSL04VBtRUGcfL2Q%3D&pass_ticket=%2BVHG5JT6TAPRJ0%2BEgPbDiMJd0tzudJm5cBMmrmzu1D1RF1aDyY2nyEIuVQLSQS23&wx_header=1&platform=win

自定义函数

https://www.alibabacloud.com/help/zh/doc-detail/69462.html

三、FlinkCDC

–>> Flink1.11中的CDC Connectors操作实践

https://mp.weixin.qq.com/s/h0RvUH8upF8Cqn7PKfo6Qw

Flink CDC 和 kafka 进行多源合并和下游同步方案

https://blog.csdn.net/qq_24505127/article/details/122761938

FlinkCDC + hudi + hive

https://www.csdn.net/tags/NtjacgzsNTgwODEtYmxvZwO0O0OO0O0O.html

—>>> hudi

https://blog.csdn.net/m0_66705151/category_11623872.html

监控

flink自定义metrics监控kafka消费

https://blog.csdn.net/zyy_2018/article/details/119537150

http://t.zoukankan.com/qiu-hua-p-13910809.html

Calcite记录

Apache Calcite 处理流程详解

http://matt33.com/2019/03/07/apache-calcite-process-flow/

flink sql 知其所以然（六）| flink sql 约会 calcite（看这篇就够了）

https://mp.weixin.qq.com/s/SxRKp368mYSKVmuduPoXFg

Calcite应用指南

https://blog.csdn.net/hxcaifly/article/details/88661252

基于Calcite自定义SQL解析器

https://blog.csdn.net/weixin_38381496/article/details/117418643

–>>> https://cloud.tencent.com/developer/article/1658572?areaSource=103001.4&traceId=KZ8pjGMtZXLpMEn4SVpTX

Calcite中定制自已SQL解析器

https://www.jianshu.com/p/e4f883a3b969
https://zhuanlan.zhihu.com/p/65345335

基于calcite做傻瓜式的sql优化(一)
基于calcite做傻瓜式的sql优化(二)
基于calcite做傻瓜式的sql优化(三)

Apache Calcite学习资料

https://zhuanlan.zhihu.com/p/76510053

Calcite系列博客

https://blog.csdn.net/jyh2005/category_9706931.html

https://blog.csdn.net/qxc1281/category_8479654.html

https://www.jianshu.com/nb/35041702

–>https://blog.csdn.net/qq_31183071/category_9427896.html

–>https://blog.csdn.net/weixin_44112790/category_10882344.html

优化

Flink 单并行度内使用多线程，提高写入性能10倍

https://blog.csdn.net/qq_23160237/article/details/103821970

flink 应用调优过程 buffer.timeout

https://www.modb.pro/db/183441

源码分析

–>https://blog.jrwang.me/tags/flink/
https://www.cnblogs.com/ljygz/tag/%E5%AE%9E%E6%97%B6%E8%AE%A1%E7%AE%97/default.html?page=1

Flink 源码分析系列文档目录

实践

Flink 容器化在唯品会的实践

https://mp.weixin.qq.com/s/Bl8HlJ_59ulbwqPNpi2CiA

美团 Flink 大作业部署与状态稳定性优化实践

https://mp.weixin.qq.com/s?search_click_id=5180439472511667885-1670808467457-9549046859&__biz=MzU3Mzg4OTMyNQ==&mid=2247495878&idx=1&sn=d63bdeb4c790faf999bc038cd19131d5&chksm=fd387e84ca4ff792a0931917d19bbe6c28026f9ea09e0f1097b3626a84c851cb9afeaf763c0f&scene=7&key=c04f33c34cf3e825767b3ab88c1266b0de1c3e1f85bd6f39425519ee42712e2764a5a15c4a2f0d9f23c13db5d6034c687c3894930faeed35efa1dea65bf0ade27f676583afd7ebe75c7badd274e0b6c68a05cf70fcd32f30eddf7eb37a98e6ebaf8ab0c93531107b9f49aaaba1bd3195997b272a0f694e46c45d99122e574039&ascene=65&uin=MjIyMTA5OTc2MQ%3D%3D&devicetype=Windows+7+x64&version=63080021&lang=zh_CN&exportkey=n_ChQIAhIQqTJzItexwUeqXXdmHmI4DRLeAQIE97dBBAEAAAAAAL4qLanELVMAAAAOpnltbLcz9gKNyK89dVj0R0%2BP4w9zfy4IWRdZup0Sn9zpf%2FM2vizWGtMcqLFlE8IaTcEyoRN4PaPKvf1B3e2nAM1JCYEy2IF7eI6pMmGzZqEgIPs5gGizLUs%2FD1fkaOE0Ks8XriLoCwBwQX2bdh%2F9KBQUko233%2FUPRYHBGLFxao%2BJXLMMj7vEhqzE1KO2Zdl5ENguywHTLklOUztDkwHBYVv7tkvzYwBFA0EIfTQNHEO2O%2F7U1DWbwxHaW6tyLjOxytOJcm4DAQ%3D%3D&acctmode=0&pass_ticket=fC6emPpwYf117xzoO0FNPq0438gHFj2P6p%2F9s9bwtwwAGySkMHpRMqhhtfpOJC%2FEXvHgQ8KLXReckNSesXYZkQ%3D%3D&wx_header=1&fontgear=2

部署

Flink on K8S

https://cloud.tencent.com/developer/article/1763130

Flink sql 消费kafka的顺序是怎么样的? 解答为什么第二次运行sql的结果和第一次不同

https://blog.51cto.com/u_15278282/4895929

–>> Flink on Yarn 远程 debug 应用提交流程及源码分析

https://zhuanlan.zhihu.com/p/444605072

Flink on yarn 源码分析

https://blog.csdn.net/jhkkhtd/article/details/118029480?spm=1001.2014.3001.5502

Flink on yarn

https://blog.csdn.net/xc_zhou/article/details/124371043?utm_term=flink%E9%A1%B5%E9%9D%A2%E6%8F%90%E4%BA%A4job&utm_medium=distribute.pc_aggpage_search_result.none-task-blog-2_allsobaiduweb~default-3-124371043-null-null&spm=3001.4430

Flink 系列博客

王知无

https://blog.csdn.net/u013411339/category_8631091.html

大数据之flink

https://www.cnblogs.com/huanghanyu/category/1758541.html

浪尖

https://blog.csdn.net/rlnlo2pnefx9c/category_10872751.html

Flink 开源项目

zhisheng

https://github.com/zhisheng17/flink-learning

flink-recommandSystem-demo

https://github.com/CheckChe0803/flink-recommandSystem-demo

Flink问题记录汇总

https://developer.aliyun.com/ask/288158?spm=a2c6h.13066369.0.0.14375f93zjXkcc
https://developer.aliyun.com/ask/258515