OpenMLDB Weekly Update(2021.9.5-2021.9.12)

OpenMLDB

Summary

本周合并 Pull requests 11个，新增Pull requests 9个，关闭 Issues 14个，新增 Issues 12个。总计57个文件修改，新增3002行代码，删除539行代码。

Merged Pull Requests

feat: refine the description of benchmark in readme#405
feat: update the benchmark images for readme#404
feat: remove junit dependencies and covert java cases as scala#396
feat: bump hadoop-common to 2.8.5 to avoid vulnerability#388
fix: fix a bug in union with instance not in window#381
feat: add batchjob module#359
feat: add task manager module#361
feat: enhance plan optimization for group and filter#350
feat: support column query with the same name in window skew optimization#349
feat: add java common lib#347
test: modify test case#368

Open Pull Requests

feat: bump junit from 4.11 to 4.13.1 in /java/openmldb-batchjob#382
WIP : create message table#383
feat: add batch, batchjob and taskmanager as java submodules#386
fix: fix some code implementation in window skew optimization#392
feat: support aggregation over the whole table#393
feat: add integration test#395
feat: support insert multiple rows into a table using a single SQL insert statement#399
feat: add kubernetes java dependencies for taskmanager#400
fix: python test and cicd#401

Close Issues

Remove junit since of the EPL license for OpenMLDB Batch#390
Avoid vulnerability by upgrading hadoop-common dependency#387
rtidb磁盘占用#389
feat: engine plan optimization for where and group with the same partition#317
是否支持批量插入#177
字段含义#378
表使用大小#373
Fix the issue to enable window skew optimization for window union case#374
Enable WindowSkewOpt by default and resolve the running issues#335
Add module for OpenMLDB Batch to run custom SQL and submit by TaskManager#351
Add TaskManager service to submit OpenMLDB Batch jobs#360
scripts: package java sdk set cmake type to release#372
Multiple columns with the same name can't execute when last join and over window#356
Add java common lib#342

Open Issues

feat: improve error message system.#406
feat: support insert multiple rows into a table using a single SQL insert statement.#398
feat: api server support parameterized query#397
Add engine test on performance insensitive mode#394
Bug: SQL INSERT Statement with multi rows does not work as expected#391
Make openmldb-batchjob and openmldb-taskmanager as submodules of openmldb-parent#385
Support AWS S3 for offline data lake storage#384
Support creating database API for NearlineTablet#380
Support hive metastore for NearlineTablet#379
Support submit and manage Flink jobs for TaskManager#376
Support submit and manage Kubernetes jobs for TaskManager#375
scripts: package java sdk set cmake type to release#371

Contributors

dl239 ([email protected])
tobe ([email protected])
Wang ZeKai ([email protected])
wuyou10206 ([email protected])
Highlights

本周新的三个Java模块，分别是openmldb-common、openmldb-batchjob和openmldb-taskmanager。通过可重用的Java模块抽象，可以快速实现新的子模块。BatchJob模块和TaskManager实现了第一个最小功能集版本，提供了基础的批处理任务管理功能。

本周通过拓展SQL语法解析器和物理计划优化器，实现了在线执行引擎的全表分组聚合和过滤功能，此功能难点在于需要在SQL语法解析阶段就区分SQL中的UDF和UDAF函数，并最终生成对应的逻辑计划和优化后的物理计划。

欢迎更多开发者关注和参与OpenMLDB开源项目。