近期关注的大数据开源项目
截止至 2022-02-08
计算引擎
| Service | Git Star | Contributors | Release | License |
|---|---|---|---|---|
| Apache Flink | 18.1k | 1005 | v1.14.3 | Apache-2.0 |
Apache Flink：https://github.com/apache/flink
数据开发
| Service | Git Star | Contributors | Release | License |
|---|---|---|---|---|
| StreamX | 810 | 23 | v1.2.1 | Apache-2.0 |
| Apache Seatunnel | 3k | 42 | v1.5.7 | Apache-2.0 |
| Apache Zeppelin | 5.6k | 362 | v0.10.1-rc1 | Apache-2.0 |
| DataSphereStudio | 1.9k | 26 | v1.0.0 | Apache-2.0 |
| Apache Beam | 5.3k | 879 | v2.36.0 | Apache-2.0 |
| LarkMidTable | 686 | 14 | Apache-2.0 |
StreamX：https://github.com/streamxhub/streamx
Apache Seatunnel（孵化中）：https://github.com/apache/incubator-seatunnel
Apache Zeppelin ：https://github.com/apache/zeppelin
DataSphereStudio：https://github.com/WeBankFinTech/DataSphereStudio
Apache Beam：https://github.com/apache/beam
LarkMidTable：https://github.com/wxgzgl/LarkMidTable
数据湖
| Service | Git Star | Contributors | Release | License |
|---|---|---|---|---|
| Apache Iceberg | 2.4k | 226 | v0.13.0 | Apache-2.0 |
| Apache Hudi | 2.8k | 238 | v0.10.1 | Apache-2.0 |
| Delta Lake | 4k | 137 | v1.1.0 | Apache-2.0 |
| Dremio | 1k | 4 | v20.0.0 | Apache-2.0 |
Apache Iceberg：https://github.com/apache/iceberg
Apache Hudi：https://github.com/apache/hudi
DeltaLake：https://github.com/delta-io/delta
Dremio（数据湖加速）：https://github.com/dremio/dremio-oss
数据编排
| Alluxio | 5.5k | 1176 | v2.7.3 | Apache-2.0 |
Alluxio： https://github.com/Alluxio/alluxio
分布式对象存储
| Service | Git Star | Contributors | Release | License |
|---|---|---|---|---|
| MinIO | 31.5k | 332 | 2022-02-07T08-17-33Z | AGPL-3.0 |
| SeaweedFS | 13.8k | 142 | v2.88 | Apache-2.0 |
| JuiceFS | 4.8k | 47 | v1.0.0-beta1 | Apache-2.0 |
| Apache Ozone | 458 | 136 | v1.2.1 | Apache-2.0 |
| LakeFS | 2.2k | 53 | v0.58.0 | Apache-2.0 |
| Ceph | 10.1k | 1169 | v12.2.14 | LGPL-2.1 or LGPL-3 |
MinIO：https://github.com/minio/minio
SeaweedFS：https://github.com/chrislusf/seaweedfs
JuiceFS：https://github.com/juicedata/juicefs
Apache Ozone：https://github.com/apache/ozone
LakeFS：https://github.com/treeverse/lakeFS
Ceph：https://github.com/ceph/ceph
查询引擎
| Service | Git Star | Contributors | Release | License |
|---|---|---|---|---|
| Clickhouse | 22k | 910 | v21.10.6.2-stable | Apache-2.0 |
| Trino | 4.9k | 558 | 370 | Apache-2.0 |
| TiDB | 30.3k | 719 | v5.0.6 | Apache-2.0 |
| Apache Pinot | 3.8k | 210 | v0.9.3 | Apache-2.0 |
| StarRocks | 2.1k | 69 | v2.0.1 | Elastic-2.0 |
| Apache Kylin | 3.2k | 188 | v4.0.1 | Apache-2.0 |
| Apache Druid | 11.5k | 476 | v0.22.1 | Apache-2.0 |
| Apache Impala | 760 | 164 | v4.0.0 | Apache-2.0 |
| Elasticsearch | 58.4k | 1712 | v7.17.0 | SSPL + Elastic-2.0 |
| Greenplum | 5k | 295 | v6.19.1 | Apache-2.0 |
| Oceanbase | 4.1k | 116 | v3.1.2_CE | MulanPubL-2.0 |
Clickhouse：https://github.com/ClickHouse/ClickHouse
Trino：https://github.com/trinodb/trino
TiDB：https://github.com/pingcap/tidb
Apache Pinot：https://github.com/apache/pinot
StarRocks：https://github.com/StarRocks/starrocks
Apache Kylin：https://github.com/apache/kylin
Apache Druid：https://github.com/apache/druid
Impala：https://github.com/apache/impala
Elasticsearch：https://github.com/elastic/elasticsearch
Greenplum：https://github.com/greenplum-db/gpdb
Oceanbase：https://github.com/oceanbase/oceanbase
数据治理
| Service | Git Star | Contributors | Release | License |
|---|---|---|---|---|
| Apache Atlas | 1.2k | 114 | v2.2.0-rc1 | Apache-2.0 |
| Amundsen | 3k | 191 | v6.4.6 | Apache-2.0 |
| Datahub | 4.7k | 174 | v0.8.25 | Apache-2.0 |
| Metacat | 1.2k | 18 | v1.2.2 | Apache-2.0 |
| Marquez | 919 | 58 | v0.20.0 | Apache-2.0 |
Apache Atlas：https://github.com/apache/atlas
Lyft Amundsen：https://github.com/amundsen-io/amundsen
Linkedin Datahub：https://github.com/linkedin/datahub
Netflix Metacat：https://github.com/Netflix/metacat
Marquez：https://github.com/MarquezProject/marquez
消息服务
| Service | Git Star | Contributors | Release | License |
|---|---|---|---|---|
| Apache pulsar | 10.3k | 495 | v2.8.2 | Apache-2.0 |
Apache pulsar：https://github.com/apache/pulsar
任务调度
| Service | Git Star | Contributors | Release | License |
|---|---|---|---|---|
| Apache DolphinScheduler | 7.3k | 302 | v2.0.3 | Apache-2.0 |
| Apache Airflow | 24.7k | 1927 | 2.2.3 | Apache-2.0 |
Apache DolphinScheduler：https://github.com/apache/dolphinscheduler
Apache Airflow：https://github.com/apache/airflow
数据抽取
| Service | Git Star | Contributors | Release | License |
|---|---|---|---|---|
| Apache Gobblin | 2k | 89 | v0.11.0 | Apache-2.0 |
Apache Gobblin：https://github.com/apache/gobblin
多租户JDBC接口
| Service | Git Star | Contributors | Release | License |
|---|---|---|---|---|
| Apache kyuubi | 914 | 61 | v1.4.1-incubating | Apache-2.0 |
Apache kyuubi（孵化中）：https://github.com/apache/incubator-kyuubi
其他
容器
| Service | Git Star | Contributors | Release | License |
|---|---|---|---|---|
| Minikube | 23.2k | 714 | v1.25.1 | Apache-2.0 |
Minikube：https://github.com/kubernetes/minikube
网关服务
| Service | Git Star | Contributors | Release | License |
|---|---|---|---|---|
| Apache APISIX | 8.3k | 270 | v2.12.0 | Apache-2.0 |
Apache APISIX：https://github.com/apache/apisix