第4节 hive spark 查询hudi 数据 环境cdh6.3.2

使用hive带where条件查询发生错误

select uid,fullname from member_ro where uid >=0 and uid <20;

发现错误

java.io.IOException: cannot find class org.apache.hudi.hadoop.realtime.HoodieParquetRealtimeInputFormat
	at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:696)
	at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.(MapTask.java:175)

在源码中查询该类HoodieParquetRealtimeInputFormat是

第4节 hive spark 查询hudi 数据 环境cdh6.3.2_第1张图片

应该是在第3节中引入到hive lib下的 hudi-hadoop-mr-bundle-0.9.0.jar

尝试在hive query中加入

add jar hdfs://nameservice1/tmp/hiveshare/hudi-hadoop-mr-bundle-0.9.0.jar

第4节 hive spark 查询hudi 数据 环境cdh6.3.2_第2张图片

add jar 后 member_ro 可以根据条件查询,通过查询两个rt和ro的表的建表语句对比,发现

rt表存储输入格式的 (merge on read 表就会出现rt与ro copy on write 表只会出现一个表名且不带rt/ro)

STORED AS INPUTFORMAT 'org.apache.hudi.hadoop.realtime.HoodieParquetRealtimeInputFormat'

ro表存储输入格式的

STORED AS INPUTFORMAT

'org.apache.hudi.hadoop.HoodieParquetInputFormat'

其实 add jar hudi-hadoop-mr-bundle-0.9.0.jar 包里面包含 HoodieParquetInputFormat 和 HoodieParquetRealtimeInputFormat 但是就是rt的时候报如下错误

Caused by: java.lang.IllegalArgumentException: HoodieRealtimeRecordReader can only work on RealtimeSplit and not with hdfs://nameservice1/tmp/hudi/hivetest/20200918/WebA/0ac945ef-b1f1-4a64-96b9-faa18d7c5563-0_7-34-123_20210918173310.parquet:0+3037944
	at org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java:40)
	at org.apache.hudi.hadoop.realtime.HoodieParquetRealtimeInputFormat.getRecordReader(HoodieParquetRealtimeInputFormat.java:116)

查看源码

@Override
public RecordReader getRecordReader(final InputSplit split, final JobConf jobConf,
                                                                 final Reporter reporter) throws IOException {
  // sanity check 爆出的异常是如下检测这部分,将这个检测注释掉 重新编译
  ValidationUtils.checkArgument(split instanceof RealtimeSplit,
      "HoodieRealtimeRecordReader can only work on RealtimeSplit and not with " + split);
  RealtimeSplit realtimeSplit = (RealtimeSplit) split;
  addProjectionToJobConf(realtimeSplit, jobConf);
  LOG.info("Creating record reader with readCols :" + jobConf.get(ColumnProjectionUtils.READ_COLUMN_NAMES_CONF_STR)
      + ", Ids :" + jobConf.get(ColumnProjectionUtils.READ_COLUMN_IDS_CONF_STR));
  return new HoodieRealtimeRecordReader(realtimeSplit, jobConf,
      super.getRecordReader(split, jobConf, reporter));
}

第4节 hive spark 查询hudi 数据 环境cdh6.3.2_第3张图片 找到linux中源码的位置

第4节 hive spark 查询hudi 数据 环境cdh6.3.2_第4张图片

修改源码

/Hudi/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime 

 rt 的 hive 表依旧无法读取,暂时不测试hive读,用spark读取,可能原因是不兼容问题,后面用apache 版本测试后再补充

使用spark-shell

[xxx@xxx Hudi]# spark-shell 
--master local[*] 
--jars packaging/hudi-spark-bundle/target/hudi-spark-bundle_2.11-0.9.0.jar 
--packages org.apache.spark:spark-avro_2.11:2.4.4,org.apache.spark:spark-sql_2.11:2.4.4    --conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer'
scala> spark.sql("select uid,fullname from member_rt limit where uid >=0  and uid <20").show();
21/09/23 11:28:24 WARN lineage.LineageWriter: Lineage directory /var/log/spark/lineage doesn't exist or is not writable. Lineage for this application will be disabled.
+---+--------+                                                                  
|uid|fullname|
+---+--------+
|  6|进行修改|
| 16|进行修改|
| 18|进行修改|
|  3|进行修改|
|  9|进行修改|
| 11|进行修改|
|  0|进行修改|
|  4|进行修改|
|  5|进行修改|
| 10|进行修改|
| 15|进行修改|
| 17|进行修改|
| 13|进行修改|
| 14|进行修改|
|  1|进行修改|
|  7|进行修改|
|  8|进行修改|
| 12|进行修改|
|  2|进行修改|
| 19|进行修改|
+---+--------+

使用spark sql就可以读取到member_rt表的变化

快照查询

使用 hudi 的快照方式查询,发现可以直接查询到最新的数据

(1)spark默认快照查询,也可以不指定类型

scala> import org.apache.hudi.DataSourceReadOptions
import org.apache.hudi.DataSourceReadOptions

scala> spark.read.format("hudi").option(DataSourceReadOptions.QUERY_TYPE_OPT_KEY, DataSourceReadOptions.QUERY_TYPE_SNAPSHOT_OPT_VAL).load("/tmp/hudi/hivetest/*/*").where("uid>=0 and uid<20").show();
+-------------------+--------------------+------------------+----------------------+--------------------+-----+----+--------+--------+-------+---+----+-------------+-------------+
|_hoodie_commit_time|_hoodie_commit_seqno|_hoodie_record_key|_hoodie_partition_path|   _hoodie_file_name|ad_id|  dn|      dt|fullname|iconurl|uid|uuid|           ts|hudipartition|
+-------------------+--------------------+------------------+----------------------+--------------------+-----+----+--------+--------+-------+---+----+-------------+-------------+
|     20210922204730| 20210922204730_7_17|                 6|         20200918/WebA|a0878fc7-7d6f-4ff...|    4|WebA|20200918|进行修改|      -|  6|   6|1632314847098|20200918/WebA|
|     20210922204730| 20210922204730_6_16|                16|         20200918/WebA|576debfb-4d83-4a8...|    6|WebA|20200918|进行修改|      -| 16|  16|1632314847098|20200918/WebA|
|     20210922204730| 20210922204730_6_18|                18|         20200918/WebA|576debfb-4d83-4a8...|    4|WebA|20200918|进行修改|      -| 18|  18|1632314847098|20200918/WebA|
|     20210922204730| 20210922204730_6_19|                 3|         20200918/WebA|576debfb-4d83-4a8...|    1|WebA|20200918|进行修改|      -|  3|   3|1632314847098|20200918/WebA|
|     20210922204730| 20210922204730_6_20|                 9|         20200918/WebA|576debfb-4d83-4a8...|    1|WebA|20200918|进行修改|      -|  9|   9|1632314847098|20200918/WebA|
|     20210922204730| 20210922204730_5_15|                11|         20200918/WebA|a5e5db4c-c190-450...|    7|WebA|20200918|进行修改|      -| 11|  11|1632314847098|20200918/WebA|
|     20210922204730|  20210922204730_4_7|                 0|         20200918/WebA|b0e74482-77e8-46c...|    5|WebA|20200918|进行修改|      -|  0|   0|1632314847098|20200918/WebA|
|     20210922204730| 20210922204730_4_10|                 4|         20200918/WebA|b0e74482-77e8-46c...|    8|WebA|20200918|进行修改|      -|  4|   4|1632314847098|20200918/WebA|
|     20210922204730|  20210922204730_3_8|                 5|         20200918/WebA|33024f1a-0f88-412...|    2|WebA|20200918|进行修改|      -|  5|   5|1632314847098|20200918/WebA|
|     20210922204730| 20210922204730_3_11|                10|         20200918/WebA|33024f1a-0f88-412...|    7|WebA|20200918|进行修改|      -| 10|  10|1632314847098|20200918/WebA|
|     20210922204730|  20210922204730_1_9|                15|         20200918/WebA|6a2c512b-4ab2-4c1...|    5|WebA|20200918|进行修改|      -| 15|  15|1632314847098|20200918/WebA|
|     20210922204730| 20210922204730_1_12|                17|         20200918/WebA|6a2c512b-4ab2-4c1...|    1|WebA|20200918|进行修改|      -| 17|  17|1632314847098|20200918/WebA|
|     20210922204730| 20210922204730_1_13|                13|         20200918/WebA|6a2c512b-4ab2-4c1...|    2|WebA|20200918|进行修改|      -| 13|  13|1632314847098|20200918/WebA|
|     20210922204730| 20210922204730_1_14|                14|         20200918/WebA|6a2c512b-4ab2-4c1...|    4|WebA|20200918|进行修改|      -| 14|  14|1632314847098|20200918/WebA|
|     20210922204730|  20210922204730_2_3|                 1|         20200918/WebA|38318074-4111-451...|    0|WebA|20200918|进行修改|      -|  1|   1|1632314847098|20200918/WebA|
|     20210922204730|  20210922204730_2_4|                 7|         20200918/WebA|38318074-4111-451...|    8|WebA|20200918|进行修改|      -|  7|   7|1632314847098|20200918/WebA|
|     20210922204730|  20210922204730_2_5|                 8|         20200918/WebA|38318074-4111-451...|    7|WebA|20200918|进行修改|      -|  8|   8|1632314847098|20200918/WebA|
|     20210922204730|  20210922204730_2_6|                12|         20200918/WebA|38318074-4111-451...|    3|WebA|20200918|进行修改|      -| 12|  12|1632314847098|20200918/WebA|
|     20210922204730|  20210922204730_0_1|                 2|         20200918/WebA|0ac945ef-b1f1-4a6...|    3|WebA|20200918|进行修改|      -|  2|   2|1632314847098|20200918/WebA|
|     20210922204730|  20210922204730_0_2|                19|         20200918/WebA|0ac945ef-b1f1-4a6...|    3|WebA|20200918|进行修改|      -| 19|  19|1632314847098|20200918/WebA|
+-------------------+--------------------+------------------+----------------------+--------------------+-----+----+--------+--------+-------+---+----+-------------+-------------+

增量查询

(1)查询前40条数据,分别包含修改的20条数据和未修改的20条数据

scala> spark.sql("select * from member_rt order where uid>=0 and uid <40 order by uid").show(40);
+-------------------+--------------------+------------------+----------------------+--------------------+-----+----+--------+--------+-------+---+----+-------------+-------------+
|_hoodie_commit_time|_hoodie_commit_seqno|_hoodie_record_key|_hoodie_partition_path|   _hoodie_file_name|ad_id|  dn|      dt|fullname|iconurl|uid|uuid|           ts|hudipartition|
+-------------------+--------------------+------------------+----------------------+--------------------+-----+----+--------+--------+-------+---+----+-------------+-------------+
|     20210922204730|  20210922204730_4_7|                 0|         20200918/WebA|b0e74482-77e8-46c...|    5|WebA|20200918|进行修改|      -|  0|   0|1632314847098|20200918/WebA|
|     20210922204730|  20210922204730_2_3|                 1|         20200918/WebA|38318074-4111-451...|    0|WebA|20200918|进行修改|      -|  1|   1|1632314847098|20200918/WebA|
|     20210922204730|  20210922204730_0_1|                 2|         20200918/WebA|0ac945ef-b1f1-4a6...|    3|WebA|20200918|进行修改|      -|  2|   2|1632314847098|20200918/WebA|
|     20210922204730| 20210922204730_6_19|                 3|         20200918/WebA|576debfb-4d83-4a8...|    1|WebA|20200918|进行修改|      -|  3|   3|1632314847098|20200918/WebA|
|     20210922204730| 20210922204730_4_10|                 4|         20200918/WebA|b0e74482-77e8-46c...|    8|WebA|20200918|进行修改|      -|  4|   4|1632314847098|20200918/WebA|
|     20210922204730|  20210922204730_3_8|                 5|         20200918/WebA|33024f1a-0f88-412...|    2|WebA|20200918|进行修改|      -|  5|   5|1632314847098|20200918/WebA|
|     20210922204730| 20210922204730_7_17|                 6|         20200918/WebA|a0878fc7-7d6f-4ff...|    4|WebA|20200918|进行修改|      -|  6|   6|1632314847098|20200918/WebA|
|     20210922204730|  20210922204730_2_4|                 7|         20200918/WebA|38318074-4111-451...|    8|WebA|20200918|进行修改|      -|  7|   7|1632314847098|20200918/WebA|
|     20210922204730|  20210922204730_2_5|                 8|         20200918/WebA|38318074-4111-451...|    7|WebA|20200918|进行修改|      -|  8|   8|1632314847098|20200918/WebA|
|     20210922204730| 20210922204730_6_20|                 9|         20200918/WebA|576debfb-4d83-4a8...|    1|WebA|20200918|进行修改|      -|  9|   9|1632314847098|20200918/WebA|
|     20210922204730| 20210922204730_3_11|                10|         20200918/WebA|33024f1a-0f88-412...|    7|WebA|20200918|进行修改|      -| 10|  10|1632314847098|20200918/WebA|
|     20210922204730| 20210922204730_5_15|                11|         20200918/WebA|a5e5db4c-c190-450...|    7|WebA|20200918|进行修改|      -| 11|  11|1632314847098|20200918/WebA|
|     20210922204730|  20210922204730_2_6|                12|         20200918/WebA|38318074-4111-451...|    3|WebA|20200918|进行修改|      -| 12|  12|1632314847098|20200918/WebA|
|     20210922204730| 20210922204730_1_13|                13|         20200918/WebA|6a2c512b-4ab2-4c1...|    2|WebA|20200918|进行修改|      -| 13|  13|1632314847098|20200918/WebA|
|     20210922204730| 20210922204730_1_14|                14|         20200918/WebA|6a2c512b-4ab2-4c1...|    4|WebA|20200918|进行修改|      -| 14|  14|1632314847098|20200918/WebA|
|     20210922204730|  20210922204730_1_9|                15|         20200918/WebA|6a2c512b-4ab2-4c1...|    5|WebA|20200918|进行修改|      -| 15|  15|1632314847098|20200918/WebA|
|     20210922204730| 20210922204730_6_16|                16|         20200918/WebA|576debfb-4d83-4a8...|    6|WebA|20200918|进行修改|      -| 16|  16|1632314847098|20200918/WebA|
|     20210922204730| 20210922204730_1_12|                17|         20200918/WebA|6a2c512b-4ab2-4c1...|    1|WebA|20200918|进行修改|      -| 17|  17|1632314847098|20200918/WebA|
|     20210922204730| 20210922204730_6_18|                18|         20200918/WebA|576debfb-4d83-4a8...|    4|WebA|20200918|进行修改|      -| 18|  18|1632314847098|20200918/WebA|
|     20210922204730|  20210922204730_0_2|                19|         20200918/WebA|0ac945ef-b1f1-4a6...|    3|WebA|20200918|进行修改|      -| 19|  19|1632314847098|20200918/WebA|
|     20210918173310|20210918173310_7_...|                20|         20200918/WebA|0ac945ef-b1f1-4a6...|    6|WebA|20200918|    王20|      -| 20|  20|1631957586912|20200918/WebA|
|     20210918173310|20210918173310_4_...|                21|         20200918/WebA|b0e74482-77e8-46c...|    6|WebA|20200918|    王21|      -| 21|  21|1631957586912|20200918/WebA|
|     20210918173310|20210918173310_3_...|                22|         20200918/WebA|a5e5db4c-c190-450...|    8|WebA|20200918|    王22|      -| 22|  22|1631957586912|20200918/WebA|
|     20210918173310|20210918173310_2_...|                23|         20200918/WebA|a0878fc7-7d6f-4ff...|    1|WebA|20200918|    王23|      -| 23|  23|1631957586912|20200918/WebA|
|     20210918173310|20210918173310_1_...|                24|         20200918/WebA|576debfb-4d83-4a8...|    5|WebA|20200918|    王24|      -| 24|  24|1631957586912|20200918/WebA|
|     20210918173310|20210918173310_4_...|                25|         20200918/WebA|b0e74482-77e8-46c...|    3|WebA|20200918|    王25|      -| 25|  25|1631957586912|20200918/WebA|
|     20210918173310|20210918173310_4_...|                26|         20200918/WebA|b0e74482-77e8-46c...|    6|WebA|20200918|    王26|      -| 26|  26|1631957586912|20200918/WebA|
|     20210918173310|20210918173310_2_...|                27|         20200918/WebA|a0878fc7-7d6f-4ff...|    1|WebA|20200918|    王27|      -| 27|  27|1631957586912|20200918/WebA|
|     20210918173310|20210918173310_5_...|                28|         20200918/WebA|33024f1a-0f88-412...|    4|WebA|20200918|    王28|      -| 28|  28|1631957586912|20200918/WebA|
|     20210918173310|20210918173310_7_...|                29|         20200918/WebA|0ac945ef-b1f1-4a6...|    4|WebA|20200918|    王29|      -| 29|  29|1631957586912|20200918/WebA|
|     20210918173310|20210918173310_1_...|                30|         20200918/WebA|576debfb-4d83-4a8...|    8|WebA|20200918|    王30|      -| 30|  30|1631957586912|20200918/WebA|
|     20210918173310|20210918173310_5_...|                31|         20200918/WebA|33024f1a-0f88-412...|    5|WebA|20200918|    王31|      -| 31|  31|1631957586912|20200918/WebA|
|     20210918173310|20210918173310_3_...|                32|         20200918/WebA|a5e5db4c-c190-450...|    6|WebA|20200918|    王32|      -| 32|  32|1631957586912|20200918/WebA|
|     20210918173310|20210918173310_3_...|                33|         20200918/WebA|a5e5db4c-c190-450...|    7|WebA|20200918|    王33|      -| 33|  33|1631957586912|20200918/WebA|
|     20210918173310|20210918173310_7_...|                34|         20200918/WebA|0ac945ef-b1f1-4a6...|    0|WebA|20200918|    王34|      -| 34|  34|1631957586912|20200918/WebA|
|     20210918173310|20210918173310_0_...|                35|         20200918/WebA|6a2c512b-4ab2-4c1...|    6|WebA|20200918|    王35|      -| 35|  35|1631957586912|20200918/WebA|
|     20210918173310|20210918173310_3_...|                36|         20200918/WebA|a5e5db4c-c190-450...|    0|WebA|20200918|    王36|      -| 36|  36|1631957586912|20200918/WebA|
|     20210918173310|20210918173310_7_...|                37|         20200918/WebA|0ac945ef-b1f1-4a6...|    2|WebA|20200918|    王37|      -| 37|  37|1631957586912|20200918/WebA|
|     20210918173310|20210918173310_3_...|                38|         20200918/WebA|a5e5db4c-c190-450...|    6|WebA|20200918|    王38|      -| 38|  38|1631957586912|20200918/WebA|
|     20210918173310|20210918173310_7_...|                39|         20200918/WebA|0ac945ef-b1f1-4a6...|    4|WebA|20200918|    王39|      -| 39|  39|1631957586912|20200918/WebA|
+-------------------+--------------------+------------------+----------------------+--------------------+-----+----+--------+--------+-------+---+----+-------------+-------------+

发现最新修改的数据的时间发生了变化,未修改的数据和修改的数据对比明显

(2)可以看到进行修改的数据和修改的数据commit时间戳是不一样的。那么hudi提供使用时间戳增量查询。指定查询类型为增量查询,并且传入时间戳,那么spark会查询时间戳以后的数据。

DataSourceReadOptions.QUERY_TYPE_INCREMENTAL_OPT_VAL 

scala> import org.apache.hudi.DataSourceReadOptions
import org.apache.hudi.DataSourceReadOptions

scala> spark.read.format("hudi").option(DataSourceReadOptions.QUERY_TYPE_OPT_KEY, DataSourceReadOptions.QUERY_TYPE_INCREMENTAL_OPT_VAL).option(DataSourceReadOptions.BEGIN_INSTANTTIME_OPT_KEY,"20210922204730").load("/tmp/hudi/hivetest/").show(40);
21/09/23 11:46:34 WARN lineage.LineageWriter: Lineage directory /var/log/spark/lineage doesn't exist or is not writable. Lineage for this application will be disabled.
+-------------------+--------------------+------------------+----------------------+-----------------+-----+---+---+--------+-------+---+----+---+-------------+
|_hoodie_commit_time|_hoodie_commit_seqno|_hoodie_record_key|_hoodie_partition_path|_hoodie_file_name|ad_id| dn| dt|fullname|iconurl|uid|uuid| ts|hudipartition|
+-------------------+--------------------+------------------+----------------------+-----------------+-----+---+---+--------+-------+---+----+---+-------------+
+-------------------+--------------------+------------------+----------------------+-----------------+-----+---+---+--------+-------+---+----+---+-------------+


scala> spark.read.format("hudi").option(DataSourceReadOptions.QUERY_TYPE_OPT_KEY, DataSourceReadOptions.QUERY_TYPE_INCREMENTAL_OPT_VAL).option(DataSourceReadOptions.BEGIN_INSTANTTIME_OPT_KEY,"20210918173310").load("/tmp/hudi/hivetest/").show(40);
+-------------------+--------------------+------------------+----------------------+--------------------+-----+----+--------+--------+-------+---+----+-------------+-------------+
|_hoodie_commit_time|_hoodie_commit_seqno|_hoodie_record_key|_hoodie_partition_path|   _hoodie_file_name|ad_id|  dn|      dt|fullname|iconurl|uid|uuid|           ts|hudipartition|
+-------------------+--------------------+------------------+----------------------+--------------------+-----+----+--------+--------+-------+---+----+-------------+-------------+
|     20210922204730|  20210922204730_2_3|                 1|         20200918/WebA|38318074-4111-451...|    0|WebA|20200918|进行修改|      -|  1|   1|1632314847098|20200918/WebA|
|     20210922204730|  20210922204730_2_6|                12|         20200918/WebA|38318074-4111-451...|    3|WebA|20200918|进行修改|      -| 12|  12|1632314847098|20200918/WebA|
|     20210922204730|  20210922204730_2_4|                 7|         20200918/WebA|38318074-4111-451...|    8|WebA|20200918|进行修改|      -|  7|   7|1632314847098|20200918/WebA|
|     20210922204730|  20210922204730_2_5|                 8|         20200918/WebA|38318074-4111-451...|    7|WebA|20200918|进行修改|      -|  8|   8|1632314847098|20200918/WebA|
|     20210922204730| 20210922204730_7_17|                 6|         20200918/WebA|a0878fc7-7d6f-4ff...|    4|WebA|20200918|进行修改|      -|  6|   6|1632314847098|20200918/WebA|
|     20210922204730| 20210922204730_6_19|                 3|         20200918/WebA|576debfb-4d83-4a8...|    1|WebA|20200918|进行修改|      -|  3|   3|1632314847098|20200918/WebA|
|     20210922204730| 20210922204730_6_16|                16|         20200918/WebA|576debfb-4d83-4a8...|    6|WebA|20200918|进行修改|      -| 16|  16|1632314847098|20200918/WebA|
|     20210922204730| 20210922204730_6_18|                18|         20200918/WebA|576debfb-4d83-4a8...|    4|WebA|20200918|进行修改|      -| 18|  18|1632314847098|20200918/WebA|
|     20210922204730| 20210922204730_6_20|                 9|         20200918/WebA|576debfb-4d83-4a8...|    1|WebA|20200918|进行修改|      -|  9|   9|1632314847098|20200918/WebA|
|     20210922204730| 20210922204730_5_15|                11|         20200918/WebA|a5e5db4c-c190-450...|    7|WebA|20200918|进行修改|      -| 11|  11|1632314847098|20200918/WebA|
|     20210922204730|  20210922204730_0_1|                 2|         20200918/WebA|0ac945ef-b1f1-4a6...|    3|WebA|20200918|进行修改|      -|  2|   2|1632314847098|20200918/WebA|
|     20210922204730|  20210922204730_0_2|                19|         20200918/WebA|0ac945ef-b1f1-4a6...|    3|WebA|20200918|进行修改|      -| 19|  19|1632314847098|20200918/WebA|
|     20210922204730|  20210922204730_4_7|                 0|         20200918/WebA|b0e74482-77e8-46c...|    5|WebA|20200918|进行修改|      -|  0|   0|1632314847098|20200918/WebA|
|     20210922204730| 20210922204730_4_10|                 4|         20200918/WebA|b0e74482-77e8-46c...|    8|WebA|20200918|进行修改|      -|  4|   4|1632314847098|20200918/WebA|
|     20210922204730| 20210922204730_1_13|                13|         20200918/WebA|6a2c512b-4ab2-4c1...|    2|WebA|20200918|进行修改|      -| 13|  13|1632314847098|20200918/WebA|
|     20210922204730| 20210922204730_1_14|                14|         20200918/WebA|6a2c512b-4ab2-4c1...|    4|WebA|20200918|进行修改|      -| 14|  14|1632314847098|20200918/WebA|
|     20210922204730|  20210922204730_1_9|                15|         20200918/WebA|6a2c512b-4ab2-4c1...|    5|WebA|20200918|进行修改|      -| 15|  15|1632314847098|20200918/WebA|
|     20210922204730| 20210922204730_1_12|                17|         20200918/WebA|6a2c512b-4ab2-4c1...|    1|WebA|20200918|进行修改|      -| 17|  17|1632314847098|20200918/WebA|
|     20210922204730|  20210922204730_3_8|                 5|         20200918/WebA|33024f1a-0f88-412...|    2|WebA|20200918|进行修改|      -|  5|   5|1632314847098|20200918/WebA|
|     20210922204730| 20210922204730_3_11|                10|         20200918/WebA|33024f1a-0f88-412...|    7|WebA|20200918|进行修改|      -| 10|  10|1632314847098|20200918/WebA|
+-------------------+--------------------+------------------+----------------------+--------------------+-----+----+--------+--------+-------+---+----+-------------+-------------+

(3)DataSourceReadOptions.BEGIN_INSTANTTIME_OPT_KEY是开始时间戳,也对应有DataSourceReadOptions.END_INSTANTTIME_OPT_KEY结束时间戳,增量查询也可以给时间范围进行查询

你可能感兴趣的:(hudi,hive,spark,big,data)