Flink On Hudi整个系列中可能遇到的问题
1、ERROR org.apache.hudi.sink.compact.CompactFunction [] - Executor executes action [Execute compaction for instant 20220331114224581 from task 0] error
ERROR org.apache.hudi.sink.compact.CompactFunction [] - Executor executes action [Execute compaction for instant 20220331114224581 from task 0] error
java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/lib/input/FileInputFormat
at org.apache.hudi.org.apache.parquet.HadoopReadOptions$Builder.
at org.apache.hudi.org.apache.parquet.HadoopReadOptions.builder(HadoopReadOptions.java:79) ~[hudi-flink-bundle_2.11-0.10.1.jar:0.10.1]
at org.apache.hudi.org.apache.parquet.hadoop.ParquetReader$Builder.
at org.apache.hudi.org.apache.parquet.avro.AvroParquetReader$Builder.
at org.apache.hudi.org.apache.parquet.avro.AvroParquetReader$Builder.
at org.apache.hudi.org.apache.parquet.avro.AvroParquetReader.builder(AvroParquetReader.java:48) ~[hudi-flink-bundle_2.11-0.10.1.jar:0.10.1]
at org.apache.hudi.io.storage.HoodieParquetReader.getRecordIterator(HoodieParquetReader.java:65) ~[hudi-flink-bundle_2.11-0.10.1.jar:0.10.1]
at org.apache.hudi.table.action.commit.FlinkMergeHelper.runMerge(FlinkMergeHelper.java:89) ~[hudi-flink-bundle_2.11-0.10.1.jar:0.10.1]
at org.apache.hudi.table.HoodieFlinkCopyOnWriteTable.handleUpdateInternal(HoodieFlinkCopyOnWriteTable.java:368) ~[hudi-flink-bundle_2.11-0.10.1.jar:0.10.1]
at org.apache.hudi.table.HoodieFlinkCopyOnWriteTable.handleUpdate(HoodieFlinkCopyOnWriteTable.java:359) ~[hudi-flink-bundle_2.11-0.10.1.jar:0.10.1]
at org.apache.hudi.table.action.compact.HoodieCompactor.compact(HoodieCompactor.java:197) ~[hudi-flink-bundle_2.11-0.10.1.jar:0.10.1]
at org.apache.hudi.sink.compact.CompactFunction.doCompaction(CompactFunction.java:104) ~[hudi-flink-bundle_2.11-0.10.1.jar:0.10.1]
at org.apache.hudi.sink.compact.CompactFunction.lambda$processElement$0(CompactFunction.java:92) ~[hudi-flink-bundle_2.11-0.10.1.jar:0.10.1]
at org.apache.hudi.sink.utils.NonThrownExecutor.lambda$execute$0(NonThrownExecutor.java:93) ~[hudi-flink-bundle_2.11-0.10.1.jar:0.10.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_291]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_291]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_291]
--解决办法
添加jar包。把hadoop目录中的share目录下的 mapredurce相关的jar包，全部引入到 flink的lib目录下
2、JobManager 报错信息
java.lang.NoClassDefFoundError: org/apache/flink/shaded/guava18/com/google/common/util/concurrent/ThreadFactoryBuilder
at com.alibaba.ververica.cdc.debezium.DebeziumSourceFunction.open(DebeziumSourceFunction.java:166) ~[flink-connector-mysql-cdc-1.4.0.jar:1.4.0]
at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:34) ~[flink-dist_2.11-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:100) ~[flink-dist_2.11-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:110) ~[flink-dist_2.11-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:711) ~[flink-dist_2.11-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.call(StreamTaskActionExecutor.java:100) ~[flink-dist_2.11-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:687) ~[flink-dist_2.11-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:654) ~[flink-dist_2.11-1.14.3.jar:1.14.3]
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958) ~[flink-dist_2.11-1.14.3.jar:1.14.3]
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:927) ~[flink-dist_2.11-1.14.3.jar:1.14.3]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766) ~[flink-dist_2.11-1.14.3.jar:1.14.3]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575) ~[flink-dist_2.11-1.14.3.jar:1.14.3]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_291]
Caused by: java.lang.ClassNotFoundException: org.apache.flink.shaded.guava18.com.google.common.util.concurrent.ThreadFactoryBuilder
at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[?:1.8.0_291]
at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_291]
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355) ~[?:1.8.0_291]
at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[?:1.8.0_291]
... 13 more
TaskManager报错信息
org.apache.hadoop.conf.Configuration [] - error parsing conf /Users/lishe.ch/Documents/Ali-work/software/hadoop-3.3.1/etc/hadoop/core-site.xml
com.ctc.wstx.exc.WstxIOException: null
at com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:578) ~[woodstox-core-5.3.0.jar:5.3.0]
at com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:633) ~[woodstox-core-5.3.0.jar:5.3.0]
at org.apache.hadoop.conf.Cofiguration.parse(Configuration.java:3002) ~[hadoop-common-3.3.1.jar:?]
at org.apache.hadoop.conf.Configuration.getStreamReader(Configuration.java:3092) ~[hadoop-common-3.3.1.jar:?]
at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3040) ~[hadoop-common-3.3.1.jar:?]
at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:3018) ~[hadoop-common-3.3.1.jar:?]
at org.apache.hadoop.conf.Configuration.loadProps(Configuration.java:2893) ~[hadoop-common-3.3.1.jar:?]
at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2875) ~[hadoop-common-3.3.1.jar:?]
at org.apache.hadoop.conf.Configuration.get(Configuration.java:1225) ~[hadoop-common-3.3.1.jar:?]
at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1279) ~[hadoop-common-3.3.1.jar:?]
at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1686) ~[hadoop-common-3.3.1.jar:?]
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:535) ~[hadoop-common-3.3.1.jar:?]
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365) ~[hadoop-common-3.3.1.jar:?]
at org.apache.hudi.common.fs.FSUtils.getFs(FSUtils.java:102) ~[hudi-flink-bundle_2.11-0.10.1.jar:0.10.1]
at org.apache.hudi.util.ViewStorageProperties.loadFromProperties(ViewStorageProperties.java:67) ~[hudi-flink-bundle_2.11-0.10.1.jar:0.10.1]
at org.apache.hudi.util.StreamerUtil.getHoodieClientConfig(StreamerUtil.java:217) ~[hudi-flink-bundle_2.11-0.10.1.jar:0.10.1]
at org.apache.hudi.util.StreamerUtil.getHoodieClientConfig(StreamerUtil.java:156) ~[hudi-flink-bundle_2.11-0.10.1.jar:0.10.1]
at org.apache.hudi.util.StreamerUtil.createWriteClient(StreamerUtil.java:386) ~[hudi-flink-bundle_2.11-0.10.1.jar:0.10.1]
at org.apache.hudi.util.StreamerUtil.createWriteClient(StreamerUtil.java:370) ~[hudi-flink-bundle_2.11-0.10.1.jar:0.10.1]
at org.apache.hudi.sink.common.AbstractStreamWriteFunction.initializeState(AbstractStreamWriteFunction.java:130) ~[hudi-flink-bundle_2.11-0.10.1.jar:0.10.1]
at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:189) ~[flink-dist_2.11-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:171) ~[flink-dist_2.11-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:94) ~[flink-dist_2.11-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:119) ~[flink-dist_2.11-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:286) ~[flink-dist_2.11-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:109) ~[flink-dist_2.11-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:711) ~[flink-dist_2.11-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55) ~[flink-dist_2.11-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:687) ~[flink-dist_2.11-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:654) ~[flink-dist_2.11-1.14.3.jar:1.14.3]
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958) [flink-dist_2.11-1.14.3.jar:1.14.3]
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:927) [flink-dist_2.11-1.14.3.jar:1.14.3]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766) [flink-dist_2.11-1.14.3.jar:1.14.3]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575) [flink-dist_2.11-1.14.3.jar:1.14.3]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_291]
Caused by: java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202) ~[?:1.8.0_291]
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:164) ~[?:1.8.0_291]
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65) ~[?:1.8.0_291]
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109) ~[?:1.8.0_291]
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) ~[?:1.8.0_291]
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) ~[?:1.8.0_291]
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) ~[?:1.8.0_291]
at java.io.BufferedInputStream.read(BufferedInputStream.java:345) ~[?:1.8.0_291]
at com.ctc.wstx.io.StreamBootstrapper.ensureLoaded(StreamBootstrapper.java:482) ~[woodstox-core-5.3.0.jar:5.3.0]
at com.ctc.wstx.io.StreamBootstrapper.resolveStreamEncoding(StreamBootstrapper.java:306) ~[woodstox-core-5.3.0.jar:5.3.0]
at com.ctc.wstx.io.StreamBootstrapper.bootstrapInput(StreamBootstrapper.java:167) ~[woodstox-core-5.3.0.jar:5.3.0]
at com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:573) ~[woodstox-core-5.3.0.jar:5.3.0]
... 34 more
2022-04-01 11:42:21,163 INFO io.debezium.connector.mysql.SnapshotReader [] - Cannot determine the privileges for 'root'
java.sql.SQLException: ResultSet is from UPDATE. No Data.
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129) ~[flink-connector-mysql-cdc-1.4.0.jar:1.4.0]
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97) ~[flink-connector-mysql-cdc-1.4.0.jar:1.4.0]
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:89) ~[flink-connector-mysql-cdc-1.4.0.jar:1.4.0]
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:63) ~[flink-connector-mysql-cdc-1.4.0.jar:1.4.0]
at com.mysql.cj.jdbc.result.ResultSetImpl.next(ResultSetImpl.java:1731) ~[flink-connector-mysql-cdc-1.4.0.jar:1.4.0]
at io.debezium.connector.mysql.SnapshotReader.lambda$logRolesForCurrentUser$20(SnapshotReader.java:1021) ~[flink-connector-mysql-cdc-1.4.0.jar:1.4.0]
at io.debezium.jdbc.JdbcConnection.query(JdbcConnection.java:556) ~[flink-connector-mysql-cdc-1.4.0.jar:1.4.0]
at io.debezium.jdbc.JdbcConnection.query(JdbcConnection.java:497) ~[flink-connector-mysql-cdc-1.4.0.jar:1.4.0]
at io.debezium.connector.mysql.SnapshotReader.logRolesForCurrentUser(SnapshotReader.java:1020) ~[flink-connector-mysql-cdc-1.4.0.jar:1.4.0]
at io.debezium.connector.mysql.SnapshotReader.execute(SnapshotReader.java:261) ~[flink-connector-mysql-cdc-1.4.0.jar:1.4.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_291]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_291]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_291]
出现如上信息，请替换下lib目录下的“flink-shaded-guava”jar包，替换成flink-shaded-guava-18.0-13.0.jar
3、
java.lang.NoSuchMethodError: org.apache.flink.api.common.functions.RuntimeContext.getMetricGroup()Lorg/apache/flink/metrics/MetricGroup;
at com.alibaba.ververica.cdc.debezium.DebeziumSourceFunction.run(DebeziumSourceFunction.java:379)
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:110)
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:67)
at org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:323)
如下这个错误，如果出现了，就去flink集群的lib目录下查看下，是否有"flink-dist"开头的jar包，如果有的话，用反编译工具查看下，里面的这个RuntimeContext类里是否有“getMetricGroup”这个方法，如果有的话，那就不是字面意思上的缺少这个方法了，那很有可能就是版本不对导致的
，最终大部分情况下，这个错误的原因就是因为使用的flink-mysql-cdc的版本不对导致的，需要替换成2.2的版本就可以了。