前置环境说明：
centos 6.10
hadoop 2.7.3
oracle jdk1.8
flink1.8.1
scala 2.11
安装完集群之后提交一个jar包中的类运行报错 , （这个类里面写了操作HDFS的内容）
2019-09-02 18:42:01,651 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job DimensionState (1b47949b019929e25b8bee89539f9f1d) switched from state FAILING to FAILED.
java.lang.RuntimeException: Error while creating FileSystem when initializing the state of the BucketingSink.
at org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink.initializeState(BucketingSink.java:379)
at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)
at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)
at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:278)
at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:738)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:289)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 'hdfs'. The scheme is not directly supported by Flink and no Hadoop file system to support this scheme could be loaded.
at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:403)
at org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink.createHadoopFileSystem(BucketingSink.java:1227)
at org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink.initFileSystem(BucketingSink.java:432)
at org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink.initializeState(BucketingSink.java:376)
... 8 more
Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Hadoop is not in the classpath/dependencies.
at org.apache.flink.core.fs.UnsupportedSchemeFactory.create(UnsupportedSchemeFactory.java:64)
at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:399)
... 11 more
2019-09-02 18:42:01,652 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Could not restart the job DimensionState (1b47949b019929e25b8bee89539f9f1d) because the restart strategy prevented it.
java.lang.RuntimeException: Error while creating FileSystem when initializing the state of the BucketingSink.
at org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink.initializeState(BucketingSink.java:379)
at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)
at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)
at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:278)
at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:738)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:289)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 'hdfs'. The scheme is not directly supported by Flink and no Hadoop file system to support this scheme could be loaded.
at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:403)
at org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink.createHadoopFileSystem(BucketingSink.java:1227)
at org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink.initFileSystem(BucketingSink.java:432)
at org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink.initializeState(BucketingSink.java:376)
... 8 more
Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Hadoop is not in the classpath/dependencies.
at org.apache.flink.core.fs.UnsupportedSchemeFactory.create(UnsupportedSchemeFactory.java:64)
at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:399)
... 11 more
2019-09-02 18:42:01,652 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Stopping checkpoint coordinator for job 1b47949b019929e25b8bee89539f9f1d.
2019-09-02 18:42:01,652 INFO org.apache.flink.runtime.checkpoint.StandaloneCompletedCheckpointStore - Shutting down
2019-09-02 18:42:01,662 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Job 1b47949b019929e25b8bee89539f9f1d reached globally terminal state FAILED.
2019-09-02 18:42:01,704 INFO org.apache.flink.runtime.jobmaster.JobMaster - Stopping the JobMaster for job DimensionState(1b47949b019929e25b8bee89539f9f1d).
2019-09-02 18:42:01,711 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Suspending SlotPool.
2019-09-02 18:42:01,711 INFO org.apache.flink.runtime.jobmaster.JobMaster - Close ResourceManager connection 0f8cea6c8e8a3904e338bfe4b4aca43b: JobManager is shutting down..
2019-09-02 18:42:01,711 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Stopping SlotPool.
2019-09-02 18:42:01,711 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager - Disconnect job manager [email protected]://flink@localhost:6123/user/jobmanager_0 for job 1b47949b019929e25b8bee89539f9f1d from the resource manager.
2019-09-02 18:42:01,712 INFO org.apache.flink.runtime.jobmaster.JobManagerRunner - JobManagerRunner already shutdown.我提取一下上面关键字 ,方便搜索引擎爬取, 以试图帮助其他人:
1. while creating FileSystem when initializing the state of the BucketingSink.
2. Could not find a file system implementation for scheme 'hdfs'. The scheme is not directly supported by Flink and no Hadoop file system to support this scheme could be loaded.
3.org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Hadoop is not in the classpath/dependencies.
1.8.1如果要操作hdfs的话，必须要在flink安装目录的 lib 下加上额外的jar包
官网有介绍:
https://ci.apache.org/projects/flink/flink-docs-release-1.8/release-notes/flink-1.8.html
所以这里原因就是在flink集群的CLASS_PATH下缺少了 HDFS相关的ja
解决办法:
jar包下载地址:
https://repo.maven.apache.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.7.5-7.0/flink-shaded-hadoop-2-uber-2.7.5-7.0.jar