spark Structured报错解决

报错,不想看原因的直接去解决方案试试

Exception in thread "main" java.lang.IllegalArgumentException: Pathname /C:/Users/Administrator/AppData/Local/Temp/1/temporary-611514af-8dc5-4b20-9237-e5f2d21fdf88/metadata from hdfs://master:8020/C:/Users/Administrator/AppData/Local/Temp/1/temporary-611514af-8dc5-4b20-9237-e5f2d21fdf88/metadata is not a valid DFS filename.
	at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:197)
	at org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:106)
	at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
	at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
	at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1426)
	at org.apache.spark.sql.execution.streaming.StreamMetadata$.read(StreamMetadata.scala:51)
	at org.apache.spark.sql.execution.streaming.StreamExecution.<init>(StreamExecution.scala:122)
	at org.apache.spark.sql.execution.streaming.MicroBatchExecution.<init>(MicroBatchExecution.scala:49)
	at org.apache.spark.sql.streaming.StreamingQueryManager.createQuery(StreamingQueryManager.scala:258)
	at org.apache.spark.sql.streaming.StreamingQueryManager.startQuery(StreamingQueryManager.scala:299)
	at org.apache.spark.sql.streaming.DataStreamWriter.start(DataStreamWriter.scala:296)
	at com.gugu.book.basespark.chapter08.StructuredNetworkWordCount$.main(StructuredNetworkWordCount.scala:25)
	at com.gugu.book.basespark.chapter08.StructuredNetworkWordCount.main(StructuredNetworkWordCount.scala)
看报错信息好像是要赋权,路径不存在,但是这个路径有点古怪

hdfs://master:8020/C:/Users/Administrator/AppData/Local/Temp/1/temporary-611514af-8dc5-4b20-9237-e5f2d21fdf88/metadata is not a valid DFS filename

按道理不应该出现C:/。。。。这个是我本地的一个地址,但是却拼接到了hdfs文件系统下,应该是什么配置的有问题
顺着报错信息

spark Structured报错解决_第1张图片
只需要checkpointRoot修改就行这个地址来源于checkpointLocation
org.apache.spark.sql.streaming.StreamingQueryManager#createQuery
spark Structured报错解决_第2张图片

也就是说直接改变“checkpointLocation”的值就行

解决方案

在start之前设置

            .option("checkpointLocation", "file:///D:\\applicationfiles\\data\\kafka")

spark Structured报错解决_第3张图片
解决,完美
环境问题,在windows下有些默认配置会失效

你可能感兴趣的:(spark,异常报错)