读取HDFS文件目录的方法

欢迎使用Markdown编辑器写博客

首先添加依赖:

//注意后面的provided,缺少会报包冲突的错误
libraryDependencies += "org.apache.hadoop" % "hadoop-hdfs" % "2.7.2"%"provided"

import org.apache.hadoop.fs.{FileSystem, FileUtil, Path}

val hdfsPath = "hdfs://0.0.0.0:9000/user/myname/classify/model"
val conf = new Configuration()
val hdfs = FileSystem.get(URI.create(hdfsPath),conf)
val fs = hdfs.listStatus(new Path(hdfsPath))
val listPath = FileUtil.stat2Paths(fs)
for (l <- listPath){
  println(l.getName)
}

输出:
建筑.model
券商.model
计算机.model