HDFS小文件合并时遇到的几个小问题

1. 如何“非正则”地遍历一个目录下某些特定的文件?

在spark中newHadoopAPI()或者FSShell其实都是支持正则话输入路径的,不要怕输入的路径太长,一定要正则化输入路径;

  • https://stackoverflow.com/questions/31782763/how-to-use-regex-to-include-exclude-some-input-files-in-sc-textfile
  • https://stackoverflow.com/questions/24224392/why-does-spark-throw-notserializableexception-org-apache-hadoop-io-nullwritable

2. 在读取->repartition->重新再写入的过程中,遇到序列化失败该如何处理?

尽量使用kyro序列化,兼容性还是比较好;
同时注意null值的处理;

  • http://apache-spark-developers-list.1001551.n3.nabble.com/NullWritable-not-serializable-td8412.html
  • https://stackoverflow.com/questions/22592811/task-not-serializable-java-io-notserializableexception-when-calling-function-ou/22594142#22594142
  • https://issues.apache.org/jira/browse/SPARK-2421
  • https://spark.apache.org/docs/latest/tuning.html

你可能感兴趣的:(HDFS小文件合并时遇到的几个小问题)