spark sql hive小文件优化

 
  
sparksession.sqlContext.setConf("hive.merge.mapfiles","true")
sparksession.sqlContext.setConf("mapred.max.split.size","256000000")
sparksession.sqlContext.setConf("mapred.min.split.size.per.node","192000000")
sparksession.sqlContext.setConf("mapred.min.split.size.per.rack","192000000")
sparksession.sqlContext.setConf("hive.input.format","org.apache.hadoop.hive.ql.io.CombineHiveInputFormat")


你可能感兴趣的:(spark)