频繁full gc 调参

Error message from spark is:java.lang.Exception: application_1678793738534_17900289 Driver Disassociated [akka.tcp://[email protected]:37931] <- [akka.tcp://[email protected]:38513] disassociated!

 

日志里频繁full gc ，可以用这里参数替换重跑看下：
set hive.exec.parallel = true;set hive.execute.engine=spark;set hive.multi.join.use.hive=false;set hive.spark.failed.retry=false;set spark.sql.relation.ignoreDefaultPartition=false;set spark.network.timeout=1200;set spark.driver.memory=16g; set spark.executor.memory=16g;set spark.partition.max=15999; set spark.task.shuffle.size = 536870910;set spark.sql.broadcastTimeout = 36000;