Spark: org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow

运行Spark程序的时候报出:
Exception in thread "main" java.sql.SQLException: org.apache.spark.SparkException: 
Job aborted due to stage failure: Task 3107 in stage 308.0 failed 4 times, most recent failure: Lost task 3107.3 in stage 308.0 (TID 620318, XXX): org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow. Available: 1572864, required: 
3236381Serialization trace:

。。。。。


解决的方法是修改这个参数的配置数量:


Spark: org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow_第1张图片


调大对应的参数即可。


你可能感兴趣的:(Java,大数据挖掘与大数据应用案例)