spark断点调试

断点调试

参考：http://danosipov.com/?p=779

测试代码

object SparkTest {
  def main(args: Array[String]): Unit = {
    // Local
    val spark = new SparkContext(new SparkConf()
      .setMaster("local").setAppName("Test")
    )
    //spark.setCheckpointDir("/user/mai/input")

    println("-------------Attach debugger now!--------------")
    Thread.sleep(8000)

    // Your job code here, with breakpoints set on the lines you want to pause
    val s = "abc1"
    print(s)
  }

}

IntelliJ Idea 生成jar

• 设置artifacts

  
  
  

  image.png
• build artifacts

提交命令

export SPARK_JAVA_OPTS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005

./bin/spark-submit.sh --class com.huaan.bd.spark.SparkTest --master local[4] --driver-memory 1G --executor-memory 1G ./testjars/mix-bdbasic.jar

连接断点

代码中有8秒钟时间等待连接