spark 并发起作业CompletableFuture

val result = tasksPrd
  .map(e => SparkJdbcConfig(url,
    e.split("\\.")(0),
    e.split("\\.")(1),
    predicates,
    properties))
  .map(e => new StatisticTask(statisticResult, spark, e))
  .map(e => {
    CompletableFuture.supplyAsync(e, threadPool)
  })

val tasksResult = CompletableFuture.allOf(result: _*)

tasksResult.join
tasksResult.get
threadPool.shutdownNow()

你可能感兴趣的:(spark 并发起作业CompletableFuture)