Spark Rest API 使用

一、地址和端口问题

spark官网 说默认端口是 4040，使用spark.ui.port 进行配置spark配置。

我们使用的是CDH，使用 spark.ui.port 查看没有找到该配置

image.png

• 搜索 port，找到如下几个端口，看着也就 18088比较像
  

  image.png

• 使用如下命令访问

curl http://xxx.xxx.xxx.xxx:18088/api/v1/applications?limit=2

得到如下结果，看着挺像那么回事，仔细一看，怎么都是历史的任务？

[ {
  "id" : "application_1618400751843_125338",
  "name" : "cat3-title-vec",
  "attempts" : [ {
    "startTime" : "2021-05-13T13:33:24.965GMT",
    "endTime" : "2021-05-13T13:34:41.551GMT",
    "lastUpdated" : "2021-05-13T13:34:46.298GMT",
    "duration" : 76586,
    "sparkUser" : "supdev",
    "completed" : true,
    "appSparkVersion" : "2.4.0-cdh6.2.0",
    "startTimeEpoch" : 1620912804965,
    "lastUpdatedEpoch" : 1620912886298,
    "endTimeEpoch" : 1620912881551
  } ]
}, {
  "id" : "application_1618400751843_125318",
  "name" : "re-list-demo",
  "attempts" : [ {
    "startTime" : "2021-05-13T13:26:58.297GMT",
    "endTime" : "2021-05-13T13:28:58.500GMT",
    "lastUpdated" : "2021-05-13T13:28:58.570GMT",
    "duration" : 120203,
    "sparkUser" : "supdev",
    "completed" : true,
    "appSparkVersion" : "2.4.0-cdh6.2.0",
    "startTimeEpoch" : 1620912418297,
    "lastUpdatedEpoch" : 1620912538570,
    "endTimeEpoch" : 1620912538500
  } ]
} ]

• 加上状态

curl "http://sj-20-207-14:18088/api/v1/applications?limit=2&status=running"

还是历史

[ {
  "id" : "application_1618400751843_99465",
  "name" : "Hive on Spark (hiveSessionId = 4f556621-4e68-40d5-a416-8f1fc6a3ea9e)",
  "attempts" : [ {
    "attemptId" : "1",
    "startTime" : "2021-05-08T09:50:00.273GMT",
    "endTime" : "1969-12-31T23:59:59.999GMT",
    "lastUpdated" : "2021-05-08T10:36:31.654GMT",
    "duration" : 0,
    "sparkUser" : "jiahang.liu",
    "completed" : false,
    "appSparkVersion" : "2.4.0-cdh6.2.0",
    "startTimeEpoch" : 1620467400273,
    "lastUpdatedEpoch" : 1620470191654,
    "endTimeEpoch" : -1
  } ]
}, {
  "id" : "application_1618400751843_93669",
  "name" : "Hive on Spark (hiveSessionId = f31a0dec-abbe-4418-bf7f-98428a7cb44a)",
  "attempts" : [ {
    "attemptId" : "1",
    "startTime" : "2021-05-07T13:30:08.725GMT",
    "endTime" : "1969-12-31T23:59:59.999GMT",
    "lastUpdated" : "2021-05-07T14:58:18.949GMT",
    "duration" : 0,
    "sparkUser" : "supdev",
    "completed" : false,
    "appSparkVersion" : "2.4.0-cdh6.2.0",
    "startTimeEpoch" : 1620394208725,
    "lastUpdatedEpoch" : 1620399498949,
    "endTimeEpoch" : -1
  } ]
} ]

未完待续