Spark Submit提交时，Json字符串作为参数

今天遇到一个把json作为参数 传入spark程序中的问题

原因如下，Spark 源码 中会对把}} 和 {{替换掉

@VisibleForTesting
  public static String expandEnvironment(String var,
      Path containerLogDir) {
    var = var.replace(ApplicationConstants.LOG_DIR_EXPANSION_VAR,
      containerLogDir.toString());
    var =  var.replace(ApplicationConstants.CLASS_PATH_SEPARATOR,
      File.pathSeparator);

    // replace parameter expansion marker. e.g. {{VAR}} on Windows is replaced
    // as %VAR% and on Linux replaced as "$VAR"
    if (Shell.WINDOWS) {
      var = var.replaceAll("(\\{\\{)|(\\}\\})", "%");
    } else {
      var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_LEFT, "$");
      var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_RIGHT, "");
    }
    return var;
  }

造成的结果就是，json传到程序中，已经变成一个不完整的JSON字符串了

20/06/04 12:21:14 ERROR yarn.ApplicationMaster: User class threw exception: com.alibaba.fastjson.JSONException: syntax error
com.alibaba.fastjson.JSONException: syntax error
    at com.alibaba.fastjson.parser.DefaultJSONParser.parseObject(DefaultJSONParser.java:528)
    at com.alibaba.fastjson.parser.DefaultJSONParser.parseObject(DefaultJSONParser.java:564)
    at com.alibaba.fastjson.parser.DefaultJSONParser.parse(DefaultJSONParser.java:1400)
    at com.alibaba.fastjson.parser.DefaultJSONParser.parse(DefaultJSONParser.java:1366)
    at com.alibaba.fastjson.JSON.parse(JSON.java:170)
    at com.alibaba.fastjson.JSON.parse(JSON.java:180)
    at com.alibaba.fastjson.JSON.parse(JSON.java:149)
    at com.alibaba.fastjson.JSON.parseObject(JSON.java:241)
    at cn.richinfo.dataclear.operator.SingleTableDataCleaningNew.enter(SingleTableDataCleaningNew.java:61)
    at cn.richinfo.dataclear.operator.MainEntrance.main(MainEntrance.java:48)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:646)
20/06/04 12:21:14 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: com.alibaba.fastjson.JSONException: syntax error)

我的解决方案就是

在}}中间加上一个空格} },之后，就不会被替换掉

Especially this line var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_RIGHT, "");. The right braces will be replaced with empty String.

So this is actually not a Spark issue, and may not easily be fixed by Yarn. But you can have many workarounds to avoid passing json String with arguments.

网上这个人说的，还有别的方式比较好的，就不得而知了