spark-submit 提示错误java.lang.IllegalArgumentException: System memory 468189184 must be at least 4.7185

在运行spark-submit时会报错，是因为内存不足导致的，但是配置了driver-memory和executor-memory时都不行，报错：

ERROR SparkContext: Error initializing SparkContext.

java.lang.IllegalArgumentException: System memory 468189184 must be at least 4.718592E8. Please use a larger heap size.

通过查看spark源码，发现源码是这么写的：

/**

   * Return the total amount of memory shared between execution and storage, in bytes.

   */

  private def getMaxMemory(conf: SparkConf): Long = {

    val systemMemory = conf.getLong("spark.testing.memory", Runtime.getRuntime.maxMemory)

    val reservedMemory = conf.getLong("spark.testing.reservedMemory",

      if (conf.contains("spark.testing")) 0 else RESERVED_SYSTEM_MEMORY_BYTES)

    val minSystemMemory = reservedMemory * 1.5

    if (systemMemory < minSystemMemory) {

      throw new IllegalArgumentException(s"System memory $systemMemory must " +

        s"be at least $minSystemMemory. Please use a larger heap size.")

    }

    val usableMemory = systemMemory - reservedMemory

    val memoryFraction = conf.getDouble("spark.memory.fraction", 0.75)

    (usableMemory * memoryFraction).toLong

  }

主要是标黑的地方的问题，获取了配置参数：spark.testing.memory参数，所以解决方法如下：

方法1：在源代码中加入 val conf=new SparkConf).setAppName("s")

                         conf.set("spark.testing.memory","2147480000") //后面的值大于512M即可。

方法2：可以在Eclipse的Run Configuration处，有一栏是Arguments，下面有VMarguments，在下面添加下面一行(值也是只要大于512m即可)

-Dspark.testing.memory=1073741824

方法3：在saprk提交方式是：yarn-cluster模式时，在spark-default.conf文件中配置参数spark.testing.memory=1073741824 

还有配置VM 的内存也是在该文件中配置

spark.driver.extraJavaOptions -XX:PermSize=256M -XX:MaxPermSize=512M