Cloudera Manager启动Spark报内存超出限制的错误：Required executor memory (1024+384 MB) is above the max threshold

启动Spark-Shell报错：
java.lang.IllegalArgumentException: Required executor memory (1024+384 MB) is above the max threshold (1024 MB) of this cluster! Please check the values of ‘yarn.scheduler.maximum-allocation-mb’ and/or ‘yarn.nodemanager.resource.memory-mb’.

解决办法：

将报错的2个参数增大到1434MB

然后重启Yarn服务

启动完毕后，重新运行Spark，报新的错误：
19/01/07 22:07:33 ERROR spark.SparkContext: Error initializing SparkContext.
org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x

于是切换成hdfs用户运行spark，这次没有任何错误，启动成功。

sudo -u hdfs spark-shell

--------------------- 

原文：https://blog.csdn.net/sunfect/article/details/86031857