启动spark的pyspark命令窗口时报错
pyspark.sql.utils.IllegalArgumentException: u"Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState':"
(1)具体命令如下:
1、切换到sbin下,具体如下
/Users/a6/Applications/spark-2.1.0-bin-hadoop2.6/sbin
2、启动spark服务
xxxxxx$ sh start-all.sh
org.apache.spark.deploy.master.Master running as process 622. Stop it first.
localhost: org.apache.spark.deploy.worker.Worker running as process 666. Stop it first.
3、切换到bin目录下,运行pyspark脚本
xxxxxx$cd /Users/a6/Applications/spark-2.1.0-bin-hadoop2.6/bin
xxxxxx$ ./pyspark
... 105 more
Traceback (most recent call last):
File "/Users/a6/Applications/spark-2.1.0-bin-hadoop2.6/python/pyspark/shell.py", line 43, in
spark = SparkSession.builder\
File "/Users/a6/Applications/spark-2.1.0-bin-hadoop2.6/python/pyspark/sql/session.py", line 179, in getOrCreate
session._jsparkSession.sessionState().conf().setConfString(key, value)
File "/Library/Python/2.7/site-packages/py4j-0.10.6-py2.7.egg/py4j/java_gateway.py", line 1160, in __call__
answer, self.gateway_client, self.target_id, self.name)
File "/Users/a6/Applications/spark-2.1.0-bin-hadoop2.6/python/pyspark/sql/utils.py", line 79, in deco
raise IllegalArgumentException(s.split(': ', 1)[1], stackTrace)
pyspark.sql.utils.IllegalArgumentException: u"Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState':"
(2)网上有很多pyspark.sql.utils.IllegalArgumentException的异常的,但是关于u"Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState':" 的页面比较少。
百度一小时,未果,认真查看报错日志,原因可能是
Caused by: ERROR XSDB6: Another instance of Derby may have already booted the database /home/hadoop/metastore_db.
于是解决方案是:重启mac,再次执行,便没有错误。
参见:
https://stackoverflow.com/questions/43551062/amazon-emr-w-spark-w-postgres-failed-to-start-database-metastore-db