spark python 上传代码包_hue通过livy提交spark jar包python文件到spark
背景：集群中有hive、spark2，没有impala
需求：通过hue访问hive以及提交spark任务
现状：hue以及通过sentry实现了对hive的权限管理
CDH5.15.1
centos7.4
livy-0.5.0-incubating-bin
目标：hue实现scala、pyspark的交互式功能、实现提交python脚本，提交spark jar包到spark 的功能
1、部署安装livy。目前博客上很多是编译安装，我试着从github上找了一下，总是编译不成功(可能是环境不对，我的是jdk1.8)，最后没有编译，直接下载了livy官网版本。
[root@dip livy]# cd conf/[root@dip conf]#scp livy-env.sh.template livy-env.sh[root@dip conf]#scp spark-blacklist.conf.template spark-blacklist.conf
[root@dip conf]#scplivy.conf.template livy.conf
[root@dip conf]#chown hdfs:hdfs livy.conf livy-env.sh spark-blacklist.conf
[root@dip conf]# ll
在HDFS上创建livy的home目录
[root@dip conf]#sudo -u hdfs hadoop fs -mkdir /user/livy
[root@dip conf]#sudo -u hdfs hadoop fs -chown hdfs:hdfs /user/livy
[root@dip conf]#sudo -u hdfs hadoop fs -ls /user
vim livy.conf
livy.spark.master = yarn
livy.spark.deploy-mode = cluster
livy.environment = production
livy.impersonation.enabled = true
livy.server.port = 8998
livy.server.session.timeout = 3600000
livy.server.recovery.mode = recovery
livy.server.recovery.state-store=filesystem
livy.server.recovery.state-store.url=/opt/cslc/livy-0.5.0-incubating-bin
vim livy-env.sh
export JAVA_HOME=/opt/cslc/jdk1.8.0_151
export HADOOP_HOME=/opt/cloudera/parcels/CDH/lib/hadoop
export SPARK_CONF_DIR=/etc/spark2/conf
export SPARK_HOME=/opt/cloudera/parcels/SPARK2-2.3.0.cloudera3-1.cdh5.13.3.p0.458809/lib/spark2
export HADOOP_CONF_DIR=/etc/hadoop/conf
export LIVY_LOG_DIR=/home/cloudera_data_lib_log_tmp/log/livy/
export LIVY_SERVER_JAVA_OPTS="-Xmx2g"
启动：
使用hdfs用户启动 ：/opt/cloudera/livy-0.5.0-incubating-bin/bin/livy-server start
2.CDH集成livy
修改 core-site.xml 的群集范围高级配置代码段(安全阀)
hadoop.proxyuser.livy.groups
*
hadoop.proxyuser.livy.hosts
*
3.hue配置livy连接spark
修改hue_safety_valve.ini 的 Hue 服务高级配置代码段(安全阀)
[desktop]
app_blacklist=[spark]
livy_server_host=dip007
livy_server_port=8998livy_server_session_kind=yarn
[notebook]
show_notebooks=trueenable_batch_execute=trueenable_query_builder=trueenable_query_scheduling=false[[interpreters]]
[[[hive]]]
# The name of the snippet.
name=Hive
# The backend connection to use to communicate with the server.
interface=hiveserver2
[[[spark]]]
name=Scala
interface=livy
[[[pyspark]]]
name=PySpark
interface=livy
[[[jar]]]
name=Spark Submit Jar
interface=livy-batch
[[[py]]]
name=Spark Submit Python
interface=livy-batch
[[[text]]]
name=Text
interface=text
[[[spark2]]]
name=Spark
interface=oozie
[[[markdown]]]
name=Markdown
interface=text
[[[java]]]
name=Java
interface=oozie
[[[mapreduce]]]
name=MapReduce
interface=oozie
[[[distcp]]]
name=Distcp
interface=oozie
[[[shell]]]
name=Shell
interface=oozie
遇到的问题：
1.livy总是去连接一个处于stand by 的name node
尝试：配置hue使用高可用的yarn 无效
尝试2：之前安装是基于普通用户，后来所有安装后都换成hdfs用户，解决