在脚本中导入pyspark的流程
import os
import sys
spark_name = os.environ.get('SPARK_HOME',None)
# SPARK_HOME即spark的安装目录，不用到bin级别，一般为/usr/local/spark
if not spark_home:
raise ValueErrorError('spark 环境没有配置好')
# sys.path是Python的第三方包查找的路径列表，将需要导入的包的路径添加进入，避免 can't find modal xxxx
# 这个方法应该同 spark-submit提交时添加参数 --py_files='/path/to/my/python/packages.zip',将依赖包打包成zip 添加进去 效果一致
sys.path.insert(0,'/root/virtualenvs/my_envs/lib/python3.6/site-packages/')
sys.path.insert(0,os.path.join(spark_name,'python')
sys.path.insert(0,os.path.join(spark_name,'python/lib/py4j-0.10.7-src.zip'))
# sys.path.insert(0,os.path.join(spark_name,'libexec/python'))
# sys.path.insert(0,os.path.join(spark_name,'libexex/python/build'))
from pyspark import SparkConf, SparkContext
设置pyspark运行时的python版本
vi ~/.bashrc
export PYSPARK_PYTHON=/usr/local/bin/python3
export PYSPARK_DRIVER_PYTHON=ipython3
编辑完保存退出
source ~/.bashrc
使用pyspark处理hbase缺少jar包时需配置环境
spark加载配置的默认目录是 SPARK_HOME/conf/spark-env.sh ,不存在此目录此文件时可自行创建
一般来说在spark-env.sh的末尾需要添加几行
export SPARK_DIST_CLASSPATH=$(/usr/local/hadoop/bin/hadoop classpath) 不添加这一行可能导致java class not found 之类的异常
export JAVA_HOME=/usr/java/jdk1.8.0_191-amd64/jre
export HADOOP_HOME=/usr/local/hadoop
export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
export SPARK_MASTER_HOST=HDP-master
export SPARK_WORKER_CORES=4 设置每个worker最多使用的核数，可设置为机器的内核数
export SPARK_WORKER_MEMORY=4g 设置每个worker最多使用的内存
spark处理hbase时需要一些hbase的jar包，可以在SPARK_HOME/jars/下新建一个hbase目录，然后将HBASE_HOME/lib/下面的相关包都复制过来
（也可单独复制lib目录下的这些包 hbase*.jar ,guava-12.0.1.jar,htrace-core-3.1.0-incubating.jar , protobuf-java-2.5.0.jar ）
另外需下载把hbase的数据转换为Python可读取的jar包 spark-example-1.6.0.jar
(下载页面地址为https://mvnrepository.com/artifact/org.apache.spark/spark-example_2.11/1.6.0-typesafe-001 )
这样就需要将spark-env.sh中的SPARK_DIST_CLASSPATH的值修改为
export SPARK_DIST_CLASSPATH=$(/usr/local/hadoop/bin/hadoop classpath):$(/usr/local/hbase/bin/hbase classpath):/usr/local/spark/jars/hbase/*
使用spark读写hbase的相关代码流程
host = 'master,slave1,slave2'
hbase_table = 'TEST:test1'
conf = {"hbase.zookeeper.quorum":host,"hbase.mapreduce.inputtable":hbase_table}
keyConv = "org.apache.spark.examples.pythonconverters.ImmutableBytesWritableToStringConverter"
valueConv = "org.apache.spark.examples.pythonconverters.HBaseResultToStringConverter"
# 读取habse表中的数据到rdd
hbase_rdd = sc.newAPIHadoopRDD("org.apache.hadoop.hbase.mapreduce.TableInputFormat","org.apache.hadoop.hbase.io.ImmutableBytesWritable",
"org.apache.hadoop.hbase.client.Result",keyConverter=keyConv,valueConverter=valueConv,conf=conf)
count = hbase_rdd.count()
one = hbase_rdd.first() 查看rdd的第一条数据tuple(rowkey,'\n'.join(str(json_value)))
one_value = one[1].split('\n')
one_value[1] 形式为'{"qualifier":"列名","timestamp":"1560533059864","columnFamily":"列簇名", "row":"0000632232_1550712079","type":"Put","value":"0"}'
写入hbase
write_table = 'student'
write_keyConv = "org.apache.spark.examples.pythonconverters.StringToImmutableBytesWritableConverter"
write_valueConv= "org.apache.spark.examples.pythonconverters.StringListToPutConverter"
conf = {"hbase.zookeeper.quorum":host,"hbase.mapred.outputtable":table,"mapreduce.outputformat.class":"org.apache.hadoop.hbase.mapreduce.TableOutputFormat",
"mapreduce.job.output.key.class":"org.apache.hadoop.habse.io.ImmutableBytesWritable","mapreduce.job.output.value.class":"org.apache.hadoop.io.Writable"}
rawData = ['3,info,age,19','4,info,age,17'] # 最后将数据改成[rowkey,[rowkey,column family, column name,value]]形式写进hbase
sc.parallelize(rawData).map(lambda x:(x[0],x.split(','))).saveAsNewAPIHadoopDataset(conf=conf,keyConverter=keyConv,valueConverter=valueConv)
spark启动后对应的进程是WORKER 和 MASTER