解压parquet格式文件到text

阅读更多

方法一：spark python实现

import sys
from pyspark import SparkConf, SparkContext
from pyspark.sql import SQLContext

inputpath=sys.argv[1]
outputpath=sys.argv[2]
sc = SparkContext(appName="Transform Pq to Csv")
sqlContext = SQLContext(sc)
df = sqlContext.read.parquet(inputpath)
df.select('*').save(outputpath,"com.databricks.spark.csv",delimiter='\001')

 

 

run：spark-submit --packages com.databricks:spark-csv_2.10:1.2.0 --master yarn-client read_pq.py /tmp/xing/20161115/1049 /tmp/xing/20161115/text/1049

 

方法二：如果是parquet表里的数据，可以通过hql查询得到数据

insert overwrite (local) inpath outputpath

select * from table1_parquet;