Spark两种转化为df的方法

1.使用Row类与StructType类
转化代码:

val frame: DataFrame = session.createDataFrame(rowRDD,logSchema.schema)

1>rowRDD
将array放进Row中形成Rdd

val rowRDD: RDD[Row] =rddAll.map(arr => Row(
      arr(0),
      arr(1),
      arr(2)

2>logSchema.schema
StructType->存放StructField

val schema = StructType(
    Array(
      StructField("MPacketHead_ATPType",StringType),
      StructField("MPacketHead_TrainID",StringType),
      StructField("MPacketHead_TrainID",StringType)

2.使用样例类
1>构造一个样例类

case class Log(
school:String,
clas:String,
gd1:Int,
)

2>在rdd中将参数放入样例类中

val rdd1 = rdd.map(x=>log(x(2),x(3),x(4)))

3>直接将接受了参数的样例类放入转化

 val frame = session.createDataFrame(rdd1 )

你可能感兴趣的:(大数据,实例)