Spark综合学习笔记（二十）SparkSQL实战2-RDD转DF

学习致谢

https://www.bilibili.com/video/BV1Xz4y1m7cv?p=57

一、使用样例类

1.要读取的文件

1 zhangsan 20
2 lisi 29
3 wangwu 25
4 zhaoliu 30
5 tianqi 35
6 kobe 40

2.代码实现

package sql

import org.apache.spark.rdd.RDD
import org.apache.spark.sql.{
     DataFrame, SparkSession}

/**
  * Author itcast
  * Desc 将RDD转为DataFrame
  */
object Demo02_RDD2DataFrame1 {
     
  def main(args: Array[String]): Unit = {
     
    //TODO 0.准备环境
    val spark = SparkSession.builder().appName("sparksql").master("local[*]").getOrCreate()
    val sc = spark.sparkContext
    sc.setLogLevel("WARN")

    //TODO 1.加载数据
    val lines:RDD[String]  = sc.textFile("data/SogouE.txt")

    //TODO 2.处理数据
    val personRDD:RDD[Person] = lines.map(line => {
     
      val arr: Array[String] = line.split(" ")
      Person(arr(0).toInt, arr(1), arr(2).toInt)
    })

    //RDD--->DF
    import spark.implicits._
    val personDF = personRDD.toDF()
    //TODO 3.输出数据
personDF.printSchema()
    personDF.show()
    //TODO 4.关闭资源

  }
    case class Person(id:Int,name:String,age:Int)
}

3.运行结果

二、指定类型+列名

1.适合用在字段较少的时候使用

2.代码实现

package sql

import org.apache.spark.rdd.RDD
import org.apache.spark.sql.SparkSession

/**
  * Author itcast
  * Desc 将RDD转为DataFrame-指定类型和列名
  */
object Demo02_RDD2DataFrame2 {
     
  def main(args: Array[String]): Unit = {
     
    //TODO 0.准备环境
    val spark = SparkSession.builder().appName("sparksql").master("local[*]").getOrCreate()
    val sc = spark.sparkContext
    sc.setLogLevel("WARN")

    //TODO 1.加载数据
    val lines:RDD[String]  = sc.textFile("data/SogouE.txt")

    //TODO 2.处理数据
    val personRDD: RDD[(Int, String, Int)] = lines.map(line => {
     
      val arr: Array[String] = line.split(" ")
      (arr(0).toInt, arr(1), arr(2).toInt)
    })
    //RDD--->DF
    import spark.implicits._
    val personDF = personRDD.toDF("id","name","age")
    //TODO 3.输出数据
personDF.printSchema()
    personDF.show()
    //TODO 4.关闭资源

  }

}

3.运行结果

三、自定义Schema

1.概述
依据RDD中数据自定义Schema，类型为StructType，每个字段的约束使用StructField定义，具体步骤如下:
1、RDD中数据类型为Row: RDD[Row];
2、针对Row中数据定义Schema: StructType;
3、使用SparkSession中方法将定义的Schema应用到RDD[Row]上;

2.代码实现

package sql



import org.apache.spark.rdd.RDD
import org.apache.spark.sql.types.{
     IntegerType, StringType, StructField, StructType}
import org.apache.spark.sql.{
     Row, SparkSession}

/**
  * Author itcast
  * Desc 将RDD转为DataFrame-指定类型和列名
  */
object Demo02_RDD2DataFrame3 {
     
  def main(args: Array[String]): Unit = {
     
    //TODO 0.准备环境
    val spark = SparkSession.builder().appName("sparksql").master("local[*]").getOrCreate()
    val sc = spark.sparkContext
    sc.setLogLevel("WARN")

    //TODO 1.加载数据
    val lines:RDD[String]  = sc.textFile("data/SogouE.txt")

    //TODO 2.处理数据
    val rowRDD: RDD[Row] = lines.map(line => {
     
      val arr: Array[String] = line.split(" ")
      Row(arr(0).toInt, arr(1), arr(2).toInt)
    })
    //RDD--->DF
    import spark.implicits._
    val schma:StructType = StructType(List(
      StructField("id", IntegerType, false),
        StructField("name", StringType, false),
        StructField("age", IntegerType, false)
     ))

    val personDF = spark.createDataFrame(rowRDD,schma)
    //TODO 3.输出数据
    personDF.printSchema()
    personDF.show()
    //TODO 4.关闭资源
    spark.close()
  }
}

3.运行结果

4.补充