Spark_SQL如何进行数据读取

从 json 文件读数据

json 文件内容示例

{"id":1, "name":"leo", "age":18}
{"id":2, "name":"jack", "age":19}
{"id":3, "name":"marry", "age":17}

从 json 文件读取

    SparkSession spark = SparkSession.builder()
    .appName("QSH")
    .master("local[*]")
    .config("spark.some.config.option", "some-value").getOrCreate();

    Dataset df = spark.read().json("D:\\Study_Space\\SparkSQLTest\\json\\student.json");

    df.show();

从MySQL数据库读取数据

从MySQL数据库读取

        SparkConf sc = new SparkConf().setAppName("QSH").setMaster("local");
        SparkContext sparkContext = new SparkContext(sc);
        SQLContext sqlContext = new SQLContext(sparkContext);

        String url = "jdbc:mysql://localhost:3306/qsh";
        String table_Name = "tb_student";

        Properties prop = new Properties();
        prop.put("username", "root");
        prop.put("password", "");
        prop.put("driver", "com.mysql.jdbc.Driver");


        Dataset ds = sqlContext.read().jdbc(url, table_Name, prop);

        ds.show();

输出效果

+------+-----+-----+
|stu_ID|class|score|
+------+-----+-----+
|     1|    A|   20|
|     2|    A|   30|
|     3|    A|   70|
|     4|    B|   60|
|     5|    B|   70|
|     6|    B|   80|
+------+-----+-----+

读取后数据类型皆为 Dataset