spark-sql 指定metastore地址、读取mysql

spark-sql-hive

spark-sql --database src --hiveconf hive.metastore.uris=thrift://hdp02:2083 -e "select * from src_db2" --driver-java-options '-Djava.net.preferIPv4Stack=true'

spark-sql-jdbc

1.启动spark-sql

## --driver-class-path：指定Driver端所需jar  
## --jars ：指定executor所需要jar
spark-sql --driver-class-path /home/dw/hubg/mysql-connector-java-5.1.45-bin.jar \
--jars /home/dw/hubg/mysql-connector-java-5.1.45-bin.jar

2.执行Spark SQL语句：

spark-sql> CREATE TEMPORARY VIEW spark_mysql USING org.apache.spark.sql.jdbc OPTIONS (  url "jdbc:mysql://xxx.xx.xx.x:xxxx/shurongtest", dbtable "dmpt_serv_object",user 'root',password 'root');
Time taken: 1.504 seconds
22/08/01 15:21:48 INFO SparkSQLCLIDriver: Time taken: 1.504 seconds

 

spark-sql> select * from spark_mysql;
1541964275789103105     第一条  4bb8df021931375dc80f056f433ba2b6        2022-06-30 12:59:59     第一条  2022-06-29 22:58:26     admin   2022-06-30 22:41:32     admin
1542701764438929409     第二条2 57119498e7649c33e2c5e4f56f2a50dc        2022-07-02 12:59:59     第二条  2022-07-01 23:48:57     yanshi_0627     2022-07-19 04:00:44     admin
1543104243022495745     服务对象1服务对象1服务对象1服务对象1服务对象1服务对象1服务对象1服务对象1服务对象1服务对象1服务对象1服务对象1    9d48f9824f9cb02190f331267966150c       2022-07-03 12:59:59      服务对象1服务对象1服务对象1服务对象1服务对象1服务对象1服务对象1服务对象1服务对象1服务对象1服务对象1服务对象1    2022-07-03 02:28:16     yhx     2022-07-03 02:28:16     yhx
1544282208364134401     11111111        c334ad6011cf3100a5e61f2832a4ae56        2022-07-06 12:59:59             2022-07-06 08:29:05     admin   2022-07-06 08:29:05     admin
1546804008885166082     111     a2184be4c6f233380e54ce128dd73bbe        2022-07-13 12:59:59     111     2022-07-13 07:29:49     admin   2022-07-13 07:29:49     admin
Time taken: 12.592 seconds, Fetched 5 row(s)
22/08/01 15:22:04 INFO SparkSQLCLIDriver: Time taken: 12.592 seconds, Fetched 5 row(s)