sparkSQL 连接Oracle

注意

1. 要将oracle.jdbc.OracleDriver ojdbc8.jar 放到spark目录D:\Users\lill001\projects\spark-2.4.3-bin-hadoop2.7\jars 下。
2. sparkSQL 连接 oracle

# -*- coding: utf-8 -*-
from pyspark.sql import SparkSession
from pyspark.sql.types import StructType, StructField,StringType,DoubleType,DateType
import os

os.environ["SPARK_HOME"] = "D:\\Users\\lill001\\projects\\spark-2.4.3-bin-hadoop2.7"
os.environ["HADOOP_HOME"] = "D:\\Users\\lill001\\projects\\hadoop-common-2.2.0-bin-master"

os.environ["HADOOP_USER_NAME"] = "hdfs"

ORACLE_DRIVER_PATH = "D:\\Users\\lill001\\softwares\\instantclient_19_3\\ojdbc8.jar"
spark = SparkSession.builder.master("local").appName("spark-oracle").getOrCreate()
print(spark)
sc=spark.sparkContext
print(sc)

#从oracle 数据库表中读取数据到spark，如果要使用sql语句，则必须添加别名alias，比如下面的t
sqltext="(select col1,col2,col3 from tablename) t"
jdbcdf = spark.read.format('jdbc').options(
 url='jdbc:oracle:thin:@//127.0.0.1/oa3',
    dbtable='OA3DATA.SYS_NEWS_MAIN',
    user='123',
    password='123',
    driver="oracle.jdbc.OracleDriver").load()
print(jdbcdf)
jdbcdf.createOrReplaceTempView('tmp_vw')

df = spark.sql("SELECT fd_id, doc_subject, doc_en_subject, fd_news_source, doc_publish_time, doc_content, doc_en_content, fd_html_content FROM tmp_vw")
df.show()
sc.stop()

JDBC中Oracle的SID和ServiceName两种方式的连接字符串格式
SID格式：

jdbc:oracle:thin:@::
如: jdbc:oracle:thin:@192.168.1.1:1521:orcl
ServiceName格式：

jdbc:oracle:thin:@//:/
如: jdbc:oracle:thin:@//192.168.1.1:1521/orcl