最近需要整理一些Oracle数据表,由于python用习惯了,直接就python转换,然后存入。在处理的过程中遇到了一些问题,好在最后都解决了,现将代码贴一下,作为自己的一个阶段性小总结把。主要处理步骤如下:
# coding=utf-8
#设置语言环境
import os
os.environ['NLS_LANG'] = 'SIMPLIFIED CHINESE_CHINA.UTF8'
#设置数据库链接
import cx_Oracle
conn_str = 'username/password@数据库服务器地址:1521/dzzwdb'
connection_oracle = cx_Oracle.Connection(conn_str)
cursor_oracle = connection_oracle.cursor()
sql = 'select distinct rule_condition from BIGDATATEAM.CASE_ASSOCIATION_RULES'
cursor_oracle.execute(sql)
data = cursor_oracle.fetchall()
#使用dataframe进行处理
import pandas as pd
df = pd.DataFrame(list(data),columns=['mxmc'])
#产生唯一编号
import uuid
uuid_list = [str(uuid.uuid1()) for i in range(df.shape[0])]
df['mxbm'] = pd.Series(uuid_list)
#生成处理时间
import datetime
time_list = [datetime.datetime.now()]* df.shape[0]
df['tjsj'] = pd.Series(time_list)
#展示前5行
df.head()
#循环逐条插入
for i in range(df.shape[0]):
sql = "insert into bigdatateam.mx1(mxbm, mxmc, tjsj, zt) values ('"+str(df['mxbm'][i])+"','"+str(df['mxmc'][i])+"',to_timestamp('"+str(df['tjsj'][i])+"','yyyy-mm-dd hh24:mi:ss:ff'), '"+'finished'+"')"
#print('sql = '+sql)
cursor_oracle.execute(sql)
connection_oracle.commit()
#关闭游标和连接
cursor_oracle.close()
connection_oracle.close()
本来插入数据的时候是批量插入的,如下代码所示。但数据库总是报错“非法的变量名/编号”,检查了很久也没发现问题在哪儿,但赶着交东西,就换成逐条插入了。后来又遇到了这个问题,重启了pl/sql和jupyter notebook就好了,迷之神奇~!
def save2oracle(data, clean_data = True):
if clean_data:
cursor_oracle.execute('truncate table bigdatateam.mx')
connection_oracle.commit()
cursor_oracle.execute('select * from bigdatateam.mx')
db_types = [d[1] for d in cursor_oracle.description]
cursor_oracle.bindarraysize = len(data)
cursor_oracle.setinputsizes(*db_types)
sql = '''insert into bigdatateam.mx
(mxbm,mxmc,mxsm,updatetime)
values (:1,:2,:3,:4)
'''
cursor_oracle.executemany(sql, data)
connection_oracle.commit()
save2oracle(insert_data, True)