让Python帮你搞定MySQL数据库

Mysql是常用的数据库之一,也是面试工作必备技能之一。

 

本文通过一个实战,将Python与Sql语句结合起来使用,搞定MySQL数据库。

 

// 实战开始 //

我们在github上下载fifa18球员数据,将这些信息存入到mysql。

①数据下载地址:

https://github.com/amanthedorkknight/fifa18-all-player-statistics

②选择:Complete->basicplayerdata.csv

 

                                                                           // 将数据从CSV导入到MySQL //

 

1)创建对应库与表

 


#创建公司数据库,编码格式utf-8
CREATE DATABASE fifa18_db DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;
#选择数据库:
use fifa18_db;
#创建球员表:id,名称,海报地址,俱乐部,年龄,薪资等,与csv文件对应
create table player(id int Primary key auto_increment, 
player_id int, name char(64), age int, poster char(64),

                    flag char(64), overall int, potential int, club char(64) default '', club_Logo char(64), 
                   value char(16), wage char(16), special int) default charset =utf8;

2)CSV读取文件

 

数据集中某些球员字段为空,插入时候需要补充默认值,使用DictReader读取。

path = '/home/linux/workdir/data/basicplayerdata.csv'#字段名称field = ['','ID','Name','Age','Photo','Nationality','Flag','Overall','Potential','Club','Club Logo','Value','Wage','Special']f = open(path)fcsv = csv.DictReader(f, fieldnames = field)#第一行去掉line = next(fcsv)#第一行有效数据line = next(fcsv)print(line) 

结果:

OrderedDict([('', '0'), ('ID', '158023'), ('Name', 'L. Messi'), ('Age', '30'), ('Photo', 'https://cdn.sofifa.org/players/4/18/158023.png'), ('Nationality', 'Argentina'), ('Flag', 'https://cdn.sofifa.org/flags/52.png'), ('Overall', '94'), ('Potential', '94'), ('Club', 'FC Barcelona'), ('Club Logo', 'https://cdn.sofifa.org/teams/2/18/light/241.png'), ('Value', '€118.5M'), ('Wage', '€565K'), ('Special', '2161')])

 

3)读取一行并整理数据格式

 

CSV读取字段与数据库对应起来,且不要CSV文件种的第一列,整理数据格式为字典,key为数据库列名,value为CSV对应内容,代码实现。

sqlfield = ['player_id','name','age','poster','nationality','flag','overall','potential','club','club_logo','value','wage','special']csvfield = ['ID','Name','Age','Photo','Nationality','Flag','Overall','Potential','Club','Club Logo','Value','Wage','Special']#转成字典:keysinfo = dict(zip(sqlfield, csvfield))#填充数据:data = {}for sfield, cfield in keysinfo.items():    ele = line.get(cfield, '')    data.setdefault(sfield, ele)print(data)

 

结果:​​​​​​​

{'player_id': '158023', 'name': 'L. Messi', 'age': '30', 'poster': 'https://cdn.sofifa.org/players/4/18/158023.png', 'nationality': 'Argentina', 'flag': 'https://cdn.sofifa.org/flags/52.png', 'overall': '94', 'potential': '94', 'club': 'FC Barcelona', 'club_logo': 'https://cdn.sofifa.org/teams/2/18/light/241.png', 'value': '€118.5M', 'wage': '€565K', 'special': '2161'}

 

4)拼接sql语句

 

尝试插入一条数据:获取插入字段与对应数据,拼接成sql语句。​​​​​​​

tablename = 'player'keys = data.keys()fields = ','.join(keys)vals = ','.join(["'%s'"% val for val in data.values()])sql = f"INSERT INTO {tablename}({fields}) VALUES({vals})"print(sql)

 

结果:​​​​​​​

INSERT INTO player(player_id,name,age,poster,nationality,flag,overall,potential,club,club_logo,value,wage,special) VALUES('158023','L. Messi','30','https://cdn.sofifa.org/players/4/18/158023.png','Argentina','https://cdn.sofifa.org/flags/52.png','94','94','FC Barcelona','https://cdn.sofifa.org/teams/2/18/light/241.png','€118.5M','€565K','2161')

 

5)连接数据库,并插入数据​​​​​​​

#编码格式:utf-8db = pymysql.connect("localhost","root","abc123","fifa18_db", charset='utf8')#获取游标cursor = db.cursor()cursor.execute(sql)#提交数据db.commit()#断开连接cursor.close()db.close()

 

查看player中信息,插入成功。

 

让Python帮你搞定MySQL数据库_第1张图片

 

基本功能都已经实现,对代码进行整理,根据功能定义类及方法。​​​​​​​

import csvimport pymysqlclass LoadDataFromCsvToMysql:    def __init__(self, csvpath, csvfield, mysqlfield, table, sqlconfig):        #初始化参数        pass    def connectSql(self):        #连接数据库,获取游标        pass    def disconnectSql(self):        #断开数据库,获取游标        pass    def processSql(self, sql):        #处理sql语句        pass    def loadCsv(self):        #打开csv文件,返回csv对象        pass    def closeCsv(self):        #关闭csv文件        pass    def gensql(self, linedata):        #生成sql语句        pass    def process(self):        #对外接口,打开文件并写数据库        pass

 

类定义完成之后,我们可以将功能明确的方法实现。​​​​​​​

import csvimport pymysqlclass LoadDataFromCsvToMysql:    def __init__(self, csvpath, csvfield, mysqlfield, table, sqlconfig):        #初始化参数        self.dbconfig = sqlconfig        self.inpath = csvpath        self.sqlfield =  mysqlfield        self.csvfield = csvfield        self.fieldmap = dict(zip(mysqlfield, csvfield[1:]))        self.tablename = table    def connectSql(self):        self.db = pymysql.connect(**self.dbconfig, charset='utf8')        self.cursor = self.db.cursor()    def disconnectSql(self):        self.cursor.close()        self.db.close()    def processSql(self, sql):        ret = self.cursor.execute(sql)        self.db.commit()        return ret    def loadCsv(self):        #打开csv文件        self.f = open(path)        fcsv = csv.DictReader(self.f, fieldnames = self.csvfield)        next(fcsv)        return fcsv    def closeCsv(self):        self.f.close()    def gensql(self, linedata):        pass    def process(self):        #测试数据库与csv文件打开关闭        self.connectSql()        print('connect sql...')        fcsv = self.loadCsv()        self.disconnectSql()        self.closeCsv()        print('disconnect sql')        path = '/home/linux/workdir/data/basicplayerdata.csv'sqlfield= ['player_id','name','age','poster','nationality','flag','overall','potential','club','club_logo','value','wage','special']csvfield = ['','ID','Name','Age','Photo','Nationality','Flag','Overall','Potential','Club','Club Logo','Value','Wage','Special']dbconfig = {'host':'localhost','port':3306,'user':'root','passwd':'abc123','db':'fifa18_db'}obj = LoadDataFromCsvToMysql(path, csvfield, sqlfield,  'player', dbconfig)obj.process()

 

输出:​​​​​​​

connect sql...load csvfiledisconnect sql

 

6)完善process与gensql

 

先完善process方法实现:​​​​​​​

 def process(self):        #测试数据库与csv文件打开关闭        self.connectSql()        print('connect sql...')        fcsv = self.loadCsv()        for line in fcsv:            sql = self.gensql(line)            print(sql)            n = self.processSql(sql)        self.disconnectSql()        self.closeCsv()        print('disconnect sql')

 

然后完善gensql方法,将前面实现添加到此方法中。​​​​​​​

def gensql(self, linedata):        #产生sql语句        data = {}        for sfield, cfield in self.fieldmap.items():            ele = linedata.get(cfield, '')            data.setdefault(sfield, ele)        tablename = self.tablename        keys = data.keys()        fields = ','.join(keys)        vals = ','.join(["'%s'"% val for val in data.values()])        sql = f"INSERT INTO {tablename}({fields}) VALUES({vals})"        return sql

 

测试前将player表中内容删除,然后使用代码先插入一条,并查看结果;如果测试没有问题,可以插入所有数据。实际运行出现问题:sql语句错误 ,字符串拼接错误。

 

出错sql语句:

"('158023','L. Messi','30','https://cdn.sofifa.org/players/4/18/158023.png',..)"

 

双引号+单引号,但是文件中,某些字符串带单引号,所以出现字段错误。

 

修改sql拼接方法:​​​​​​

    def gensql(self, linedata):        #产生sql语句        data = {}        for sfield, cfield in self.fieldmap.items():            ele = linedata.get(cfield, '')            data.setdefault(sfield, ele)        tablename = self.tablename        keys = data.keys()        fields = ','.join(keys)        vals = ','.join(['"%s"'% val for val in data.values()])        sql = f'INSERT INTO {tablename}({fields}) VALUES({vals})'        return sql

 

运行结果:​​​​​​​

dstram","18","https://cdn.sofifa.org/players/4/18/238813.png","England","https://cdn.sofifa.org/flags/14.png","47","65","Crewe Alexandra","https://cdn.sofifa.org/teams/2/18/light/121.png","€60K","€1K","1305")INSERT INTO player(player_id,name,age,poster,nationality,flag,overall,potential,club,club_logo,value,wage,special) VALUES("238306","A. Conway","19","https://cdn.sofifa.org/players/4/18/238306.png","Republic of Ireland","https://cdn.sofifa.org/flags/25.png","47","63","Galway United","https://cdn.sofifa.org/teams/2/18/light/1571.png","€60K","€1K","1314")disconnect sql1555235373.5942054 1555235392.2122808

 

插入18000条数据,花费时间大概为20S左右;后续优化:每次插入500条数据,然后在查看花费时间,这个大家可以参考前面案例自己实现。

 

// 在SQL中查询数据 //

 

需求:查询player表中阿根廷国家球员姓名,年龄,头像信息。读者朋友可以自己尝试去实现,考虑使用继承。

 

①sql中查询数据我们步骤:连接数据库,执行sql语句,关闭数据库;

②查询与写入很多方法通用,考虑继承LoadDataFromCsvToMysql类;

③需要重载init,process,processSql方法;

 

代码实现:​​​​​​​

class QueryMysql(LoadDataFromCsvToMysql):    def __init__(self, sqlconfig):        #调用父类方法,初始化传一些无效参数        super(QueryMysql, self).__init__('',[], [], '', dbconfig)    def genSql(self, table, fields, condition=None):        #查询语句生成        fds = ','.join(fields)        cond = ''        print(condition)        if condition:            cond = f' where {condition}'        sql = f'select {fds} from {table}{cond}'        return sql    def process(self, tablename, fields, condition=None):        #对外接口        #调用父类中的连接数据库,关闭数据库方法        self.connectSql()        sql = self.genSql(tablename, fields, condition)        self.processSql(sql)        items = self.cursor.fetchall()        for item in items:            print(item)        print('all Argentina player:', len(items))        self.disconnectSql()        dbconfig= {'host':'localhost','port':3306,'user':'root','passwd':'abc123','db':'fifa18_db'}tablename = 'player'files = ['name','poster', 'age']obj = QueryMysql(dbconfig)cond = 'nationality="Argentina"'obj.process(tablename, files, cond)

 

结果:​​​​​​​

('L. Messi', 'https://cdn.sofifa.org/players/4/18/158023.png', 30)('G. Higuaín', 'https://cdn.sofifa.org/players/4/18/167664.png', 29)('P. Dybala', 'https://cdn.sofifa.org/players/4/18/211110.png', 23)...('A. Miño', 'https://cdn.sofifa.org/players/4/18/243298.png', 23)('T. Durso', 'https://cdn.sofifa.org/players/4/18/240955.png', 18)('K. Humeler', 'https://cdn.sofifa.org/players/4/18/240291.png', 20)('J. Mendive', 'https://cdn.sofifa.org/players/4/18/241584.png', 20)all Argentina player: 966

 

结果:

 

数据集中一共有966名Argentina球员。

你可能感兴趣的:(mysql,python,数据库,sql)