python提取百万数据到csv文件

今天有需求,需要把系统所有用户注册的id和邮箱等信息导出来提供给他们,在mysql里面count了下,大概有350万左右

就尝试了下用python实现,顺带练习下python写csv的功能,本来想用工具的,但想了下速度会很慢,

整个导出过程大概就3分钟左右,还是蛮快的,毕竟有三百多万,导完后有150M左右

下面是我的脚本deal_csv.py,由于需要连接mysql数据库,脚本依赖MySQLdb模块

__author__ = 'chunyang.wu'
# -*- coding: utf-8 -*-
#!/usr/bin/env python
import MySQLdb
import os
os.environ['NLS_LANG'] = 'SIMPLIFIED CHINESE_CHINA.UTF8'
import sys
reload(sys)
sys.setdefaultencoding('utf-8')
import csv

class Handle:
    def __init_db(self):
        self._mysql_db = MySQLdb.connect(host="172.16.1.55",user="test",passwd="123456",port=3306,db="test",unix_socket="/tmp/mysql5.sock")
        self.mysql_cur=self._mysql_db.cursor()
        self.seq = 0

    def __init__(self):
        self.__init_db()

    def _release_db(self):
        self.mysql_cur.close()
        self._mysql_db.close()

    def _do(self):
        self.mysql_cur.arraysize = 50
        select_sql = "SELECT id,email,FROM_UNIXTIME(create_time) AS create_time FROM test.tbl_member "
        print select_sql
        self.mysql_cur.execute(select_sql)
        count = 0
        csvfile = file('all_user.csv', 'wb')
        print dir(csv)
        writers = csv.writer(csvfile)
        writers.writerow(['uid', 'email', 'createtime'])
        while 1:
            lines = self.mysql_cur.fetchmany(50)
            if len(lines)==0:
                break
            for i in lines:
                print i
                writers.writerows([i])
        csvfile.close()


def main():
    p = Handle()
    p._do()
    p._release_db()

if __name__=="__main__":
    main()
csv文件结构如下图

python提取百万数据到csv文件_第1张图片

---------------------------------------------------------------------------------------------------------------------------------------------------------

版权所有,文章允许转载,但必须以链接方式注明源地址,否则追究法律责任!
QQ:       380968195
Email:    [email protected]
Blog:     http://blog.csdn.net/selectdb

URL:      http://blog.csdn.net/selectdb/article/details/16337621



你可能感兴趣的:(mysql,python)