python使用pymysql批量插入数据库5000W条复杂的数据，如何用python脚本实现和用MySQL自定义存储过程实现

python使用pymysql批量插入数据库5000W条复杂的数据，如何用python脚本实现和用MySQL自定义存储过程实现

一、使用Python脚本实现批量插入

import pymysql

# 连接数据库
db = pymysql.connect(host='localhost', user='username', password='password', database='dbname')
cursor = db.cursor()

# 准备数据
data = [
    (1, 'John', 'Doe'),
    (2, 'Jane', 'Smith'),
    # 更多数据...
]

# 生成插入语句
insert_query = "INSERT INTO table_name (id, first_name, last_name) VALUES (%s, %s, %s)"

# 执行批量插入
cursor.executemany(insert_query, data)

# 提交更改
db.commit()

# 关闭连接
cursor.close()
db.close()

当处理大量数据的批量插入时，确保脚本具有错误处理和日志记录是非常重要的。以下是优化后的示例代码，包括错误处理和日志记录：

import pymysql
import logging

# 设置日志
logging.basicConfig(filename='insert_log.log', level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')

try:
    # 连接数据库
    db = pymysql.connect(host='localhost', user='username', password='password', database='dbname')
    cursor = db.cursor()

    # 准备数据
    data = [
        (1, 'John', 'Doe'),
        (2, 'Jane', 'Smith'),
        # 更多数据...
    ]

    # 生成插入语句
    insert_query = "INSERT INTO table_name (id, first_name, last_name) VALUES (%s, %s, %s)"

    # 执行批量插入
    cursor.executemany(insert_query, data)

    # 提交更改
    db.commit()

    # 关闭连接
    cursor.close()
    db.close()

    logging.info("批量插入成功")

except Exception as e:
    logging.error(f"批量插入失败: {str(e)}")

二、使用MySQL自定义存储过程实现批量插入

DELIMITER //

CREATE PROCEDURE bulk_insert_data()
BEGIN
    DECLARE i INT DEFAULT 1;
    WHILE i <= 500000 DO
        INSERT INTO table_name (id, first_name, last_name) VALUES (i, CONCAT('First', i), CONCAT('Last', i));
        SET i = i + 1;
    END WHILE;
END //

DELIMITER ;

CALL bulk_insert_data();

优化后的存储过程代码，增加了错误处理和日志记录：

DELIMITER //

CREATE PROCEDURE bulk_insert_data()
BEGIN
    DECLARE i INT DEFAULT 1;
    DECLARE continue_handling BOOLEAN DEFAULT TRUE;
    
    DECLARE CONTINUE HANDLER FOR SQLEXCEPTION
    BEGIN
        SET continue_handling = FALSE;
        ROLLBACK;
        INSERT INTO error_log (error_message) VALUES (CONCAT('Error occurred at iteration ', i, ': ',  MYSQL_ERRNO(), ' - ',  MYSQL_ERROR()));
    END;

    START TRANSACTION;
    WHILE i <= 500000 DO
        BEGIN
            INSERT INTO table_name (id, first_name, last_name) VALUES (i, CONCAT('First', i), CONCAT('Last', i));
            SET i = i + 1;
        END;
    END WHILE;

    IF continue_handling THEN
        COMMIT;
        INSERT INTO success_log (success_message) VALUES ('Bulk insert completed successfully');
    END IF;
    
END //

DELIMITER ;