Pandas批量拆分合并Excel文件

目标

1. 将一个大Excel等份拆成多个Excel
2. 将多个小Excel合并成一个大Excel并标记来源

数据准备

import pandas as pd
import os
work_dir=r"D:\node\nd\Pandas_study\pandas_test"
splits_dir=f"{work_dir}/splits"
# print(splits_dir)

if not os.path.exists(splits_dir):
    os.mkdir(splits_dir)

df_source = pd.read_excel(f"{work_dir}/crazyant_blog_articles_merged.xlsx")
print(df_source.head())
print(df_source.index)
print(df_source.shape)

total_row_count = df_source.shape[0]
print("总行数为：",total_row_count)

image.png

一、将一个大Excel等份拆成多个Excel

1. 使用df.iloc方法，将一个大的dataframe，拆分成多个小dataframe
2. 将使用dataframe.to_excel保存每个小Excel

#计算拆分后每个excel行数
#打excel，会拆分给如下几个人
user_names = ["xiao_shuai", "xiao_wang", "xiao_ming", "xiao_lei", "xiao_bo", "xiao_hong"]
#每个人的任务数量
split_size = total_row_count //len(user_names)

if total_row_count % len(user_names) != 0:
    split_size + 1
print(split_size)

#拆分成多个dataframe
df_subs = []
for idx,user_name in enumerate(user_names):
    #开始索引
    begin = idx * split_size
    #结束缩影
    end = begin +split_size
    #实现df按照iloc拆分
    df_sub = df_source.iloc[begin:end]
    #将每个子df存入列表
    df_subs.append((idx,user_name,df_sub))

#将每个datafame存入excel
for idx,user_name,df_sub in df_subs:
    file_name = f"{splits_dir}/crazyant_blog_articles_{idx}_{user_name}.xlsx"
    df_sub.to_excel(file_name, index=False)

image.png

image.png

二、合并多个小Excel到一个大Excel

1. 遍历文件夹，得到要合并的Excel文件列表
2. 分别读取到dataframe，给每个df添加一列用于标记来源
3. 使用pd.concat进行df批量合并
4. 将合并后的dataframe输出到excel

import os
import pandas as pd

work_dir=r"D:\node\nd\Pandas_study\pandas_test"
splits_dir=f"{work_dir}/splits"

#遍历文件夹，得到要合并的excel名称列表
excel_names = []
for excel_name in os.listdir(splits_dir):
    excel_names.append(excel_name)

#分别读取datagrame
df_list = []
for excel_name in excel_names:
    #读取每个excel到df
    excel_path = f"{splits_dir}/{excel_name}"
    df_split = pd.read_excel(excel_path)
    #得到username
    username = excel_name.replace("crazyant_blog_articles_","").replace(".xlsx","")[2:]
    #给每个df添加一列，即用户名
    df_split["username"] = username
    df_list.append(df_split)
#
#使用pd.concat 进行合并
df_merged = pd.concat(df_list)
print(df_merged.shape)
print(df_merged.head())
print(df_merged["username"].value_counts())

#将合并后的dataframe输出到excel
df_merged.to_excel(f"{work_dir}/crazyant_blog_articles_merged.xlsx",index=False)

image.png

多个excel合并大的excel