巧用groupby解决Dataframe筛选分组效率慢问题

原代码:

for name in list_valid_perfor_inventory:
    time_stamp = time.time()
    df_tmp1 = df_all_performance[df_all_performance['res_ins_id'] == name] ###170万行,该语句大约需要2S
    if df_tmp1.empty:
        continue
    del df_tmp1['res_ins_id']
    print('choose time ')
    print(str(time.time() - time_stamp))
    time_stamp = time.time()
    df_tmp1.to_csv(path_or_buf=os.path.join(cs.max_avg_busy_dir, str(name) + '.csv'))
    print(str(time.time() - time_stamp))

优化后代码:

groups = df_all_performance.groupby('res_ins_id')  ##先分组
for name in list_valid_perfor_inventory:
    time_stamp = time.time()
    df_tmp1 = groups.get_group(name) ##再取每组的值,返回dataframe
    if df_tmp1.empty:
        continue

    del df_tmp1['res_ins_id']
    df_tmp1.to_csv(path_or_buf=os.path.join(cs.max_avg_busy_dir, str(name) + '.csv'))



你可能感兴趣的:(python)