python项目应用实例(二)制作数据集相关: label更改与批量化图片处理

背景：制作数据集时，时常需要对图片进行重命名、格式转换、以及批量处理等等。python可以方便的实现这些操作。我们对这种程序进行汇总，以便以后查阅。先汇总程序，然后汇总各种模块化的功能如何实现。

博主代码地址：https://github.com/Xingxiangrui/Batch_data_preprocess

目录

一、字符串相关

1.1 ground truth格式转换

1.2 预测生成bounding box

1.3 图片及路径写入txt文件

1.4 split分开为不同元素

二、图片处理相关

2.1 图片重命名写入写出

2.2查看图像任一点坐标

2.3 批量重命名

三、序列乱序（拆训练集与验证集）

序列乱序

list拆为两个list

图像与标签写入

图像读出与写入

标签的生成

四、字典元素更改

---

一、字符串相关

程序不难看懂。

1.1 ground truth格式转换

转换背景：COCO与YOLO的数据集的bounding box的标注方式是不同的。其中一个是[ left，top，right，bottom ]，另一个是 [ x,y , width, height ]，这就要涉及转换，及写入。这个程序就是解决这个问题。

除了背景转换之外，还需要将图片读出并根据图片的名称创建相应的标注文档 txt。

"""
author:xing xiangrui
time  :2018.9.26   10:34
copy image and change name to the new directory
write image name and labels to the list.txt
"""
import os
import shutil
oldDir = "data"
newDir = "head_data"
width = 352 
height = 288
fw = open('list.txt','w')
for root,dirs,files in os.walk(oldDir):
    if 'images' in root:
        continue
    for file in files:
        if 'jpg' in file:
            # copy jpg
            oldPic = root + os.path.sep + file
            newPic = newDir + os.path.sep + root.split(os.path.sep)[-2] + "_" + root.split(os.path.sep)[-1] + "_" + file
            shutil.copyfile(oldPic,newPic)
            
            fw.write(newPic)
            # write txt
            oldTxt = root + os.path.sep + file.replace('jpg','txt')
            fo = open(oldTxt,'r')
            lines = fo.readlines()
            for line in lines:
                line = line.split()
                cx, cy, w, h = float(line[1])*width, float(line[2])*height, float(line[3])*width, float(line[4])*height
                #[x, y, w, h] = [float(line[i]) for i in range(1,5)]
                x1 = round(cx - w/2, 2)
                y1 = round(cy - h/2, 2)
                x2 = round(cx + w/2, 2)
                y2 = round(cy + h/2, 2)
                print(x1,y1,x2,y2)
                fw.write(" " + str(x1) + " " + str(y1) + " " + str(x2) + " " + str(y2))
            fw.write("\n")
            fo.close()
            #print("fine")
print("done")
fw.close()

1.2 预测生成bounding box

程序目的: 将网络预测结果按照mAP测试的格式写入文件，然后用系统调用mAP运行的程序运行测得mAP

                for tmp_file in jpg_list:
                    img=cv2.imread(tmp_file)
                    # add ROI region
                    ROI=img[ROI_idx[0]:ROI_idx[1],ROI_idx[2]:ROI_idx[3]]
                    ROI_temp=ROI.copy()
                    img[:,:,:]=0
                    img[ROI_idx[0]:ROI_idx[1],ROI_idx[2]:ROI_idx[3]]=ROI_temp
                    #create txt file
                    tmp_file=tmp_file.replace("jpg","txt")
                    txt_filename=tmp_file.replace("images","predicted")
                    print("LOACTION!!!predict:"+tmp_file)
                    
#                    start_time = time.time()
                    #print("LOCATION!!!detect_face function start"+"\n")
                    rectangles, points = detect_face(img, args.minsize,
                                                     pnet_fun, rnet_fun, onet_fun,
                                                     args.threshold, args.factor)
                    #print("LOCATION!!!idetect_face function done"+"\n")
#                    duration = time.time() - start_time
    
#                    print("duration:"+str(duration))
                    #print(type(rectangles))
                    points = np.transpose(points)
                    #print("LOCATION!!!loop rectangles"+"\n")
                    with open(txt_filename,'w') as result_file:
                        for rectangle in rectangles:
                            result_file.write("head" + " " + str(rectangle[4]) + " " + str(rectangle[0]) + " " + str(rectangle[1]) + " " + str(rectangle[2]) +  " " + str(rectangle[3])+"\n")
                    #print("LOCATION!!!Write done!"+"\n")
                print(ROI_idx)
                os.chdir("mAP/")
                os.system("python main.py -na")

更改文件夹，用os.chdir即可。然后运行程序直接 os.system("需要的命令行")

1.3 图片及路径写入txt文件

运用见这篇文章。就是将文件夹之中所有jpg格式的文件路径写入train.txt 方便后续训练。

 YOLOv3：Darknet代码解析（四）结构更改与训练

"""
Created on Wed Dec 13, 15:14:53 2017
批量写入 train.txt
@auther xingxiangrui
"""        
import os
#root = os.getcwd()
#picpath= '\\0\\1_f\\'
#filename=os.listdir(root+picpath)
#for file in filename:
#    if 'jpg' in file:
#        print(file)
openfile=open('train.txt','w')

for root,dirs,files in os.walk(os.getcwd())
    for item in files:
        if 'jpg' in item:
            openfile.write(os.path.join(root,item)+'\n')
        if '.mat' in item or '.m' in item
            os.remove(os.path.join(root,item))
openfile.close()

1.4 split分开为不同元素

split的用法，通过符号对字符串进行分割--gpu_ids 0,1,2,3

        # set gpu ids
        str_ids = opt.gpu_ids.split(',')
        opt.gpu_ids = []
        for str_id in str_ids:
            id = int(str_id)
            if id >= 0:
                opt.gpu_ids.append(id)
        if len(opt.gpu_ids) > 0:
            torch.cuda.set_device(opt.gpu_ids[0])

二、图片处理相关

2.1 图片重命名写入写出

# -*- coding: utf-8 -*
"""
Created by Xingxiangrui on 2019.4.15
This code is created to copy and rename images from the source_dir
 to the target_dir
"""

#import matplotlib.pyplot as plt
import numpy as np
from PIL import Image
import os

#image source directory
#---------------------here need to be changed---------dir location---------
source_dir="/Users/baidu/Desktop/trainingSet/pired-images/lambda_10_idt_0.5/tear/"
target_dir="/Users/baidu/Desktop/trainingSet/pired-images/lambda_10_idt_0.5/tear/augmented-tear/"
if not os.path.isdir(target_dir):
    os.makedirs(target_dir)

#images list in source directory
source_imgs_list = os.listdir(source_dir)
print source_imgs_list

#for each image in the images list
for source_img_name in source_imgs_list:
    if '.jpg' in source_img_name:
        print(source_img_name)

        #read images
        path_source_img = os.path.join(source_dir, source_img_name)
        src_img = Image.open(path_source_img) 
        #src_img.show()

        #---------------------here need to be changed---------rename--------
        target_img_name=source_img_name.replace('.jpg','_real_A.png')
        path_target_img=os.path.join(target_dir, target_img_name)
        print(path_target_img)
        #save new reanmed img to the target dir
        src_img.save(path_target_img)

判断文件夹是否存在，如果不存在;则创建文件夹：

if not os.path.isdir(target_dir):
    os.makedirs(target_dir)

2.2查看图像任一点坐标

import matplotlib.pyplot as plt
import matplotlib as mpl

mpl.rcParams['font.family'] = 'sans-serif'
mpl.rcParams['font.sans-serif'] = 'NSimSun,Times New Roman'
path = input('please enter your pic path: ')
img = plt.imread(path)
plt.imshow(img)
plt.show()

2.3 批量重命名

https://blog.csdn.net/weixin_42854038/article/details/81332779

#coding=gbk
import os
import sys
def rename():
    path=input("请输入路径(例如D:\\\\picture)：")
    name=input("请输入开头名:")
    startNumber=input("请输入开始数:")
    fileType=input("请输入后缀名（如 .jpg、.txt等等）:")
    print("正在生成以"+name+startNumber+fileType+"迭代的文件名")
    count=0
    filelist=os.listdir(path)
    for files in filelist:
        Olddir=os.path.join(path,files)
        if os.path.isdir(Olddir):
            continue
        Newdir=os.path.join(path,name+str(count+int(startNumber))+fileType)
        os.rename(Olddir,Newdir)
        count+=1
    print("一共修改了"+str(count)+"个文件")

rename() 

三、序列乱序（拆训练集与验证集）

https://blog.csdn.net/matrix_google/article/details/72803741

序列乱序

参考：https://blog.csdn.net/amy_wumanfang/article/details/64483340

https://blog.csdn.net/matrix_google/article/details/72803741

直接 random.shuffle(list)，直接讲list乱序后存入自己，很方便。

例如：

# -*- coding: utf-8 -*-
import random
# 对list洗牌，在原list上做改变
list = range(10)
print list
random.shuffle(list)
print "随机排序列表 : ",  list

list拆为两个list

拆为训练集和验证集，分别1/4和3/4

# train list and val list
source_train_list=[]
source_val_list=[]
for idx in range(len(source_image_list)):
    if idx

图像与标签写入

图像读出与写入

图像存于src_img之中，图像重命名用后用save写入。

    # read dource images and rename
    path_source_img = os.path.join(source_image_dir, source_image_name)
    src_img = Image.open(path_source_img)
    full_image_name=prefix+"_train_"+source_image_name
    print(full_image_name)
    # save renamed image to the target dir
    target_image_path=os.path.join(target_image_dir, full_image_name)
    src_img.save(target_image_path)

标签的生成

创建文件：a表示追加写入

# create label_file or write label file
txt_file_train_name="train.txt"
txt_file_val_name="val.txt"
txt_file_train_path=os.path.join(txt_file_dir, txt_file_train_name)
txt_file_val_path=os.path.join(txt_file_dir, txt_file_val_name)
train_txt_file= open(txt_file_train_path,"a")
val_txt_file= open(txt_file_val_path,"a")

有必要对每行加一个"\n"进行结尾

    # write image names and labels
    line_strings= full_image_name+"\t"+str(class_label)+"\n"
    train_txt_file.write(line_strings)

四、字典元素更改

https://www.cnblogs.com/volcao/p/8780725.html

字典key_value容易更改，确定key_value更改key不太容易更改

可以用pop方法进行更改key的值：

思路：先删除原键值对，保存值，然后以新键插入字典

格式：dict[newkey] = dict.pop(key)

d = {'a':1, 'aa':11}
d['b'] = d.pop('a')
d['bb'] = d.pop('aa')
print(d)
#输出：{'b': 1, 'bb': 11}

#执行语句：d.pop('a')，删除 'a' 所对应的键值对，返回 'a' 对应的值；d['b']，相当于给字典新添加一个key，其value为d.pop('a')返回的值。

d = {'a':1, 'aa':11}
c = d.pop('a')
print(c)
#输出：1