十万~百万~千万级 图片去重/筛选/搜重复图完整实现--10毫秒以内，平均约0.5ms
Image-repeat-image-filter
abstract
通过提取描述图片信息的hash值（phash、whash、dhash等）转化成10进制hash索引，采用字符转化方式可逆转化为原始hash值，通过hash索引组成的图片信息库进行重复图查询和图片信息插入更新，使用多线程信息提取和重复查询、二分位置查找，信息库建立和查询过程同步，使得重复图查找/搜索与图片信息库大小无关，查询时间稳定在0~10 ms，受图片信息提取时resize耗时影响，均时约0.5 ms。
后期可以在信息库的存储方式上进行优化，目前以json文件存取，50万图片信息约在25M，随着数据量增多不建议使用文件存取的方式。
keywords: Hash， 重复图， 二分法， 多线程
实现算法思路
1.多线程提取图片的hash值；
2.hash转化成32~64位长索引值, 在信息库中顺序查找位置索引，相同索引为重复图；
3.如无重复，将提取图片信息按索引顺序位置插入信息库；
- requirement:
cv2
PIL
concurrent
imagehash or use my phash only
- 信息库格式
image_infobase.json
{"info_base": [[hash_index, image_path], ...]}
信息库大小会影响读写时的速度，基本不影响查找时间，信息库详细格式如下：
{
"info_base":
[
[3010302070400050200020702050207030301, "E:/2019/image_rm-dui/train250_silimar/376-19_50325220.jpg"],
[3010302070400050200020702050207030301, "E:/2019/image_rm-dui/train250_silimar/377-19_50393399.jpg"],
[5040305050303020703050105020602060600, "E:/2019/image_rm-dui/train250_silimar/952-69_50475587.jpg"],
[106000103040107020705050602010605020001, "E:/2019/image_rm-dui/train250_silimar/800-4_47983604.jpg"],
[106020300060505070507050101030504010701, "E:/2019/image_rm-dui/train250_silimar/30-53_48706252.jpg"],
[106020300060505070507050101030504010701, "E:/2019/image_rm-dui/train250_silimar/31-53_49151624.jpg"],
[107040005020001010304020606070106020301, "E:/2019/image_rm-dui/train250_silimar/1228-22_50197080.jpg"],
[107040005020001010304020606070106020301, "E:/2019/image_rm-dui/train250_silimar/1229-22_50248780.jpg"],
[202070502070004060200010603070007050400, "E:/2019/image_rm-dui/train250_silimar/1920-40_49714099.jpg"]
]
}
代码拆解
1、提取待查重图片信息
提取图片的phash，这里自写phash提取（也可采用其他库hash提取，如：imagehash），采取图片不同RGB通道不同位置区域提取，如下图（就是图片这段代码哈，方便看）：
def pHash(img, read=False):
if read:
# img = io.imread(img)
# img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
img = cv2.imread(img)
img = cv2.resize(img, (32, 32), interpolation=cv2.INTER_CUBIC)
vis = np.zeros_like(img)
if len(img.shape)==3:
vis[..., 0] = cv2.dct(cv2.dct(np.float32(img[..., 0])))
vis[..., 1] = cv2.dct(cv2.dct(np.float32(img[..., 1])))
vis[..., 2] = cv2.dct(cv2.dct(np.float32(img[..., 2])))
vis1 = (vis[6:14, 6:14, 0] + vis[6:14, 18:26, 1] + vis[18:26, 6:14, 2] + vis[18:26, 18:26, 0] + vis[12:20, 12:20, 0] + vis[12:20, 12:20, 1] + vis[12:20, 12:20, 2])/7.0
else:
vis[..., 0] = cv2.dct(cv2.dct(np.float32(img[..., 0])))
vis1 = (vis[6:14, 6:14, 0] + vis[6:14, 18:26, 0] + vis[18:26, 6:14, 0] + vis[18:26, 18:26, 0] + vis[12:20, 12:20, 0]) / 5.0
img_list = vis1.reshape(-1).tolist()
avg = np.mean(img_list)
avg_list = ['0' if i < avg else '1' for i in img_list]
phash = ['%d' % int("".join(avg_list[x:x + 3]), 2) for x in range(0, 8 * 8, 3)]
int_hash = ''
for i in phash:
if int(i) < 10:
# 如果为各位数十位补零
int_hash += '0' + str(i)
else:
int_hash += str(i)
int_hash = int(int_hash)
return int_hash
提取后的hash值先逐位转化为二进制字符，后通过字符型转换成中间十位数补0的hash索引序列，如：
['12', '13', '12', '4', '1', '15', '12', '5', '14', '7', '6', '5', '2', '0', '4', '5']
--> 12131204011512051407060502000405
2、图片在信息库位置查找
图片hash索引在信息库中位置查找，二分法查找target在List（信息库）中位置
def find_Sorted_Position(List, target):
"""
# 二分法查找100000000以内,查找耗时不超过1ms
:param List: the list waiting to lookup
:param target: target number
:return: target's position
"""
if target < List[0]:
return 0
elif target > List[-1]:
return len(List)
else:
low = 0
high = len(List) - 1
while low <= high:
mid = (high + low) // 2
if high - low <= 1:
return mid + 1
elif target == List[mid]:
return mid
elif target < List[mid]:
high = mid
else:
low = mid
return low + 1
3、更新图片信息库
def info_update(self, update=False):
import time
tm_min = time.localtime(time.time())[5]
# 每30min更新一次图片信息库
if tm_min % 31 == 0 or update:
time_start = time.time()
with open(self.infobase, 'w') as write_f:
img_info = {"info_base": self.img_dict}
json.dump(img_info, write_f)
print("写/更新的图片信息库耗时： {:0.4f} s".format(time.time() - time_start, len(self.img_dict)))
with open("similar_imgs.txt", 'a')as file_similar:
for similar_img in self.similar:
file_similar.write(similar_img[0] + ' ' + similar_img[1] + '\n')
- 本地测试结果展示
读取图片信息库数据字典耗时: 0.6860 s
当前信息库图片数量： 436776
10-57_50157089.jpg is similar to E:/2019/image_rm-dui/train0/57_50157089.jpg !
100-57_49992915.jpg is similar to E:/2019/image_rm-dui/train0/57_49992915.jpg !
0-69_50002503.jpg is similar to E:/2019/image_rm-dui/train0/69_50002503.jpg !
1-69_50362572.jpg is similar to E:/2019/image_rm-dui/train0/69_50002503.jpg !
1002-69_49593728.jpg is similar to E:/2019/image_rm-dui/train0/69_49593728.jpg !
1000-69_49924978.jpg is similar to E:/2019/image_rm-dui/train0/69_49924978.jpg !
1001-69_49926153.jpg is similar to E:/2019/image_rm-dui/train0/69_49924978.jpg !
1003-69_49614853.jpg is similar to E:/2019/image_rm-dui/train0/69_49593728.jpg !
重复图查找平均耗时: 0.2556 ms
写/更新的图片信息库耗时： 1.3470 s
- 觉得还行，点个赞，或者github点个star呗，有问题欢迎留言咨询， 奉上github完整demo地址link
https://github.com/tao-ht/Image-repeat-image-filter
Copyright hengtao tao. All Rights Reserved.