matlab parfor如何传递large共享变量

cvpr2019在超分辨率重建方面,旷视觉科技有一篇论文:Meta-SR,实现单一模型任意缩放因子超分,非常感谢作者的付出和贡献.

作者已经开放了源码,下面是我的分支,由于自己改的比较随意,所以今后的一些修改将在这个分支上继续:

https://github.com/idealboy/Meta-SR-Pytorch.git

在prepare_dataset的时候,源码里没有采用并行parfor,所以处理会比较慢,关键代码中的循环如下:

%% DIV2K_HR中存储所有图片,是一个站内存很大的数组
%%  如果将这个大数组传入parfor的每个worker中,效率会相当慢,启动不了parfor循环
%%  一个每个worker都会拷贝一份这个变量

for idx_im = 1:nb_im
    fprintf('Read HR :%d\n', idx_im);
    ImHR = imread(fullfile(path_src, filepaths(idx_im).name));
    DIV2K_HR{idx_im} = ImHR;
end
 
%% generate and save LR via imresize() with Bicubic

%% parfor只支持integer做循环索引,所以这里改变一下循环的形式

scales = 1.0:0.1:4.0;
scale_num = length(scales);

parfor i = 2:1:scale_num   %% 这里,DIV2K_HR会传入到parfor的每个worker中,效率很差

    scale = scales(i)

    FolderLR = fullfile(path_save,'LR_bicubic', sprintf('X%.2f',scale));
    
    if ~exist(FolderLR,'dir')  %%这里,需要'dir'这个参数,否则在parfor中会报错
        mkdir(FolderLR)
    end
    for IdxIm = 1:nb_im
        fprintf('IdxIm=%d\n', IdxIm);
        ImHR = DIV2K_HR{IdxIm};
        [h,w,n]=size(ImHR);
        if mod(h,scale) ~= 0
            h = h - 4;
        end
        if mod(w,scale) ~= 0
            w = w - 4;
        end
        ImHR =ImHR(1:h,1:w,:);
        ImLR = imresize(ImHR, 1/scale, 'bicubic');
        % name image
        fileName = filepaths(IdxIm).name
        NameLR = fullfile(FolderLR, fileName);
        % save image
        imwrite(ImLR, NameLR, 'png');
    end
end

    原始代码中,parfor循环中每个worker会传入大的数组变量DIV2K_HR,并拷贝,造成负载(load)很大,所以效率很慢,那么,怎么办?

    调整内外for循环,使得传入parfor workers中的变量体积变小即可.

    调整的代码详见如下(多线程parfor版本):

    https://github.com/idealboy/Meta-SR-Pytorch/blob/master/prepare_dataset/generate_LR_metasr_X1_X4_mt.m

    

for idx_im = 1:nb_im
    fprintf('Read HR :%d\n', idx_im);
    ImHR = imread(fullfile(path_src, filepaths(idx_im).name));
    DIV2K_HR{idx_im} = ImHR;
end

FolderLR_bicubic = fullfile(path_save,'LR_bicubic')

if ~exist(FolderLR_bicubic)
    mkdir(FolderLR_bicubic)
end
    
%% generate and save LR via imresize() with Bicubic

scales = 1.0:0.1:4.0;

scale_num = length(scales);

%% look at the original source code, when applying parfor on outter loop, the large variable 'DIV2K_HR' wiil be
%% passed to each workers, and it will be very very slow just for start up the parfor.

%% we just exchange the original outter-loop and inner-loop,
%% so that, when applying parfor on current inner-loop('parfor i = 2:1:scale_num'),
%% it will not cause a heavy load to each workers when passing the shared variable,(image = ImHR;)
%% and, now, for each image, all the resized images (with diffrent scale) will be processed parallelly(speed up)

%% 内外循环切换,每张图片,生成所有的scale图片后,再处理下一张图片
for IdxIm = 1:nb_im
    fprintf('IdxIm=%d\n', IdxIm);
    ImHR = DIV2K_HR{IdxIm};
    
    %% 每张图像的ImHR将在每个parfor woker中拷贝,load开销会很小
    parfor i = 2:1:scale_num
        image = ImHR;
        scale = scales(i);
        FolderLR = fullfile(path_save,'LR_bicubic', sprintf('X%.2f',scale));
        
        if ~exist(FolderLR, 'dir')
            mkdir(FolderLR)
        end

        [h,w,n]=size(image);
        if mod(h,scale) ~= 0
            h = h - 4;
        end
        if mod(w,scale) ~= 0
            w = w - 4;
        end
        image =image(1:h,1:w,:);

        image= imresize(image, 1/scale, 'bicubic');
        % name image
        fileName = filepaths(IdxIm).name
        NameLR = fullfile(FolderLR, fileName);
        % save image
        imwrite(image, NameLR, 'png');
    end
end

 

你可能感兴趣的:(图像复原,matlab编程技术,meta,super,resolution,parfor,share,large,data,array,matlab)