利用梯度上升可视化卷积核:基于torch实现

利用梯度上升可视化卷积核

文章目录

  • 前言
      • 基本原理
      • 版本和包
      • 结果展示
  • 简单绘图
  • 修改源码绘图
      • 方法一
    • 方法二(推荐)
  • 报错解决
  • 总结


前言

基于梯度上升的可视化是一种常用的技术,用于理解卷积神经网络(CNN)中的卷积核是如何对输入图像进行特征提取的。该方法可以通过最大化卷积层输出的激活值来生成图像,从而使得卷积核对特定特征更加敏感。

基本原理

以下是基于梯度上升可视化卷积核的基本原理步骤:

  1. 选择目标卷积层和卷积核:首先,选择你希望可视化的目标卷积层和对应的卷积核。通常选择靠近网络顶部的卷积层,因为这些层对于更高级别的特征有更强的响应。
  2. 定义损失函数:为了最大化卷积层的输出激活值,需要定义一个损失函数。通常使用该卷积层的平均激活值作为损失函数。
  3. 随机生成图像:从随机噪声图像开始,作为初始输入图像。
  4. 前向传播:将生成的图像输入到CNN网络中,并进行前向传播,记录目标卷积层的输出激活值。
  5. 反向传播:计算损失函数对于输入图像的梯度,即目标卷积层输出激活值对于输入图像的影响。
  6. 更新输入图像:根据梯度信息更新输入图像,使得激活值增大。
  7. 重复步骤4-6:重复进行前向传播、反向传播和图像更新,直到达到预定的迭代次数或满足特定的停止条件。
  8. 可视化结果:最终生成的图像即为可视化结果,它代表了卷积核对于特定特征的响应模式。

通过这个过程,可以逐步调整输入图像,使得卷积核对于特定特征更加敏感。这种方法可以帮助我们理解卷积神经网络学习到的特征,以及卷积层如何对输入图像进行处理。


版本和包

python=3.7
torch=1.7
flashtorch=0.1.3

其中flashtorch就是用来求取中间层梯度上升的包

结果展示

利用梯度上升可视化卷积核:基于torch实现_第1张图片

利用梯度上升可视化卷积核:基于torch实现_第2张图片

利用梯度上升可视化卷积核:基于torch实现_第3张图片

简单绘图

官方代码如下所示:
(以vgg16为例子)

import torchvision.models as models
from flashtorch.activmax import GradientAscent

model = models.vgg16(pretrained=True)
g_ascent = GradientAscent(model.features)

# specify layer and filter info
conv5_1 = model.features[24]
conv5_1_filters = [45, 271, ]## 这里调节要展示的卷积核(绘出子图的个数),最大值是对应权重的通道数

vis = g_ascent.visualize(conv5_1, conv5_1_filters, title="VGG16: conv5_1",return_output=True)

以上代码,用vis即可画图。
vis =【output45,output271】的列表,每个都包含很多层,源代码是用的最后一层进行绘图(下图为源码截图):
利用梯度上升可视化卷积核:基于torch实现_第4张图片


修改源码绘图

方法一

由于官方代码画不出来图,所以修改from flashtorch.activmax import GradientAscent 的源码,把plt返回即可:
找到GradientAscent函数(gradient_ascent.py) 第211行
利用梯度上升可视化卷积核:基于torch实现_第5张图片
修改为:

if return_output:
    return self.output, self.plt

方法二(推荐)

最好的方法是:在同级目录下新建文件grad_as.py
直接把以下代码复制到grad_as.py里:

import numpy as np
import matplotlib.pyplot as plt

import torch
import torch.nn as nn

from flashtorch.utils import (apply_transforms,
                              format_for_plotting,
                              load_image,
                              standardize_and_clip)


class GradientAscent:
    """Provides an interface for activation maximization via gradient descent.

    This class implements the gradient ascent algorithm in order to perform
    activation maximization with convolutional neural networks (CNN).

    `Activation maximization `_
    is one form of feature visualization that allows us to visualize what CNN
    filters are "looking for", by applying each filter to an input image and
    updating the input image so as to maximize the activation of the filter of
    interest (i.e. treating it as a gradient ascent task with activation as the
    loss). The implementation is inspired by `this demo `_
    by Francois Chollet.

    Args:
        model: A neural network model from `torchvision.models
            `_,
            typically without the fully-connected part of the network.
            e.g. torchvisions.alexnet(pretrained=True).features
        img_size (int, optional, default=224): The size of an input image to be
            optimized.
        lr (float, optional, default=1.): The step size (or learning rate) of
            the gradient ascent.
        use_gpu (bool, optional, default=False): Use GPU if set to True and
            `torch.cuda.is_available()`.

    """ # noqa

    ####################
    # Public interface #
    ####################

    def __init__(self, model, img_size=224, lr=1., use_gpu=False):
        self.model = model
        self._img_size = img_size
        self._lr = lr
        self._use_gpu = use_gpu

        self.num_layers = len(list(self.model.named_children()))
        self.activation = None
        self.gradients = None

        self.handlers = []

        self.output = None
        self.plt = None

    @property
    def lr(self):
        return self._lr

    @lr.setter
    def lr(self, lr):
        self._lr = lr

    @property
    def img_size(self):
        return self._img_size

    @img_size.setter
    def img_size(self, img_size):
        self._img_size = img_size

    @property
    def use_gpu(self):
        return self._use_gpu

    @use_gpu.setter
    def use_gpu(self, use_gpu):
        self._use_gpu = use_gpu

    def optimize(self, layer, filter_idx, input_=None, num_iter=30):
        """Generates an image that maximally activates the target filter.

        Args:
            layer (torch.nn.modules.conv.Conv2d): The target Conv2d layer from
                which the filter to be chosen, based on `filter_idx`.
            filter_idx (int): The index of the target filter.
            num_iter (int, optional, default=30): The number of iteration for
                the gradient ascent operation.

        Returns:
            output (list of torch.Tensor): With dimentions
                :math:`(num_iter, C, H, W)`. The size of the image is
                determined by `img_size` attribute which defaults to 224.

        """

        # Validate the type of the layer

        if type(layer) != nn.modules.conv.Conv2d:
            raise TypeError('The layer must be nn.modules.conv.Conv2d.')

        # Validate filter index

        num_total_filters = layer.out_channels
        self._validate_filter_idx(num_total_filters, filter_idx)

        # Inisialize input (as noise) if not provided

        if input_ is None:
            input_ = np.uint8(np.random.uniform(
                150, 180, (self._img_size, self._img_size, 3)))
            input_ = apply_transforms(input_, size=self._img_size)

        if torch.cuda.is_available() and self.use_gpu:
            self.model = self.model.to('cuda')
            input_ = input_.to('cuda')

        # Remove previous hooks if any

        while len(self.handlers) > 0:
            self.handlers.pop().remove()

        # Register hooks to record activation and gradients

        self.handlers.append(self._register_forward_hooks(layer, filter_idx))
        self.handlers.append(self._register_backward_hooks())

        # Inisialize gradients

        self.gradients = torch.zeros(input_.shape)

        # Optimize

        return self._ascent(input_, num_iter)

    def visualize(self, layer, filter_idxs=None, lr=1., num_iter=30,
                  num_subplots=4, figsize=(4, 4), title='Conv2d',
                  return_output=False):
        """Optimizes for the target layer/filter and visualizes the output.

        A method that combines optimization and visualization. There are
        mainly 3 types of operations, given a target layer:

        1. If `filter_idxs` is provided as an integer, it optimizes for the
            filter specified and plots the output.
        2. If `filter_idxs` is provided as a list of integers, it optimizes for
            all the filters specified and plots the output.
        3. if `filter_idx` is not provided, i.e. None, it randomly chooses
            `num_subplots` number of filters from the layer provided and
            plots the output.

        It also returns the output of the optimization, if specified with
        `return_output=True`.

        Args:
            layer (torch.nn.modules.conv.Conv2d): The target Conv2d layer from
                which the filter to be chosen, based on `filter_idx`.
            filter_idxs (int or list of int, optional, default=None): The index
                or indecies of the target filter(s).
            lr (float, optional, default=.1): The step size of optimization.
            num_iter (int, optional, default=30): The number of iteration for
                the gradient ascent operation.
            num_subplots (int, optional, default=4): The number of filters to
                optimize for and visualize. Relevant in case 3 above.
            figsize (tuple, optional, default=(4, 4)): The size of the plot.
                Relevant in case 1 above.
            title (str, optional default='Conv2d'): The title of the plot.
            return_output (bool, optional, default=False): Returns the
                output(s) of optimization if set to True.


        Returns:
            For a single optimization (i.e. case 1 above):
                output (list of torch.Tensor): With dimentions
                    :math:`(num_iter, C, H, W)`. The size of the image is
                    determined by `img_size` attribute which defaults to 224.
            For multiple optimization (i.e. case 2 or 3 above):
                output (list of list of torch.Tensor): With dimentions
                    :math:`(num_subplots, num_iter, C, H, W)`. The size of the
                    image is determined by `img_size` attribute which defaults
                    to 224.

        """

        self._lr = lr

        if (type(filter_idxs) == int):
            self._visualize_filter(layer,
                                   filter_idxs,
                                   num_iter=num_iter,
                                   figsize=figsize,
                                   title=title)
        else:
            num_total_filters = layer.out_channels

            if filter_idxs is None:
                num_subplots = min(num_total_filters, num_subplots)
                filter_idxs = np.random.choice(range(num_total_filters),
                                               size=num_subplots)

            self._visualize_filters(layer,
                                    filter_idxs,
                                    num_iter,
                                    len(filter_idxs),
                                    title=title)

        if return_output:
            return self.output, self.plt

    def deepdream(self, img_path, layer, filter_idx, lr=.1, num_iter=20,
                  figsize=(4, 4), title='DeepDream', return_output=False):
        """Creates DeepDream.

        It applies the optimization on the image provided. The image is loaded
        and made into a torch.Tensor that is compatible as the input to the
        network.

        Read the original blog post by Google for more information on
        `DeepDream `_.

        Args:
            img_path (str): A path to the image you want to apply DeepDream on
            layer (torch.nn.modules.conv.Conv2d): The target Conv2d layer from
                which the filter to be chosen, based on `filter_idx`.
            filter_idx (int): The index of the target filter.
            lr (float, optional, default=.1): The step size of optimization.
            num_iter (int, optional, default=30): The number of iteration for
                the gradient ascent operation.
            figsize (tuple, optional, default=(4, 4)): The size of the plot.
                Relevant in case 1 above.
            title (str, optional default='Conv2d'): The title of the plot.
            return_output (bool, optional, default=False): Returns the
                output(s) of optimization if set to True.

        Returns:
            output (list of torch.Tensor): With dimentions
                :math:`(num_iter, C, H, W)`. The size of the image is
                determined by `img_size` attribute which defaults to 224.

        """ # noqa

        input_ = apply_transforms(load_image(img_path), self.img_size)

        self._lr = lr
        output = self.optimize(layer, filter_idx, input_, num_iter=num_iter)

        plt.figure(figsize=figsize)
        plt.axis('off')
        plt.title(title)

        plt.imshow(format_for_plotting(
            standardize_and_clip(output[-1],
                                 saturation=0.15,
                                 brightness=0.7))); # noqa

        if return_output:
            return output

    #####################
    # Private interface #
    #####################

    def _register_forward_hooks(self, layer, filter_idx):
        def _record_activation(module, input_, output):
            self.activation = torch.mean(output[:, filter_idx, :, :])

        return layer.register_forward_hook(_record_activation)

    def _register_backward_hooks(self):
        def _record_gradients(module, grad_in, grad_out):
            if self.gradients.shape == grad_in[0].shape:
                self.gradients = grad_in[0]

        for _, module in self.model.named_modules():
            if isinstance(module, nn.modules.conv.Conv2d) and \
                    module.in_channels == 3:
                return module.register_backward_hook(_record_gradients)

    def _ascent(self, x, num_iter):
        output = []

        for i in range(num_iter):
            self.model(x)

            self.activation.backward()

            self.gradients /= (torch.sqrt(torch.mean(
                torch.mul(self.gradients, self.gradients))) + 1e-5)

            x = x + self.gradients * self._lr
            output.append(x)

        return output

    def _validate_filter_idx(self, num_filters, filter_idx):
        if not np.issubdtype(type(filter_idx), np.integer):
            raise TypeError('Indecies must be integers.')
        elif (filter_idx < 0) or (filter_idx > num_filters):
            raise ValueError(f'Filter index must be between 0 and \
                             {num_filters - 1}.')

    def _visualize_filter(self, layer, filter_idx, num_iter, figsize, title):
        self.output = self.optimize(layer, filter_idx, num_iter=num_iter)

        plt.figure(figsize=figsize)
        plt.axis('off')
        plt.title(title)

        plt.imshow(format_for_plotting(
            standardize_and_clip(self.output[-1],
                                 saturation=0.15,
                                 brightness=0.7))); # noqa

    def _visualize_filters(self, layer, filter_idxs, num_iter, num_subplots,
                           title):
        # Prepare the main plot

        num_cols = 4
        num_rows = int(np.ceil(num_subplots / num_cols))

        fig = plt.figure(figsize=(16, num_rows * 5))
        plt.title(title)
        plt.axis('off')

        self.output = []

        # Plot subplots

        for i, filter_idx in enumerate(filter_idxs):
            output = self.optimize(layer, filter_idx, num_iter=num_iter)

            self.output.append(output)

            ax = fig.add_subplot(num_rows, num_cols, i+1)
            ax.set_xticks([])
            ax.set_yticks([])
            ax.set_title(f'filter {filter_idx}')

            ax.imshow(format_for_plotting(
                standardize_and_clip(output[-1],
                                     saturation=0.15,
                                     brightness=0.7)))

        plt.subplots_adjust(wspace=0, hspace=0); # noqa
        self.plt = plt```

然后使用以下代码绘图:

import torchvision.models as models
from grad_as import GradientAscent

model = models.vgg16(pretrained=True) # 这里可以改成其它模型

g_ascent = GradientAscent(model.features)


# specify layer and filter info
conv5_1 = model.features[24]  # 相应的这里也要改,可以用print(model)代码查看模型结构进行修改
conv5_1_filters = [45, 271, ]

vis,plt = g_ascent.visualize(conv5_1, conv5_1_filters, title="VGG16: conv5_1",return_output=True)

plt.show()



报错解决

报错如下

from importlib_resources import path
ImportError: cannot import name 'path' from 'importlib_resources'

找到报错处的源文件,尝试在代码中手动导入 path,而不是从 importlib_resources 直接导入:

from importlib_resources import path

替换为:

from importlib_resources import files

def path(package, resource):
    return files(package).joinpath(resource)

这样,代码将使用 files 函数来导入 path,而不是直接从 importlib_resources 中导入。


总结

以上代码均经过本人亲测可用。

你可能感兴趣的:(深度学习,cnn,深度学习,神经网络)