【MindSpore易点通】网络构建经验总结下篇

MindSpore实现梯度不回传以及梯度回传后不更新权重

背景信息

训练中经常会用到某层的梯度不回传（比如互学习）或者梯度回传但是不更新权重（Fine-tuning）

经验总结

1. 梯度不回传使用stop_gradient接口实现，代码示例如下：

import mindspore.nn as nnfrom mindspore.ops 
import operations as Pfrom mindspore.ops 
import functional as Ffrom mindspore.nn.loss.loss 
import _Lossfrom mindspore 
import Tensor, Parameterfrom mindspore.common 
import dtype as mstypefrom mindspore.ops.functional 
import stop_gradient

class Contrastive(_Loss):
    def __init__(self, args):
        super(Contrastive, self).__init__()
        self.args = args
        self.stride_slice = P.StridedSlice()
        self.pow = P.Pow()
        self.sum = P.CumSum()
        self.dist_weight = Tensor(4, dtype=mstype.float32)
        emb_list = list(range(args.per_batch_size))
        emb1_list = emb_list[0::2]
        emb2_list = emb_list[1::2]
        self.emb1_param = Tensor(emb1_list, dtype=mstype.int32)
        self.emb2_param = Tensor(emb2_list, dtype=mstype.int32)
        self.add = P.TensorAdd()
        self.div = P.RealDiv()
        self.cast = P.Cast()
        self.gatherv2 = P.GatherV2()

    def construct(self, nembeddings):
        nembeddings_shape = F.shape(nembeddings)
        emb1 = self.gatherv2(nembeddings, self.emb1_param, 0)
        emb2 = self.gatherv2(nembeddings, self.emb2_param, 0)
        emb2_detach = stop_gradient(emb2)      //阻止emb2的梯度回传
        emb3 = emb1 - emb2_detach
        pow_emb3 = emb3 * emb3
        dist = self.sum(pow_emb3, 1)

        return self.div(dist*self.dist_weight, self.cast(F.scalar_to_array(nembeddings_shape[0]), mstype.float32))

1. 梯度回传后不更新权重，使用requires_grad=False来实现，代码示例如下（假设要把名字为conv1的层权重冻结）：

for param in net.trainable_params():
    if 'conv1' in param.name:
        param.requires_grad = False
    else:
        param.requires_grad = True

MindSpore中使用Loss Scale（Feed模式下）关于sens参数的配置

背景信息

D芯片的卷积只有FP16精度，所以用D芯片训练一定是在跑混合精度。为避免梯度下溢，需要使用Loss Scale。

经验总结

Feed模式流程下，接口中Optimizer和TrainOneStepCell的sens需要手动设置成同一数值

opt = nn.Momentum(params=train_net.trainable_params(),
                  learning_rate=lr_iter,
                  momentum=0.9,
                  weight_decay=0.0001,
                  loss_scale=1000.0)

train_net = TrainOneStepCell(train_net, opt, sens=1000.0)

MindSpore中使用SequentialCell的输入必须为nn.Cell组成的List

背景信息

PyTorch在网络定义中经常使用torch.nn.Sequential来构造算子的列表，在MindSpore中要使用mindspore.nn.SequentialCell来实现这个功能。

经验总结

mindspore.nn.SequentialCell的输入和PyTorch的Sequential有所不同，输入必须为Cell组成的List，否则会有不符合预期的错误。 使用示例如下：

class MyNet(nn.Cell):
    def __init__(self):
        super(MyNet, self).__init__()
        self.conv = nn.Conv2d(16, 64, 3, pad_mode='pad', padding=0, dilation=2)
        self.bn = nn.BatchNorm2d(64)
        self.relu = nn.ReLU()
        self.seq = nn.SequentialCell([self.conv, self.bn, self.relu])   #这里必须把nn.Cell的对象包装为List作为SequentialCell的输入

    def construct(self, x):
        x = self.seq(x)
        return x

Transformer中Positional Encoding的MindSpore简单实现

背景信息

《Attention Is All You Need》中的位置编码方法，Transformer中较为常用。公式如下：

经验总结

为了适用于动态shape的输入，又由于mindspore.nn.Cell.construct中不便于进行numpy操作，采用先生成一个足够长的positional encodding向量再根据输入长度进行截取的方法。

import mindspore.ops.operations as Pimport mindspore.nn as nnfrom mindspore.common 
import dtype as mstypefrom mindspore import Tensorimport numpy as npimport math

class PositionalEncoding(nn.Cell):
    """Positional encoding as in Sec 3.5 https://arxiv.org/pdf/1706.03762.pdf
    :param int dim: dimension of input
    :param int maxlen: upper limit of sequence length
    :param float dropout_rate: dropout rate
    """

    def __init__(self, dim, maxlen=10000, dropout_rate=0.1):
        """Construct an PositionalEncoding object."""
        super(PositionalEncoding, self).__init__()

        xscale = math.sqrt(dim)
        self.dropout = nn.Dropout(1 - dropout_rate)
        self.mul = P.Mul()
        self.add = P.TensorAdd()
        self.shape = P.Shape()
        self.pe = self.postion_encoding_table(maxlen, dim)
        self.te = Tensor([xscale, ], mstype.float32)

    def construct(self, x):
        """
        Add positional encoding
        :param mindspore.Tensor x: batches of inputs (B, len, dim)
        :return: Encoded x (B, len, dim)
        """
        (_, l, _) = self.shape(x)
        pos = self.pe[:, :l, :]
        x = self.mul(x, self.te)
        x = self.add(x, pos)
        x = self.dropout(x)
        return x

    def postion_encoding_table(self, max_length, dims):
        pe = np.zeros((max_length, dims))
        position = np.arange(0, max_length).reshape((max_length, 1))
        div_term = np.exp(np.arange(0, dims, 2) * (-(math.log(10000.0) / dims)))
        div_term = div_term.reshape((1, div_term.shape[0]))
        pe[:, 0::2] = np.sin(np.matmul(position, div_term))
        pe[:, 1::2] = np.cos(np.matmul(position, div_term))
        pe = pe.reshape((1, max_length, dims))
        pe = Tensor(pe, mstype.float32)
        return pe