关于Pytorch的入门学习笔记

关于Anaconda的一些命令

原文
https://blog.csdn.net/qq_42257666/article/details/121361983
https://blog.csdn.net/ITLearnHall/article/details/81708148?spm=1001.2101.3001.6650.1&utm_medium=distribute.pc_relevant.none-task-blog-2%7Edefault%7ECTRLIST%7Edefault-1.pc_relevant_default&depth_1-utm_source=distribute.pc_relevant.none-task-blog-2%7Edefault%7ECTRLIST%7Edefault-1.pc_relevant_default&utm_relevant_index=2

cmd命令行
    查看CUDA版本
        nvidia-smi
    进入Anaconda
        conda
        activate
    在Anaconda安装 pytorch
        conda install pytorch torchvision cudatoolkit=10.1
    创建conda虚拟环境
        conda create --name pytorch1 python=3.8
    进入创建的conda
        conda activate pytorch1
    移除创建的conda
        conda remove -n pytorch1 --all
    展示在Anaconda中创建的环境
        conda env list

Dataset

类继承Dataset
    # MyData 类
    class MyData(Dataset):
        def __init__(self, root_dir, label_dir):
            self.root_dir = root_dir
            self.label_dir = label_dir
            self.path = os.path.join(self.root_dir, self.label_dir)
            self.img_path_list = os.listdir(self.path)
    
        def __getitem__(self, index):
            # 得到dataset图片
            img_name = self.img_path_list[index]
            img_item_path = os.path.join(self.root_dir,self.label_dir,img_name)
            img = Image.open(img_item_path)
            label = self.label_dir
            return img,label
    
        def __len__(self):
            return len(self.img_path_list)
使用
    # 实例
    root_dir = "dataset1/train"
    
    # 获得ants数据集
    ants_label_dir = "ants_image"
    ant_dataset = MyData(root_dir=root_dir, label_dir=ants_label_dir)
    
    # 获得bees数据集
    bees_label_dir = "bees_image"
    bee_dataset = MyData(root_dir=root_dir, label_dir=bees_label_dir)
    
    # 整合bees和anys 的数据集
    train_dataset = ant_dataset + bee_dataset

tensorboard

使用tensorboard
    from torch.utils.tensorboard import SummaryWriter
    import numpy as np
    from PIL import Image
    
    writer = SummaryWriter("logs")
    
    img_path = "dataset1/train/bees_image/92663402_37f379e57a.jpg"
    img = Image.open(img_path)
    img_array = np.array(img)
    # 图像数据类型 img_tensor
    writer.add_image("img_array1",img_array,2,dataformats='HWC')
    for i in range(100):
        # Args:
        #   tag (string): Data title
        #   scalar_value : y Value
        #   global_step (int): x Value
        writer.add_scalar("y=x",3*i,i) # 添加标量
    
    writer.close()


打开tensorboard生成的事务文件
Terminal 
    tensorboard --logdir=logs 
        -port=6007

将图片的格式PIL.JpegImagePlugin.JpegImageFile转成add_image接受的格式
    pip install opencv-python
    numpy.array
        img_array = np.array(img)

transforms

transforms.ToTensor
    作用
        将数据转换成神经网络需要的格式
        将数据转换成为 Tensor
    使用
        img_path = "dataset1/train/bees_image/92663402_37f379e57a.jpg"
        img = Image.open(img_path)
        # img_array = np.array(img)
        tensor_trans = transforms.ToTensor()
        tensor_img = tensor_trans(img)

transforms.Normalize
    output[channel] = (input[channel] - mean[channel]) / std[channel]
    例子
        0 < input[channel] < 1
        mean =[0.5,0.5,0.5]  #len(mean) = channel 
        std =[0.5,0.5,0.5]
        output[channel=0] = input[channel=0] * 2 - 1 
        -1 < output[channel=0] < 1

transforms.Resize & transforms.Compose
    # 使用Resize 对图片进行任意的缩放 512,512
    print(img.size)
    trans_resize = transforms.Resize((512, 512))
    img_resize1 = trans_resize(img)
    print(img_resize1)
    # PIL To TOTENSOR
    img_resize1 = trans_totensor(img_resize1)
    writer.add_image("img_resize1", img_resize1)
    
    # 使用Resize 对图片进行等比例的缩放 735 550
    print(img.size)
    trans_resize = transforms.Resize(550)
    trans_compose = transforms.Compose([trans_resize, trans_totensor])
        # img_resize2 = trans_resize(img)
        # img_resize2 = trans_totensor(img_resize2) 

    # PIL To TOTENSOR
    img_resize2 = trans_compose1(img)
    print(img_resize2.size)
    writer.add_image("img_resize2", img_resize2)
transforms.RandomCrop
    # 使用RandomCrop 随机裁剪指定大小的图片
    trans_randomCrop = transforms.RandomCrop([30,30])
    trans_compose2 = transforms.Compose([trans_randomCrop, trans_totensor])
    for i in range(10):
        img_randomCrop = trans_compose2(img)
        print(img_randomCrop.size)
        # PIL To TOTENSOR
        writer.add_image("img_randomCrop", img_randomCrop,i+1)
使用方法总结
    关注方法描述的 输入和输出
    关注方法的参数

torchvision中的数据集使用

CIFAR10
https://pytorch.org/vision/stable/generated/torchvision.datasets.CIFAR10.html#torchvision.datasets.CIFAR10

Parameters
- root (string) 
    – Root directory of dataset where directory cifar-10-batches-py exists or will be saved to if download is set to True.
- train (bool, optional) 
    – If True, creates dataset from training set, otherwise creates from test set.
- transform (callable, optional) 
    – A function/transform that takes in an PIL image and returns a transformed version. E.g, transforms.RandomCrop
- target_transform (callable, optional) 
    – A function/transform that takes in the target and transforms it.
- download (bool, optional) 
    – If true, downloads the dataset from the internet and puts it in root directory. If dataset is already downloaded, it is not downloaded again.


code
    import torchvision
    from torch.utils.tensorboard import SummaryWriter
    import ssl
    
    ssl._create_default_https_context = ssl._create_unverified_context
    
    dataset_transform = torchvision.transforms.Compose([torchvision.transforms.ToTensor()])
    
    train_set = torchvision.datasets.CIFAR10(root="./dataset3", transform=dataset_transform, train=True, download=True)
    test_set = torchvision.datasets.CIFAR10(root="./dataset3", transform=dataset_transform, train=False, download=True)
    
    # 写事务中
    writer = SummaryWriter("log_CIFAR10")
    for i in range(10):
        img, target = test_set[i]
        writer.add_image("test_set", img, i)
    writer.close()

DataLoader中的数据集使用

https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader

torch.utils.data.DataLoader(dataset, batch_size=1, shuffle=False, 
                            sampler=None, batch_sampler=None, num_workers=0, 
                            collate_fn=None, pin_memory=False, drop_last=False, 
                            timeout=0, worker_init_fn=None, multiprocessing_context=None,
                            generator=None, *, prefetch_factor=2, persistent_workers=False)
Parameters
    dataset (Dataset) – dataset from which to load the data.
    batch_size (int, optional) 随机从dataset中抓取batch_size数量的 对应数据
        – how many samples per batch to load (default: 1).
    shuffle (bool, optional) 决定每次DataLoader的结果是否一样 
        – set to True to have the data reshuffled at every epoch (default: False).
    sampler (Sampler or Iterable, optional) – defines the strategy to draw samples from the dataset. Can be any Iterable with __len__ implemented. If specified, shuffle must not be specified.
    batch_sampler (Sampler or Iterable, optional) – like sampler, but returns a batch of indices at a time. Mutually exclusive with batch_size, shuffle, sampler, and drop_last.
    num_workers (int, optional) 控制进程数进程
        – how many subprocesses to use for data loading. 0 means that the data will be loaded in the main process. (default: 0)
    collate_fn (callable, optional) – merges a list of samples to form a mini-batch of Tensor(s). Used when using batched loading from a map-style dataset.
    pin_memory (bool, optional) – If True, the data loader will copy Tensors into CUDA pinned memory before returning them. If your data elements are a custom type, or your collate_fn returns a batch that is a custom type, see the example below.
    drop_last (bool, optional) 处理多出来的数据 
        – set to True to drop the last incomplete batch, if the dataset size is not divisible by the batch size. If False and the size of dataset is not divisible by the batch size, then the last batch will be smaller. (default: False)
    timeout (numeric, optional) – if positive, the timeout value for collecting a batch from workers. Should always be non-negative. (default: 0)
    worker_init_fn (callable, optional) – If not None, this will be called on each worker subprocess with the worker id (an int in [0, num_workers - 1]) as input, after seeding and before data loading. (default: None)

torch.nn

nn.Module的使用

Python API https://pytorch.org/docs/stable/nn.html

NN -Neural Networks
    Containers
        容器、骨架
        模块
            Module
                Base class for all neural network modules.
                提供模板
                import torch.nn as nn
                import torch.nn.functional as F
                class Model(nn.Module):
                    def __init__(self):
                        super().__init__()
                        self.conv1 = nn.Conv2d(1, 20, 5)
                        self.conv2 = nn.Conv2d(20, 20, 5)
                    
                    # 处理输入的函数 
                    # 定义每次调用时执行的计算。
                    # 应该被所有子类覆盖。
                    def forward(self, x):
                        # self.conv1(x) 对输入 x, 进行卷积
                        #  F.relu() 非线性处理 
                        x = F.relu(self.conv1(x))
                        # 二次卷积self.conv2(x)
                        return F.relu(self.conv2(x))
            Sequential
                A sequential container.
            ModuleList
                Holds submodules in a list.
            ModuleDict
                Holds submodules in a dictionary.
            ParameterList
                Holds parameters in a list.
            ParameterDict
                Holds parameters in a dictionary.

    

例子
    import torch
    from torch import nn
    
    class Test_Neural_Networks(nn.Module):
        # 参数的设置
        def __init__(self):
            super().__init__()
        # 使用该类
        def forward(self, input):
            output = input + 1
            return output
    
    
    TNN = Test_Neural_Networks()
    tensor_x = torch.tensor(1.0)
    output = TNN(tensor_x)
    print(output)

Sequential()

code
    self.sequential1 = Sequential(
        # c=3 32*32
        Conv2d(in_channels=3, out_channels=32, kernel_size=(5, 5), stride=(1, 1), padding=2),
        # c=32 32*32
        MaxPool2d(kernel_size=2),
        # c=32 16*16
        Conv2d(in_channels=32, out_channels=32, kernel_size=(5, 5), stride=(1, 1), padding=2),
        # c=32 16*16
        MaxPool2d(kernel_size=2),
        # c=32 8*8
        Conv2d(in_channels=32, out_channels=64, kernel_size=(5, 5), stride=(1, 1), padding=2),
        # c=64 8*8
        MaxPool2d(kernel_size=2),
        # c=64 4*4
        Flatten(),
        # c=1 1*1024 高度 = 1
        Linear(1024, 64),
        # c=1 1*64 高度 = 1
        Linear(64, 10)
        # c=1 1*10 高度 = 1
    )
     def forward(self, input):
        input = self.conv1(input)
        input = self.maxpool1(input)
        input = self.conv2(input)
        input = self.maxpool2(input)
        input = self.conv3(input)
        input = self.maxpool3(input)
        input = self.flatten1(input)
        hidden = self.linear1(input)
        output = self.linear2(hidden)
        return output

    等效于

    # c=3 32*32
    self.conv1 = Conv2d(in_channels=3, out_channels=32, kernel_size=(5, 5), stride=(1, 1), padding=2)
    # c=32 32*32
    self.maxpool1 = MaxPool2d(kernel_size=2)
    # c=32 16*16
    self.conv2 = Conv2d(in_channels=32, out_channels=32, kernel_size=(5, 5), stride=(1, 1), padding=2)
    # c=32 16*16
    self.maxpool2 = MaxPool2d(kernel_size=2)
    # c=32 8*8
    self.conv3 = Conv2d(in_channels=32, out_channels=64, kernel_size=(5, 5), stride=(1, 1), padding=2)
    # c=64 8*8
    self.maxpool3 = MaxPool2d(kernel_size=2)
    # c=64 4*4
    self.flatten1 = Flatten()
    # c=1 1*1024 高度 = 1
    self.linear1 = Linear(1024, 64)
    # c=1 1*64 高度 = 1
    self.linear2 = Linear(64, 10)
    # c=1 1*10 高度 = 1

    def forward(self, input):
        output = self.sequential1(input)
        return output

torch.nn 是 torch.nn.functional封装

torch.nn.functional
    conv1d
        Applies a 1D convolution over an input signal composed of several input planes.
    conv2d
        对2D卷积进行操作
        Applies a 2D convolution over an input image composed of several input planes.
        Pameters
            input – input tensor of shape (\text{minibatch} , \text{in\_channels} , iH , iW)(minibatch,in_channels,iH,iW)
            weight 权重、卷积核
                groups 通常设定为1
                – filters of shape (\text{out\_channels} , \frac{\text{in\_channels}}{\text{groups}} , kH , kW)(out_channels,groupsin_channels ,kH,kW)
            bias 编制
                – optional bias tensor of shape (\text{out\_channels})(out_channels). Default: None
            stride 卷积核 的步幅和方向(右，右下) 
                – the stride of the convolving kernel. Can be a single number or a tuple (sH, sW). Default: 1
                卷积核的移动为 在输入矩阵上
                    向右移动sW直到尽头
                    向下移动SH的左边
                    向右移动sW直到尽头
                    向下移动SH的左边
            padding 对输入的矩阵两边进行填充，宽度为1 例如 2*2 -> 3*3
                介绍
                    –implicit paddings on both sides of the input. Can be a string {‘valid’, ‘same’}, single number or a tuple (padH, padW). Default: 0 padding='valid' is the same as no padding. padding='same' pads the input so the output has the same shape as the input. However, this mode doesn’t support any stride values other than 1
                例子
                    [[1,2],
                    [3,4]] -> padding = 1

                    [[0,0,0,0],
                    [0,1,2,0],
                    [0,3,4,0],
                    [0,0,0,0]]
    conv3d
        Applies a 3D convolution over an input image composed of several input planes.
    conv_transpose1d
        Applies a 1D transposed convolution operator over an input signal composed of several input planes, sometimes also called “deconvolution”.
    conv_transpose2d
        Applies a 2D transposed convolution operator over an input image composed of several input planes, sometimes also called “deconvolution”.
    conv_transpose3d
        Applies a 3D transposed convolution operator over an input image composed of several input planes, sometimes also called “deconvolution”
    unfold
        Extracts sliding local blocks from a batched input tensor.
    fold
        Combines an array of sliding local blocks into a large containing tensor.

code
    import torch
    from torch import nn
    import torch.nn.functional as F

    input_matrix_2D = [[1, 2, 0, 3, 1],
                       [0, 1, 2, 3, 1],
                       [1, 2, 1, 0, 0],
                       [5, 2, 3, 1, 1],
                       [2, 1, 0, 1, 1], ]
    input_kernel_2D = [[1, 2, 1],
                       [0, 1, 0],
                       [2, 1, 0], ]
    matrix_torch = torch.tensor(input_matrix_2D)
    kernel_torch = torch.tensor(input_kernel_2D)
    
    # input tensor of shape (\text{minibatch} , \text{in\_channels} , iH , iW)(minibatch,in_channels,iH,iW)
    # 2.根据要求做输入矩阵的 变换
    
    # input, (channel,batch_size, matrisx_h, matrisx_w)
    matrix_torch = torch.reshape(matrix_torch, (1, 1, 5, 5))
    kernel_torch = torch.reshape(kernel_torch, (1, 1, 3, 3))
    
    # out_{i}{j} =  input1*w1 + input2*w2+ ... + inputn*wn
    output = F.conv2d(input=matrix_torch, weight=kernel_torch, bias=None, stride=1)
    print(output)
    
    output2 = F.conv2d(input=matrix_torch, weight=kernel_torch, bias=None, stride=2)
    print(output2)
    
    output2 = F.conv2d(input=matrix_torch, weight=kernel_torch, bias=None, stride=1, padding=1)
    print(output2)

torch.nn.CONV2D

https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html#torch.nn.Conv2d

以Layers为结尾应该是 往骨架中填充的程序
Convolution Layers
        卷积层
        torch.nn.Conv2d(in_channels, out_channels, kernel_size, 
                        stride=1, padding=0, dilation=1, groups=1,
                        bias=True, padding_mode='zeros', device=None, dtype=None)
            
Parameters
    in_channels (int) – Number of channels in the input image
    out_channels (int) – Number of channels produced by the convolution
    kernel_size (int or tuple) – Size of the convolving kernel
    stride (int or tuple, optional) – Stride of the convolution. Default: 1
    padding (int, tuple or str, optional) – Padding added to all four sides of the input. Default: 0
    padding_mode (string, optional) – 'zeros', 'reflect', 'replicate' or 'circular'. Default: 'zeros'
    dilation (int or tuple, optional) – Spacing between kernel elements. Default: 1
    groups (int, optional) – Number of blocked connections from input channels to output channels. Default: 1
    bias (bool, optional) – If True, adds a learnable bias to the output. Default: True

code
    import torch
    import torchvision
    from torch import nn
    from torch.nn import Conv2d
    from torch.utils.data import DataLoader
    from torch.utils.tensorboard import SummaryWriter
    
    import ssl
    
    ssl._create_default_https_context = ssl._create_unverified_context
    
    dataset_transform = torchvision.transforms.Compose([torchvision.transforms.ToTensor()])
    train_set = torchvision.datasets.CIFAR10(root="./dataset3", transform=dataset_transform, train=True, download=True)
    test_set = torchvision.datasets.CIFAR10(root="./dataset3", transform=dataset_transform, train=False, download=True)
    
    dataloader = DataLoader(test_set, batch_size=64, shuffle=True)
    
    
    # 将32*32像素的图片，经过3*3卷积核，转换未30*30的图片
    class simple_neural_network(nn.Module):
        def __init__(self):
            # 完成simple_neural_network的父类 nn.Module 的初始化
            super(simple_neural_network, self).__init__()
            self.conv1 = Conv2d(in_channels=3, out_channels=6,
                                kernel_size=3, stride=1, padding=0)
    
        def forward(self, x):
            x = self.conv1(x)
            return x
    
    # 使用simple_neural_network
    
    neural_network1 = simple_neural_network()
    print(neural_network1)
    
    # 写入tensorboard
    writer = SummaryWriter("logs_conv2D")
    tage = 0
    for data_loder_item in dataloader:
        imgs, targets = data_loder_item
        # 进行卷积，实施上就是对图片每个 组成像素点的编码的RGB段，这3给channel进行变换，对图片进行放大或缩小（变换）
        output = neural_network1(imgs)
        # 64 images, 3 is RGB, 32 is height, 32 is width
        print(imgs.shape)
        print(output.shape)
        print("----------------------------------")
        writer.add_images("imgs", imgs, tage)
        # -1 为无视 batch_size=64,根据reshape来进行计算
        output = torch.reshape(output, (-1, 3, 30, 30))
        writer.add_images("output", output, tage)
    
        tage = tage + 1

Pooling layers的nn.MaxPool2d

https://pytorch.org/docs/stable/generated/torch.nn.MaxPool2d.html#torch.nn.MaxPool2d

作用
    根据n*n的池化核，从输入的矩阵中，选出最大的值
    取数据集中更加重要的数据，以达到减少数据量的目的   
Parameters
    kernel_size – the size of the window to take a max over
    stride 默认值为池化核的大小kernel_size，不重复比较值
        – the stride of the window. Default value is kernel_size
    padding – implicit zero padding to be added on both sides
    dilation – a parameter that controls the stride of elements in the window
    return_indices – if True, will return the max indices along with the outputs. Useful for torch.nn.MaxUnpool2d later
    ceil_mode 向下取整floor，向上取整ceiling 
        – when True, will use ceil instead of floor to compute the output shape

Padding layer

作用
    对输入的tensor进行填充
    用的少

Non-linear Activations

作用
    给神经网咯引入非线性的特特征

code
    class simple_nerual_network(nn.Module):
    def __init__(self):
        super(simple_nerual_network, self).__init__()
        self.relu1 = ReLU()
        self.sigmoid1 = Sigmoid()

    def forward(self, input):
        output = self.relu1(input)
        return output

Normalization Layers

作用
    加快神经网络的训练速度

code
    class simple_nerual_network(nn.Module):
    def __init__(self):
        super(simple_nerual_network, self).__init__()
        self.batchNorm2d1 = BatchNorm2d(3)


    def forward(self, input):
        output = self.batchNorm2d1(input)
        return output

Recurrent Layers

作用
    适用于文字识别中，适用于特定的网络结构，适用于特定网络

Transformer Layers

作用
    适用于特定的网络结构，适用于特定网络

Linear Layers

https://pytorch.org/docs/stable/generated/torch.nn.Linear.html#torch.nn.Linear

作用
    使用较多，适用于特定网络
方法
    torch.nn.Linear(in_features, out_features, bias=True, device=None, dtype=None)
    nn.Identity
        不区分参数的占位符标识运算符。

g1 = k1*x1 + k2*x2 + ... + kn*xn + b
    k_i \in (1,n) 为权重 由模型根据输入进行数学运算获得初始化的值
    b is bias，由模型根据输入进行数学运算获得初始化的值
    g1 为hidden layer的input
    insight
        神经网络的k_i和b进行调整，让神经网络的预测更加合理

Dropout Layers

作用
    防止过拟合

方法
    nn.Dropout
        在训练过程中，使用来自伯努利分布的样本，以概率p随机地将输入张量的一些元素归零。 

Sparse Layers

作用
    使用于NLP，适用于特定网络

Distance Functions

作用
    计算两个值之间的误差

Loss Functions

作用
    衡量module的output和实际的target、label(目标)的差距loss
    然后我们可以根据loss，指导module让它的输出更加接近 target(反向传播)
    loss越小越好

反向传播
    尝试如何调整神经网络的参数，使loss变小
    因为是从loss的方向计算，和网络的输出的顺序相反
    输出
        给每个需要调优的参数一个梯度（grab）
        优化的过程中根据梯度选择优化器(optimizer)对参数进行调优

nn.L1Loss
    https://pytorch.org/docs/stable/generated/torch.nn.L1Loss.html#torch.nn.L1Loss
    Creates a criterion that measures the mean absolute error (MAE) between each element in the input x and target y.
            

Vision Layers

Shuffle Layers

DataParallel Layers (multi-GPU, distributed)

Utilities

Quantized Functions

Lazy Modules Initialization

TORCH.OPTIM

https://pytorch.org/docs/stable/optim.html

code
    # lr 是 学习速率
    # 构造优化器
    optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.9)
    optimizer = optim.Adam([var1, var2], lr=0.0001)

    for input, target in dataset:
        # 将grad清零
        optimizer.zero_grad()
        output = model(input)
        loss = loss_fn(output, target)
        loss.backward()
        optimizer.step()

使用和修改现有的网络模型

https://pytorch.org/vision/stable/index.html

下载数据集
    安装下载数据集的工具
        pip install scipy

Parameters
    pretrained (bool) – If True, returns a model pre-trained on ImageNet
    progress (bool) – If True, displays a progress bar of the download to stderr

VGG模型
    定义
        VGG 16-layer model (configuration “D”) “Very Deep Convolutional Networks For Large-Scale Image Recognition”. The required minimum input size of the model is 32x32.
        VGG 16 层模型（配置“D”）“用于大规模图像识别的非常深的卷积网络”。模型所需的最小输入大小为 32x32。

# 在module中添加layer
vgg16_true.classifier.add_module('add_linear',nn.Linear(1000,10))
print(vgg16_true)
# 在module中修改已有的ayer: 修改名称为vgg16_false的module中classifier的索引为6的layer
vgg16_false.classifier[6] = nn.Linear(4096,10)

网络模型的保存和读取

保存
    vgg16_false = torchvision.models.vgg16(pretrained='false')

    # 保存方法 1 保存模型的结构 和 模型的参数 适用小模型
    # torch.save(vgg16_false,"./models1_vgg16.pth")
    
    # 保存方法 2 将vgg16_false的状态中的参数保存成字典，不保存结构 (官方推荐) 适用大模型
    torch.save(vgg16_false.state_dict(),"./models2_vgg16.pth")
    
    # 保存方式1的缺点
    class simple_nerual_network(nn.Module):
        def __init__(self):
            。。。
            )
    
        def forward(self, input):
            output = self.sequential1(input)
            return output
    
    nerual_network = simple_nerual_network()
    torch.save(nerual_network,"./models3_nerual_network.pth")

    保存方法1的缺点
    AttributeError: Can't get attribute 'simple_nerual_network' 
加载
    # 加载方法 1 加载模型的结构 和 模型的参数 适用小模型
    vgg16_false1 = torch.load("./models1_vgg16.pth")
    # print(vgg16_false1)
    
    # 加载方法 2 加载模型的参数
    vgg16_false2 = torchvision.models.vgg16(pretrained='false')
    
    vgg16_false2.load_state_dict(torch.load("./models2_vgg16.pth"))
    # print(vgg16_false2)
    
    
    # 保存方式 1 的缺点
    # 需要确保正确的模型
    
    nerual_network = torch.load("./models3_nerual_network.pth")
    print(nerual_network)

模型训练的流程

import torchvision
from torch.utils.data import DataLoader
from torch.utils.tensorboard import SummaryWriter

from L22_model import *

# 1. 准备数据集 10个标签 每个图片只有一种确定的标签
train_dataset = torchvision.datasets.CIFAR10("./dataset3", train=True,
                                             transform=torchvision.transforms.ToTensor(), download=True)
test_dataset = torchvision.datasets.CIFAR10("./dataset3", train=False,
                                            transform=torchvision.transforms.ToTensor(), download=True)
# 2. 输出数据集的长度
print("训练数据的 长度： {}".format(len(train_dataset)))
print("测试数据的 长度： {}".format(len(test_dataset)))

# 3. 使用DataLoader加载数据集
dataloder_train_set = DataLoader(train_dataset, batch_size=64)
dataloder_test_set = DataLoader(test_dataset, batch_size=64)

# 4. tensorboard记录数据
writer = SummaryWriter("./log_train")

# 5. 引入 实例化的、需要训练的模型
nerual_network = simple_nerual_network()
# 6. 引入损失函数
loss_fn = nn.CrossEntropyLoss()
# 7. 引入优化器 optim
learning_rate = 1e-2
optimizer = torch.optim.SGD(nerual_network.parameters(), lr=learning_rate)

# 8. 设置训练网络的一些参数
# 记录训练的次数
total_train_step = 0
# 记录测试的次数
total_test_step = 0
# 设置训练的轮数
epoch = 20

for i in range(epoch):
    print("------------NO {} train is START------------".format(i))
    loss_epoch_train = 0
    # 9 开始训练模型
    nerual_network.train()
    for data_item in dataloder_train_set:
        img, target = data_item
        outputs = nerual_network(img)
        loss = loss_fn(outputs, target)
        loss_epoch_train = loss_epoch_train + loss.item()
        # 使用优化器
        optimizer.zero_grad()
        # 使用loss进行反向传播, 获得每个参数节点的梯度grab
        loss.backward()
        # 优化器根据 梯度grab 对模型进行优化
        optimizer.step()
        print("NO {}, total_train_step: {}, loss: {}:  ".format(i, total_train_step, loss.item()))
        # writer记录数据
        writer.add_scalar("train_loss", loss.item(), global_step=total_train_step)
        total_train_step = total_train_step + 1
    print("NO {} 整体训练数据集 loss_epoch_train is :        {}".format(i, loss_epoch_train))
    writer.add_scalar("loss_epoch_train", loss.item(), global_step=total_train_step)


    # 测试 过程中不需要调优 torch.no_grab()
    loss_epoch_test = 0
    accuracy_epoch_test = 0
    with torch.no_grad():
        # 10 开始测试模型，在训练后的模型中 使用测试数据集 计算loss
        nerual_network.eval()
        for data_item in dataloder_test_set:
            img, target = data_item
            outputs = nerual_network(img)

            loss = loss_fn(outputs, target)
            loss_epoch_test = loss_epoch_test + loss.item()

            # 计算正确率
            accuracy = (outputs.argmax(1)==target).sum()
            accuracy_epoch_test = accuracy_epoch_test + accuracy

    print("NO {} 整体测试数据集 loss_epoch_test is :            {}".format(epoch, loss_epoch_test))
    print("NO {} 整体测试数据集 正确率 is :        {}".format(epoch, accuracy_epoch_test / len(test_dataset)))
    # writer记录数据
    writer.add_scalar("loss_epoch_test", loss_epoch_test, total_test_step)
    writer.add_scalar("accuracy_epoch_test", accuracy_epoch_test / len(test_dataset), total_test_step)
    total_test_step = total_test_step + 1

    # 11 保存模型 每一轮的模型
    torch.save(nerual_network, "model22_train_{}.pth".format(i))
    print("NO.{} 模型已经保存".format(i))

writer.close()

关于模型训练中的细节

module应该是model中的模块
model.eval()
    Sets the module in evaluation mode. 
model.train()
    定义
        Sets the module in training mode.
    介绍
        This has any effect only on certain modules. See documentations of
        particular modules for details of their behaviors in training/evaluation
        mode, if they are affected, e.g. :class:`Dropout`, :class:`BatchNorm`,
        etc.
    

##利用GPU训练模式

方式一
    class simple_nerual_network(nn.Module):
        def __init__(self):
            super(simple_nerual_network, self).__init__()
            # 使用Sequential
            self.sequential1 = Sequential(
                # c=3 32*32
                Conv2d(in_channels=3, out_channels=32, kernel_size=(5, 5), stride=(1, 1), padding=2),
                # c=32 32*32
                MaxPool2d(kernel_size=2),
                # c=32 16*16
                Conv2d(in_channels=32, out_channels=32, kernel_size=(5, 5), stride=(1, 1), padding=2),
                # c=32 16*16
                MaxPool2d(kernel_size=2),
                # c=32 8*8
                Conv2d(in_channels=32, out_channels=64, kernel_size=(5, 5), stride=(1, 1), padding=2),
                # c=64 8*8
                MaxPool2d(kernel_size=2),
                # c=64 4*4
                Flatten(),
                # c=1 1*1024 高度 = 1
                Linear(1024, 64),
                # c=1 1*64 高度 = 1
                Linear(64, 10)
                # c=1 1*10 高度 = 1
            )
    
        def forward(self, input):
            output = self.sequential1(input)
            return output
    
    # 利用GPU训练模型 G1
    # 1. 准备数据集 10个标签 每个图片只有一种确定的标签
    train_dataset = torchvision.datasets.CIFAR10("./dataset3", train=True,
                                                 transform=torchvision.transforms.ToTensor(), download=True)
    test_dataset = torchvision.datasets.CIFAR10("./dataset3", train=False,
                                                transform=torchvision.transforms.ToTensor(), download=True)
    # 2. 输出数据集的长度
    print("训练数据的 长度： {}".format(len(train_dataset)))
    print("测试数据的 长度： {}".format(len(test_dataset)))
    
    # 3. 使用DataLoader加载数据集
    dataloder_train_set = DataLoader(train_dataset, batch_size=64)
    dataloder_test_set = DataLoader(test_dataset, batch_size=64)
    
    # 4. tensorboard记录数据
    writer = SummaryWriter("./log_GPU1_train")
    
    
    # 5. 引入 实例化的、需要训练的模型
    nerual_network = simple_nerual_network()
    # G1.1 将网络模型转移到cuda上
    if torch.cuda.is_available():
        nerual_network = nerual_network.cuda()
    # 6. 引入损失函数
    loss_fn = nn.CrossEntropyLoss()
    # G1.2 将损失函数转移到cuda上
    if torch.cuda.is_available():
        loss_fn = loss_fn.cuda()
    # 7. 引入优化器 optim
    learning_rate = 1e-2
    optimizer = torch.optim.SGD(nerual_network.parameters(), lr=learning_rate)
    
    # 8. 设置训练网络的一些参数
    # 记录训练的次数
    total_train_step = 0
    # 记录测试的次数
    total_test_step = 0
    # 设置训练的轮数
    epoch = 20
    
    for i in range(epoch):
        print("------------NO {} train is START------------".format(i))
        loss_epoch_train = 0
        # 9 开始训练模型
        nerual_network.train()
        for data_item in dataloder_train_set:
            imgs, targets = data_item
            # G1.3 将input、label 转移到cuda上
            if torch.cuda.is_available():
                imgs = imgs.cuda()
                targets = targets.cuda()
            # 使用模型
            outputs = nerual_network(imgs)
            loss = loss_fn(outputs, targets)
            loss_epoch_train = loss_epoch_train + loss.item()
            # 使用优化器
            optimizer.zero_grad()
            # 使用loss进行反向传播, 获得每个参数节点的梯度grab
            loss.backward()
            # 优化器根据 梯度grab 对模型进行优化
            optimizer.step()
            print("NO {}, total_train_step: {}, loss: {}:  ".format(i, total_train_step, loss.item()))
            # writer记录数据
            writer.add_scalar("train_loss", loss.item(), global_step=total_train_step)
            total_train_step = total_train_step + 1
        print("NO {} 整体训练数据集 loss_epoch_train is :        {}".format(i, loss_epoch_train))
        writer.add_scalar("loss_epoch_train", loss.item(), global_step=total_train_step)
    
    
        # 测试 过程中不需要调优 torch.no_grab()
        loss_epoch_test = 0
        accuracy_epoch_test = 0
        with torch.no_grad():
            # 10 开始测试模型，在训练后的模型中 使用测试数据集 计算loss
            nerual_network.eval()
            for data_item in dataloder_test_set:
                imgs, targets = data_item
                # G1.3 将input、label 转移到cuda上
                if torch.cuda.is_available():
                    imgs = imgs.cuda()
                    targets = targets.cuda()
                # 使用模型
                outputs = nerual_network(imgs)
                loss = loss_fn(outputs, targets)
                loss_epoch_test = loss_epoch_test + loss.item()
    
                # 计算正确率
                accuracy = (outputs.argmax(1)==targets).sum()
                accuracy_epoch_test = accuracy_epoch_test + accuracy
    
        print("NO {} 整体测试数据集 loss_epoch_test is :            {}".format(epoch, loss_epoch_test))
        print("NO {} 整体测试数据集 正确率 is :        {}".format(epoch, accuracy_epoch_test / len(test_dataset)))
        # writer记录数据
        writer.add_scalar("loss_epoch_test", loss_epoch_test, total_test_step)
        writer.add_scalar("accuracy_epoch_test", accuracy_epoch_test / len(test_dataset), total_test_step)
        total_test_step = total_test_step + 1
    
        # 11 保存模型 每一轮的模型
        torch.save(nerual_network, "model22_train_{}.pth".format(i))
        print("NO.{} 模型已经保存".format(i))
    
    writer.close()

方式二
    class simple_nerual_network(nn.Module):
        def __init__(self):
            super(simple_nerual_network, self).__init__()
            # 使用Sequential
            self.sequential1 = Sequential(
                # c=3 32*32
                Conv2d(in_channels=3, out_channels=32, kernel_size=(5, 5), stride=(1, 1), padding=2),
                # c=32 32*32
                MaxPool2d(kernel_size=2),
                # c=32 16*16
                Conv2d(in_channels=32, out_channels=32, kernel_size=(5, 5), stride=(1, 1), padding=2),
                # c=32 16*16
                MaxPool2d(kernel_size=2),
                # c=32 8*8
                Conv2d(in_channels=32, out_channels=64, kernel_size=(5, 5), stride=(1, 1), padding=2),
                # c=64 8*8
                MaxPool2d(kernel_size=2),
                # c=64 4*4
                Flatten(),
                # c=1 1*1024 高度 = 1
                Linear(1024, 64),
                # c=1 1*64 高度 = 1
                Linear(64, 10)
                # c=1 1*10 高度 = 1
            )
    
        def forward(self, input):
            output = self.sequential1(input)
            return output
    
    
    # 利用GPU训练模型 G2
    
    # G2.1 定义训练device cpu\ cuda:0
    device = torch.device("cuda:0" if torch.cuda.is_available() else "cuda")
    # 1. 准备数据集 10个标签 每个图片只有一种确定的标签
    train_dataset = torchvision.datasets.CIFAR10("./dataset3", train=True,
                                                 transform=torchvision.transforms.ToTensor(), download=True)
    test_dataset = torchvision.datasets.CIFAR10("./dataset3", train=False,
                                                transform=torchvision.transforms.ToTensor(), download=True)
    # 2. 输出数据集的长度
    print("训练数据的 长度： {}".format(len(train_dataset)))
    print("测试数据的 长度： {}".format(len(test_dataset)))
    
    # 3. 使用DataLoader加载数据集
    dataloder_train_set = DataLoader(train_dataset, batch_size=64)
    dataloder_test_set = DataLoader(test_dataset, batch_size=64)
    
    # 4. tensorboard记录数据
    writer = SummaryWriter("./log_GPU1_train")
    
    # 5. 引入 实例化的、需要训练的模型
    nerual_network = simple_nerual_network()
    # G2.1 将网络模型转移到cuda上
    nerual_network = nerual_network.to(device)
    # 6. 引入损失函数
    loss_fn = nn.CrossEntropyLoss()
    # G2.2 将损失函数转移到cuda上
    loss_fn = loss_fn.to(device)
    # 7. 引入优化器 optim
    learning_rate = 1e-2
    optimizer = torch.optim.SGD(nerual_network.parameters(), lr=learning_rate)
    
    # 8. 设置训练网络的一些参数
    # 记录训练的次数
    total_train_step = 0
    # 记录测试的次数
    total_test_step = 0
    # 设置训练的轮数
    epoch = 20
    
    for i in range(epoch):
        print("------------NO {} train is START------------".format(i))
        loss_epoch_train = 0
        # 9 开始训练模型
        nerual_network.train()
        for data_item in dataloder_train_set:
            imgs, targets = data_item
            # G2.3 将input、label 转移到cuda上
            imgs = imgs.to(device)
            targets = targets.to(device)
            # 使用模型
            outputs = nerual_network(imgs)
            loss = loss_fn(outputs, targets)
            loss_epoch_train = loss_epoch_train + loss.item()
            # 使用优化器
            optimizer.zero_grad()
            # 使用loss进行反向传播, 获得每个参数节点的梯度grab
            loss.backward()
            # 优化器根据 梯度grab 对模型进行优化
            optimizer.step()
            print("NO {}, total_train_step: {}, loss: {}:  ".format(i, total_train_step, loss.item()))
            # writer记录数据
            writer.add_scalar("train_loss", loss.item(), global_step=total_train_step)
            total_train_step = total_train_step + 1
        print("NO {} 整体训练数据集 loss_epoch_train is :        {}".format(i, loss_epoch_train))
        writer.add_scalar("loss_epoch_train", loss.item(), global_step=total_train_step)
    
        # 测试 过程中不需要调优 torch.no_grab()
        loss_epoch_test = 0
        accuracy_epoch_test = 0
        with torch.no_grad():
            # 10 开始测试模型，在训练后的模型中 使用测试数据集 计算loss
            nerual_network.eval()
            for data_item in dataloder_test_set:
                imgs, targets = data_item
                # G1.3 将input、label 转移到cuda上
                if torch.cuda.is_available():
                    imgs = imgs.cuda()
                    targets = targets.cuda()
                # 使用模型
                outputs = nerual_network(imgs)
                loss = loss_fn(outputs, targets)
                loss_epoch_test = loss_epoch_test + loss.item()
    
                # 计算正确率
                accuracy = (outputs.argmax(1) == targets).sum()
                accuracy_epoch_test = accuracy_epoch_test + accuracy
    
        print("NO {} 整体测试数据集 loss_epoch_test is :            {}".format(epoch, loss_epoch_test))
        print("NO {} 整体测试数据集 正确率 is :        {}".format(epoch, accuracy_epoch_test / len(test_dataset)))
        # writer记录数据
        writer.add_scalar("loss_epoch_test", loss_epoch_test, total_test_step)
        writer.add_scalar("accuracy_epoch_test", accuracy_epoch_test / len(test_dataset), total_test_step)
        total_test_step = total_test_step + 1
    
        # 11 保存模型 每一轮的模型
        torch.save(nerual_network, "model22_train_{}.pth".format(i))
        print("NO.{} 模型已经保存".format(i))
    
    writer.close()

关于没有GPU的解决办法
    https://colab.research.google.com/notebooks/welcome.ipynb

Question & Answer

Python中with的使用

https://blog.csdn.net/Ego_Bai/article/details/80873242

file = open("/tmp/foo.txt")
try:
    data = file.read()
finally:
    file.close()
两者等效
with open("/tmp/foo.txt") as file:
    data = file.read()
 

RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same or input should be a MKLDNN tensor and weight is a dense tensor

原因
    RuntimeError: 输入类型 (torch.FloatTensor) 和权重类型 (torch.cuda.FloatTensor) 应该相同，或者输入应该是 MKLDNN 张量并且权重是密集张量
    保存的模型是使用GPU训练的，输入和标注的数据也应该使用GPU
解决方法
    原理
        让网络模型、数据标注、损失函数在同样的环境中GPU或CPU
    一 让在GPU中的模型对应到CPU
        nerual_network = torch.load("./model22_train_6.pth",map_location=torch.device("cpu"))
    二 把数据转到cuda(GPU)中
        device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
        img = img.to(device)
        targets = targets.to(device)
    

###NotImplementedError

解决方法
    def foward(self, input):
    改为
    def forward(self, input):

使用writer.add_images发生AssertionError

原因
    add_images只能处理channel为3
解决方法
    使用reshape把channel改为3
    output = torch.reshape(output, (-1, 3, 30, 30))

AssertionError: size of input tensor and input format are different. tensor shape: (64, 3, 32, 32), input_format: CHW

解决方法
    writer.add_image("log_DataLoader1", imgs, step)
    改为
    writer.add_images("log_DataLoader1", imgs, step)

urllib.error.URLError:

解决方法
    添加以下代码至程序中
    import ssl
    ssl._create_default_https_context = ssl._create_unverified_context
原因
    可能是目标网站的安全证书失效

Python中下划线在变量名和方法名中的含义

原文链接：https://blog.csdn.net/cnds123/article/details/118707525

python中的下划线主要分为如下几类：

前置单下划线，如_var，
    主要是用于将变量或方法定义为私有属性。它对于程序员而言是一种提示，这里的私有属性并非如java中的私有，而是一种约定俗成，想强行访问仍旧可以正常访问。
后置单下划线，如var_,
    主要用于将变量区别，如变量名称和关键字冲突，后面加个下划线区别开来。
前置双下划线，如__var,
    类中带前置双下划线的变量或方法，都会触发python解释器的名称修饰，是不能直接通过该名称去访问到的，实现了一定程度上的隔离。
前后都有双下划线，如__var__,
    常见的有__init__()对象构造函数，这类方法是python中的魔法方法（特殊方法），用于特殊用途。
单下划线本身，如_ 在Python REPLs如IDLE Shell中是一个特殊变量（可以表示一个临时值），它表示解释器计算的最后一个表达式的结果。

AttributeError: type object ‘Image’ has no attribute ‘open’

原因
    from PIL.Image import Image
解决方法
    from PIL import Image

关于Python Console中设置相对路径出现乱码问题

Python Console
    定义
        Python程序的Script
问题描述
    如设置 img = "dataset\train\ants"
原因
    '\' 导致字符转义
解决方法
    img = "dataset\train\ants"
    改为
    1. img = "dataset/train/ants"
    2. img = "dataset\\train\\ants"

add_image方法出现TypeError: Cannot handle this data type: (1, 1, 512), |u1

原因
Shape:
    img_tensor: Default is :math:`(3, H, W)`. You can use ``torchvision.utils.make_grid()`` to
    convert a batch of tensor into 3xHxW format or call ``add_images`` and let us do the job.
    Tensor with :math:`(1, H, W)`, :math:`(H, W)`, :math:`(H, W, 3)` is also suitable as long as
    corresponding ``dataformats`` argument is passed, e.g. ``CHW``, ``HWC``, ``HW``.
解决方法
    使用type(img)查看dataformats
    使用add_image时设置对应的dataformats
    例如
        writer.add_image("img_array1",img_array,1,dataformats='HWC')