羊城迷鹿

pytorch搭建神经网络玩转mnist

文章目录

引入必要的包
构建分类模型
- MNIST介绍
- 设置网络结构
- 重写_init_和forward方法，完成结构的搭建和前向传播
- 训练过程
- - 设置超参数
  - 设法使weight和bias在初始化时拥有不同的参数分布
  - - 默认型
    - 正态分布
    - 初始化为常数
    - 初始化为xaveir_uniform来保持每一层的梯度大小都差不多相同，在tanh中表现的很好
    - kaiming是针对于Relu的初始化方法，pytorch也是使用kaiming 初始化卷积层参数的
  - 对数据进行预处理
  - - 上面的图片恰好是16*8的大小，8是nrow的默认值，128是batch_size
  - 正式训练
  - 保存训练好的模型参数
  - - 打印模型参数
    - 将参数保存在本地
    - 加载模型参数
  - 观察l2_norm对结果的影响
  - 观察dropout对结果的影响
  - 观察batch_normalization对结果的影响
  - 观察数据增广对结果的影响
  - 使用tensorboard进行可视化
  - 检查梯度消失和爆炸问题
  - - 正常情况
    - 梯度消失（学习率超小）
    - 梯度爆炸(学习率很大)
  - 使用模型进行单图像识别

引入必要的包

# Load all necessary modules here, for clearness
import torch
import numpy as np
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
# from torchvision.datasets import MNIST
import torchvision
from torchvision import transforms
from torch.optim import lr_scheduler
from tensorboardX import SummaryWriter
from collections import OrderedDict
import matplotlib.pyplot as plt
from tqdm import tqdm

cuda = torch.device('cuda:0')

构建分类模型

MNIST介绍

The MNIST database (Modified National Institute of Standards and Technology database) is a large database of handwritten digits that is commonly used for training various image processing systems.

The MNIST database contains 60,000 training images and 10,000 testing images. Each class has 5000 traning images and 1000 test images.

Each image is 32x32.

我们将定义前向传播的神经网络来完成分类任务

设置网络结构

Inputs: For every batch

[batchSize, channels, height, width] -> [B,C,H,W]

Outputs: prediction scores of each images, eg. [0.001, 0.0034 …, 0.3]

[batchSize, classes]

Network Strutrue

    Inputs                Linear/Function        Output
    [128, 1, 28, 28]   -> Linear(28*28, 100) -> [128, 100]  # first hidden lyaer
                       -> ReLU               -> [128, 100]  # relu activation function, may sigmoid
                       -> Linear(100, 100)   -> [128, 100]  # second hidden lyaer
                       -> ReLU               -> [128, 100]  # relu activation function, may sigmoid
                       -> Linear(100, 100)   -> [128, 100]  # third hidden lyaer
                       -> ReLU               -> [128, 100]  # relu activation function, may sigmoid
                       -> Linear(100, 10)    -> [128, 10]   # Classification Layer

如果隐层的层数过多，需要用循环来设置层的话，那么要注意把新增的层设为神经网络的属性。

fc = nn.Linear(input_size, 10)
setattr(self, 'fc%i' % i, fc)

重写_init_和forward方法，完成结构的搭建和前向传播

class FeedForwardNeuralNetwork(nn.Module):
    """
    Inputs                Linear/Function        Output
    [128, 1, 28, 28]   -> Linear(28*28, 100) -> [128, 100]  # first hidden lyaer
                       -> ReLU               -> [128, 100]  # relu activation function, may sigmoid
                       -> Linear(100, 100)   -> [128, 100]  # second hidden lyaer
                       -> ReLU               -> [128, 100]  # relu activation function, may sigmoid
                       -> Linear(100, 100)   -> [128, 100]  # third hidden lyaer
                       -> ReLU               -> [128, 100]  # relu activation function, may sigmoid
                       -> Linear(100 , 10)    -> [128, 10]   # Classification Layer                                                          
   """
    def __init__(self, input_size, hidden_size, output_size, activation_function='RELU'):
        super(FeedForwardNeuralNetwork, self).__init__()
        self.use_dropout = False
        self.use_bn = False
        self.hidden1 = nn.Linear(input_size, hidden_size)  # Linear function 1: 784 --> 100 
        self.hidden2 = nn.Linear(hidden_size, hidden_size) # Linear function 2: 100 --> 100
        self.hidden3 = nn.Linear(hidden_size, hidden_size) # Linear function 3: 100 --> 100
        # Linear function 4 (readout): 100 --> 10
        self.classification_layer = nn.Linear(hidden_size, output_size)
        self.dropout = nn.Dropout(p=0.5) # Drop out with prob = 0.5
        self.hidden1_bn = nn.BatchNorm1d(hidden_size) # Batch Normalization 
        self.hidden2_bn = nn.BatchNorm1d(hidden_size)
        self.hidden3_bn = nn.BatchNorm1d(hidden_size)
        
        # 定义激活函数
        if activation_function == 'SIGMOID':
            self.activation_function1 = nn.Sigmoid()
            self.activation_function2 = nn.Sigmoid()
            self.activation_function3 = nn.Sigmoid()
        elif activation_function == 'RELU':
            self.activation_function1 = nn.ReLU()
            self.activation_function2 = nn.ReLU()
            self.activation_function3 = nn.ReLU()
        
    def forward(self, x):
        """Defines the computation performed at every call.
           Should be overridden by all subclasses.
        Args:
            x: [batch_size, channel, height, width], input for network
        Returns:
            out: [batch_size, n_classes], output from network
        """
        
        x = x.view(x.size(0), -1) # 将x的大小调整为 [batch_size, 784]
        out = self.hidden1(x)
        out = self.activation_function1(out) # Non-linearity 1
        if self.use_bn == True:
            out = self.hidden1_bn(out)
        out = self.hidden2(out)
        out = self.activation_function2(out)
        if self.use_bn == True:
            out = self.hidden2_bn(out)
        out = self.hidden3(out)
        if self.use_bn == True:
            out = self.hidden3_bn(out)
        out = self.activation_function3(out)
        if self.use_dropout == True:
            out = self.dropout(out)
        out = self.classification_layer(out)
        return out
    
    def set_use_dropout(self, use_dropout):
        """Whether to use dropout. Auxiliary function for our exp, not necessary.
        Args:
            use_dropout: True, False
        """
        self.use_dropout = use_dropout
        
    def set_use_bn(self, use_bn):
        """Whether to use batch normalization. Auxiliary function for our exp, not necessary.
        Args:
            use_bn: True, False
        """
        self.use_bn = use_bn
        
    def get_grad(self):
        """Return average grad for hidden2, hidden3. Auxiliary function for our exp, not necessary.
        """
        hidden2_average_grad = np.mean(np.sqrt(np.square(self.hidden2.weight.grad.detach().numpy())))
        hidden3_average_grad = np.mean(np.sqrt(np.square(self.hidden3.weight.grad.detach().numpy())))
        return hidden2_average_grad, hidden3_average_grad

训练过程

设置超参数

### Hyper parameters

batch_size = 128 # batch size is 128
n_epochs = 5 # train for 5 epochs
learning_rate = 0.01 # learning rate is 0.01
input_size = 28*28 # input image has size 28x28
hidden_size = 100 # hidden neurons is 100 for each layer
output_size = 10 # classes of prediction
l2_norm = 0 # not to use l2 penalty
dropout = False # not to use
get_grad = False # not to obtain grad

# create a model object
model = FeedForwardNeuralNetwork(input_size=input_size, hidden_size=hidden_size, output_size=output_size)
# Cross entropy
loss_fn = torch.nn.CrossEntropyLoss()
# weight)decay的含义是权重衰减，或者说是采用l2正则化
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate, weight_decay=l2_norm)

设法使weight和bias在初始化时拥有不同的参数分布

默认型

def show_weight_bias(model):
    # Create a figure and a set of subplots
    fig, axs = plt.subplots(2,3, sharey=False, tight_layout=True)
    
    # weight and bias for every hidden layer
    h1_w = model.hidden1.weight.detach().numpy().flatten()
    h1_b = model.hidden1.bias.detach().numpy().flatten()
    h2_w = model.hidden2.weight.detach().numpy().flatten()
    h2_b = model.hidden2.bias.detach().numpy().flatten()
    h3_w = model.hidden3.weight.detach().numpy().flatten()
    h3_b = model.hidden3.bias.detach().numpy().flatten()
    
    axs[0,0].hist(h1_w)
    axs[0,1].hist(h2_w)
    axs[0,2].hist(h3_w)
    axs[1,0].hist(h1_b)
    axs[1,1].hist(h2_b)
    axs[1,2].hist(h3_b)
    
    # set title for every sub plots
    axs[0,0].set_title('hidden1_weight')
    axs[0,1].set_title('hidden2_weight')
    axs[0,2].set_title('hidden3_weight')
    axs[1,0].set_title('hidden1_bias')
    axs[1,1].set_title('hidden2_bias')
    axs[1,2].set_title('hidden3_bias')

# Show default initialization for every hidden layer by pytorch
# it's uniform distribution 
show_weight_bias(model)

正态分布

# If you want to use other intialization method, you can use code below
# and define your initialization below

def weight_bias_reset(model):
    """Custom initialization, you can use your favorable initialization method.
    """
    for m in model.modules():
        if isinstance(m, nn.Linear):
            # initialize linear layer with mean and std
            mean, std = 0, 0.1 
            
            # Initialization method
            torch.nn.init.normal_(m.weight, mean, std)
            torch.nn.init.normal_(m.bias, mean, std)
            
#             Another way to initialize
#             m.weight.data.normal_(mean, std)
#             m.bias.data.normal_(mean, std)

weight_bias_reset(model) # reset parameters for each hidden layer
show_weight_bias(model) # show weight and bias distribution, normal distribution now.

初始化为常数

# TODO

def weight_bias_reset_constant(model):
    """Constant initalization
    """
    for m in model.modules():
        if isinstance(m, nn.Linear):
            # initialize linear layer with mean and std
            val = 1
            
            # Initialization method
            torch.nn.init.constant_(m.weight, val)
            torch.nn.init.constant_(m.bias, val)

weight_bias_reset_constant(model) # reset parameters for each hidden layer
show_weight_bias(model)

初始化为xaveir_uniform来保持每一层的梯度大小都差不多相同，在tanh中表现的很好

# TODO

def weight_bias_reset_xavier_uniform(model):
    """xaveir_uniform, gain=1
    """
    for m in model.modules():
        if isinstance(m, nn.Linear):
            # initialize linear layer with mean and std
            gain = 1
            
            # Initialization method
            torch.nn.init.xavier_uniform_(m.weight, gain)
#             torch.nn.init.xavier_uniform_(m.bias, gain)
#             print(m.bias.shape)

weight_bias_reset_xavier_uniform(model) # reset parameters for each hidden layer
show_weight_bias(model)

kaiming是针对于Relu的初始化方法，pytorch也是使用kaiming 初始化卷积层参数的

# TODO

def weight_bias_reset_kaiming_uniform(model):
    """xaveir_uniform, gain=1
    """
    for m in model.modules():
        if isinstance(m, nn.Linear):
            # initialize linear layer with mean and std
            a = 0
            
            # Initialization method
            torch.nn.init.kaiming_uniform_(m.weight, a, mode='fan_in')
#             torch.nn.init.kaiming_uniform_(m.bias, a, mode='fan_in')
#             print(m.bias.shape)

weight_bias_reset_kaiming_uniform(model) # reset parameters for each hidden layer
show_weight_bias(model)

对数据进行预处理

可以用transforms进行以下操作：

PIL.Image/numpy.ndarray与Tensor的相互转化；
归一化；
对PIL.Image进行裁剪、缩放等操作。
transforms.Compose就是将transforms组合在一起；
取值范围为[0, 255]的PIL.Image，转换成形状为[C, H, W]，取值范围是[0, 1.0]的torch.FloadTensor；
形状为[H, W, C]的numpy.ndarray，转换成形状为[C, H, W]，取值范围是[0, 1.0]的torch.FloadTenso
transforms.Normalize使用如下公式进行归一化：channel=（channel-mean）/std

train_transform = transforms.Compose([
    transforms.ToTensor(), # transforms.ToTensor() 将 PIL.Image/numpy.ndarray 数据转化为torch.FloadTensor，并归一化到[0, 1.0]，大小为3*H*W
    # Normalize a tensor image with mean 0.1307 and standard deviation 0.3081
    transforms.Normalize((0.1307,), (0.3081,))
])

test_transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.1307,), (0.3081,))
])

torchvision.datasets的使用说明：https://pytorch-cn.readthedocs.io/zh/latest/torchvision/torchvision-datasets/

root指定了数据集存放的路径，transform指定导入数据集时需要进行何种变换操作，train设置为True说明导入的是训练集合，否则为测试集合。

# use MNIST provided by torchvision

# torchvision.datasets provide MNIST dataset for classification

train_dataset = torchvision.datasets.MNIST(root='./data/MNIST/', 
                            train=True, 
                            transform=train_transform,
                            download=False)

test_dataset = torchvision.datasets.MNIST(root='./data/MNIST/', 
                           train=False, 
                           transform=test_transform,
                           download=False)

# pay attention to this, train_dataset doesn't load any data
# It just defined some method and store some message to preprocess data
train_dataset

Dataset MNIST
    Number of datapoints: 60000
    Split: train
    Root Location: ./data/MNIST/
    Transforms (if any): Compose(
                             ToTensor()
                             Normalize(mean=(0.1307,), std=(0.3081,))
                         )
    Target Transforms (if any): None

PyTorch中数据读取的一个重要接口是torch.utils.data.DataLoader，该接口定义在dataloader.py脚本中，只要是用PyTorch来训练模型基本都会用到该接口，该接口主要用来将自定义的数据读取接口的输出或者PyTorch已有的数据读取接口的输入按照batch size封装成Tensor，后续只需要再包装成Variable即可作为模型的输入。

__init__中的几个重要的输入：

dataset，这个就是PyTorch已有的数据读取接口（比如torchvision.datasets.ImageFolder）或者自定义的数据接口的输出，该输出要么是torch.utils.data.Dataset类的对象，要么是继承自torch.utils.data.Dataset类的自定义类的对象。
batch_size，根据具体情况设置即可。
shuffle，一般在训练数据中会采用。
collate_fn，是用来处理不同情况下的输入dataset的封装，一般采用默认即可，除非你自定义的数据读取输出非常少见。
batch_sampler，从注释可以看出，其和batch_size、shuffle等参数是互斥的，一般采用默认。
sampler，从代码可以看出，其和shuffle是互斥的，一般默认即可。
num_workers，从注释可以看出这个参数必须大于等于0，0的话表示数据导入在主进程中进行，其他大于0的数表示通过多个进程来导入数据，可以加快数据导入速度。
pin_memory，注释写得很清楚了： pin_memory (bool, optional): If True, the data loader will copy tensors into CUDA pinned memory before returning them. 也就是一个数据拷贝的问题。
timeout，是用来设置数据读取的超时时间的，但超过这个时间还没读取到数据的话就会报错。

# Data loader. 

# Combines a dataset and a sampler, 
# and provides single- or multi-process iterators over the dataset.

train_loader = torch.utils.data.DataLoader(dataset=train_dataset, 
                                           batch_size=batch_size, 
                                           shuffle=False)

test_loader = torch.utils.data.DataLoader(dataset=test_dataset, 
                                          batch_size=batch_size, 
                                          shuffle=False)

# functions to show an image

def imshow(img):
    """show some imgs in datasets
        !!YOU CAN READ THIS CODE LATER!! """
    
    npimg = img.numpy() # convert tensor to numpy
    plt.imshow(np.transpose(npimg, (1, 2, 0))) # [channel, height, width] -> [height, width, channel]
    plt.show()

make_grid的作用是将若干幅图像拼成一幅图像。其中padding的作用就是子图像与子图像之间的pad有多宽。
使用：torchvision.utils.make_grid(tensor, nrow=8, padding=2, normalize=False, range=None, scale_each=False, pad_value=0)

tensor (Tensor or list) – 4D mini-batch Tensor of shape (B x C x H x W) or a list of images all of the same size.

nrow (int, optional) – Number of images displayed in each row of the grid. The Final grid size is (B / nrow, nrow). Default is 8.

padding (int, optional) – amount of padding. Default is 2.

normalize (bool, optional) – If True, shift the image to the range (0, 1), by subtracting the minimum and dividing by the maximum pixel value.

range (tuple, optional) – tuple (min, max) where min and max are numbers, then these numbers are used to normalize the image. By default, min and max are computed from the tensor.

scale_each (bool, optional) – If True, scale each image in the batch of images separately rather than the (min, max) over all images.

pad_value (float, optional) – Value for the padded pixels.

# get some random training images by batch

dataiter = iter(train_loader)
images, labels = dataiter.next() # get a batch of images

# show images
print(images.shape)
print(labels)
print(torchvision.utils.make_grid(images).shape)
imshow(torchvision.utils.make_grid(images))

torch.Size([128, 1, 28, 28])
tensor([5, 0, 4, 1, 9, 2, 1, 3, 1, 4, 3, 5, 3, 6, 1, 7, 2, 8, 6, 9, 4, 0, 9, 1,
        1, 2, 4, 3, 2, 7, 3, 8, 6, 9, 0, 5, 6, 0, 7, 6, 1, 8, 7, 9, 3, 9, 8, 5,
        9, 3, 3, 0, 7, 4, 9, 8, 0, 9, 4, 1, 4, 4, 6, 0, 4, 5, 6, 1, 0, 0, 1, 7,
        1, 6, 3, 0, 2, 1, 1, 7, 9, 0, 2, 6, 7, 8, 3, 9, 0, 4, 6, 7, 4, 6, 8, 0,
        7, 8, 3, 1, 5, 7, 1, 7, 1, 1, 6, 3, 0, 2, 9, 3, 1, 1, 0, 4, 9, 2, 0, 0,
        2, 0, 2, 7, 1, 8, 6, 4])
torch.Size([3, 482, 242])


Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).

上面的图片恰好是16*8的大小，8是nrow的默认值，128是batch_size

通常通过 module.train() 和 module.eval() 来切换模型的训练测试阶段。train的基本单位是epoch。

def train(train_loader, model, loss_fn, optimizer, get_grad=False):
    """train model using loss_fn and optimizer. When thid function is called, model trains for one epoch.
    Args:
        train_loader: train data
        model: prediction model
        loss_fn: loss function to judge the distance between target and outputs
        optimizer: optimize the loss function
        get_grad: True, False
    Returns:
        total_loss: loss
        average_grad2: average grad for hidden 2 in this epoch
        average_grad3: average grad for hidden 3 in this epoch
    """
    
    # set the module in training model, affecting module e.g., Dropout, BatchNorm, etc.
    model.train()
    
    total_loss = 0
    grad_2 = 0.0 # store sum(grad) for hidden 3 layer
    grad_3 = 0.0 # store sum(grad) for hidden 3 layer
    
    for batch_idx, (data, target) in enumerate(train_loader):
        optimizer.zero_grad() # clear gradients of all optimized torch.Tensors'
        outputs = model(data) # make predictions 
        loss = loss_fn(outputs, target) # compute loss 
        total_loss += loss.item() # accumulate every batch loss in a epoch
        loss.backward() # compute gradient of loss over parameters 
        
        if get_grad == True:
            g2, g3 = model.get_grad() # get grad for hiddern 2 and 3 layer in this batch
            grad_2 += g2 # accumulate grad for hidden 2
            grad_3 += g3 # accumulate grad for hidden 2
            
        optimizer.step() # update parameters with gradient descent 
            
    average_loss = total_loss / batch_idx # average loss in this epoch
    average_grad2 = grad_2 / batch_idx # average grad for hidden 2 in this epoch
    average_grad3 = grad_3 / batch_idx # average grad for hidden 3 in this epoch
    
    return average_loss, average_grad2, average_grad3

model.eval()，让model变成测试模式，此时pytorch会自动把BN和DropOut固定住，不会取平均，而是用训练好的值。不然的话，一旦test的batch_size过小，很容易就会被BN层导致生成图片颜色失真极大。

def evaluate(loader, model, loss_fn):
    """test model's prediction performance on loader.  
    When thid function is called, model is evaluated.
    Args:
        loader: data for evaluation
        model: prediction model
        loss_fn: loss function to judge the distance between target and outputs
    Returns:
        total_loss
        accuracy
    """
    
    # context-manager that disabled gradient computation
    with torch.no_grad():
        
        # set the module in evaluation mode
        model.eval()
        
        correct = 0.0 # account correct amount of data
        total_loss = 0  # account loss
        
        for batch_idx, (data, target) in enumerate(loader):
            outputs = model(data) # make predictions 
            # return the maximum value of each row of the input tensor in the 
            # given dimension dim, the second return vale is the index location
            # of each maxium value found(argmax)
            _, predicted = torch.max(outputs, 1)
            # Detach: Returns a new Tensor, detached from the current graph.
            #The result will never require gradient.
            correct += (predicted == target).sum().detach().numpy()
            loss = loss_fn(outputs, target)  # compute loss 
            total_loss += loss.item() # accumulate every batch loss in a epoch
            
        accuracy = correct*100.0 / len(loader.dataset) # accuracy in a epoch
        
    return total_loss, accuracy

fit函数训练多轮epoch完成整个的训练过程

def fit(train_loader, val_loader, model, loss_fn, optimizer, n_epochs, get_grad=False):
    """train and val model here, we use train_epoch to train model and 
    val_epoch to val model prediction performance
    Args: 
        train_loader: train data
        val_loader: validation data
        model: prediction model
        loss_fn: loss function to judge the distance between target and outputs
        optimizer: optimize the loss function
        n_epochs: training epochs
        get_grad: Whether to get grad of hidden2 layer and hidden3 layer
    Returns:
        train_accs: accuracy of train n_epochs, a list
        train_losses: loss of n_epochs, a list
    """
    
    grad_2 = [] # save grad for hidden 2 every epoch
    grad_3 = [] # save grad for hidden 3 every epoch
    
    train_accs = [] # save train accuracy every epoch
    train_losses = [] # save train loss every epoch
    
    for epoch in range(n_epochs): # train for n_epochs 
        # train model on training datasets, optimize loss function and update model parameters 
        train_loss, average_grad2, average_grad3 = train(train_loader, model, loss_fn, optimizer, get_grad)
        
        # evaluate model performance on train dataset
        _, train_accuracy = evaluate(train_loader, model, loss_fn)
        message = 'Epoch: {}/{}. Train set: Average loss: {:.4f}, Accuracy: {:.4f}'.format(epoch+1, \
                                                                n_epochs, train_loss, train_accuracy)
        print(message)
    
        # save loss, accuracy, grad
        train_accs.append(train_accuracy)
        train_losses.append(train_loss)
        grad_2.append(average_grad2)
        grad_3.append(average_grad3)
    
        # evaluate model performance on val dataset
        val_loss, val_accuracy = evaluate(val_loader, model, loss_fn)
        message = 'Epoch: {}/{}. Validation set: Average loss: {:.4f}, Accuracy: {:.4f}'.format(epoch+1, \
                                                                n_epochs, val_loss, val_accuracy)
        print(message)
        
        
    # Whether to get grad for showing
    if get_grad == True:
        fig, ax = plt.subplots() # add a set of subplots to this figure
        ax.plot(grad_2, label='Gradient for Hidden 2 Layer') # plot grad 2 
        ax.plot(grad_3, label='Gradient for Hidden 3 Layer') # plot grad 3 
        plt.ylim(top=0.004)
        # place a legend on axes
        legend = ax.legend(loc='best', shadow=True, fontsize='x-large')
    
    return train_accs, train_losses

def show_curve(ys, title):
    """plot curlve for Loss and Accuacy
    
    !!YOU CAN READ THIS LATER, if you are interested
    
    Args:
        ys: loss or acc list
        title: Loss or Accuracy
    """
    x = np.array(range(len(ys)))
    y = np.array(ys)
    plt.plot(x, y, c='b')
    plt.axis()
    plt.title('{} Curve:'.format(title))
    plt.xlabel('Epoch')
    plt.ylabel('{} Value'.format(title))
    plt.show()

正式训练

### Hyper parameters

batch_size = 128 # batch size is 128
n_epochs = 5 # train for 5 epochs
learning_rate = 0.01 # learning rate is 0.01
input_size = 28*28 # input image has size 28x28
hidden_size = 100 # hidden neurons is 100 for each layer
output_size = 10 # classes of prediction
l2_norm = 0 # not to use l2 penalty
dropout = False # not to use
get_grad = False # not to obtain grad

# declare a model
model = FeedForwardNeuralNetwork(input_size=input_size, hidden_size=hidden_size, output_size=output_size)
# Cross entropy
loss_fn = torch.nn.CrossEntropyLoss()
# l2_norm can be done in SGD
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate, weight_decay=l2_norm)

train_accs, train_losses = fit(train_loader, test_loader, model, loss_fn, optimizer, n_epochs, get_grad)

Epoch: 1/5. Train set: Average loss: 1.8444, Accuracy: 76.8783
Epoch: 1/5. Validation set: Average loss: 69.3857, Accuracy: 77.4200
Epoch: 2/5. Train set: Average loss: 0.5882, Accuracy: 87.0650
Epoch: 2/5. Validation set: Average loss: 33.3306, Accuracy: 87.5500
Epoch: 3/5. Train set: Average loss: 0.3835, Accuracy: 89.4333
Epoch: 3/5. Validation set: Average loss: 26.6955, Accuracy: 89.6800
Epoch: 4/5. Train set: Average loss: 0.3243, Accuracy: 90.8367
Epoch: 4/5. Validation set: Average loss: 23.5091, Accuracy: 90.9600
Epoch: 5/5. Train set: Average loss: 0.2885, Accuracy: 91.8267
Epoch: 5/5. Validation set: Average loss: 21.1413, Accuracy: 91.9300

show_curve(train_accs, "Accu")
show_curve(train_losses, "Loss")

并未出现过拟合现象，因为验证集的准确率一直在上升，损失函数值一直在下降，与训练集的变化趋势保持一致。

Ans: 并未出现过拟合现象，因为验证集的准确率一直在上升，损失函数值一直在下降，与训练集的变化趋势保持一致。另外一方面，将学习率调整到原来的十倍之后，由于学习速度大大加快，产生了过拟合的现象，第六轮Epoch之后模型在验证集上的准确率有所下降，小于epoch5的水平。

保存训练好的模型参数

打印模型参数

# show parameters in model

# Print model's state_dict
print("Model's state_dict:")
for param_tensor in model.state_dict():
    print(param_tensor, "\t", model.state_dict()[param_tensor].size())

# Print optimizer's state_dict
print("\nOptimizer's state_dict:")
for var_name in optimizer.state_dict():
    print(var_name, "\t", optimizer.state_dict()[var_name])

Model's state_dict:
hidden1.weight 	 torch.Size([100, 784])
hidden1.bias 	 torch.Size([100])
hidden2.weight 	 torch.Size([100, 100])
hidden2.bias 	 torch.Size([100])
hidden3.weight 	 torch.Size([100, 100])
hidden3.bias 	 torch.Size([100])
classification_layer.weight 	 torch.Size([10, 100])
classification_layer.bias 	 torch.Size([10])
hidden1_bn.weight 	 torch.Size([100])
hidden1_bn.bias 	 torch.Size([100])
hidden1_bn.running_mean 	 torch.Size([100])
hidden1_bn.running_var 	 torch.Size([100])
hidden1_bn.num_batches_tracked 	 torch.Size([])
hidden2_bn.weight 	 torch.Size([100])
hidden2_bn.bias 	 torch.Size([100])
hidden2_bn.running_mean 	 torch.Size([100])
hidden2_bn.running_var 	 torch.Size([100])
hidden2_bn.num_batches_tracked 	 torch.Size([])
hidden3_bn.weight 	 torch.Size([100])
hidden3_bn.bias 	 torch.Size([100])
hidden3_bn.running_mean 	 torch.Size([100])
hidden3_bn.running_var 	 torch.Size([100])
hidden3_bn.num_batches_tracked 	 torch.Size([])

Optimizer's state_dict:
state 	 {}
param_groups 	 [{'lr': 0.01, 'momentum': 0, 'dampening': 0, 'weight_decay': 0, 'nesterov': False, 'params': [2361592445112, 2361592446120, 2361592445184, 2361595752312, 2361596540464, 2361596526808, 2361596528032, 2361596528320, 2361590420680, 2361590421328, 2361590420320, 2361590421544, 2361590420104, 2361590421040]}]

将参数保存在本地

# save model

save_path = './model.pt'
torch.save(model.state_dict(), save_path)

加载模型参数

# load parameters from files
saved_parametes = torch.load(save_path)
print(saved_parametes)

# initailze model by saved parameters
new_model = FeedForwardNeuralNetwork(input_size, hidden_size, output_size)
new_model.load_state_dict(saved_parametes)

观察l2_norm对结果的影响

we could minimize the regularization term below by use $weight\_decay$ in SGD optimizer
\begin{equation}
L_norm = {\sum_{i=1}^{{m}{\theta_{i}}{2}}}
\end{equation}

使用l2_norm=0.01训练模型

### Hyper parameters
batch_size = 128
n_epochs = 5
learning_rate = 0.01
input_size = 28*28
hidden_size = 100
output_size = 10
l2_norm = 0.01 # use l2 penalty
get_grad = False

# declare a model
model = FeedForwardNeuralNetwork(input_size=input_size, hidden_size=hidden_size, output_size=output_size)
# Cross entropy
loss_fn = torch.nn.CrossEntropyLoss()
# l2_norm can be done in SGD
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate, weight_decay=l2_norm)

train_accs, train_losses = fit(train_loader, test_loader, model, loss_fn, optimizer, n_epochs, get_grad)

Epoch: 1/5. Train set: Average loss: 1.9078, Accuracy: 72.7167
Epoch: 1/5. Validation set: Average loss: 78.3702, Accuracy: 72.9900
Epoch: 2/5. Train set: Average loss: 0.6579, Accuracy: 86.0950
Epoch: 2/5. Validation set: Average loss: 36.8969, Accuracy: 86.3800
Epoch: 3/5. Train set: Average loss: 0.4124, Accuracy: 89.0433
Epoch: 3/5. Validation set: Average loss: 28.6882, Accuracy: 89.3300
Epoch: 4/5. Train set: Average loss: 0.3507, Accuracy: 90.2167
Epoch: 4/5. Validation set: Average loss: 25.6519, Accuracy: 90.4100
Epoch: 5/5. Train set: Average loss: 0.3207, Accuracy: 90.9800
Epoch: 5/5. Validation set: Average loss: 23.7643, Accuracy: 91.3100

使用 l2_norm = 1, 训练模型

### Hyper parameters
batch_size = 128
n_epochs = 5
learning_rate = 0.01
input_size = 28*28
hidden_size = 100
output_size = 10
l2_norm = 1 # use l2 penalty
get_grad = False

# declare a model
model = FeedForwardNeuralNetwork(input_size=input_size, hidden_size=hidden_size, output_size=output_size)
# Cross entropy
loss_fn = torch.nn.CrossEntropyLoss()
# l2_norm can be done in SGD
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate, weight_decay=l2_norm)

train_accs, train_losses = fit(train_loader, test_loader, model, loss_fn, optimizer, n_epochs, get_grad)

Epoch: 1/5. Train set: Average loss: 2.3069, Accuracy: 11.2367
Epoch: 1/5. Validation set: Average loss: 181.8871, Accuracy: 11.3500
Epoch: 2/5. Train set: Average loss: 2.3073, Accuracy: 11.2367
Epoch: 2/5. Validation set: Average loss: 181.8870, Accuracy: 11.3500
Epoch: 3/5. Train set: Average loss: 2.3073, Accuracy: 11.2367
Epoch: 3/5. Validation set: Average loss: 181.8871, Accuracy: 11.3500
Epoch: 4/5. Train set: Average loss: 2.3073, Accuracy: 11.2367
Epoch: 4/5. Validation set: Average loss: 181.8871, Accuracy: 11.3500
Epoch: 5/5. Train set: Average loss: 2.3073, Accuracy: 11.2367
Epoch: 5/5. Validation set: Average loss: 181.8871, Accuracy: 11.3500

从结果中可以看出，将l2_norm调整为1之后准确率大幅下降，说明正则化的权重过大，导致模型对于减少特征的数量发展出了强烈的偏好，使得从特征中可利用的特征和信息数减少，模型的预测能力下降。

观察dropout对结果的影响

During training, randomly zeroes some of the elements of the input tensor with probability p using samples from a Bernoulli distribution.

Each channel will be zeroed out independently on every forward call.

### Hyper parameters
batch_size = 128
n_epochs = 5
learning_rate = 0.01
input_size = 28*28
hidden_size = 100
output_size = 10
l2_norm = 0 # without using l2 penalty
get_grad = False

# declare a model
model = FeedForwardNeuralNetwork(input_size=input_size, hidden_size=hidden_size, output_size=output_size)
# Cross entropy
loss_fn = torch.nn.CrossEntropyLoss()
# l2_norm can be done in SGD
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate, weight_decay=l2_norm)

# Set dropout to True and probability = 0.5
model.set_use_dropout(True)

train_accs, train_losses = fit(train_loader, test_loader, model, loss_fn, optimizer, n_epochs, get_grad)

Epoch: 1/5. Train set: Average loss: 2.0297, Accuracy: 74.1700
Epoch: 1/5. Validation set: Average loss: 89.8412, Accuracy: 74.3100
Epoch: 2/5. Train set: Average loss: 0.8378, Accuracy: 86.6600
Epoch: 2/5. Validation set: Average loss: 36.5149, Accuracy: 87.0900
Epoch: 3/5. Train set: Average loss: 0.5397, Accuracy: 88.9917
Epoch: 3/5. Validation set: Average loss: 28.3120, Accuracy: 89.1800
Epoch: 4/5. Train set: Average loss: 0.4486, Accuracy: 90.5850
Epoch: 4/5. Validation set: Average loss: 24.2983, Accuracy: 90.5500
Epoch: 5/5. Train set: Average loss: 0.3940, Accuracy: 91.5483
Epoch: 5/5. Validation set: Average loss: 21.7570, Accuracy: 91.4200

观察batch_normalization对结果的影响

Batch normalization is a technique for improving the performance and stability of artificial neural networks

\begin{equation}
y=\frac{x-E[x]}{\sqrt{Var[x]+\epsilon}} * \gamma + \beta,
\end{equation}

$\gamma$ and $\beta$ are learnable parameters

### Hyper parameters
batch_size = 128
n_epochs = 5
learning_rate = 0.01
input_size = 28*28
hidden_size = 100
output_size = 10
l2_norm = 0 # without using l2 penalty
get_grad = False

# declare a model
model = FeedForwardNeuralNetwork(input_size=input_size, hidden_size=hidden_size, output_size=output_size)
# Cross entropy
loss_fn = torch.nn.CrossEntropyLoss()
# l2_norm can be done in SGD
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate, weight_decay=l2_norm)

model.set_use_bn(True)

model.use_bn

True

train_accs, train_losses = fit(train_loader, test_loader, model, loss_fn, optimizer, n_epochs, get_grad)

Epoch: 1/5. Train set: Average loss: 1.0543, Accuracy: 91.7083
Epoch: 1/5. Validation set: Average loss: 35.6068, Accuracy: 92.0500
Epoch: 2/5. Train set: Average loss: 0.3326, Accuracy: 94.5767
Epoch: 2/5. Validation set: Average loss: 19.0148, Accuracy: 94.4000
Epoch: 3/5. Train set: Average loss: 0.2085, Accuracy: 95.9867
Epoch: 3/5. Validation set: Average loss: 13.7753, Accuracy: 95.5100
Epoch: 4/5. Train set: Average loss: 0.1552, Accuracy: 96.8933
Epoch: 4/5. Validation set: Average loss: 11.2369, Accuracy: 96.2400
Epoch: 5/5. Train set: Average loss: 0.1243, Accuracy: 97.4067
Epoch: 5/5. Validation set: Average loss: 9.7330, Accuracy: 96.5500

观察数据增广对结果的影响

torchvision.transforms.RandomHorizontalFlip:随机水平翻转给定的PIL.Image,概率为0.5。即：一半的概率翻转，一半的概率不翻转。
class torchvision.transforms.RandomCrop(size, padding=0)，随机裁剪，size可以是tuple也可以是Integer。

详见 https://pytorch-cn.readthedocs.io/zh/latest/torchvision/torchvision-transform/

# only add random horizontal flip
train_transform_1 = transforms.Compose([
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(), # Convert a PIL Image or numpy.ndarray to tensor.
    # Normalize a tensor image with mean and standard deviation
    transforms.Normalize((0.1307,), (0.3081,))
])

# only add random crop
train_transform_2 = transforms.Compose([
    transforms.RandomCrop(size=[32,32], padding=4),
    transforms.ToTensor(), # Convert a PIL Image or numpy.ndarray to tensor.
    # Normalize a tensor image with mean and standard deviation
    transforms.Normalize((0.1307,), (0.3081,))
])

# add random horizontal flip and random crop
train_transform_3 = transforms.Compose([
    transforms.RandomHorizontalFlip(),
    transforms.RandomCrop(size=[32,32], padding=4),
    transforms.ToTensor(), # Convert a PIL Image or numpy.ndarray to tensor.
    # Normalize a tensor image with mean and standard deviation
    transforms.Normalize((0.1307,), (0.3081,))
])

# reload train_loader using trans

train_dataset_1 = torchvision.datasets.MNIST(root='./data', 
                            train=True, 
                            transform=train_transform_1,
                            download=False)

train_loader_1 = torch.utils.data.DataLoader(dataset=train_dataset_1, 
                                           batch_size=batch_size, 
                                           shuffle=True)

print(train_dataset_1)

Dataset MNIST
    Number of datapoints: 60000
    Split: train
    Root Location: ./data
    Transforms (if any): Compose(
                             RandomHorizontalFlip(p=0.5)
                             ToTensor()
                             Normalize(mean=(0.1307,), std=(0.3081,))
                         )
    Target Transforms (if any): None

### Hyper parameters
batch_size = 128
n_epochs = 5
learning_rate = 0.01
input_size = 28*28
hidden_size = 100
output_size = 10
l2_norm = 0 # without using l2 penalty
get_grad = False

# declare a model
model = FeedForwardNeuralNetwork(input_size=input_size, hidden_size=hidden_size, output_size=output_size)
# Cross entropy
loss_fn = torch.nn.CrossEntropyLoss()
# l2_norm can be done in SGD
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate, weight_decay=l2_norm)

train_accs, train_losses = fit(train_loader_1, test_loader, model, loss_fn, optimizer, n_epochs, get_grad)

Epoch: 1/5. Train set: Average loss: 2.0279, Accuracy: 68.6250
Epoch: 1/5. Validation set: Average loss: 92.1617, Accuracy: 69.8500
Epoch: 2/5. Train set: Average loss: 0.8009, Accuracy: 79.2333
Epoch: 2/5. Validation set: Average loss: 49.5853, Accuracy: 79.8800
Epoch: 3/5. Train set: Average loss: 0.6014, Accuracy: 82.1367
Epoch: 3/5. Validation set: Average loss: 42.1372, Accuracy: 82.8200
Epoch: 4/5. Train set: Average loss: 0.5312, Accuracy: 83.8583
Epoch: 4/5. Validation set: Average loss: 37.6811, Accuracy: 84.7300
Epoch: 5/5. Train set: Average loss: 0.4776, Accuracy: 85.8750
Epoch: 5/5. Validation set: Average loss: 33.9977, Accuracy: 86.6100

# reload train_loader using trans
test_transform = transforms.Compose([
    transforms.RandomCrop(size=[32,32], padding=4),
    transforms.ToTensor(),
    transforms.Normalize((0.1307,), (0.3081,))
])
test_dataset = torchvision.datasets.MNIST(root='./data/MNIST/', 
                           train=False, 
                           transform=test_transform,
                           download=False)
test_loader = torch.utils.data.DataLoader(dataset=test_dataset, 
                                          batch_size=batch_size, 
                                          shuffle=False)
train_dataset_2 = torchvision.datasets.MNIST(root='./data', 
                            train=True, 
                            transform=train_transform_2,
                            download=False)

train_loader_2 = torch.utils.data.DataLoader(dataset=train_dataset_2, 
                                           batch_size=batch_size, 
                                           shuffle=True)
### Hyper parameters
batch_size = 128
n_epochs = 5
learning_rate = 0.01
input_size = 32*32
hidden_size = 100
output_size = 10
l2_norm = 0 # without using l2 penalty
get_grad = False

# declare a model
model = FeedForwardNeuralNetwork(input_size=input_size, hidden_size=hidden_size, output_size=output_size)
# Cross entropy
loss_fn = torch.nn.CrossEntropyLoss()
# l2_norm can be done in SGD
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate, weight_decay=l2_norm) 
train_accs, train_losses = fit(train_loader_2, test_loader, model, loss_fn, optimizer, n_epochs, get_grad)

Epoch: 1/5. Train set: Average loss: 2.1805, Accuracy: 46.8083
Epoch: 1/5. Validation set: Average loss: 143.3776, Accuracy: 47.1100
Epoch: 2/5. Train set: Average loss: 1.2855, Accuracy: 68.1933
Epoch: 2/5. Validation set: Average loss: 77.3718, Accuracy: 68.7600
Epoch: 3/5. Train set: Average loss: 0.8984, Accuracy: 75.2133
Epoch: 3/5. Validation set: Average loss: 60.4608, Accuracy: 76.2300
Epoch: 4/5. Train set: Average loss: 0.7193, Accuracy: 81.0950
Epoch: 4/5. Validation set: Average loss: 49.5156, Accuracy: 81.4100
Epoch: 5/5. Train set: Average loss: 0.5722, Accuracy: 85.3317
Epoch: 5/5. Validation set: Average loss: 38.4313, Accuracy: 85.7600

# reload train_loader using trans

train_dataset_3 = torchvision.datasets.MNIST(root='./data', 
                            train=True, 
                            transform=train_transform_3,
                            download=False)

train_loader_3 = torch.utils.data.DataLoader(dataset=train_dataset_3, 
                                           batch_size=batch_size, 
                                           shuffle=True)
### Hyper parameters
batch_size = 128
n_epochs = 5
learning_rate = 0.01
input_size = 32*32
hidden_size = 100
output_size = 10
l2_norm = 0 # without using l2 penalty
get_grad = False

# declare a model
model = FeedForwardNeuralNetwork(input_size=input_size, hidden_size=hidden_size, output_size=output_size)
# Cross entropy
loss_fn = torch.nn.CrossEntropyLoss()
# l2_norm can be done in SGD
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate, weight_decay=l2_norm) 
train_accs, train_losses = fit(train_loader_3, test_loader, model, loss_fn, optimizer, n_epochs, get_grad)

Epoch: 1/5. Train set: Average loss: 2.2192, Accuracy: 36.5367
Epoch: 1/5. Validation set: Average loss: 153.8687, Accuracy: 36.2300
Epoch: 2/5. Train set: Average loss: 1.4885, Accuracy: 59.1383
Epoch: 2/5. Validation set: Average loss: 93.1653, Accuracy: 60.1300
Epoch: 3/5. Train set: Average loss: 1.1089, Accuracy: 65.2250
Epoch: 3/5. Validation set: Average loss: 79.0645, Accuracy: 66.8100
Epoch: 4/5. Train set: Average loss: 0.9566, Accuracy: 70.8733
Epoch: 4/5. Validation set: Average loss: 68.2114, Accuracy: 71.9500
Epoch: 5/5. Train set: Average loss: 0.8243, Accuracy: 74.8700
Epoch: 5/5. Validation set: Average loss: 58.6696, Accuracy: 76.3400

使用tensorboard进行可视化

示例教程1 教程2

with SummaryWriter(comment="simplenet") as s:
    s.add_graph(model, (images,))
    s.close()

接下来进入run文件夹同级的terminal，输入：tensorboard --logdir runs，双击之后可以查看细节的。

检查梯度消失和爆炸问题

正常情况

### Hyper parameters
batch_size = 128
n_epochs = 15
learning_rate = 0.01
input_size = 28*28
hidden_size = 100
output_size = 10
l2_norm = 0 # use l2 penalty
get_grad = True

# declare a model
model = FeedForwardNeuralNetwork(input_size=input_size, hidden_size=hidden_size, output_size=output_size)
# Cross entropy
loss_fn = torch.nn.CrossEntropyLoss()
# l2_norm can be done in SGD
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate, weight_decay=l2_norm)

fit(train_loader, test_loader, model, loss_fn, optimizer, n_epochs, get_grad)

Epoch: 1/15. Train set: Average loss: 1.8366, Accuracy: 78.8000
Epoch: 1/15. Validation set: Average loss: 63.6212, Accuracy: 79.8800
Epoch: 2/15. Train set: Average loss: 0.5435, Accuracy: 87.6750
Epoch: 2/15. Validation set: Average loss: 31.8640, Accuracy: 88.1300
Epoch: 3/15. Train set: Average loss: 0.3700, Accuracy: 89.9000
Epoch: 3/15. Validation set: Average loss: 26.0366, Accuracy: 90.0600
Epoch: 4/15. Train set: Average loss: 0.3155, Accuracy: 91.2067
Epoch: 4/15. Validation set: Average loss: 23.0006, Accuracy: 91.3300
Epoch: 5/15. Train set: Average loss: 0.2810, Accuracy: 92.1017
Epoch: 5/15. Validation set: Average loss: 20.7663, Accuracy: 92.1800
Epoch: 6/15. Train set: Average loss: 0.2538, Accuracy: 92.8850
Epoch: 6/15. Validation set: Average loss: 18.9108, Accuracy: 92.9700
Epoch: 7/15. Train set: Average loss: 0.2308, Accuracy: 93.5200
Epoch: 7/15. Validation set: Average loss: 17.3296, Accuracy: 93.5200
Epoch: 8/15. Train set: Average loss: 0.2110, Accuracy: 94.0817
Epoch: 8/15. Validation set: Average loss: 16.0059, Accuracy: 93.9200
Epoch: 9/15. Train set: Average loss: 0.1941, Accuracy: 94.5667
Epoch: 9/15. Validation set: Average loss: 14.8797, Accuracy: 94.2600
Epoch: 10/15. Train set: Average loss: 0.1795, Accuracy: 94.9167
Epoch: 10/15. Validation set: Average loss: 13.9058, Accuracy: 94.7200
Epoch: 11/15. Train set: Average loss: 0.1670, Accuracy: 95.3200
Epoch: 11/15. Validation set: Average loss: 13.0572, Accuracy: 95.1200
Epoch: 12/15. Train set: Average loss: 0.1561, Accuracy: 95.6383
Epoch: 12/15. Validation set: Average loss: 12.3507, Accuracy: 95.4000
Epoch: 13/15. Train set: Average loss: 0.1465, Accuracy: 95.9200
Epoch: 13/15. Validation set: Average loss: 11.7505, Accuracy: 95.5900
Epoch: 14/15. Train set: Average loss: 0.1381, Accuracy: 96.1933
Epoch: 14/15. Validation set: Average loss: 11.2232, Accuracy: 95.7700
Epoch: 15/15. Train set: Average loss: 0.1305, Accuracy: 96.3700
Epoch: 15/15. Validation set: Average loss: 10.7843, Accuracy: 95.9200





([78.8,
  87.675,
  89.9,
  91.20666666666666,
  92.10166666666667,
  92.885,
  93.52,
  94.08166666666666,
  94.56666666666666,
  94.91666666666667,
  95.32,
  95.63833333333334,
  95.92,
  96.19333333333333,
  96.37],
 [1.8365707892892706,
  0.5434680474269339,
  0.36995476882299805,
  0.31546069086234796,
  0.2810159547604684,
  0.25381433282397753,
  0.23079734468339091,
  0.2109861063540109,
  0.19405433198866937,
  0.17953249415717062,
  0.1669697950506567,
  0.15606886974703044,
  0.14652140330300373,
  0.13807939636544922,
  0.1305215704676687])

梯度消失（学习率超小）

### Hyper parameters
batch_size = 128
n_epochs = 15
learning_rate = 1e-20
input_size = 28*28
hidden_size = 100
output_size = 10
l2_norm = 0 # use l2 penalty
get_grad = True

# declare a model
model = FeedForwardNeuralNetwork(input_size=input_size, hidden_size=hidden_size, output_size=output_size)
# Cross entropy
loss_fn = torch.nn.CrossEntropyLoss()
# l2_norm can be done in SGD
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate, weight_decay=l2_norm)

fit(train_loader, test_loader, model, loss_fn, optimizer, n_epochs, get_grad=get_grad)

Epoch: 1/15. Train set: Average loss: 2.3062, Accuracy: 9.9650
Epoch: 1/15. Validation set: Average loss: 181.8200, Accuracy: 9.6200
Epoch: 2/15. Train set: Average loss: 2.3062, Accuracy: 9.9650
Epoch: 2/15. Validation set: Average loss: 181.8200, Accuracy: 9.6200
Epoch: 3/15. Train set: Average loss: 2.3062, Accuracy: 9.9650
Epoch: 3/15. Validation set: Average loss: 181.8200, Accuracy: 9.6200
Epoch: 4/15. Train set: Average loss: 2.3062, Accuracy: 9.9650
Epoch: 4/15. Validation set: Average loss: 181.8200, Accuracy: 9.6200
Epoch: 5/15. Train set: Average loss: 2.3062, Accuracy: 9.9650
Epoch: 5/15. Validation set: Average loss: 181.8200, Accuracy: 9.6200
Epoch: 6/15. Train set: Average loss: 2.3062, Accuracy: 9.9650
Epoch: 6/15. Validation set: Average loss: 181.8200, Accuracy: 9.6200
Epoch: 7/15. Train set: Average loss: 2.3062, Accuracy: 9.9650
Epoch: 7/15. Validation set: Average loss: 181.8200, Accuracy: 9.6200
Epoch: 8/15. Train set: Average loss: 2.3062, Accuracy: 9.9650
Epoch: 8/15. Validation set: Average loss: 181.8200, Accuracy: 9.6200
Epoch: 9/15. Train set: Average loss: 2.3062, Accuracy: 9.9650
Epoch: 9/15. Validation set: Average loss: 181.8200, Accuracy: 9.6200
Epoch: 10/15. Train set: Average loss: 2.3062, Accuracy: 9.9650
Epoch: 10/15. Validation set: Average loss: 181.8200, Accuracy: 9.6200
Epoch: 11/15. Train set: Average loss: 2.3062, Accuracy: 9.9650
Epoch: 11/15. Validation set: Average loss: 181.8200, Accuracy: 9.6200
Epoch: 12/15. Train set: Average loss: 2.3062, Accuracy: 9.9650
Epoch: 12/15. Validation set: Average loss: 181.8200, Accuracy: 9.6200
Epoch: 13/15. Train set: Average loss: 2.3062, Accuracy: 9.9650
Epoch: 13/15. Validation set: Average loss: 181.8200, Accuracy: 9.6200
Epoch: 14/15. Train set: Average loss: 2.3062, Accuracy: 9.9650
Epoch: 14/15. Validation set: Average loss: 181.8200, Accuracy: 9.6200
Epoch: 15/15. Train set: Average loss: 2.3062, Accuracy: 9.9650
Epoch: 15/15. Validation set: Average loss: 181.8200, Accuracy: 9.6200





([9.965,
  9.965,
  9.965,
  9.965,
  9.965,
  9.965,
  9.965,
  9.965,
  9.965,
  9.965,
  9.965,
  9.965,
  9.965,
  9.965,
  9.965],
 [2.3061594305894313,
  2.3061594305894313,
  2.3061594305894313,
  2.3061594305894313,
  2.3061594305894313,
  2.3061594305894313,
  2.3061594305894313,
  2.3061594305894313,
  2.3061594305894313,
  2.3061594305894313,
  2.3061594305894313,
  2.3061594305894313,
  2.3061594305894313,
  2.3061594305894313,
  2.3061594305894313])

梯度爆炸(学习率很大)

### Hyper parameters
batch_size = 128
n_epochs = 15
learning_rate = 10.168
input_size = 28*28
hidden_size = 100
output_size = 10
l2_norm = 0 # not to use l2 penalty
get_grad = True

# declare a model
model = FeedForwardNeuralNetwork(input_size=input_size, hidden_size=hidden_size, output_size=output_size)
# Cross entropy
loss_fn = torch.nn.CrossEntropyLoss()
# l2_norm can be done in SGD
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate, weight_decay=l2_norm)

fit(train_loader, test_loader, model, loss_fn, optimizer, n_epochs, get_grad=True)

E:\train\anaconda\envs\torch\lib\site-packages\ipykernel_launcher.py:80: RuntimeWarning: overflow encountered in square
E:\train\anaconda\envs\torch\lib\site-packages\ipykernel_launcher.py:81: RuntimeWarning: overflow encountered in square


Epoch: 1/15. Train set: Average loss: nan, Accuracy: 9.8717
Epoch: 1/15. Validation set: Average loss: nan, Accuracy: 9.8000
Epoch: 2/15. Train set: Average loss: nan, Accuracy: 9.8717
Epoch: 2/15. Validation set: Average loss: nan, Accuracy: 9.8000
Epoch: 3/15. Train set: Average loss: nan, Accuracy: 9.8717
Epoch: 3/15. Validation set: Average loss: nan, Accuracy: 9.8000
Epoch: 4/15. Train set: Average loss: nan, Accuracy: 9.8717
Epoch: 4/15. Validation set: Average loss: nan, Accuracy: 9.8000
Epoch: 5/15. Train set: Average loss: nan, Accuracy: 9.8717
Epoch: 5/15. Validation set: Average loss: nan, Accuracy: 9.8000
Epoch: 6/15. Train set: Average loss: nan, Accuracy: 9.8717
Epoch: 6/15. Validation set: Average loss: nan, Accuracy: 9.8000
Epoch: 7/15. Train set: Average loss: nan, Accuracy: 9.8717
Epoch: 7/15. Validation set: Average loss: nan, Accuracy: 9.8000
Epoch: 8/15. Train set: Average loss: nan, Accuracy: 9.8717
Epoch: 8/15. Validation set: Average loss: nan, Accuracy: 9.8000
Epoch: 9/15. Train set: Average loss: nan, Accuracy: 9.8717
Epoch: 9/15. Validation set: Average loss: nan, Accuracy: 9.8000
Epoch: 10/15. Train set: Average loss: nan, Accuracy: 9.8717
Epoch: 10/15. Validation set: Average loss: nan, Accuracy: 9.8000
Epoch: 11/15. Train set: Average loss: nan, Accuracy: 9.8717
Epoch: 11/15. Validation set: Average loss: nan, Accuracy: 9.8000
Epoch: 12/15. Train set: Average loss: nan, Accuracy: 9.8717
Epoch: 12/15. Validation set: Average loss: nan, Accuracy: 9.8000
Epoch: 13/15. Train set: Average loss: nan, Accuracy: 9.8717
Epoch: 13/15. Validation set: Average loss: nan, Accuracy: 9.8000
Epoch: 14/15. Train set: Average loss: nan, Accuracy: 9.8717
Epoch: 14/15. Validation set: Average loss: nan, Accuracy: 9.8000
Epoch: 15/15. Train set: Average loss: nan, Accuracy: 9.8717
Epoch: 15/15. Validation set: Average loss: nan, Accuracy: 9.8000





([9.871666666666666,
  9.871666666666666,
  9.871666666666666,
  9.871666666666666,
  9.871666666666666,
  9.871666666666666,
  9.871666666666666,
  9.871666666666666,
  9.871666666666666,
  9.871666666666666,
  9.871666666666666,
  9.871666666666666,
  9.871666666666666,
  9.871666666666666,
  9.871666666666666],
 [nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan])

梯度消失：反向传播时，当梯度从后往前传时，梯度不断减小，最后变为零，此时，浅层的神经网络权重得不到更新，这种现象称为“饱和”（saturate）,整个网络学习速度非常慢。梯度消失导致后层的权重更新的快，而前层网络由于梯度传递不过去而得不到更新。这样在网络很深的时候，学习的速度很慢或者无法学习。

梯度爆炸：这是一种与梯度消失相反的情况，当进行反向传播时，梯度从后往前传时，梯度不断增大，导致权重更新太大，以致不断波动，使网络在最优点之间波动。

使用模型进行单图像识别

input_data = images[0]
test_img = torchvision.utils.make_grid(input_data)
imshow(test_img)

Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).

output = model(input_data)
output = F.softmax(output, dim=1)
output = output.detach().numpy().tolist()[0]
for label, prob in enumerate(output):
    print('预测数字为%d的概率为: %f' % (label, prob))

预测数字为0的概率为: 0.010734
预测数字为1的概率为: 0.000016
预测数字为2的概率为: 0.001222
预测数字为3的概率为: 0.262739
预测数字为4的概率为: 0.000002
预测数字为5的概率为: 0.722390
预测数字为6的概率为: 0.000050
预测数字为7的概率为: 0.001125
预测数字为8的概率为: 0.001215
预测数字为9的概率为: 0.000506

你可能感兴趣的:(神经网络,mnist,pytorch,神经网络,tensorboardx)

ai绘画工具midjourney怎么下载？附作品管理教程设计师早上好
Midjourney是一款功能强大的AI绘画工具，它使用机器学习技术和深度神经网络等算法，可以生成各种艺术风格的绘画作品。在创意设计、广告宣传等方面有着广泛的应用前景。那么，ai绘画工具midjourney怎么下载？本文将为您介绍Midjourney的下载以及作品的相关管理。一、Midjourney下载Midjourney的下载非常简单，只需打开Midjourney官网（点击“GetMidjour
[实践应用] 深度学习之优化器 YuanDaima2048 深度学习工具使用 pytorch 深度学习人工智能机器学习 python 优化器
文章总览：YuanDaiMa2048博客文章总览深度学习之优化器1.随机梯度下降（SGD）2.动量优化（Momentum）3.自适应梯度（Adagrad）4.自适应矩估计（Adam）5.RMSprop总结其他介绍在深度学习中，优化器用于更新模型的参数，以最小化损失函数。常见的优化函数有很多种，下面是几种主流的优化器及其特点、原理和PyTorch实现：1.随机梯度下降（SGD）原理:随机梯度下降通过
吴恩达深度学习笔记(30)-正则化的解释极客Array
正则化（Regularization）深度学习可能存在过拟合问题——高方差，有两个解决方法，一个是正则化，另一个是准备更多的数据，这是非常可靠的方法，但你可能无法时时刻刻准备足够多的训练数据或者获取更多数据的成本很高，但正则化通常有助于避免过拟合或减少你的网络误差。如果你怀疑神经网络过度拟合了数据，即存在高方差问题，那么最先想到的方法可能是正则化，另一个解决高方差的方法就是准备更多数据，这也是非常
个人学习笔记7-6：动手学深度学习pytorch版-李沐浪子L 深度学习深度学习笔记计算机视觉 python 人工智能神经网络 pytorch
#人工智能##深度学习##语义分割##计算机视觉##神经网络#计算机视觉13.11全卷积网络全卷积网络（fullyconvolutionalnetwork，FCN）采用卷积神经网络实现了从图像像素到像素类别的变换。引入l转置卷积（transposedconvolution）实现的，输出的类别预测与输入图像在像素级别上具有一一对应关系：通道维的输出即该位置对应像素的类别预测。13.11.1构造模型下
计算机视觉中，Pooling的作用 Wils0nEdwards 计算机视觉人工智能
在计算机视觉中，Pooling（池化）是一种常见的操作，主要用于卷积神经网络（CNN）中。它通过对特征图进行下采样，减少数据的空间维度，同时保留重要的特征信息。Pooling的作用可以归纳为以下几个方面：1.降低计算复杂度与内存需求Pooling操作通过对特征图进行下采样，减少了特征图的空间分辨率（例如，高度和宽度）。这意味着网络需要处理的数据量会减少，从而降低了计算量和内存需求。这对大型神经网络
神经网络-损失函数红米煮粥神经网络人工智能深度学习
文章目录一、回归问题的损失函数1.均方误差（MeanSquaredError,MSE）2.平均绝对误差（MeanAbsoluteError,MAE）二、分类问题的损失函数1.0-1损失函数（Zero-OneLossFunction）2.交叉熵损失（Cross-EntropyLoss）3.合页损失（HingeLoss）三、总结在神经网络中，损失函数（LossFunction）扮演着至关重要的角色，它
BP神经网络的传递函数大胜归来19 MATLAB
BP网络一般都是用三层的，四层及以上的都比较少用；传输函数的选择，这个怎么说，假设你想预测的结果是几个固定值，如1,0等，满足某个条件输出1，不满足则0的话，首先想到的是hardlim函数，阈值型的，当然也可以考虑其他的；然后，假如网络是用来表达某种线性关系时，用purelin---线性传输函数；若是非线性关系的话，用别的非线性传递函数，多层网络时，每层不一定要用相同的传递函数，可以是三种配合，可
神经网络传递函数sigmoid,神经网络传递函数作用快乐的小荣荣神经网络机器学习深度学习人工智能
神经网络传递函数选取不同会有特别大差别嘛？只是最后一层，但前面层是非线性，那么可能存在区别不大的情况。线性函数f(a*input)=af(input),一般来说，input为向量，最简化情况下，可以假设input的各个维度，a1=a2=a3。。。意味着你线性层只是简单的对输入做了scale~而神经网络能起作用的原因，在于通过足够复杂的非线性函数，来模拟任何的分布。所以，神经网络必须要用非线性函数。
【安装环境】配置MMTracking环境 xuanyu22 安装环境机器学习神经网络深度学习 python
版本v0.14.0安装torchnumpy的版本不能太高，否则后面安装时会发生冲突。先安装numpy，因为pytorch的安装会自动配置高版本numpy。condainstallnumpy=1.21.5mmtracking支持的torch版本有限，需要找到合适的condainstallpytorch==1.11.0torchvision==0.12.0cudatoolkit=10.2-cpytor
Python和R均方根误差平均绝对误差算法模型亚图跨际 Python 交叉知识 R 回归模型误差指标归一化均方根误差生态状态指标神经网络成本误差气体排放气候模型多项式拟合
要点回归模型误差评估指标归一化均方根误差生态状态指标神经网络成本误差计算气体排放气候算法模型Python误差指标均方根误差和平均绝对误差均方根偏差或均方根误差是两个密切相关且经常使用的度量值之一，用于衡量真实值或预测值与观测值或估计值之间的差异。估计器θ^\hat{\theta}θ^相对于估计参数θ\thetaθ的RMSD定义为均方误差的平方根：RMSD⁡(θ^)=MSE⁡(θ^)=E((θ^−θ
Python(PyTorch)和MATLAB及Rust和C++结构相似度指数测量导图亚图跨际 Python 交叉知识算法量化检查图像压缩质量低分辨率多光谱峰值信噪比端到端优化图像压缩手术机器人三维实景实时可微分渲染重建三维可视化
要点量化检查图像压缩质量低分辨率多光谱和高分辨率图像实现超分辨率分析图像质量图像索引/多尺度结构相似度指数和光谱角映射器及视觉信息保真度多种指标峰值信噪比和结构相似度指数测量结构相似性图像分类PNG和JPEG图像相似性近似算法图像压缩，视频压缩、端到端优化图像压缩、神经图像压缩、GPU变速图像压缩手术机器人深度估计算法重建三维可视化推理图像超分辨率算法模型三维实景实时可微分渲染算法MATLAB结构
【深度学习】训练过程中一个OOM的问题，太难查了 weixin_40293999 深度学习深度学习人工智能
现象：各位大佬又遇到过ubuntu的这个问题么？现象是在训练过程中，ssh上不去了，能ping通，没死机，但是ubunutu的pc侧的显示器，鼠标啥都不好用了。只能重启。问题原因：OOM了95G，尼玛！！！！pytorch爆内存了，然后journald假死了，在journald被watchdog干掉之后，系统就崩溃了。这种规模的爆内存一般，即使被oomkill了，也要卡半天的，确实会这样，能不能配
Pyorch中 nn.Conv1d 与 nn.Linear 的区别迪三 #NN_Layer 神经网络
即一维卷积层和全联接层的区别nn.Conv1d和nn.Linear都是PyTorch中的层，它们用于不同的目的，主要区别在于它们处理输入数据的方式和执行的操作类型。nn.Conv1d通过应用滑动过滤器来捕捉序列数据中的局部模式，适用于处理具有时间或序列结构的数据。nn.Linear通过将每个输入与每个输出相连接，捕捉全局关系，适用于将输入数据作为整体处理的任务。1.维度与输入nn.Conv1d（一
图片中的上采样，下采样和通道融合(up-sample, down-sample, channel confusion) 迪三 #图像处理_PyTorch 计算机视觉深度学习人工智能
前言以conv2d为例（即图片），Pytorch中输入的数据格式为tensor，格式为:[N,C,W,H,W]第一维N.代表图片个数，类似一个batch里面有N张图片第二维C.代表通道数，在模型中输入如果为彩色，常用RGB三色图，那么就是3维，即C=3。如果是黑白的，即灰度图，那么只有一个通道，即C=1第三维H.代表图片的高度，H的数量是图片像素的列数第四维W.代表图片的宽度，W的数量是图片像素的
【NLP5-RNN模型、LSTM模型和GRU模型】一蓑烟雨紫洛 nlp rnn lstm gru nlp
RNN模型、LSTM模型和GRU模型1、什么是RNN模型RNN（RecurrentNeuralNetwork)中文称为循环神经网络，它一般以序列数据为输入，通过网络内部的结构设计有效捕捉序列之间的关系特征，一般也是以序列形式进行输出RNN的循环机制使模型隐层上一时间步产生的结果，能够作为当下时间步输入的一部分（当下时间步的输入除了正常的输入外还包括上一步的隐层输出）对当下时间步的输出产生影响2、R
基于深度学习的农作物病害检测 SEU-WYL 深度学习dnn 深度学习人工智能
基于深度学习的农作物病害检测利用卷积神经网络（CNN）、生成对抗网络（GAN）、Transformer等深度学习技术，自动识别和分类农作物的病害，帮助农业工作者提高作物管理效率、减少损失。1.农作物病害检测的挑战病害种类繁多：农作物病害的类型多样，不同病害在同一作物上的表现差异很大，同时同一种病害在不同生长阶段的症状也可能不同。环境影响：天气、光照、湿度等外部环境因素会影响农作物的表现，使得病害检
深度学习--对抗生成网络（GAN, Generative Adversarial Network） Ambition_LAO 深度学习生成对抗网络
对抗生成网络（GAN,GenerativeAdversarialNetwork）是一种深度学习模型，由IanGoodfellow等人在2014年提出。GAN主要用于生成数据，通过两个神经网络相互对抗，来生成以假乱真的新数据。以下是对GAN的详细阐述，包括其概念、作用、核心要点、实现过程、代码实现和适用场景。1.概念GAN由两个神经网络组成：生成器（Generator）和判别器（Discrimina
深度学习：怎么看pth文件的参数奥利给少年深度学习人工智能
.pth文件是PyTorch模型的权重文件，它通常包含了训练好的模型的参数。要查看或使用这个文件，你可以按照以下步骤操作：1.确保你有模型的定义你需要有创建这个.pth文件时所用的模型的代码。这意味着你需要有模型的类定义和架构。2.加载模型权重使用PyTorch的load_state_dict方法来加载权重。这里是如何操作的：importtorchimporttorch.nnasnn#定义模型结构
chatgpt赋能python：如何在Python中安装Keras库？ turensu ChatGpt python chatgpt keras 计算机
如何在Python中安装Keras库？Keras是一个简单易用的神经网络库，由FrançoisChollet编写。它在Python编程语言中实现了深度学习的功能，可以使您更轻松地构建和试验不同类型的神经网络。如果您是一名Python开发人员，肯定会想知道如何在您的Python项目中安装Keras库。在本文中，我们将向您展示如何安装和配置Keras库。步骤1：安装Python要使用Keras库，您需
如何理解深度学习的训练过程奋斗的草莓熊深度学习人工智能 python scikit-learn virtualenv numpy pandas
文章目录1.训练是干什么？2.预训练模型进行训练，主要更改的是预训练模型的什么东西？1.训练是干什么？以yolov5为例子，训练的目的是把一组输入猫狗图像放到神经网络中，得到一个输出模型，这个模型下次可以直接用来识别哪个是猫，哪个是狗2.预训练模型进行训练，主要更改的是预训练模型的什么东西？超参数（Hyperparameters）：这是模型结构中定义的参数，比如：卷积核大小（kernel_size
Keras深度学习框架入门及实战指南司莹嫣Maude
Keras深度学习框架入门及实战指南keraskeras-team/keras:是一个基于Python的深度学习库，它没有使用数据库。适合用于深度学习任务的开发和实现，特别是对于需要使用Python深度学习库的场景。特点是深度学习库、Python、无数据库。项目地址:https://gitcode.com/gh_mirrors/ke/keras一、项目介绍Keras简介Keras是一款高级神经网络
每天五分钟玩转深度学习PyTorch：模型参数优化器torch.optim 幻风_huanfeng 深度学习框架pytorch 深度学习 pytorch 人工智能神经网络机器学习优化算法
本文重点在机器学习或者深度学习中，我们需要通过修改参数使得损失函数最小化(或最大化)，优化算法就是一种调整模型参数更新的策略。在pytorch中定义了优化器optim，我们可以使用它调用封装好的优化算法，然后传递给它神经网络模型参数，就可以对模型进行优化。本文是学习第6步(优化器)，参考链接pytorch的学习路线随机梯度下降算法在深度学习和机器学习中，梯度下降算法是最常用的参数更新方法，它的公式
如何有效的学习AI大模型？ Python程序员罗宾学习人工智能语言模型自然语言处理架构
学习AI大模型是一个系统性的过程，涉及到多个学科的知识。以下是一些建议，帮助你更有效地学习AI大模型：基础知识储备：数学基础：学习线性代数、概率论、统计学和微积分等，这些是理解机器学习算法的数学基础。编程技能：掌握至少一种编程语言，如Python，因为大多数AI模型都是用Python实现的。理论学习：机器学习基础：了解监督学习、非监督学习、强化学习等基本概念。深度学习：学习神经网络的基本结构，如卷
【深度学习】【OnnxRuntime】【Python】模型转化、环境搭建以及模型部署的详细教程牙牙要健康深度学习 onnx onnxruntime 深度学习 python 人工智能
【深度学习】【OnnxRuntime】【Python】模型转化、环境搭建以及模型部署的详细教程提示:博主取舍了很多大佬的博文并亲测有效,分享笔记邀大家共同学习讨论文章目录【深度学习】【OnnxRuntime】【Python】模型转化、环境搭建以及模型部署的详细教程前言模型转换--pytorch转onnxWindows平台搭建依赖环境onnxruntime调用onnx模型ONNXRuntime推理核
【3.6 python中的numpy编写一个“手写数字识”的神经网络】 wang151038606 深度学习入门 python numpy 神经网络
3.6python中的numpy编写一个“手写数字识”的神经网络要使用Python中的NumPy库从头开始编写一个“手写数字识别”的神经网络，我们通常会处理MNIST数据集，这是一个广泛使用的包含手写数字的图像数据集。但是，完全用NumPy来实现神经网络（包括数据的加载、预处理、模型定义、前向传播、损失计算、反向传播和权重更新）是一个相当复杂的任务，因为NumPy本身不提供自动微分或高级优化算法（
yolov5单目测距+速度测量+目标跟踪 cv_2025 YOLO 目标跟踪人工智能计算机视觉机器学习图像处理 opencv
要在YOLOv5中添加测距和测速功能，您需要了解以下两个部分的原理：单目测距算法单目测距是使用单个摄像头来估计场景中物体的距离。常见的单目测距算法包括基于视差的方法（如立体匹配）和基于深度学习的方法（如神经网络）。基于深度学习的方法通常使用卷积神经网络（CNN）来学习从图像到深度图的映射关系。单目测距代码单目测距涉及到坐标转换，代码如下：defconvert_2D_to_3D(point2D,R,
天下苦英伟达久矣！PyTorch官方免CUDA加速推理，Triton时代要来？诗者才子酒中仙物联网 /互联网 /人工智能 /其他 pytorch 人工智能 python
在做大语言模型（LLM）的训练、微调和推理时，使用英伟达的GPU和CUDA是常见的做法。在更大的机器学习编程与计算范畴，同样严重依赖CUDA，使用它加速的机器学习模型可以实现更大的性能提升。虽然CUDA在加速计算领域占据主导地位，并成为英伟达重要的护城河之一。但其他一些工作的出现正在向CUDA发起挑战，比如OpenAI推出的Triton，它在可用性、内存开销、AI编译器堆栈构建等方面具有一定的优势
pytorch安装(windows) m0_62244898 windows 人工智能
（1）下载pycharmPyCharm:thePythonIDEforProfessionalDevelopersbyJetBrains(2)下载anacondaAnaconda|TheWorld'sMostPopularDataSciencePlatform(3)创建一个新环境：torchcondacreate-ntorch-y(4)进入新环境condaactivatetorch(5)加入清华源
深度学习入门篇：PyTorch实现手写数字识别 AI_Guru人工智能深度学习 pytorch 人工智能
深度学习作为机器学习的一个分支，近年来在图像识别、自然语言处理等领域取得了显著的成就。在众多的深度学习框架中，PyTorch以其动态计算图、易用性强和灵活度高等特点，受到了广泛的喜爱。本篇文章将带领大家使用PyTorch框架，实现一个手写数字识别的基础模型。手写数字识别简介手写数字识别是计算机视觉领域的一个经典问题，目的是让计算机能够识别并理解手写数字图像。这个问题通常作为深度学习入门的练习，因为
探索深度学习的奥秘：从理论到实践的奇幻之旅小周不想卷深度学习
目录引言：穿越智能的迷雾一、深度学习的奇幻起源：从感知机到神经网络1.1感知机的启蒙1.2神经网络的诞生与演进1.3深度学习的崛起二、深度学习的核心魔法：神经网络架构2.1前馈神经网络（FeedforwardNeuralNetwork,FNN）2.2卷积神经网络（CNN）2.3循环神经网络（RNN）及其变体（LSTM,GRU）2.4生成对抗网络（GAN）三、深度学习的魔法秘籍：算法与训练3.1损失
怎么样才能成为专业的程序员？ cocos2d-x小菜编程 PHP
如何要想成为一名专业的程序员？仅仅会写代码是不够的。从团队合作去解决问题到版本控制，你还得具备其他关键技能的工具包。当我们询问相关的专业开发人员，那些必备的关键技能都是什么的时候，下面是我们了解到的情况。关于如何学习代码，各种声音很多，然后很多人就被误导为成为专业开发人员懂得一门编程语言就够了？！呵呵，就像其他工作一样，光会一个技能那是远远不够的。如果你想要成为
java web开发高并发处理 BreakingBad java Web 并发开发处理高
java处理高并发高负载类网站中数据库的设计方法（java教程,java处理大量数据，java高负载数据）一：高并发高负载类网站关注点之数据库没错,首先是数据库,这是大多数应用所面临的首个SPOF。尤其是Web2.0的应用，数据库的响应是首先要解决的。一般来说MySQL是最常用的，可能最初是一个mysql主机，当数据增加到100万以上，那么，MySQL的效能急剧下降。常用的优化措施是M-S（
mysql批量更新 ekian mysql
mysql更新优化：一版的更新的话都是采用update set的方式，但是如果需要批量更新的话，只能for循环的执行更新。或者采用executeBatch的方式，执行更新。无论哪种方式，性能都不见得多好。三千多条的更新，需要3分多钟。查询了批量更新的优化，有说replace into的方式，即： replace into tableName(id,status) values
微软BI（3） 18289753290 微软BI SSIS
1) Q：该列违反了完整性约束错误；已获得 OLE DB 记录。源:“Microsoft SQL Server Native Client 11.0” Hresult: 0x80004005 说明:“不能将值 NULL 插入列 'FZCHID'，表 'JRB_EnterpriseCredit.dbo.QYFZCH'；列不允许有 Null 值。INSERT 失败。”。 A：一般这类问题的存在是
Java中的List g21121 java
List是一个有序的 collection（也称为序列）。此接口的用户可以对列表中每个元素的插入位置进行精确地控制。用户可以根据元素的整数索引（在列表中的位置）访问元素，并搜索列表中的元素。与 set 不同，列表通常允许重复
读书笔记永夜-极光读书笔记
1. K是一家加工厂,需要采购原材料,有A,B,C,D 4家供应商,其中A给出的价格最低,性价比最高,那么假如你是这家企业的采购经理,你会如何决策? 传统决策: A:100%订单 B,C,D:0% &nbs
centos 安装 Codeblocks 随便小屋 codeblocks
1.安装gcc,需要c和c++两部分,默认安装下,CentOS不安装编译器的,在终端输入以下命令即可yum install gccyum install gcc-c++ 2.安装gtk2-devel,因为默认已经安装了正式产品需要的支持库,但是没有安装开发所需要的文档.yum install gtk2* 3. 安装wxGTK yum search w
23种设计模式的形象比喻 aijuans 设计模式
1、ABSTRACT FACTORY—追MM少不了请吃饭了，麦当劳的鸡翅和肯德基的鸡翅都是MM爱吃的东西，虽然口味有所不同，但不管你带MM去麦当劳或肯德基，只管向服务员说“来四个鸡翅”就行了。麦当劳和肯德基就是生产鸡翅的Factory 　　工厂模式：客户类和工厂类分开。消费者任何时候需要某种产品，只需向工厂请求即可。消费者无须修改就可以接纳新产品。缺点是当产品修改时，工厂类也要做相应的修改。如：
开发管理 CheckLists aoyouzi 开发管理 CheckLists
开发管理 CheckLists(23) -使项目组度过完整的生命周期开发管理 CheckLists(22) -组织项目资源开发管理 CheckLists(21) -控制项目的范围开发管理 CheckLists(20) -项目利益相关者责任开发管理 CheckLists(19) -选择合适的团队成员开发管理 CheckLists(18) -敏捷开发 Scrum Master 工作开发管理 C
js实现切换百合不是茶 JavaScript 栏目切换
js主要功能之一就是实现页面的特效,窗体的切换可以减少页面的大小,被门户网站大量应用思路: 1,先将要显示的设置为display:bisible 否则设为none 2,设置栏目的id ,js获取栏目的id,如果id为Null就设置为显示 3,判断js获取的id名字;再设置是否显示代码实现: html代码: <di
周鸿祎在360新员工入职培训上的讲话 bijian1013 感悟项目管理人生职场
这篇文章也是最近偶尔看到的，考虑到原博客发布者可能将其删除等原因，也更方便个人查找，特将原文拷贝再发布的。“学东西是为自己的，不要整天以混的姿态来跟公司博弈，就算是混，我觉得你要是能在混的时间里，收获一些别的有利于人生发展的东西，也是不错的，看你怎么把握了”，看了之后，对这句话记忆犹新。 &
前端Web开发的页面效果 Bill_chen html Web Microsoft
1.IE6下png图片的透明显示： <img src="图片地址" border="0" style="Filter.Alpha(Opacity)=数值(100),style=数值(3)"/> 或在<head></head>间加一段JS代码让透明png图片正常显示。 2.<li>标
【JVM五】老年代垃圾回收：并发标记清理GC(CMS GC) bit1129 垃圾回收
CMS概述并发标记清理垃圾回收(Concurrent Mark and Sweep GC）算法的主要目标是在GC过程中，减少暂停用户线程的次数以及在不得不暂停用户线程的请夸功能，尽可能短的暂停用户线程的时间。这对于交互式应用，比如web应用来说，是非常重要的。 CMS垃圾回收针对新生代和老年代采用不同的策略。相比同吞吐量垃圾回收，它要复杂的多。吞吐量垃圾回收在执
Struts2技术总结白糖_ struts2
必备jar文件早在struts2.0.*的时候，struts2的必备jar包需要如下几个： commons-logging-*.jar Apache旗下commons项目的log日志包 freemarker-*.jar
Jquery easyui layout应用注意事项 bozch jquery 浏览器 easyui layout
在jquery easyui中提供了easyui-layout布局，他的布局比较局限，类似java中GUI的border布局。下面对其使用注意事项作简要介绍：如果在现有的工程中前台界面均应用了jquery easyui，那么在布局的时候最好应用jquery eaysui的layout布局，否则在表单页面（编辑、查看、添加等等）在不同的浏览器会出
java-拷贝特殊链表：有一个特殊的链表，其中每个节点不但有指向下一个节点的指针pNext，还有一个指向链表中任意节点的指针pRand，如何拷贝这个特殊链表？ bylijinnan java
public class CopySpecialLinkedList { /** * 题目：有一个特殊的链表，其中每个节点不但有指向下一个节点的指针pNext，还有一个指向链表中任意节点的指针pRand，如何拷贝这个特殊链表？拷贝pNext指针非常容易，所以题目的难点是如何拷贝pRand指针。假设原来链表为A1 -> A2 ->... -> An，新拷贝
color Chen.H JavaScript html css
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"> <HTML> <HEAD>&nbs
[信息与战争]移动通讯与网络 comsci 网络
两个坚持:手机的电池必须可以取下来光纤不能够入户,只能够到楼宇建议大家找这本书看看:<&
oracle flashback query(闪回查询) daizj oracle flashback query flashback table
在Oracle 10g中，Flash back家族分为以下成员： Flashback Database Flashback Drop Flashback Table Flashback Query(分Flashback Query,Flashback Version Query，Flashback Transaction Query) 下面介绍一下Flashback Drop 和Flas
zeus持久层DAO单元测试 deng520159 单元测试
zeus代码测试正紧张进行中,但由于工作比较忙,但速度比较慢.现在已经完成读写分离单元测试了,现在把几种情况单元测试的例子发出来,希望有人能进出意见,让它走下去. 本文是zeus的dao单元测试: 1.单元测试直接上代码 package com.dengliang.zeus.webdemo.test; import org.junit.Test; import o
C语言学习三printf函数和scanf函数学习 dcj3sjt126com c printf scanf language
printf函数 /* 2013年3月10日20:42:32 地点：北京潘家园功能：目的：测试%x %X %#x %#X的用法 */ # include <stdio.h> int main(void) { printf("哈哈！\n"); // \n表示换行 int i = 10; printf
那你为什么小时候不好好读书? dcj3sjt126com life
dady, 我今天捡到了十块钱, 不过我还给那个人了 good girl! 那个人有没有和你讲thank you啊没有啦....他拉我的耳朵我才把钱还给他的, 他哪里会和我讲thank you 爸爸, 如果地上有一张5块一张10块你拿哪一张呢.... 当然是拿十块的咯... 爸爸你很笨的, 你不会两张都拿爸爸为什么上个月那个人来跟你讨钱, 你告诉他没
iptables开放端口 Fanyucai linux iptables 端口
1，找到配置文件 vi /etc/sysconfig/iptables 2，添加端口开放，增加一行，开放18081端口 -A INPUT -m state --state NEW -m tcp -p tcp --dport 18081 -j ACCEPT 3，保存 ESC :wq! 4，重启服务 service iptables
Ehcache（05）——缓存的查询 234390216 排序 ehcache 统计 query
缓存的查询目录 1. 使Cache可查询 1.1 基于Xml配置 1.2 基于代码的配置 2 指定可搜索的属性 2.1 可查询属性类型 2.2 &
通过hashset找到数组中重复的元素 jackyrong hashset
如何在hashset中快速找到重复的元素呢?方法很多，下面是其中一个办法： int[] array = {1,1,2,3,4,5,6,7,8,8}; Set<Integer> set = new HashSet<Integer>(); for(int i = 0
使用ajax和window.history.pushState无刷新改变页面内容和地址栏URL lanrikey history
后退时关闭当前页面 <script type="text/javascript"> jQuery(document).ready(function ($) { if (window.history && window.history.pushState) {
应用程序的通信成本 netkiller.github.com 虚拟机应用服务器陈景峰 netkiller neo
应用程序的通信成本什么是通信一个程序中两个以上功能相互传递信号或数据叫做通信。什么是成本这是是指时间成本与空间成本。时间就是传递数据所花费的时间。空间是指传递过程耗费容量大小。都有哪些通信方式全局变量线程间通信共享内存共享文件管道 Socket 硬件（串口，USB）等等全局变量全局变量是成本最低通信方法，通过设置
一维数组与二维数组的声明与定义恋洁e生二维数组一维数组定义声明初始化
/** * */ package test20111005; /** * @author FlyingFire * @date:2011-11-18 上午04:33:36 * @author ：代码整理 * @introduce :一维数组与二维数组的初始化 *summary： */ public c
Spring Mybatis独立事务配置 toknowme mybatis
在项目中有很多地方会使用到独立事务，下面以获取主键为例（1）修改配置文件spring-mybatis.xml  <tx:annotation-driven transaction-manager="transactionManager" /> &n
更新Anadroid SDK Tooks之后，Eclipse提示No update were found xp9802 eclipse
使用Android SDK Manager 更新了Anadroid SDK Tooks 之后，打开eclipse提示 This Android SDK requires Android Developer Toolkit version 23.0.0 or above, 点击Check for Updates 检测一会后提示 No update were found