Pytorch学习：张量和自动求导——搭建一个简单的神经网络

参考pytorch官方文档：https://pytorch.org/tutorials/beginner/pytorch_with_examples.html#id17  （可能需要科学上网）

当使用自动求导时(autograd)，网络的向前传播过程将会定义一个计算图，图中的结点由张量(tensor)构成，而运算过程将构成图的边(edge)。通过图的后向传播将会很容易实现对梯度的计算。

如上所述，每一个张量在计算图中表示一个结点(node)，如果x是一个张量，并且有x.requires_grad=Ture，那么x.grad则是表示x关于某个标量的梯度张量。

下边是利用自动求导搭建的一个简单的神经网络：

# -*- coding: utf-8 -*-
import torch

dtype = torch.float
device = torch.device("cpu")

# N is batch size; D_in is input dimension;
# H is hidden dimension; D_out is output dimension.
N, D_in, H, D_out = 64, 1000, 100, 10

# Create random Tensors to hold input and outputs.
# Setting requires_grad=False indicates that we do not need to compute gradients
# with respect to these Tensors during the backward pass.
x = torch.randn(N, D_in, device=device, dtype=dtype)
y = torch.randn(N, D_out, device=device, dtype=dtype)

# Create random Tensors for weights.
# Setting requires_grad=True indicates that we want to compute gradients with
# respect to these Tensors during the backward pass.
w1 = torch.randn(D_in, H, device=device, dtype=dtype, requires_grad=True)
w2 = torch.randn(H, D_out, device=device, dtype=dtype, requires_grad=True)

learning_rate = 1e-6
for t in range(500):
    # Forward pass: compute predicted y using operations on Tensors; these
    # are exactly the same operations we used to compute the forward pass using
    # Tensors, but we do not need to keep references to intermediate values since
    # we are not implementing the backward pass by hand.
    y_pred = x.mm(w1).clamp(min=0).mm(w2)

    # Compute and print loss using operations on Tensors.
    # Now loss is a Tensor of shape (1,)
    # loss.item() gets the a scalar value held in the loss.
    loss = (y_pred - y).pow(2).sum()
    print(t, loss.item())

    #利用backward命令进行向后传播，loss.backward()将会计算所有与loss有关且requires_grad=True的变量的梯度
    #在本例中将会计算loss对w1和w2的梯度，并可由w1.grad和w2.grad获取
    loss.backward()

    #w1和w2都是requires_grad=Ture,但是在更新w1和w2的时候我们并不希望计算这一步骤的梯度
    #因此在更新w时，要用到torch.no_grad()
    #这一步也可以通过torch.optim.SGD自动实现
    with torch.no_grad():
        w1 -= learning_rate * w1.grad
        w2 -= learning_rate * w2.grad

        # 在更新完权重后，将梯度值进行重置
        w1.grad.zero_()
        w2.grad.zero_()