通过示例学习pytorch

本教程通过自包含的示例介绍了PyTorch的基本概念。

在其核心，PyTorch提供了两个主要功能：

1)一个n维张量，类似于numpy但可以在gpu上运行

2)构建和训练神经网络的自动微分

我们将使用一个完全连接的ReLU网络作为我们的运行示例。该网络将有一个单独的隐藏层，

并将通过最小化网络输出和真实输出之间的欧氏距离来对随机数据进行训练。

Tensors

在介绍PyTorch之前，我们将首先使用numpy实现网络。

Numpy提供了一个n维数组对象，以及许多用于操纵这些数组的函数。Numpy是一个用于科学计

算的通用框架;它不知道任何关于计算图、深度学习或渐变的知识。然而，我们可以很容易地使

用numpy来将一个两层的网络连接到随机的数据，通过使用numpy操作来手动实现向前和向后的

通过网络：

# -*- coding: utf-8 -*-
import numpy as np

# N is batch size; D_in is input dimension;
# H is hidden dimension; D_out is output dimension.
N, D_in, H, D_out = 64, 1000, 100, 10

# Create random input and output data
x = np.random.randn(N, D_in)  #dimention:64*1000
y = np.random.randn(N, D_out)  #dimention:64*10

# Randomly initialize weights
w1 = np.random.randn(D_in, H)  #dimention:1000*100
w2 = np.random.randn(H, D_out)  #dimention:100*10

learning_rate = 1e-6
for t in range(500):
    # Forward pass: compute predicted y
    h = x.dot(w1)    #dimention:64*100
    h_relu = np.maximum(h, 0) #dimention:64*100
    y_pred = h_relu.dot(w2) #dimention:64*10

    # Compute and print loss
    loss = np.square(y_pred - y).sum()  #dimention:64*10 - 100*10
    print(t, loss)

    # Backprop to compute gradients of w1 and w2 with respect to loss
    grad_y_pred = 2.0 * (y_pred - y)
    grad_w2 = h_relu.T.dot(grad_y_pred)
    grad_h_relu = grad_y_pred.dot(w2.T)
    grad_h = grad_h_relu.copy()
    grad_h[h < 0] = 0
    grad_w1 = x.T.dot(grad_h)

    # Update weights
    w1 -= learning_rate * grad_w1
    w2 -= learning_rate * grad_w2

pytorch： Tensors

Numpy是一个很好的框架，但是它不能利用gpu来加速它的数值计算。对于现代的深层神经网络来说，gpu常常提供50倍或更大的速度，所以不幸的是，numpy对于现代深度学习来说是不够的。

这里我们介绍了最基本的PyTorch概念：张量。PyTorch张量在概念上与numpy阵列相同：一个张量是一个n维数组，而PyTorch提供了许多在这些张量上操作的函数。在幕后，张量可以跟踪计算图和梯度，但它们作为科学计算的通用工具也很有用。

与numpy不同的是，PyTorch张量可以利用gpu加速它们的数值计算。要在GPU上运行一个PyTorch张量，你只需要把它转换成新的数据类型。

在这里，我们使用PyTorch张量来将一个两层的网络连接到随机的数据中。就像上面的numpy示例一样，我们需要手动实现通过网络的向前和向后传递：

# -*- coding: utf-8 -*-
import torch

dtype = torch.float
device = torch.device("cpu")
# device = torch.device("cuda:0") # Uncomment this to run on GPU

# N is batch size; D_in is input dimension;
# H is hidden dimension; D_out is output dimension.
N, D_in, H, D_out = 64, 1000, 100, 10

# Create random input and output data
x = torch.randn(N, D_in, device=device, dtype=dtype)
y = torch.randn(N, D_out, device=device, dtype=dtype)

# Randomly initialize weights
w1 = torch.randn(D_in, H, device=device, dtype=dtype)
w2 = torch.randn(H, D_out, device=device, dtype=dtype)

learning_rate = 1e-6
for t in range(500):
    # Forward pass: compute predicted y
    h = x.mm(w1)
    h_relu = h.clamp(min=0)
    y_pred = h_relu.mm(w2)

    # Compute and print loss
    loss = (y_pred - y).pow(2).sum().item()
    print(t, loss)

    # Backprop to compute gradients of w1 and w2 with respect to loss
    grad_y_pred = 2.0 * (y_pred - y)
    grad_w2 = h_relu.t().mm(grad_y_pred)
    grad_h_relu = grad_y_pred.mm(w2.t())
    grad_h = grad_h_relu.clone()
    grad_h[h < 0] = 0
    grad_w1 = x.t().mm(grad_h)

    # Update weights using gradient descent
    w1 -= learning_rate * grad_w1
    w2 -= learning_rate * grad_w2

AUTOGRAD

PyTorch: Tensors and autograd

在上面的例子中，我们必须手动地实现我们的神经网络的向前和向后的传递。对于一个小型的两层网络来说，手动实现向后传递并不是什么大问题，但是对于大型复杂的网络来说，它很快就会变得非常麻烦。

值得庆幸的是，我们可以使用自动微分来自动计算神经网络的后传。PyTorch的autograd包提供了这种功能。当使用autograd时，您的网络的前向传播将定义一个计算图;图中的节点将是张量，而边缘部分将是从输入张量中产生输出张量。通过这个图进行反向传播，可以方便地计算梯度。

这听起来很复杂，在实践中使用起来很简单。每个张量代表一个计算图中的一个节点。如果x是一个张量且x.requires_grad=True，然后x.grad是另一个张量，是x关于某个标量值的梯度。

在这里，我们使用PyTorch张量和autograd来实现我们的两层网络;现在，我们不再需要手动地通过网络实现逆向传递：

与numpy不同的是，PyTorch张量可以利用gpu加速它们的数值计算。要在GPU上运行一个PyTorch张量，你只需要把它转换成新的数据类型。

在这里，我们使用PyTorch张量来将一个两层的网络连接到随机的数据中。就像上面的numpy示例一样，我们需要手动实现通过网络的向前和向后传递：

# -*- coding: utf-8 -*-
import torch

dtype = torch.float
device = torch.device("cpu")
# device = torch.device("cuda:0") # Uncomment this to run on GPU

# N is batch size; D_in is input dimension;
# H is hidden dimension; D_out is output dimension.
N, D_in, H, D_out = 64, 1000, 100, 10

# Create random Tensors to hold input and outputs.
# Setting requires_grad=False indicates that we do not need to compute gradients
# with respect to these Tensors during the backward pass.
x = torch.randn(N, D_in, device=device, dtype=dtype)
y = torch.randn(N, D_out, device=device, dtype=dtype)

# Create random Tensors for weights.
# Setting requires_grad=True indicates that we want to compute gradients with
# respect to these Tensors during the backward pass.
w1 = torch.randn(D_in, H, device=device, dtype=dtype, requires_grad=True)
w2 = torch.randn(H, D_out, device=device, dtype=dtype, requires_grad=True)

learning_rate = 1e-6
for t in range(500):
    # Forward pass: compute predicted y using operations on Tensors; these
    # are exactly the same operations we used to compute the forward pass using
    # Tensors, but we do not need to keep references to intermediate values since
    # we are not implementing the backward pass by hand.
    y_pred = x.mm(w1).clamp(min=0).mm(w2)

    # Compute and print loss using operations on Tensors.
    # Now loss is a Tensor of shape (1,)
    # loss.item() gets the a scalar value held in the loss.
    loss = (y_pred - y).pow(2).sum()
    print(t, loss.item())

    # Use autograd to compute the backward pass. This call will compute the
    # gradient of loss with respect to all Tensors with requires_grad=True.
    # After this call w1.grad and w2.grad will be Tensors holding the gradient
    # of the loss with respect to w1 and w2 respectively.
    loss.backward()

    # Manually update weights using gradient descent. Wrap in torch.no_grad()
    # because weights have requires_grad=True, but we don't need to track this
    # in autograd.
    # An alternative way is to operate on weight.data and weight.grad.data.
    # Recall that tensor.data gives a tensor that shares the storage with
    # tensor, but doesn't track history.
    # You can also use torch.optim.SGD to achieve this.
    with torch.no_grad():
        w1 -= learning_rate * w1.grad
        w2 -= learning_rate * w2.grad

        # Manually zero the gradients after updating weights
        w1.grad.zero_()
        w2.grad.zero_()

PyTorch: Defining new autograd functions

在引擎盖下，每个原始的autograd操作符实际上是两个作用于张量的函数。正向传播函数计算来自输入张量的输出张量。该反向传播函数接受与某个标量值有关的输出张量的梯度，并计算出与相同标量值有关的输入张量的梯度。

在PyTorch中，我们可以非常容易的通过torch.autograd.Function的子类定义我们自己的autograd 运算来实现前向和反向传播运算函数。然后我们可以使用新的autograd操作符构造一个实例，可以向调用函数一样调用它传递包含输入数据的张量。

在这个例子中，我们定义了自己的自定义autograd函数来实现ReLU非线性，并使用它来实现我们的两层网络：

# -*- coding: utf-8 -*-
import torch


class MyReLU(torch.autograd.Function):
    """
    We can implement our own custom autograd Functions by subclassing
    torch.autograd.Function and implementing the forward and backward passes
    which operate on Tensors.
    """

    @staticmethod
    def forward(ctx, input):
        """
        In the forward pass we receive a Tensor containing the input and return
        a Tensor containing the output. ctx is a context object that can be used
        to stash information for backward computation. You can cache arbitrary
        objects for use in the backward pass using the ctx.save_for_backward method.
        """
        ctx.save_for_backward(input)
        return input.clamp(min=0)

    @staticmethod
    def backward(ctx, grad_output):
        """
        In the backward pass we receive a Tensor containing the gradient of the loss
        with respect to the output, and we need to compute the gradient of the loss
        with respect to the input.
        """
        input, = ctx.saved_tensors
        grad_input = grad_output.clone()
        grad_input[input < 0] = 0
        return grad_input


dtype = torch.float
device = torch.device("cpu")
# device = torch.device("cuda:0") # Uncomment this to run on GPU

# N is batch size; D_in is input dimension;
# H is hidden dimension; D_out is output dimension.
N, D_in, H, D_out = 64, 1000, 100, 10

# Create random Tensors to hold input and outputs.
x = torch.randn(N, D_in, device=device, dtype=dtype)
y = torch.randn(N, D_out, device=device, dtype=dtype)

# Create random Tensors for weights.
w1 = torch.randn(D_in, H, device=device, dtype=dtype, requires_grad=True)
w2 = torch.randn(H, D_out, device=device, dtype=dtype, requires_grad=True)

learning_rate = 1e-6
for t in range(500):
    # To apply our Function, we use Function.apply method. We alias this as 'relu'.
    relu = MyReLU.apply

    # Forward pass: compute predicted y using operations; we compute
    # ReLU using our custom autograd operation.
    y_pred = relu(x.mm(w1)).mm(w2)

    # Compute and print loss
    loss = (y_pred - y).pow(2).sum()
    print(t, loss.item())

    # Use autograd to compute the backward pass.
    loss.backward()

    # Update weights using gradient descent
    with torch.no_grad():
        w1 -= learning_rate * w1.grad
        w2 -= learning_rate * w2.grad

        # Manually zero the gradients after updating weights
        w1.grad.zero_()
        w2.grad.zero_()

TensorFlow: Static Graphs

PyTorch autograd看起来很像TensorFlow：在这两个框架中，我们定义了一个计算图，并使用自动微分来计算梯度。两者之间最大的区别是，TensorFlow的计算图是静态的，而PyTorch使用动态计算图。

在TensorFlow中，我们对计算图进行一次定义，然后一次又一次地执行相同的计算图，可以将不同的输入数据输入到图表中。在PyTorch中，每个前向传播都定义一个新的计算图。

静态图很好，因为你可以预先优化计算图;例如，一个框架为了提高效率可能拒绝一些计算图的运算，或者提出一个策略，以便在许多gpu或许多机器上实现分布式的计算图运算。如果您一次又一次地重用相同的图表，那么这个潜在的昂贵的预先优化就可以被摊销，因为相同的图一次又一次地重新运行。

静态和动态图不同的一个方面是控制流。对于某些模型，我们可能希望对每个数据点执行不同的计算;例如，对于每个数据点，可以为不同的时间步骤打开一个循环网络;这个展开可以作为一个循环来实现。有了静态图，循环结构需要成为图的一部分;由于这个原因，TensorFlow提供了诸如tf之类的操作符。扫描在图中嵌入循环。有了动态图，情况就更简单了：由于我们为每个例子都动态地构建图表，所以我们可以使用普通的命令流控制来执行不同于每个输入的计算。

与上面的PyTorch autograd示例相比，这里我们使用TensorFlow来安装一个简单的两层网络：

# -*- coding: utf-8 -*-
import tensorflow as tf
import numpy as np

# First we set up the computational graph:

# N is batch size; D_in is input dimension;
# H is hidden dimension; D_out is output dimension.
N, D_in, H, D_out = 64, 1000, 100, 10

# Create placeholders for the input and target data; these will be filled
# with real data when we execute the graph.
x = tf.placeholder(tf.float32, shape=(None, D_in))
y = tf.placeholder(tf.float32, shape=(None, D_out))

# Create Variables for the weights and initialize them with random data.
# A TensorFlow Variable persists its value across executions of the graph.
w1 = tf.Variable(tf.random_normal((D_in, H)))
w2 = tf.Variable(tf.random_normal((H, D_out)))

# Forward pass: Compute the predicted y using operations on TensorFlow Tensors.
# Note that this code does not actually perform any numeric operations; it
# merely sets up the computational graph that we will later execute.
h = tf.matmul(x, w1)
h_relu = tf.maximum(h, tf.zeros(1))
y_pred = tf.matmul(h_relu, w2)

# Compute loss using operations on TensorFlow Tensors
loss = tf.reduce_sum((y - y_pred) ** 2.0)

# Compute gradient of the loss with respect to w1 and w2.
grad_w1, grad_w2 = tf.gradients(loss, [w1, w2])

# Update the weights using gradient descent. To actually update the weights
# we need to evaluate new_w1 and new_w2 when executing the graph. Note that
# in TensorFlow the the act of updating the value of the weights is part of
# the computational graph; in PyTorch this happens outside the computational
# graph.
learning_rate = 1e-6
new_w1 = w1.assign(w1 - learning_rate * grad_w1)
new_w2 = w2.assign(w2 - learning_rate * grad_w2)

# Now we have built our computational graph, so we enter a TensorFlow session to
# actually execute the graph.
with tf.Session() as sess:
    # Run the graph once to initialize the Variables w1 and w2.
    sess.run(tf.global_variables_initializer())

    # Create numpy arrays holding the actual data for the inputs x and targets
    # y
    x_value = np.random.randn(N, D_in)
    y_value = np.random.randn(N, D_out)
    for _ in range(500):
        # Execute the graph many times. Each time it executes we want to bind
        # x_value to x and y_value to y, specified with the feed_dict argument.
        # Each time we execute the graph we want to compute the values for loss,
        # new_w1, and new_w2; the values of these Tensors are returned as numpy
        # arrays.
        loss_value, _, _ = sess.run([loss, new_w1, new_w2],
                                    feed_dict={x: x_value, y: y_value})
        print(loss_value)