PyTorch学习笔记(14) ——PyTorch 1.0 的C++ FrontEnd初体验

在去年12月份,我尝试了一下PyTorch 1.0的C++前端, 当时官方负责PyTorch的C++前端的老哥是: Peter Goldsborough, 当时的C++前端还不够稳定,官方文档提供的demo无法跑通.
所以为了避免后面的同学再次入坑. 本部分将会手把手教你如何用PyTorch1.0 跑通C++ Mnist模型的训练~

0. PyTorch C++接口哲学

PyTorch’s C++ frontend was designed with the idea that the Python frontend is great, and should be used when possible; but in some settings, performance and portability requirements make the use of the Python interpreter infeasible（不可实行的）.

For example, Python is a poor choice for low latency, high performance or multithreaded environments, such as video games or production servers. The goal of the C++ frontend is to address these use cases, while not sacrificing the user experience of the Python frontend.

显然, C++前端的目的是为了提供一种跟Python相差不多的接口, 却比Python接口效率高出许多. 在不牺牲用户的开发体验（即跟Python中的实现方式相近，使得用户上手更加方便。）

如此, C++前端在开发时候就遵循了两个哲学:

• ① Closely model the Python frontend in its design（在设计中与Python前端的实现进行尽可能贴近）
  在设计中与Python前端的实现进行尽可能贴近, 这意味着C++前端兼具了功能性(functionality)和便捷性(conventions). 虽然C++前端和Python前端可能有一些差异 (e.g., 我们在C++前端中放弃了不赞成的特性或修复了Python前端的“缺陷”), 我们保证, 移植python到C++上仅仅需要做编程语言层面的工作, 而不涉及修改功能或者行为.

• ② Prioritize flexibility and user-friendliness over micro-optimization（将灵活性和用户友好性置于微观优化之上）
  在C++中, 用户通常可以获得最优化的代码(虽然代价是编写困难和调试问题等(静态语言的特点)). 但是, 灵活性(Flexibility)和动态性(dynamism)是PyTorch的灵魂, 所以我们的C++前端尝试保留这种PyTorch的灵魂-----这意味着在某些情况下可能会为了保持灵活性和动态性牺牲性能.
  我们的目的是让不以使用C++为生的研究人员也可以容易的使用C++前端!!!

注意: Python前端不一定会比C++慢!!

Python前端中, 将所有的computationally expensive运算都放到C++里面计算, 因为这些操作会占用程序的大部分时间. 如果您更喜欢编写Python，并且有能力编写Python，我们建议使用Python接口来编写PyTorch. 然而, 无论出于什么原因(不管你是个C++ programmer还是单纯像我一样想尝试一些PyTorch1.0的新feature), 与Python前端类似, C++前端也非常方便、灵活、友好和直观。

最后, 作者认为: 这两种前端的目标不同, 但是他们携手工作(work hand in hand), 没有任何一方可以说无条件的替换对方。

1. PyTorch C++ Mnist例子

刚开始, 我是上官网找的例子: C++ Mnist minimal example, 但是!!!跑步起来, 为了解决这个问题, 我去官方论坛问了一下.

【PyTorch1.0】 C++ mnist demo Error

最后终于解决了, 下面我直接贴代码和环境, 并把当时遇到的坑附上~

1.1 环境

• gcc: 5.5+ (我最开始用的gcc是4.8.5的,会报段错误!!!)
  
• g++: 5.5+
• cuda: 9.0
• system: ubuntu18.04
• libtorch: https://download.pytorch.org/libtorch/nightly/cu90/libtorch-shared-with-deps-latest.zip
• cmake: 3.13

1.2 CMakeLists.txt

cmake_minimum_required(VERSION 3.1 FATAL_ERROR)
project(mnist)
SET(CMAKE_BUILD_TYPE "Release")
SET(CMAKE_CXX_FLAGS_RELEASE "$ENV{CXXFLAGS} -O3 -Wall")

find_package(Torch REQUIRED)

option(DOWNLOAD_MNIST "Download the MNIST dataset from the internet" ON)
if (DOWNLOAD_MNIST)
  message(STATUS "Downloading MNIST dataset")
  execute_process(
    COMMAND python ${CMAKE_CURRENT_LIST_DIR}/download_mnist.py
      -d ${CMAKE_BINARY_DIR}/data
    ERROR_VARIABLE DOWNLOAD_ERROR)
  if (DOWNLOAD_ERROR)
    message(FATAL_ERROR "Error downloading MNIST dataset: ${DOWNLOAD_ERROR}")
  endif()
endif()


add_executable(mnist mnist.cpp)
target_compile_features(mnist PUBLIC cxx_range_for)
target_link_libraries(mnist ${TORCH_LIBRARIES})

1.3 download_mnist.py脚本

这个代码是下载所需要的mnist数据集的脚本代码.

from __future__ import division
from __future__ import print_function

import argparse
import gzip
import os
import sys
import urllib

try:
    from urllib.error import URLError
    from urllib.request import urlretrieve
except ImportError:
    from urllib2 import URLError
    from urllib import urlretrieve

RESOURCES = [
    'train-images-idx3-ubyte.gz',
    'train-labels-idx1-ubyte.gz',
    't10k-images-idx3-ubyte.gz',
    't10k-labels-idx1-ubyte.gz',
]


def report_download_progress(chunk_number, chunk_size, file_size):
    if file_size != -1:
        percent = min(1, (chunk_number * chunk_size) / file_size)
        bar = '#' * int(64 * percent)
        sys.stdout.write('\r0% |{:<64}| {}%'.format(bar, int(percent * 100)))


def download(destination_path, url, quiet):
    if os.path.exists(destination_path):
        if not quiet:
            print('{} already exists, skipping ...'.format(destination_path))
    else:
        print('Downloading {} ...'.format(url))
        try:
            hook = None if quiet else report_download_progress
            urlretrieve(url, destination_path, reporthook=hook)
        except URLError:
            raise RuntimeError('Error downloading resource!')
        finally:
            if not quiet:
                # Just a newline.
                print()


def unzip(zipped_path, quiet):
    unzipped_path = os.path.splitext(zipped_path)[0]
    if os.path.exists(unzipped_path):
        if not quiet:
            print('{} already exists, skipping ... '.format(unzipped_path))
        return
    with gzip.open(zipped_path, 'rb') as zipped_file:
        with open(unzipped_path, 'wb') as unzipped_file:
            unzipped_file.write(zipped_file.read())
            if not quiet:
                print('Unzipped {} ...'.format(zipped_path))

def main():
    parser = argparse.ArgumentParser(
        description='Download the MNIST dataset from the internet')
    parser.add_argument(
        '-d', '--destination', default='.', help='Destination directory')
    parser.add_argument(
        '-q',
        '--quiet',
        action='store_true',
        help="Don't report about progress")
    options = parser.parse_args()

    if not os.path.exists(options.destination):
        os.makedirs(options.destination)

    try:
        for resource in RESOURCES:
            path = os.path.join(options.destination, resource)
            url = 'http://yann.lecun.com/exdb/mnist/{}'.format(resource)
            download(path, url, options.quiet)
            unzip(path, options.quiet)
    except KeyboardInterrupt:
        print('Interrupted')


if __name__ == '__main__':
    main()

1.4 mnist.cpp

关键的训练代码:

#include 

#include 
#include 
#include 
#include 

struct Net : torch::nn::Module {
        Net()
                : conv1(torch::nn::Conv2dOptions(1, 10, /*kernel_size=*/5)),
                conv2(torch::nn::Conv2dOptions(10, 20, /*kernel_size=*/5)),
                fc1(320, 50),
                fc2(50, 10) {
                register_module("conv1", conv1);
                register_module("conv2", conv2);
                register_module("conv2_drop", conv2_drop);
                register_module("fc1", fc1);
                register_module("fc2", fc2);
        }

        torch::Tensor forward(torch::Tensor x) {
                x = torch::relu(torch::max_pool2d(conv1->forward(x), 2));
                x = torch::relu(
                        torch::max_pool2d(conv2_drop->forward(conv2->forward(x)), 2));
                x = x.view({ -1, 320 });
                x = torch::relu(fc1->forward(x));
                x = torch::dropout(x, /*p=*/0.5, /*training=*/is_training());
                x = fc2->forward(x);
                return torch::log_softmax(x, /*dim=*/1);
        }

        torch::nn::Conv2d conv1;
        torch::nn::Conv2d conv2;
        torch::nn::FeatureDropout conv2_drop;
        torch::nn::Linear fc1;
        torch::nn::Linear fc2;
};

struct Options {
        std::string data_root{ "data" };
        int32_t batch_size{ 64 };
        int32_t epochs{ 10 };
        double lr{ 0.01 };
        double momentum{ 0.5 };
        bool no_cuda{ false };
        int32_t seed{ 1 };
        int32_t test_batch_size{ 1000 };
        int32_t log_interval{ 10 };
};

template <typename DataLoader>
void train(
        int32_t epoch,
        const Options& options,
        Net& model,
        torch::Device device,
        DataLoader& data_loader,
        torch::optim::SGD& optimizer,
        size_t dataset_size) {
        model.train();
        size_t batch_idx = 0;
        for (auto& batch : data_loader) {
                auto data = batch.data.to(device), targets = batch.target.to(device);
                optimizer.zero_grad();
                auto output = model.forward(data);
                auto loss = torch::nll_loss(output, targets);
                loss.backward();
                optimizer.step();

                if (batch_idx++ % options.log_interval == 0) {
                        std::cout << "Train Epoch: " << epoch << " ["
                                << batch_idx * batch.data.size(0) << "/" << dataset_size
                                << "]\tLoss: " << loss.template item<float>() << std::endl;
                }
        }
}

template <typename DataLoader>
void test(
        Net& model,
        torch::Device device,
        DataLoader& data_loader,
        size_t dataset_size) {
        torch::NoGradGuard no_grad;
        model.eval();
        double test_loss = 0;
        int32_t correct = 0;
        for (const auto& batch : data_loader) {
                auto data = batch.data.to(device), targets = batch.target.to(device);
                auto output = model.forward(data);
                test_loss += torch::nll_loss(
                        output,
                        targets,
                        /*weight=*/{},
                        Reduction::Sum)
                        .template item<float>();
                auto pred = output.argmax(1);
                correct += pred.eq(targets).sum().template item<int64_t>();
        }
test_loss /= dataset_size;
        std::cout << "Test set: Average loss: " << test_loss
                << ", Accuracy: " << correct << "/" << dataset_size << std::endl;
}

struct Normalize : public torch::data::transforms::TensorTransform<> {
        Normalize(float mean, float stddev)
                : mean_(torch::tensor(mean)), stddev_(torch::tensor(stddev)) {}
        torch::Tensor operator()(torch::Tensor input) {
                return input.sub_(mean_).div_(stddev_);
        }
        torch::Tensor mean_, stddev_;
};

auto main(int argc, const char* argv[]) -> int {
        torch::manual_seed(0);

        Options options;
        torch::DeviceType device_type;
        if (torch::cuda::is_available() && !options.no_cuda) {
                std::cout << "CUDA available! Training on GPU" << std::endl;
                device_type = torch::kCUDA;
        }
        else {
                std::cout << "Training on CPU" << std::endl;
                device_type = torch::kCPU;
        }
        torch::Device device(device_type);

        Net model;
        model.to(device);

        auto train_dataset =
                torch::data::datasets::MNIST(
                        options.data_root, torch::data::datasets::MNIST::Mode::kTrain)
                .map(Normalize(0.1307, 0.3081))
                .map(torch::data::transforms::Stack<>());
        const auto dataset_size = train_dataset.size();

        auto train_loader = torch::data::make_data_loader(
                std::move(train_dataset), options.batch_size);

        auto test_loader = torch::data::make_data_loader(
                torch::data::datasets::MNIST(
                        options.data_root, torch::data::datasets::MNIST::Mode::kTest)
                .map(Normalize(0.1307, 0.3081))
                .map(torch::data::transforms::Stack<>()),
                options.batch_size);

        torch::optim::SGD optimizer(
                model.parameters(),
                torch::optim::SGDOptions(options.lr).momentum(options.momentum));
for (int32_t epoch = 1; epoch <= options.epochs; ++epoch) {
            // 将train和test的最后参数改为dataset_size.value())
                train(
                        epoch, options, model, device, *train_loader, optimizer, dataset_size.value());
                test(model, device, *test_loader, dataset_size.value());
        }
}

1.5 编译&运行

注意, 上面的3个文件需要放在同一级目录下, 然后在这级目录建一个build文件夹, 用常规的cmake ..即可.

2. 参考资料

[1] PyTorch1.0: C++ mnist demo Error
[2] TonyzBi: libtorch-mnist例子