树莓派4B安装paddle-lite跑yolo3项目全过程

硬件环境：树莓派4B
软件环境：python3.8.10，ubuntu20.4，cmake3.10.3，opencv4.2.0
Paddlelite2.8（自己在ARM端编译）

目录：

          1.系统环境准备

          2.安装编译环境

          3.运行官方demo

          4.运行yolo模型

---

1.系统环境准备

1.系统环境准备
刷ubuntu20.4镜像。
读卡器打开TF卡，直接添加txt文件，把文件名，后缀全删除，改成ssh。
用网线连接电脑，电脑网络设置共享

 cmd 后用  arp-a  查看开发板的ip地址
用putty连接，登录，默认用户名ubuntu，密码Ubuntu

登陆后切换root用户：
              sudo passwd root (设置root密码)
              su root (切换root用户) 

登录后：更换源。最好是先不换，等系统软件更新完再更换。实测清华源下载不如ubuntu官方快。
更换步骤：
     sudo nano /etc/apt/sources.list
把所有的网站改成：
     deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports bionic main restricted
     deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports bionic main restricted
     ……
升级软件，很慢，因为设置systemd，会突然后会断开ssh的连接。重连就好.
     sudo apt update
     sudo apt upgrade

下面开始安装桌面，安装桌面的时候，会出现依赖的问题，先安装aptitude，用aptitude能够自动安装所缺失的依赖：
     sudo apt-get install aptitude

安装桌面：
     sudo aptitude install xubuntu-desktop
如果中途ssh断开或者是没有下完：
     sudo apt-get update –fix-missing

使用xrdp远程桌面连接操作树莓派，如果可以直接用树莓派连接显示器操作更好，就不需要这一步。
     sudo apt-get install tightvncserver xrdp
安装完后，将xfce4-session写入到文件.xsession中：
     echo xfce4-session >~/.xsession
修改/etc/xrdp/startwm.sh中
     sudo nano /etc/xrdp/startwm.sh
在/etc/X11/Xsession上边一行添加xfce4-session

修改配置文件 /etc/X11/Xsession
在文件最上面添加xfce4-session


重新启动xrdp服务：
  sudo service xrdp restart

然后我们就可以使用Windows的mstsc登录了，界面如下：

2.安装编译环境

---

2.安装编译环境
  （1）编译环境准备：
         1.安装工具，安装交叉编译环境，ubuntu20.4自带的python3.8.10:，按以下步骤安装：
             sudo apt update
             sudo apt-get install -y gcc g++ make wget python3.8 unzip patchelf python3.8-dev
             sudo apt-get install -y g++-arm-linux-gnueabihf gcc-arm-linux-gnueabihf
 
         2.安装python依赖项：
             sudo apt-get install python3-pip
             pip3 install --upgrade pip
             sudo apt-get install patchelf

        3.安装cmake：( 3.10 以上的版本支持较好） 
             wget https://www.cmake.org/files/v3.10/cmake-3.10.3.tar.gz
             tar -zxvf cmake-3.10.3.tar.gz
             cd cmake-3.10.3
             ./configure --prefix=/usr/local/cmake-3.1.0
             make
             sudo make install
       查看cmake版本号：  cmake –version:  表示cmake安装成功。 

安装时可能出现的问题：

1.安装完成后没有cmake指令：

       如果没有cmake指令：遵循下面步骤，很重要的一步：将cmake添加到系统的环境变量：
              sudo vi /etc/profile
       在末尾加入一行：
              export PATH="$PATH:/usr/local/cmake-3.1.0/bin"
              source /etc/profile
       查看一下，PATH里已经有了：
              echo $PATH
       刚刚安装的cmake已被永久添加到系统PATH。

 (2)编译安装paddle-lite的python Whl包：

        下载paddle-lite源码，并切换分支，从gitee上下载，比较快，github也都一样。
             git clone https://gitee.com/paddlepaddle/paddle-lite

             cd paddle-lite

             git checkout release/v2.8

         删除此目录，编译脚本会自动从国内CDN下载第三方库文件：

             rm –rf third-party

         编译:

             sudo

            ./lite/tools/build_linux.sh --with_python=ON --python_version=3.8 --with_cv=ON

        编译时更多可以参考这篇文章：

            https://blog.csdn.net/weixin_40973138/article/details/114780090

         安装：

            cd paddle-lite /build.lite.linux.armv8.gcc/inference_lite_lib.armlinux.armv8/python/install/dist

            pip3 install xxxxxx.whl

编译，安装时可能出现的问题：

         找不到python指令，没有python版本：因为ubuntu18.04默认安装了python2.7和python3.6.当系统中安装多个python版本时，输入python默认调用的是python2，这就需要我们把python3添加到环境变量：  

           echo alias python=python3 >> ~/.bashrc

更新环境变量：

           source ~/.bashrc

           sudo ln -s /usr/bin/python3 /usr/bin/python

（3）.安装opencv：

    （1）编译安装，参考下面文章：

           https://www.pianshen.com/article/874994362/

    （2）非编译安装：

           sudo apt-get install libopencv-dev

           sudo apt-get install python-opencv

3.运行官方demo

---

3.运行官方demo：

          下载paddle-lite-demo：

          直接在github或者gitee上下载paddle-lite-demo：

          解压;

 

 在此打开终端，下载demo的组件：  

       sh download_models_and_libs.sh

用安装版本的paddle-lite内核替换demo的paddle-lite：

      在此界面进入paddle-lite：

      另外打开root目录下的paddle-lite：

           Paddle-lite/build.lite.linux.armv8.gcc/inference_lite_lib.armlinux.armv8/cxx

 将include替换demo中paddle-lite中的include文件夹。

将lib内的libpaddle_light_api_share.so分别替换demo/paddle_lite/libs/armv7hf和/armv8中的同名文件。

进入/root/Paddle-Lite-Demo-master/Paddle-Lite-armlinux-demo/object_detection_demo.

打开object_detection_demo.cc修改：

复制下面代码替换源代码：

// Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//     http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#include 
#include "paddle_api.h"
#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 

int WARMUP_COUNT = 0;
int REPEAT_COUNT = 1;
const int CPU_THREAD_NUM = 2;
const paddle::lite_api::PowerMode CPU_POWER_MODE =
    paddle::lite_api::PowerMode::LITE_POWER_HIGH;
const std::vector INPUT_SHAPE = {1, 3, 300, 300};
const std::vector INPUT_MEAN = {0.5f, 0.5f, 0.5f};
const std::vector INPUT_STD = {0.5f, 0.5f, 0.5f};
const float SCORE_THRESHOLD = 0.5f;

struct RESULT {
  std::string class_name;
  float score;
  float left;
  float top;
  float right;
  float bottom;
};

inline int64_t get_current_us() {
  struct timeval time;
  gettimeofday(&time, NULL);
  return 1000000LL * (int64_t)time.tv_sec + (int64_t)time.tv_usec;
}

std::vector load_labels(const std::string &path) {
  std::ifstream file;
  std::vector labels;
  file.open(path);
  while (file) {
    std::string line;
    std::getline(file, line);
    labels.push_back(line);
  }
  file.clear();
  file.close();
  return labels;
}

void preprocess(cv::Mat &input_image, const std::vector &input_mean,
                const std::vector &input_std, int input_width,
                int input_height, float *input_data) {
  cv::Mat resize_image;
  cv::resize(input_image, resize_image, cv::Size(input_width, input_height), 0, 0);
  if (resize_image.channels() == 4) {
    cv::cvtColor(resize_image, resize_image, CV_BGRA2RGB);
  }
  cv::Mat norm_image;
  resize_image.convertTo(norm_image, CV_32FC3, 1 / 255.f);
  // NHWC->NCHW
  int image_size = input_height * input_width;
  const float *image_data = reinterpret_cast(norm_image.data);
  float32x4_t vmean0 = vdupq_n_f32(input_mean[0]);
  float32x4_t vmean1 = vdupq_n_f32(input_mean[1]);
  float32x4_t vmean2 = vdupq_n_f32(input_mean[2]);
  float32x4_t vscale0 = vdupq_n_f32(1.0f / input_std[0]);
  float32x4_t vscale1 = vdupq_n_f32(1.0f / input_std[1]);
  float32x4_t vscale2 = vdupq_n_f32(1.0f / input_std[2]);
  float *input_data_c0 = input_data;
  float *input_data_c1 = input_data + image_size;
  float *input_data_c2 = input_data + image_size * 2;
  int i = 0;
  for (; i < image_size - 3; i += 4) {
    float32x4x3_t vin3 = vld3q_f32(image_data);
    float32x4_t vsub0 = vsubq_f32(vin3.val[0], vmean0);
    float32x4_t vsub1 = vsubq_f32(vin3.val[1], vmean1);
    float32x4_t vsub2 = vsubq_f32(vin3.val[2], vmean2);
    float32x4_t vs0 = vmulq_f32(vsub0, vscale0);
    float32x4_t vs1 = vmulq_f32(vsub1, vscale1);
    float32x4_t vs2 = vmulq_f32(vsub2, vscale2);
    vst1q_f32(input_data_c0, vs0);
    vst1q_f32(input_data_c1, vs1);
    vst1q_f32(input_data_c2, vs2);
    image_data += 12;
    input_data_c0 += 4;
    input_data_c1 += 4;
    input_data_c2 += 4;
  }
  for (; i < image_size; i++) {
    *(input_data_c0++) = (*(image_data++) - input_mean[0]) / input_std[0];
    *(input_data_c1++) = (*(image_data++) - input_mean[1]) / input_std[1];
    *(input_data_c2++) = (*(image_data++) - input_mean[2]) / input_std[2];
  }
}

std::vector postprocess(const float *output_data, int64_t output_size,
                                const std::vector &word_labels,
                                const float score_threshold,
                                cv::Mat &output_image, double time) {
  std::vector results;
  std::vector colors = {
      cv::Scalar(237, 189, 101), cv::Scalar(0, 0, 255), cv::Scalar(102, 153, 153),
      cv::Scalar(255, 0, 0), cv::Scalar(9, 255, 0), cv::Scalar(0, 0, 0),
      cv::Scalar(51, 153, 51)};
  for (int64_t i = 0; i < output_size; i += 6) {
    if (output_data[i + 1] < score_threshold) {
      continue;
    }
    int class_id = static_cast(output_data[i]);
    float score = output_data[i + 1];
    RESULT result;
    std::string class_name = "Unknown";
    if (word_labels.size() > 0 && class_id >= 0 && 
        class_id < word_labels.size()) {
      class_name = word_labels[class_id];
    }
    result.class_name = class_name;
    result.score = score;
    result.left = output_data[i + 2];
    result.top = output_data[i + 3];
    result.right = output_data[i + 4];
    result.bottom = output_data[i + 5];
    int lx = static_cast(result.left * output_image.cols);
    int ly = static_cast(result.top * output_image.rows);
    int w = static_cast(result.right * output_image.cols) - lx;
    int h = static_cast(result.bottom * output_image.rows) - ly;
    cv::Rect bounding_box = cv::Rect(lx, ly, w, h) &
        cv::Rect(0, 0, output_image.cols, output_image.rows);
    if (w > 0 && h > 0 && score <= 1) {
      cv::Scalar color = colors[results.size() % colors.size()];
      cv::rectangle(output_image, bounding_box, color);
      cv::rectangle(output_image, cv::Point2d(lx, ly), cv::Point2d(lx + w, ly - 10),
                    color, -1);
      cv::putText(output_image,
                  std::to_string(results.size()) + "." + class_name + ":" +
                      std::to_string(score),
                  cv::Point2d(lx, ly), cv::FONT_HERSHEY_PLAIN, 1, cv::Scalar(255, 255, 255));
      results.push_back(result);
    }
  }
  return results;
}

cv::Mat process(cv::Mat &input_image,
                std::vector &word_labels,
                std::shared_ptr &predictor) {
  // Preprocess image and fill the data of input tensor
  std::unique_ptr input_tensor(
      std::move(predictor->GetInput(0)));
  input_tensor->Resize(INPUT_SHAPE);
  int input_width = INPUT_SHAPE[3];
  int input_height = INPUT_SHAPE[2];
  auto *input_data = input_tensor->mutable_data();
  double preprocess_start_time = get_current_us();
  preprocess(input_image, INPUT_MEAN, INPUT_STD, input_width, input_height,
             input_data);
  double preprocess_end_time = get_current_us();
  double preprocess_time = (preprocess_end_time - preprocess_start_time) / 1000.0f;

  double prediction_time;
  // Run predictor
  // warm up to skip the first inference and get more stable time, remove it in
  // actual products
  for (int i = 0; i < WARMUP_COUNT; i++) {
    predictor->Run();
  }
  // repeat to obtain the average time, set REPEAT_COUNT=1 in actual products
  double max_time_cost = 0.0f;
  double min_time_cost = std::numeric_limits::max();
  double total_time_cost = 0.0f;
  for (int i = 0; i < REPEAT_COUNT; i++) {
    auto start = get_current_us();
    predictor->Run();
    auto end = get_current_us();
    double cur_time_cost = (end - start) / 1000.0f;
    if (cur_time_cost > max_time_cost) {
      max_time_cost = cur_time_cost;
    }
    if (cur_time_cost < min_time_cost) {
      min_time_cost = cur_time_cost;
    }
    total_time_cost += cur_time_cost;
    prediction_time = total_time_cost / REPEAT_COUNT;
    printf("iter %d cost: %f ms\n", i, cur_time_cost);
  }
  printf("warmup: %d repeat: %d, average: %f ms, max: %f ms, min: %f ms\n",
         WARMUP_COUNT, REPEAT_COUNT, prediction_time,
         max_time_cost, min_time_cost);

  // Get the data of output tensor and postprocess to output detected objects
  std::unique_ptr output_tensor(
      std::move(predictor->GetOutput(0)));
  const float *output_data = output_tensor->mutable_data();
  int64_t output_size = 1;
  for (auto dim : output_tensor->shape()) {
    output_size *= dim;
  }
  cv::Mat output_image = input_image.clone();
  double postprocess_start_time = get_current_us();
  std::vector results = postprocess(
      output_data, output_size, word_labels, SCORE_THRESHOLD, output_image, prediction_time);
  double postprocess_end_time = get_current_us();
  double postprocess_time = (postprocess_end_time - postprocess_start_time) / 1000.0f;

  //printf("results: %d\n", results.size());
  for (int i = 0; i < results.size(); i++) {
    printf("[%d] %s - %f %f,%f,%f,%f\n", i, results[i].class_name.c_str(),
           results[i].score, results[i].left, results[i].top, results[i].right,
           results[i].bottom);
  }
  printf("Preprocess time: %f ms\n", preprocess_time);
  printf("Prediction time: %f ms\n", prediction_time);
  printf("Postprocess time: %f ms\n\n", postprocess_time);

  return output_image;
}

int main(int argc, char **argv) {
  if (argc < 3 || argc == 4) {
    printf(
        "Usage: \n"
        "./object_detection_demo model_dir label_path [input_image_path] [output_image_path]"
        "use images from camera if input_image_path and input_image_path isn't provided.");
    return -1;
  }

  std::string model_path = argv[1];
  std::string label_path = argv[2];

  std::vector word_labels = load_labels(label_path);

  paddle::lite_api::MobileConfig config;
  config.set_model_from_file(model_path);
  config.set_threads(CPU_THREAD_NUM);
  config.set_power_mode(CPU_POWER_MODE);

  std::shared_ptr predictor =
      paddle::lite_api::CreatePaddlePredictor(config);

  if (argc > 3) {
    WARMUP_COUNT = 1;
    REPEAT_COUNT = 5;
    std::string input_image_path = argv[3];
    std::string output_image_path = argv[4];
    cv::Mat input_image = cv::imread(input_image_path);
    cv::Mat output_image = process(input_image, word_labels, predictor);
    cv::imwrite(output_image_path, output_image);
    cv::imshow("Object Detection Demo", output_image);
    cv::waitKey(0);
  } else {
    cv::VideoCapture cap(-1);
    cap.set(cv::CAP_PROP_FRAME_WIDTH, 640);
    cap.set(cv::CAP_PROP_FRAME_HEIGHT, 480);
    if (!cap.isOpened()) {
      return -1;
    }
    while (1) {
      cv::Mat input_image;
      cap >> input_image;
      cv::Mat output_image = process(input_image, word_labels, predictor);
      cv::imshow("Object Detection Demo", output_image);
      if (cv::waitKey(1) == char('q')) {
        break;
      }
    }
    cap.release();
    cv::destroyAllWindows();
  }
  return 0;
}

 在当前位置打开命令行：./run.sh

可以看到demo已经运行成功。

 4.运行yolo模型

---

4.运行yolo模型：

     1.用paddlelite自带的工具将模型转化为支持paddle-lite的.nb文件

         paddle_lite_opt --model_file=./inference_model_yolo/__model__（__model__的路径）

                                   --param_file=./inference_model_yolo/__params__（__params__的路径） 

                                   --valid_targets=arm

                                   --optimize_out=yolo（模型名）

1. 将生成的.nb模型替换原来的models文件夹下的model。
2. 将object_detection_demo.cc替换成如下代码：

// Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//     http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#include 
#include 
#include 
#include 
#include "opencv2/core.hpp"
#include "opencv2/imgcodecs.hpp"
#include "opencv2/imgproc.hpp"
#include "paddle_api.h"  // NOLINT
/
// If this demo is linked to static library:libpaddle_api_light_bundled.a
// , you should include `paddle_use_ops.h` and `paddle_use_kernels.h` to
// avoid linking errors such as `unsupport ops or kernels`.
/
// #include "paddle_use_kernels.h"  // NOLINT
// #include "paddle_use_ops.h"      // NOLINT

using namespace paddle::lite_api;  // NOLINT
using namespace std;
struct Object {
  cv::Rect rec;
  int class_id;
  float prob;
};

int64_t ShapeProduction(const shape_t& shape) {
  int64_t res = 1;
  for (auto i : shape) res *= i;
  return res;
}

const char* class_names[] = {"自己的分类"};  记得修改

// fill tensor with mean and scale and trans layout: nhwc -> nchw, neon speed up
void neon_mean_scale(const float* din,
                     float* dout,
                     int size,
                     const std::vector mean,
                     const std::vector scale) {
  if (mean.size() != 3 || scale.size() != 3) {
    std::cerr << "[ERROR] mean or scale size must equal to 3\n";
    exit(1);
  }
  float32x4_t vmean0 = vdupq_n_f32(mean[0]);
  float32x4_t vmean1 = vdupq_n_f32(mean[1]);
  float32x4_t vmean2 = vdupq_n_f32(mean[2]);
  float32x4_t vscale0 = vdupq_n_f32(1.f / scale[0]);
  float32x4_t vscale1 = vdupq_n_f32(1.f / scale[1]);
  float32x4_t vscale2 = vdupq_n_f32(1.f / scale[2]);

  float* dout_c0 = dout;
  float* dout_c1 = dout + size;
  float* dout_c2 = dout + size * 2;

  int i = 0;
  for (; i < size - 3; i += 4) {
    float32x4x3_t vin3 = vld3q_f32(din);
    float32x4_t vsub0 = vsubq_f32(vin3.val[0], vmean0);
    float32x4_t vsub1 = vsubq_f32(vin3.val[1], vmean1);
    float32x4_t vsub2 = vsubq_f32(vin3.val[2], vmean2);
    float32x4_t vs0 = vmulq_f32(vsub0, vscale0);
    float32x4_t vs1 = vmulq_f32(vsub1, vscale1);
    float32x4_t vs2 = vmulq_f32(vsub2, vscale2);
    vst1q_f32(dout_c0, vs0);
    vst1q_f32(dout_c1, vs1);
    vst1q_f32(dout_c2, vs2);

    din += 12;
    dout_c0 += 4;
    dout_c1 += 4;
    dout_c2 += 4;
  }
  for (; i < size; i++) {
    *(dout_c0++) = (*(din++) - mean[0]) * scale[0];
    *(dout_c0++) = (*(din++) - mean[1]) * scale[1];
    *(dout_c0++) = (*(din++) - mean[2]) * scale[2];
  }
}

void pre_process(const cv::Mat& img, int width, int height, float* data) {
  cv::Mat rgb_img;
  //cv::cvtColor(resize_image, resize_image, CV_BGRA2RGB);
  cv::cvtColor(img, rgb_img, cv::COLOR_BGR2RGB);
  cv::resize(
      rgb_img, rgb_img, cv::Size(width, height), 0.f, 0.f, cv::INTER_CUBIC);
  cv::Mat imgf;
  rgb_img.convertTo(imgf, CV_32FC3, 1 / 255.f);
  std::vector mean = {0.485f, 0.456f, 0.406f};
  std::vector scale = {0.229f, 0.224f, 0.225f};
  const float* dimg = reinterpret_cast(imgf.data);
  neon_mean_scale(dimg, data, width * height, mean, scale);
}

std::vector