深度学习部署——vue webassembly ncnn demo 流程记录

说明

      看大佬nihui/ncnn-webassembly-nanodet的demo，直接使用本机算力，只需要传输一个模型文件，不需要通过网络传输视频流，极大的节省了流量；就想wasm+vue日后用到项目中，是深度学习不错部署方式，就直接开干；本人前端菜鸡，不是很熟悉webassembly，有写的不好欢迎指正；有帮助的话帮忙点赞收藏哈！

vue完整代码：
https://github.com/VITA-Alchemy/vue-webassembly-ncnn

环境

emsdk 2.0.8
ncnn-20220729-webassembly
vue 2.6.11

更新日志

2022-09-26 创建

流程

1.安装 emscripten

git clone https://github.com/emscripten-core/emsdk.git
cd emsdk
./emsdk install 2.0.8
./emsdk activate 2.0.8
source emsdk/emsdk_env.sh

2.下载ncnn-webassembly-nanodet与ncnn-wabsaaembly

git clone https://github.com/nihui/ncnn-webassembly-nanodet.git
cd ncnn-webassembly-nanodet
wget https://github.com/Tencent/ncnn/releases/download/20220216/ncnn-20220729-webassembly.zip
unzip ncnn-20220729-webassembly.zip

3.编译生成wasm文件

source emsdk/emsdk_env.sh
cd ncnn-webassembly-nanodet
mkdir build
cd build
# $EMSDK指emsdk项目包路径
cmake -DCMAKE_TOOLCHAIN_FILE=$EMSDK/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake -DWASM_FEATURE=basic ..
make -j16
cmake -DCMAKE_TOOLCHAIN_FILE=$EMSDK/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake -DWASM_FEATURE=simd ..
make -j16
cmake -DCMAKE_TOOLCHAIN_FILE=$EMSDK/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake -DWASM_FEATURE=threads ..
make -j16
cmake -DCMAKE_TOOLCHAIN_FILE=$EMSDK/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake -DWASM_FEATURE=simd-threads ..
make -j16

3.构建web运行文件

cd ncnn-webassembly-nanodet
mkdir deploy
cp index.html ./delpoy/
cp wasmFeatureDetect.js ./delpoy/
cp ./build/nanodet* ./delpoy/

deploy结构：
deploy/
├── index.html
├── nanodet-basic.data
├── nanodet-basic.js
├── nanodet-basic.wasm
├── nanodet-simd.data
├── nanodet-simd.js
├── nanodet-simd-threads.data
├── nanodet-simd-threads.js
├── nanodet-simd-threads.wasm
├── nanodet-simd-threads.worker.js
├── nanodet-simd.wasm
├── nanodet-threads.data
├── nanodet-threads.js
├── nanodet-threads.wasm
├── nanodet-threads.worker.js
└── wasmFeatureDetect.js

python测试

python3 -m http.server --directory deploy

4.改写vue文件

• 创建一个vue项目
• 将deploy下所有文件copy到vue项目的public下
• 改写HelloWorld.vue

<template>
  <div class="hello">
    <h3>vue ncnn webassembly nanodet demo</h3>
    <button @click="open" style="height: 30px">Start</button>
    <button @click="close" style="height: 30px">End</button>
    <div>
      <canvas id="canvas" width="640"></canvas>
    </div>
    <video id="video" playsinline autoplay></video>
  </div>
</template>

<script>
import { simd, threads } from "./../../public/wasmFeatureDetect.js";

export default {
  name: "HandModel",
  data: function () {
    return {
      ctx: null,
      video: null,
      isopen: true,
    };
  },
  mounted() {
    this.checkWasmSupportAndLoad();
  },
  methods: {
    async checkWasmSupportAndLoad() {
      var Module = {};
      var w = 420;
      var h = 320;
      let img = new Image();
      img.src = require("./../assets/logo.png");
      img.onload = () => {
        var canvas = document.getElementById("canvas");
        canvas.setAttribute("width", w);
        canvas.setAttribute("height", h);
        var ctx = canvas.getContext("2d");
        ctx.drawImage(img, 0, 0, w, h);
      };
      var has_simd;
      var has_threads;
      simd().then((simdSupported) => {
        has_simd = simdSupported;
        threads().then((threadsSupported) => {
          has_threads = threadsSupported;
          if (has_simd) {
            if (has_threads) {
              var nanodet_module_name = "nanodet-simd-threads";
            } else {
              nanodet_module_name = "nanodet-simd";
            }
          } else {
            if (has_threads) {
              nanodet_module_name = "nanodet-threads";
            } else {
              nanodet_module_name = "nanodet-basic";
            }
          }
          console.log("load " + nanodet_module_name);
          var nanodetwasm = nanodet_module_name + ".wasm";
          var nanodetjs = nanodet_module_name + ".js";
          fetch(nanodetwasm)
            .then((response) => response.arrayBuffer())
            .then((buffer) => {
              Module.wasmBinary = buffer;
              var script = document.createElement("script");
              script.src = nanodetjs;
              script.onload = () => {
                console.log("Emscripten  loaded.");
              };
              document.body.appendChild(script);
            });
          console.log(Module);
        });
      });
    },
    async open() {
      var wasmModuleLoadedCallbacks = [];
      Module.onRuntimeInitialized = function () {
        wasmModuleLoaded = true;
        for (var i = 0; i < wasmModuleLoadedCallbacks.length; i++) {
          wasmModuleLoadedCallbacks[i]();
        }
      };
      var w = 640;
      var h = 480;
      var wasmModuleLoaded = true;
      var dst = null;
      var isStreaming = false;
      var video = document.getElementById("video");
      var canvas = document.getElementById("canvas");
      var ctx = canvas.getContext("2d");

      // Wait until the video stream canvas play
      video.addEventListener(
        "canplay",
        function (e) {
          if (!isStreaming) {
            // videoWidth isn't always set correctly in all browsers
            if (video.videoWidth > 0)
              h = video.videoHeight / (video.videoWidth / w);
            canvas.setAttribute("width", w);
            canvas.setAttribute("height", h);
            isStreaming = true;
          }
        },
        false
      );
      // Wait for the video to start to play
      video.addEventListener("play", function () {
        //Setup image memory
        var id = ctx.getImageData(0, 0, canvas.width, canvas.height);
        var d = id.data;
        if (wasmModuleLoaded) {
          mallocAndCallSFilter();
        } else {
          wasmModuleLoadedCallbacks.push(mallocAndCallSFilter);
        }
        function mallocAndCallSFilter() {
          if (dst != null) {
            _free(dst);
            dst = null;
          }
          dst = _malloc(d.length);
          sFilter();
        }
      });
      this.capture();

      function ncnn_nanodet() {
        var canvas = document.getElementById("canvas");
        var ctx = canvas.getContext("2d");
        var imageData = ctx.getImageData(0, 0, canvas.width, canvas.height);
        var data = imageData.data;
        HEAPU8.set(data, dst);
        _nanodet_ncnn(dst, canvas.width, canvas.height);
        var result = HEAPU8.subarray(dst, dst + data.length);
        imageData.data.set(result);
        ctx.putImageData(imageData, 0, 0);
      }
      //Request Animation Frame function
      var sFilter = function () {
        console.log(video.ended, video.paused);
        if (video.paused || video.ended) return;
        ctx.fillRect(0, 0, w, h);
        ctx.drawImage(video, 0, 0, w, h);
        ncnn_nanodet();
        window.requestAnimationFrame(sFilter);
      };
    },
    capture() {
      this.video = document.querySelector("video");
      var constraints = {
        audio: false,
        video: {
          width: 640,
          height: 480,
          facingMode: "user",
        },
      };
      navigator.mediaDevices
        .getUserMedia(constraints)
        .then(function (mediaStream) {
          var stream = mediaStream;
          video.srcObject = mediaStream;
          video.onloadedmetadata = function (e) {
            video.play();
          };
        })
        .catch(function (err) {
          console.log(err.message);
        });
    },
    close() {
      this.video.srcObject.getTracks().forEach((track) => {
        track.stop();
      });
      this.video.srcObject = null;
      this.checkWasmSupportAndLoad();
    },
  },
};
</script>

5.flask后端启动

• vue项目打包

npm run build

• vue项目下创建index.py启动脚本

from flask import Flask
from flask import render_template #引入模板插件
from flask import request, jsonify, render_template, Response

app = Flask(__name__,
static_folder='./dist',  #设置静态文件夹目录
template_folder = "./dist",
static_url_path="")  #设置vue编译输出目录dist文件夹，为Flask模板文件目录

@app.route('/')
def index():
    return render_template('index.html',name='index') #使用模板插件，引入index.html。此处会自动Flask模板文件目录寻找index.html文件。

if __name__ == '__main__':
    app.run(
      debug=True,  # 调试打开
      host='0.0.0.0',  # ip
      port=5000,  # 端口
      ssl_context='adhoc',  # 默认SSL证书，实现https
      threaded=True,  # 多线程
    )

• 启动

python index.py

爬坑记录

1. 调用问题

在.vue代码中如何调用webassembly的胶水代码（nanodet-simd.js）和wasm文件（nanodet-simd.wasm），爬了几天，可能是太菜了；原来初始化加载wasm和js函数前只要加异步（async）即可。

执行推理函数也得加上异步。

2. wasm文件位置

需要将emsdk编译好的wasm文件（.js和.wasm）放到vue项目的public下。

参考

https://github.com/nihui/ncnn-webassembly-nanodet
https://github.com/hanFengSan/realcugan-ncnn-webassembly
https://juejin.cn/post/7014379128837275678