花花少年

TensorRT实现EfficientDet推理加速（一）

一、参考资料

EfficientDet tensorrt

二、相关环境

1. 系统环境

Environment
Operating System + Version: Ubuntu + 16.04
TensorRT Version: 8.0.1.6
GPU Type: GeForce GTX1650,4GB
Nvidia Driver Version: 470.63.01
CUDA Version: 11.0.194
CUDNN Version: 8.0.5
Python Version (if applicable): 3.7.11
Anaconda Version：4.10.3
gcc：7.5.0
g++：7.5.0

2. requirements-gpu.txt

absl-py==0.13.0
appdirs==1.4.4
astunparse==1.6.3
attrs==21.2.0
cached-property==1.5.2
cachetools==4.2.2
certifi==2021.5.30
charset-normalizer==2.0.5
clang==5.0
cycler==0.10.0
Cython==0.29.24
dm-tree==0.1.6
flatbuffers==1.12
gast==0.4.0
google-auth==1.35.0
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
graphsurgeon @ file:///.../TensorRT-8.0.1.6/graphsurgeon/graphsurgeon-0.4.5-py2.py3-none-any.whl
grpcio==1.34.1
h5py==3.1.0
idna==3.2
importlib-metadata==4.8.1
keras==2.6.0
keras-nightly==2.5.0.dev2021032900
Keras-Preprocessing==1.1.2
kiwisolver==1.3.2
lxml==4.6.3
Mako==1.1.5
Markdown==3.3.4
MarkupSafe==2.0.1
matplotlib==3.4.3
neural-structured-learning==1.3.1
numpy==1.19.5
oauthlib==3.1.1
onnx==1.8.1
onnx-graphsurgeon==0.3.11
onnxruntime==1.8.0
opt-einsum==3.3.0
Pillow==8.3.2
protobuf==3.18.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycocotools==2.0
pycuda==2021.1
pyparsing==2.4.7
python-dateutil==2.8.2
pytools==2021.2.8
PyYAML==5.4.1
requests==2.26.0
requests-oauthlib==1.3.0
rsa==4.7.2
scipy==1.7.1
six==1.15.0
tensorboard==2.6.0
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.0
tensorflow-addons==0.14.0
tensorflow-estimator==2.5.0
tensorflow-gpu @ file:///.../tensorflow_gpu-2.5.0-cp37-cp37m-manylinux2010_x86_64.whl
tensorflow-hub==0.12.0
tensorflow-model-optimization==0.6.0
tensorrt @ file:///.../TensorRT-8.0.1.6/python/tensorrt-8.0.1.6-cp37-none-linux_x86_64.whl
termcolor==1.1.0
tf2onnx==1.8.1
typeguard==2.12.1
typing-extensions==3.7.4.3
uff @ file:///.../TensorRT-8.0.1.6/uff/uff-0.6.9-py2.py3-none-any.whl
urllib3==1.26.6
Werkzeug==2.0.1
wrapt==1.12.1
zipp==3.5.0

3. tensorRT-efficientdet.yaml

name: tensorRT-efficientdet
channels:
  - http://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
  - http://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r
  - http://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/msys2
dependencies:
  - _libgcc_mutex=0.1=main
  - _openmp_mutex=4.5=1_gnu
  - ca-certificates=2021.7.5=h06a4308_1
  - certifi=2021.5.30=py37h06a4308_0
  - ld_impl_linux-64=2.35.1=h7274673_9
  - libffi=3.3=he6710b0_2
  - libgcc-ng=9.3.0=h5101ec6_17
  - libgomp=9.3.0=h5101ec6_17
  - libstdcxx-ng=9.3.0=hd4cf53a_17
  - ncurses=6.2=he6710b0_1
  - openssl=1.1.1l=h7f8727e_0
  - pip=21.0.1=py37h06a4308_0
  - python=3.7.11=h12debd9_0
  - readline=8.1=h27cfd23_0
  - setuptools=58.0.4=py37h06a4308_0
  - sqlite=3.36.0=hc218d9a_0
  - tk=8.6.10=hbc83047_0
  - wheel=0.37.0=pyhd3eb1b0_1
  - xz=5.2.5=h7b6447c_0
  - zlib=1.2.11=h7b6447c_3
  - pip:
    - absl-py==0.13.0
    - appdirs==1.4.4
    - astunparse==1.6.3
    - attrs==21.2.0
    - cached-property==1.5.2
    - cachetools==4.2.2
    - charset-normalizer==2.0.5
    - clang==5.0
    - cycler==0.10.0
    - cython==0.29.24
    - dm-tree==0.1.6
    - flatbuffers==1.12
    - gast==0.4.0
    - google-auth==1.35.0
    - google-auth-oauthlib==0.4.6
    - google-pasta==0.2.0
    - graphsurgeon==0.4.5
    - grpcio==1.34.1
    - h5py==3.1.0
    - idna==3.2
    - importlib-metadata==4.8.1
    - keras==2.6.0
    - keras-nightly==2.5.0.dev2021032900
    - keras-preprocessing==1.1.2
    - kiwisolver==1.3.2
    - lxml==4.6.3
    - mako==1.1.5
    - markdown==3.3.4
    - markupsafe==2.0.1
    - matplotlib==3.4.3
    - neural-structured-learning==1.3.1
    - numpy==1.19.5
    - oauthlib==3.1.1
    - onnx==1.8.1
    - onnx-graphsurgeon==0.3.11
    - onnxruntime==1.8.0
    - opt-einsum==3.3.0
    - pillow==8.3.2
    - protobuf==3.18.0
    - pyasn1==0.4.8
    - pyasn1-modules==0.2.8
    - pycocotools==2.0
    - pycuda==2021.1
    - pyparsing==2.4.7
    - python-dateutil==2.8.2
    - pytools==2021.2.8
    - pyyaml==5.4.1
    - requests==2.26.0
    - requests-oauthlib==1.3.0
    - rsa==4.7.2
    - scipy==1.7.1
    - six==1.15.0
    - tensorboard==2.6.0
    - tensorboard-data-server==0.6.1
    - tensorboard-plugin-wit==1.8.0
    - tensorflow-addons==0.14.0
    - tensorflow-estimator==2.5.0
    - tensorflow-gpu==2.5.0
    - tensorflow-hub==0.12.0
    - tensorflow-model-optimization==0.6.0
    - tensorrt==8.0.1.6
    - termcolor==1.1.0
    - tf2onnx==1.8.1
    - typeguard==2.12.1
    - typing-extensions==3.7.4.3
    - uff==0.6.9
    - urllib3==1.26.6
    - werkzeug==2.0.1
    - wrapt==1.12.1
    - zipp==3.5.0
prefix: /home/yichao/miniconda3/envs/tensorRT-efficientdet

三、注意事项

If you are performing this conversion to run inference on the edge, such as for NVIDIA Jetson devices, it might be easier to do the ONNX conversion on a PC first.
The interface to map a .pb model into .uff is called graphsurgeon
自定义 tensorRT plugin 插件

四、相关介绍

预处理与后处理

Pre-processing and inference (output: class confidence and anchor-based bounding-box offset prediction of 5 different scales feature map).
Post-processing written in TensorFlow 1.x api which can be transfer to tensorflow graphdef. Post-process pipeline is described below:
- merge all output into one tensor;
- select topK confidence result;
- decode class-label and bbox coordinate from the TopK result;
- doing NMS for each class;

正常流程

方式一：model.index/model.meta/model.data-00000-of-00001 --> frozen_inference_graph.pb --> model.onnx --> model.trt --> infer。
方式二：frozen_inference_graph.pb -> UFF -> TensorRT。

五、关键步骤

1. 下载官方 efficientdet ，安装相关的依赖包

git clone https://github.com/google/automl.git

cd /home/yichao/MyDocuments/automl/efficientdet

pip install -r requirements.txt

pip install pyyaml
pip install tensorflow-model-optimization
pip install matplotlib

2. 下载预训练模型并解压

efficientdet-d0

# /home/yichao/Downloads/efficientdet-d0

├── efficientdet-d0
│   ├── checkpoint
│   ├── d0_coco_test-dev2017.txt
│   ├── d0_coco_val.txt
│   ├── model.data-00000-of-00001
│   ├── model.index
│   └── model.meta

3. 模型转换：转成pb文件

cd /home/yichao/MyDocuments/automl/efficientdet

python model_inspect.py \
    --runmode saved_model \
    --model_name efficientdet-d0 \
    --ckpt_path /home/yichao/Downloads/efficientdet-d0 \
    --saved_model_dir /home/yichao/Downloads/saved_model

# /home/yichao/Downloads/saved_model

├── saved_model
│   ├── efficientdet-d0_frozen.pb
│   ├── saved_model.pb
│   └── variables

# saved_model.pb，7.3MB

4. 下载 tensorRT 官方提供的 EfficientDet，生成onnx模型

git clone https://github.com/NVIDIA/TensorRT.git

cd /home/yichao/MyDocuments/TensorRT/samples/python/efficientdet

pip install -r requirements.txt

pip install onnx-graphsurgeon --index-url https://pypi.ngc.nvidia.com

cd /home/yichao/MyDocuments/TensorRT/samples/python/efficientdet

python create_onnx.py \
    --input_shape '1,512,512,3' \
    --saved_model /home/yichao/Downloads/saved_model \
    --onnx /home/yichao/Downloads/saved_model_onnx/model.onnx

# /home/yichao/Downloads/saved_model_onnx

├── saved_model_onnx
│   └── model.onnx

# model.onnx，16.5MB

5. 生成engine引擎

tensorRT FP32

cd /home/yichao/MyDocuments/TensorRT/samples/python/efficientdet

python build_engine.py \
    --onnx /home/yichao/Downloads/saved_model_onnx/model.onnx \
    --engine /home/yichao/Downloads/saved_model_trt_fp32/engine.trt \
    --precision fp32

(tensorRT-efficientdet) yichao@yichao:~/MyDocuments/TensorRT/samples/python/efficientdet$ time python build_engine.py \
>     --onnx /home/yichao/Downloads/saved_model_onnx/model.onnx \
>     --engine /home/yichao/Downloads/saved_model_trt_fp32/engine.trt \
>     --precision fp32
[TensorRT] INFO: [MemUsageChange] Init CUDA: CPU +320, GPU +0, now: CPU 343, GPU 476 (MiB)
[TensorRT] WARNING: onnx2trt_utils.cpp:364: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[TensorRT] INFO: No importer registered for op: EfficientNMS_TRT. Attempting to import as plugin.
[TensorRT] INFO: Searching for plugin: EfficientNMS_TRT, plugin_version: 1, plugin_namespace: 
[TensorRT] INFO: Successfully created plugin: EfficientNMS_TRT
INFO:EngineBuilder:Network Description
INFO:EngineBuilder:Input 'image_arrays:0' with shape (1, 512, 512, 3) and dtype DataType.FLOAT
INFO:EngineBuilder:Output 'num_detections' with shape (1, 1) and dtype DataType.INT32
INFO:EngineBuilder:Output 'detection_boxes' with shape (1, 100, 4) and dtype DataType.FLOAT
INFO:EngineBuilder:Output 'detection_scores' with shape (1, 100) and dtype DataType.FLOAT
INFO:EngineBuilder:Output 'detection_classes' with shape (1, 100) and dtype DataType.INT32
INFO:EngineBuilder:Building fp32 Engine in /home/yichao/Downloads/saved_model_trt_fp32/engine.trt
build_engine.py:203: DeprecationWarning: Use build_serialized_network instead.
  with self.builder.build_engine(self.network, self.config) as engine, open(engine_path, "wb") as f:
[TensorRT] INFO: [MemUsageSnapshot] Builder begin: CPU 361 MiB, GPU 476 MiB
[TensorRT] WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.5.1 but loaded cuBLAS/cuBLAS LT 11.1.0
[TensorRT] INFO: [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +357, GPU +148, now: CPU 721, GPU 624 (MiB)
[TensorRT] INFO: [MemUsageChange] Init cuDNN: CPU +118, GPU +148, now: CPU 839, GPU 772 (MiB)
[TensorRT] WARNING: TensorRT was linked against cuDNN 8.2.1 but loaded cuDNN 8.0.5
[TensorRT] WARNING: Detected invalid timing cache, setup a local cache instead
[TensorRT] INFO: Detected 1 inputs and 4 output network tensors.
[TensorRT] INFO: Total Host Persistent Memory: 321392
[TensorRT] INFO: Total Device Persistent Memory: 16182272
[TensorRT] INFO: Total Scratch Memory: 106066176
[TensorRT] INFO: [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 12 MiB, GPU 8 MiB
[TensorRT] WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.5.1 but loaded cuBLAS/cuBLAS LT 11.1.0
[TensorRT] INFO: [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 1267, GPU 970 (MiB)
[TensorRT] INFO: [MemUsageChange] Init cuDNN: CPU +1, GPU +8, now: CPU 1268, GPU 978 (MiB)
[TensorRT] WARNING: TensorRT was linked against cuDNN 8.2.1 but loaded cuDNN 8.0.5
[TensorRT] INFO: [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +0, now: CPU 1268, GPU 962 (MiB)
[TensorRT] INFO: [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +0, now: CPU 1267, GPU 944 (MiB)
[TensorRT] INFO: [MemUsageSnapshot] Builder end: CPU 1266 MiB, GPU 944 MiB
INFO:EngineBuilder:Serializing engine to file: /home/yichao/Downloads/saved_model_trt_fp32/engine.trt

real	1m51.712s
user	1m33.469s
sys	0m3.339s

# engine.trt，29.7MB

显存占用情况

Sat Sep 18 11:39:08 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.63.01    Driver Version: 470.63.01    CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0  On |                  N/A |
| 27%   42C    P0    19W /  75W |   1750MiB /  3903MiB |      2%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1465      G   /usr/lib/xorg/Xorg                285MiB |
|    0   N/A  N/A     20266      C   python                           1461MiB |
+-----------------------------------------------------------------------------+

tensorRT FP16

cd /home/yichao/MyDocuments/TensorRT/samples/python/efficientdet

python build_engine.py \
    --onnx /home/yichao/Downloads/saved_model_onnx/model.onnx \
    --engine /home/yichao/Downloads/saved_model_trt_fp16/engine.trt \
    --precision fp16

(tensorRT-efficientdet) yichao@yichao:~/MyDocuments/TensorRT/samples/python/efficientdet$ time python build_engine.py \
>     --onnx /home/yichao/Downloads/saved_model_onnx/model.onnx \
>     --engine /home/yichao/Downloads/saved_model_trt_fp16/engine.trt \
>     --precision fp16
[TensorRT] INFO: [MemUsageChange] Init CUDA: CPU +320, GPU +0, now: CPU 343, GPU 496 (MiB)
[TensorRT] WARNING: onnx2trt_utils.cpp:364: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[TensorRT] INFO: No importer registered for op: EfficientNMS_TRT. Attempting to import as plugin.
[TensorRT] INFO: Searching for plugin: EfficientNMS_TRT, plugin_version: 1, plugin_namespace: 
[TensorRT] INFO: Successfully created plugin: EfficientNMS_TRT
INFO:EngineBuilder:Network Description
INFO:EngineBuilder:Input 'image_arrays:0' with shape (1, 512, 512, 3) and dtype DataType.FLOAT
INFO:EngineBuilder:Output 'num_detections' with shape (1, 1) and dtype DataType.INT32
INFO:EngineBuilder:Output 'detection_boxes' with shape (1, 100, 4) and dtype DataType.FLOAT
INFO:EngineBuilder:Output 'detection_scores' with shape (1, 100) and dtype DataType.FLOAT
INFO:EngineBuilder:Output 'detection_classes' with shape (1, 100) and dtype DataType.INT32
INFO:EngineBuilder:Building fp16 Engine in /home/yichao/Downloads/saved_model_trt_fp16/engine.trt
build_engine.py:203: DeprecationWarning: Use build_serialized_network instead.
  with self.builder.build_engine(self.network, self.config) as engine, open(engine_path, "wb") as f:
[TensorRT] INFO: [MemUsageSnapshot] Builder begin: CPU 361 MiB, GPU 496 MiB
[TensorRT] WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.5.1 but loaded cuBLAS/cuBLAS LT 11.1.0
[TensorRT] INFO: [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +357, GPU +148, now: CPU 721, GPU 644 (MiB)
[TensorRT] INFO: [MemUsageChange] Init cuDNN: CPU +118, GPU +148, now: CPU 839, GPU 792 (MiB)
[TensorRT] WARNING: TensorRT was linked against cuDNN 8.2.1 but loaded cuDNN 8.0.5
[TensorRT] WARNING: Detected invalid timing cache, setup a local cache instead
[TensorRT] INFO: Detected 1 inputs and 4 output network tensors.
[TensorRT] INFO: Total Host Persistent Memory: 295824
[TensorRT] INFO: Total Device Persistent Memory: 8877568
[TensorRT] INFO: Total Scratch Memory: 88388864
[TensorRT] INFO: [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 12 MiB, GPU 8 MiB
[TensorRT] WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.5.1 but loaded cuBLAS/cuBLAS LT 11.1.0
[TensorRT] INFO: [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +1, GPU +8, now: CPU 1294, GPU 984 (MiB)
[TensorRT] INFO: [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 1294, GPU 992 (MiB)
[TensorRT] WARNING: TensorRT was linked against cuDNN 8.2.1 but loaded cuDNN 8.0.5
[TensorRT] INFO: [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +0, now: CPU 1294, GPU 976 (MiB)
[TensorRT] INFO: [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +0, now: CPU 1293, GPU 958 (MiB)
[TensorRT] INFO: [MemUsageSnapshot] Builder end: CPU 1290 MiB, GPU 958 MiB
INFO:EngineBuilder:Serializing engine to file: /home/yichao/Downloads/saved_model_trt_fp16/engine.trt

real	12m14.915s
user	10m47.976s
sys	0m17.289s

# engine.trt，23.4MB

显存占用情况

Sat Sep 18 11:20:07 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.63.01    Driver Version: 470.63.01    CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0  On |                  N/A |
| 28%   42C    P0    19W /  75W |   2640MiB /  3903MiB |     10%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1465      G   /usr/lib/xorg/Xorg                305MiB |
|    0   N/A  N/A     19261      C   python                           2331MiB |
+-----------------------------------------------------------------------------+

tensorRT INT8

cd /home/yichao/MyDocuments/TensorRT/samples/python/efficientdet

python build_engine.py \
    --onnx /home/yichao/Downloads/saved_model_onnx/model.onnx \
    --engine /home/yichao/Downloads/saved_model_trt_int8/engine.trt \
    --precision int8 \
    --calib_input /home/yichao/Downloads/coco_calib \
    --calib_cache /home/yichao/Downloads/calibration/calibration.cache

(tensorRT-efficientdet) yichao@yichao:~/MyDocuments/TensorRT/samples/python/efficientdet$ time python build_engine.py \
>     --onnx /home/yichao/Downloads/saved_model_onnx/model.onnx \
>     --engine /home/yichao/Downloads/saved_model_trt_int8/engine.trt \
>     --precision int8 \
>     --calib_input /home/yichao/Downloads/coco_calib \
>     --calib_cache /home/yichao/Downloads/calibration/calibration.cache
[TensorRT] INFO: [MemUsageChange] Init CUDA: CPU +320, GPU +0, now: CPU 343, GPU 476 (MiB)
[TensorRT] WARNING: onnx2trt_utils.cpp:364: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[TensorRT] INFO: No importer registered for op: EfficientNMS_TRT. Attempting to import as plugin.
[TensorRT] INFO: Searching for plugin: EfficientNMS_TRT, plugin_version: 1, plugin_namespace: 
[TensorRT] INFO: Successfully created plugin: EfficientNMS_TRT
INFO:EngineBuilder:Network Description
INFO:EngineBuilder:Input 'image_arrays:0' with shape (1, 512, 512, 3) and dtype DataType.FLOAT
INFO:EngineBuilder:Output 'num_detections' with shape (1, 1) and dtype DataType.INT32
INFO:EngineBuilder:Output 'detection_boxes' with shape (1, 100, 4) and dtype DataType.FLOAT
INFO:EngineBuilder:Output 'detection_scores' with shape (1, 100) and dtype DataType.FLOAT
INFO:EngineBuilder:Output 'detection_classes' with shape (1, 100) and dtype DataType.INT32
INFO:EngineBuilder:Building int8 Engine in /home/yichao/Downloads/saved_model_trt/engine.trt
build_engine.py:203: DeprecationWarning: Use build_serialized_network instead.
  with self.builder.build_engine(self.network, self.config) as engine, open(engine_path, "wb") as f:
[TensorRT] INFO: [MemUsageSnapshot] Builder begin: CPU 361 MiB, GPU 500 MiB
[TensorRT] WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.5.1 but loaded cuBLAS/cuBLAS LT 11.1.0
[TensorRT] INFO: [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +357, GPU +148, now: CPU 720, GPU 648 (MiB)
[TensorRT] INFO: [MemUsageChange] Init cuDNN: CPU +128, GPU +148, now: CPU 848, GPU 796 (MiB)
[TensorRT] WARNING: TensorRT was linked against cuDNN 8.2.1 but loaded cuDNN 8.0.5
[TensorRT] INFO: Detected 1 inputs and 4 output network tensors.
[TensorRT] INFO: Total Host Persistent Memory: 34896
[TensorRT] INFO: Total Device Persistent Memory: 0
[TensorRT] INFO: Total Scratch Memory: 106066176
[TensorRT] INFO: [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 0 MiB, GPU 8 MiB
[TensorRT] WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.5.1 but loaded cuBLAS/cuBLAS LT 11.1.0
[TensorRT] INFO: [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 1258, GPU 990 (MiB)
[TensorRT] INFO: [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 1258, GPU 998 (MiB)
[TensorRT] WARNING: TensorRT was linked against cuDNN 8.2.1 but loaded cuDNN 8.0.5
[TensorRT] INFO: [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +0, now: CPU 1258, GPU 982 (MiB)
[TensorRT] INFO: [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +0, now: CPU 1257, GPU 964 (MiB)
[TensorRT] INFO: [MemUsageSnapshot] ExecutionContext creation begin: CPU 1257 MiB, GPU 964 MiB
[TensorRT] WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.5.1 but loaded cuBLAS/cuBLAS LT 11.1.0
[TensorRT] INFO: [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +1, GPU +10, now: CPU 1258, GPU 974 (MiB)
[TensorRT] INFO: [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 1258, GPU 982 (MiB)
[TensorRT] WARNING: TensorRT was linked against cuDNN 8.2.1 but loaded cuDNN 8.0.5
[TensorRT] INFO: [MemUsageSnapshot] ExecutionContext creation end: CPU 1258 MiB, GPU 1138 MiB
[TensorRT] INFO: Starting Calibration.
INFO:EngineBuilder:Calibrating image 8 / 1000
[TensorRT] INFO:   Calibrated batch 0 in 0.299519 seconds.
INFO:EngineBuilder:Calibrating image 16 / 1000
...
[TensorRT] INFO:   Calibrated batch 123 in 0.275864 seconds.
INFO:EngineBuilder:Calibrating image 1000 / 1000
[TensorRT] INFO:   Calibrated batch 124 in 0.289 seconds.
INFO:EngineBuilder:Finished calibration batches
[TensorRT] INFO: [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +0, now: CPU 1349, GPU 1122 (MiB)
[TensorRT] INFO:   Post Processing Calibration data in 81.7665 seconds.
[TensorRT] INFO: Calibration completed in 129.299 seconds.
[TensorRT] INFO: Writing Calibration Cache for calibrator: TRT-8001-EntropyCalibration2
INFO:EngineBuilder:Writing calibration cache data to: /home/yichao/Downloads/calibration/calibration.cache
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 82) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 177) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 180) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 263) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 266) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 270) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 273) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 277) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 284) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 297) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 300) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 305) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 308) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 314) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 329) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 332) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 338) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 341) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 356) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 361) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 366) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 370) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 385) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 390) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 395) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 398) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 413) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 418) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 421) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 425) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 434) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 440) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 449) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 453) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 462) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 465) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 469) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 478) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 481) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 485) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 494) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 497) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 501) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 510) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 515) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 524) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 530) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 539) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 545) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 554) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 560) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 572) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 600) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 644) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: Missing scale and zero-point for tensor (Unnamed Layer* 692) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[TensorRT] WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.5.1 but loaded cuBLAS/cuBLAS LT 11.1.0
[TensorRT] INFO: [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +10, now: CPU 1322, GPU 958 (MiB)
[TensorRT] INFO: [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 1322, GPU 966 (MiB)
[TensorRT] WARNING: TensorRT was linked against cuDNN 8.2.1 but loaded cuDNN 8.0.5
[TensorRT] WARNING: Detected invalid timing cache, setup a local cache instead
[TensorRT] INFO: Detected 1 inputs and 4 output network tensors.
[TensorRT] INFO: Total Host Persistent Memory: 306784
[TensorRT] INFO: Total Device Persistent Memory: 6846464
[TensorRT] INFO: Total Scratch Memory: 88388864
[TensorRT] INFO: [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 20 MiB, GPU 158 MiB
[TensorRT] WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.5.1 but loaded cuBLAS/cuBLAS LT 11.1.0
[TensorRT] INFO: [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 1369, GPU 986 (MiB)
[TensorRT] INFO: [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 1369, GPU 994 (MiB)
[TensorRT] WARNING: TensorRT was linked against cuDNN 8.2.1 but loaded cuDNN 8.0.5
[TensorRT] INFO: [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +0, now: CPU 1369, GPU 978 (MiB)
[TensorRT] INFO: [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +0, now: CPU 1368, GPU 962 (MiB)
[TensorRT] INFO: [MemUsageSnapshot] Builder end: CPU 1364 MiB, GPU 962 MiB
INFO:EngineBuilder:Serializing engine to file: /home/yichao/Downloads/saved_model_trt/engine.trt

real	25m27.581s
user	23m12.319s
sys	0m43.315s

# engine.trt，22.2MB

显存占用情况

Sat Sep 18 09:48:55 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.63.01    Driver Version: 470.63.01    CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0  On |                  N/A |
| 28%   42C    P0    18W /  75W |   1784MiB /  3903MiB |      1%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1465      G   /usr/lib/xorg/Xorg                285MiB |
|    0   N/A  N/A     15585      C   python                           1495MiB |
+-----------------------------------------------------------------------------+

6. 推理

tensorRT FP32

cd /home/yichao/MyDocuments/TensorRT/samples/python/efficientdet

python infer.py \
    --engine /home/yichao/Downloads/saved_model_trt_fp32/engine.trt \
    --input /home/yichao/Downloads/coco_calib \
    --output /home/yichao/Downloads/infer_fp32

(tensorRT-efficientdet) yichao@yichao:/media/yichao/蚁巢文件/YOYOFile/YOYOFile/PyProjects/TensorRT/samples/python/efficientdet$ time python infer.py     --engine /media/yichao/蚁巢文件/YOYOFile/EfficientDet/saved_model_trt_fp32/engine.trt     --input /media/yichao/蚁巢文件/YOYOFile/EfficientDet/coco_calib     --output /media/yichao/蚁巢文件/YOYOFile/EfficientDet/infer_fp32
len(batch_images): ['/media/yichao/蚁巢文件/YOYOFile/EfficientDet/coco_calib/COCO_train2014_000000000009.jpg']
time_1: 511Image 1 / 1000
len(images): 1
detections: [[{'ymin': -10.355606079101562, 'xmin': 299.74897384643555, 'ymax': 240.54559707641602, 'xmax': 637.5265502929688, 'score': 0.6842542, 'class': 50}, {'ymin': 80.77005386352539, 'xmin': -1.2041282653808594, 'ymax': 436.90185546875, 'xmax': 459.16343688964844, 'score': 0.62642604, 'class': 50}, {'ymin': 188.0276870727539, 'xmin': 27.04761505126953, 'ymax': 471.3081741333008, 'xmax': 601.4738464355469, 'score': 0.57705814, 'class': 50}, {'ymin': 222.76105880737305, 'xmin': 249.59787368774414, 'ymax': 473.05328369140625, 'xmax': 562.2936248779297, 'score': 0.5430005, 'class': 55}, {'ymin': 69.5292615890503, 'xmin': 388.29090118408203, 'ymax': 141.84276580810547, 'xmax': 470.1576614379883, 'score': 0.45278752, 'class': 54}, {'ymin': 6.308698654174805, 'xmin': 19.28295135498047, 'ymax': 293.5408401489258, 'xmax': 427.21588134765625, 'score': 0.4251722, 'class': 50}]]
time_2: 159
len(batch_images): ['/media/yichao/蚁巢文件/YOYOFile/EfficientDet/coco_calib/COCO_train2014_000000000151.jpg']
time_1: 15 Image 2 / 1000
len(images): 1
detections: [[{'ymin': 326.95018768310547, 'xmin': 211.58906936645508, 'ymax': 365.53333282470703, 'xmax': 250.12731552124023, 'score': 0.84729725, 'class': 12}, {'ymin': 38.47867965698242, 'xmin': 200.59566497802734, 'ymax': 634.6967315673828, 'xmax': 473.6790084838867, 'score': 0.7619643, 'class': 6}]]
time_2: 108
...
...
len(batch_images): ['/media/yichao/蚁巢文件/YOYOFile/EfficientDet/coco_calib/COCO_train2014_000000580385.jpg']
time_1: 14 Image 998 / 1000
len(images): 1
detections: [[{'ymin': 47.34233856201172, 'xmin': 125.20095825195312, 'ymax': 359.3656539916992, 'xmax': 525.5948638916016, 'score': 0.9364282, 'class': 6}]]
time_2: 117
len(batch_images): ['/media/yichao/蚁巢文件/YOYOFile/EfficientDet/coco_calib/COCO_train2014_000000581218.jpg']
time_1: 15 Image 999 / 1000
len(images): 1
detections: [[{'ymin': 309.59455490112305, 'xmin': 241.9521713256836, 'ymax': 348.5172653198242, 'xmax': 271.9435501098633, 'score': 0.6911016, 'class': 4}, {'ymin': 147.70557403564453, 'xmin': 271.3910484313965, 'ymax': 188.7246322631836, 'xmax': 300.4056739807129, 'score': 0.6408937, 'class': 4}, {'ymin': 269.51610565185547, 'xmin': 260.4315185546875, 'ymax': 310.3202438354492, 'xmax': 289.44976806640625, 'score': 0.63147306, 'class': 4}, {'ymin': 234.6424102783203, 'xmin': 308.0194282531738, 'ymax': 274.24530029296875, 'xmax': 339.48490142822266, 'score': 0.52650064, 'class': 4}, {'ymin': 273.0849838256836, 'xmin': 298.9488983154297, 'ymax': 313.12862396240234, 'xmax': 325.66226959228516, 'score': 0.4886203, 'class': 4}, {'ymin': 233.1707763671875, 'xmin': 282.28904724121094, 'ymax': 268.44730377197266, 'xmax': 312.5209426879883, 'score': 0.41873103, 'class': 4}]]
time_2: 169
len(batch_images): ['/media/yichao/蚁巢文件/YOYOFile/EfficientDet/coco_calib/COCO_train2014_000000581766.jpg']
time_1: 15 Image 1000 / 1000
len(images): 1
detections: [[{'ymin': 148.56109023094177, 'xmin': 204.04024422168732, 'ymax': 283.86542201042175, 'xmax': 297.4337339401245, 'score': 0.8775655, 'class': 69}, {'ymin': 153.7458896636963, 'xmin': 17.941415309906006, 'ymax': 285.7228219509125, 'xmax': 127.41473317146301, 'score': 0.81447136, 'class': 69}, {'ymin': 146.68381214141846, 'xmin': 373.2069432735443, 'ymax': 278.8565158843994, 'xmax': 481.57575726509094, 'score': 0.75143397, 'class': 69}]]
time_2: 111

get_batch time: 158031


Finished Processing

real	2m40.346s
user	2m30.624s
sys	0m4.989s

结论：tensorRT FP 32平均耗时 15ms/张，即 66fps。

显存占用情况

Sat Sep 18 11:44:30 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.63.01    Driver Version: 470.63.01    CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0  On |                  N/A |
| 27%   40C    P0    20W /  75W |   1146MiB /  3903MiB |     14%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1465      G   /usr/lib/xorg/Xorg                305MiB |
|    0   N/A  N/A     20470      C   python                            837MiB |
+-----------------------------------------------------------------------------+

tensorRT FP16

cd /home/yichao/MyDocuments/TensorRT/samples/python/efficientdet

python infer.py \
    --engine /home/yichao/Downloads/saved_model_trt_fp16/engine.trt \
    --input /home/yichao/Downloads/coco_calib \
    --output /home/yichao/Downloads/infer_fp16

(tensorRT-efficientdet) yichao@yichao:/media/yichao/蚁巢文件/YOYOFile/YOYOFile/P
yProjects/TensorRT/samples/python/efficientdet$ time python infer.py \
>     --engine /media/yichao/蚁巢文件/YOYOFile/EfficientDet/saved_model_trt_fp16/engine.trt \
>     --input /media/yichao/蚁巢文件/YOYOFile/EfficientDet/coco_calib \
>     --output /media/yichao/蚁巢文件/YOYOFile/EfficientDet/infer_fp16
len(batch_images): ['/media/yichao/蚁巢文件/YOYOFile/EfficientDet/coco_calib/COCO_train2014_000000000009.jpg']
time_1: 3582mage 1 / 1000
len(images): 1
detections: [[{'ymin': -10.390625, 'xmin': 299.84375, 'ymax': 240.625, 'xmax': 637.5, 'score': 0.6845703, 'class': 50}, {'ymin': 80.9375, 'xmin': -1.25, 'ymax': 436.875, 'xmax': 459.0625, 'score': 0.625, 'class': 50}, {'ymin': 188.125, 'xmin': 27.1875, 'ymax': 471.25, 'xmax': 601.25, 'score': 0.57666016, 'class': 50}, {'ymin': 222.8125, 'xmin': 249.6875, 'ymax': 472.8125, 'xmax': 562.1875, 'score': 0.54296875, 'class': 55}, {'ymin': 69.53125, 'xmin': 388.125, 'ymax': 141.875, 'xmax': 470.0, 'score': 0.453125, 'class': 54}, {'ymin': 6.328125, 'xmin': 19.375, 'ymax': 293.4375, 'xmax': 427.1875, 'score': 0.42456055, 'class': 50}]]
time_2: 230
len(batch_images): ['/media/yichao/蚁巢文件/YOYOFile/EfficientDet/coco_calib/COCO_train2014_000000000151.jpg']
time_1: 10 Image 2 / 1000
len(images): 1
detections: [[{'ymin': 326.875, 'xmin': 211.5625, 'ymax': 365.625, 'xmax': 250.0, 'score': 0.84716797, 'class': 12}, {'ymin': 38.28125, 'xmin': 200.625, 'ymax': 635.0, 'xmax': 473.75, 'score': 0.7636719, 'class': 6}]]
time_2: 100
...
...
len(batch_images): ['/media/yichao/蚁巢文件/YOYOFile/EfficientDet/coco_calib/COCO_train2014_000000580385.jpg']
time_1: 11 Image 998 / 1000
len(images): 1
detections: [[{'ymin': 47.265625, 'xmin': 125.15625, 'ymax': 359.375, 'xmax': 525.625, 'score': 0.9370117, 'class': 6}]]
time_2: 114
len(batch_images): ['/media/yichao/蚁巢文件/YOYOFile/EfficientDet/coco_calib/COCO_train2014_000000581218.jpg']
time_1: 11 Image 999 / 1000
len(images): 1
detections: [[{'ymin': 309.6875, 'xmin': 241.875, 'ymax': 348.4375, 'xmax': 271.875, 'score': 0.6899414, 'class': 4}, {'ymin': 147.734375, 'xmin': 271.40625, 'ymax': 188.75, 'xmax': 300.46875, 'score': 0.640625, 'class': 4}, {'ymin': 269.375, 'xmin': 260.46875, 'ymax': 310.3125, 'xmax': 289.53125, 'score': 0.6303711, 'class': 4}, {'ymin': 234.53125, 'xmin': 308.125, 'ymax': 274.21875, 'xmax': 339.375, 'score': 0.52734375, 'class': 4}, {'ymin': 273.125, 'xmin': 299.0625, 'ymax': 313.125, 'xmax': 325.625, 'score': 0.48632812, 'class': 4}, {'ymin': 233.125, 'xmin': 282.1875, 'ymax': 268.4375, 'xmax': 312.5, 'score': 0.41870117, 'class': 4}]]
time_2: 170
len(batch_images): ['/media/yichao/蚁巢文件/YOYOFile/EfficientDet/coco_calib/COCO_train2014_000000581766.jpg']
time_1: 10 Image 1000 / 1000
len(images): 1
detections: [[{'ymin': 148.5595703125, 'xmin': 204.1015625, 'ymax': 283.69140625, 'xmax': 297.36328125, 'score': 0.8774414, 'class': 69}, {'ymin': 153.80859375, 'xmin': 17.9443359375, 'ymax': 285.64453125, 'xmax': 127.44140625, 'score': 0.81396484, 'class': 69}, {'ymin': 146.728515625, 'xmin': 373.291015625, 'ymax': 278.80859375, 'xmax': 481.689453125, 'score': 0.7519531, 'class': 69}]]
time_2: 113

get_batch time: 158293


Finished Processing

real	3m5.879s
user	2m28.488s
sys	0m5.079s

结论：tesorRT FP 16平均耗时 11ms/张，即 90fps。

显存占用情况

Sat Sep 18 11:22:24 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.63.01    Driver Version: 470.63.01    CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0  On |                  N/A |
| 27%   40C    P0    20W /  75W |   1094MiB /  3903MiB |     11%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1465      G   /usr/lib/xorg/Xorg                305MiB |
|    0   N/A  N/A     19579      C   python                            785MiB |
+-----------------------------------------------------------------------------+

tensorRT INT8

cd /home/yichao/MyDocuments/TensorRT/samples/python/efficientdet

python infer.py \
    --engine /home/yichao/Downloads/saved_model_trt_int8/engine.trt \
    --input /home/yichao/Downloads/coco_calib \
    --output /home/yichao/Downloads/infer_int8

(tensorRT-efficientdet) yichao@yichao:/media/yichao/蚁巢文件/YOYOFile/YOYOFile/PyProjects/TensorRT/samples/python/efficientdet$ time python infer.py \
>     --engine /media/yichao/蚁巢文件/YOYOFile/EfficientDet/saved_model_trt_int8/engine.trt \
>     --input /media/yichao/蚁巢文件/YOYOFile/EfficientDet/coco_calib \
>     --output /media/yichao/蚁巢文件/YOYOFile/EfficientDet/infer_int8
len(batch_images): ['/media/yichao/蚁巢文件/YOYOFile/EfficientDet/coco_calib/COCO_train2014_000000000009.jpg']
time_1: 499Image 1 / 1000
len(images): 1
detections: [[{'ymin': 224.6875, 'xmin': 251.40625, 'ymax': 476.5625, 'xmax': 561.875, 'score': 0.57666016, 'class': 55}, {'ymin': 1.09375, 'xmin': 311.5625, 'ymax': 238.75, 'xmax': 630.3125, 'score': 0.50146484, 'class': 50}, {'ymin': 68.125, 'xmin': -3.4375, 'ymax': 458.125, 'xmax': 527.5, 'score': 0.49902344, 'class': 50}]]
time_2: 165
len(batch_images): ['/media/yichao/蚁巢文件/YOYOFile/EfficientDet/coco_calib/COCO_train2014_000000000151.jpg']
time_1: 6g Image 2 / 1000
len(images): 1
detections: [[{'ymin': 18.28125, 'xmin': 204.6875, 'ymax': 639.375, 'xmax': 505.9375, 'score': 0.5888672, 'class': 6}]]
time_2: 104
...
...
len(batch_images): ['/media/yichao/蚁巢文件/YOYOFile/EfficientDet/coco_calib/COCO_train2014_000000580385.jpg']
time_1: 6g Image 998 / 1000
len(images): 1
detections: [[{'ymin': 45.0, 'xmin': 122.03125, 'ymax': 361.875, 'xmax': 525.625, 'score': 0.9316406, 'class': 6}]]
time_2: 120
len(batch_images): ['/media/yichao/蚁巢文件/YOYOFile/EfficientDet/coco_calib/COCO_train2014_000000581218.jpg']
time_1: 6g Image 999 / 1000
len(images): 1
detections: [[]]
time_2: 177
len(batch_images): ['/media/yichao/蚁巢文件/YOYOFile/EfficientDet/coco_calib/COCO_train2014_000000581766.jpg']
time_1: 6g Image 1000 / 1000
len(images): 1
detections: [[{'ymin': 147.705078125, 'xmin': 204.1015625, 'ymax': 284.423828125, 'xmax': 297.36328125, 'score': 0.7817383, 'class': 69}, {'ymin': 151.3671875, 'xmin': 17.63916015625, 'ymax': 281.982421875, 'xmax': 128.173828125, 'score': 0.7714844, 'class': 69}, {'ymin': 145.99609375, 'xmin': 373.779296875, 'ymax': 277.34375, 'xmax': 482.666015625, 'score': 0.7338867, 'class': 69}]]
time_2: 108

get_batch time: 153597


Finished Processing

real	2m37.937s
user	2m25.907s
sys	0m2.906s

结论：tensorRT INT8平均耗时 7ms/张，即 142fps。

显存占用情况

Tue Sep 28 18:35:05 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.63.01    Driver Version: 470.63.01    CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0  On |                  N/A |
| 28%   43C    P0    18W /  75W |    935MiB /  3903MiB |     10%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1553      G   /usr/lib/xorg/Xorg                156MiB |
|    0   N/A  N/A      7282      C   python                            775MiB |
+-----------------------------------------------------------------------------+

7. 评估指标

tensorRT FP32

cd /home/yichao/MyDocuments/TensorRT/samples/python/efficientdet

python eval_coco.py \
    --engine /home/yichao/Downloads/saved_model_trt_fp32/engine.trt \
    --input /home/yichao/Downloads/COCO/val2017 \
    --annotations /home/yichao/Downloads/COCO/annotations/instances_val2017.json \
    --automl_path /home/yichao/MyDocuments/automl

(tensorRT-efficientdet) yichao@yichao:~/MyDocuments/TensorRT/samples/python/efficientdet$ time python eval_coco.py \
>     --engine /home/yichao/Downloads/saved_model_trt_fp32/engine.trt \
>     --input /home/yichao/Downloads/COCO/val2017 \
>     --annotations /home/yichao/Downloads/COCO/annotations/instances_val2017.json \
>     --automl_path /home/yichao/MyDocuments/automl
2021-09-18 11:47:43.730693: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
Processing Image 5000 / 5000
loading annotations into memory...
Done (t=0.60s)
creating index...
index created!
Loading and preparing results...
Converting ndarray to lists...
(18145, 7)
0/18145
DONE (t=0.13s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=8.57s).
Accumulating evaluation results...
DONE (t=1.46s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.282
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.397
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.315
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.053
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.319
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.494
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.242
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.315
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.316
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.052
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.349
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.562

real	3m11.921s
user	2m3.908s
sys	0m17.163s

显存占用情况

Sat Sep 18 11:47:57 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.63.01    Driver Version: 470.63.01    CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0  On |                  N/A |
| 27%   45C    P0    38W /  75W |   1146MiB /  3903MiB |     34%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1465      G   /usr/lib/xorg/Xorg                305MiB |
|    0   N/A  N/A     20581      C   python                            837MiB |
+-----------------------------------------------------------------------------+

tensorRT FP16

cd /home/yichao/MyDocuments/TensorRT/samples/python/efficientdet

python eval_coco.py \
    --engine /home/yichao/Downloads/saved_model_trt_fp16/engine.trt \
    --input /home/yichao/Downloads/COCO/val2017 \
    --annotations /home/yichao/Downloads/COCO/annotations/instances_val2017.json \
    --automl_path /home/yichao/MyDocuments/automl

(tensorRT-efficientdet) yichao@yichao:~/MyDocuments/TensorRT/samples/python/efficientdet$ python eval_coco.py \
>     --engine /home/yichao/Downloads/saved_model_trt_fp16/engine.trt \
>     --input /home/yichao/Downloads/COCO/val2017 \
>     --annotations /home/yichao/Downloads/COCO/annotations/instances_val2017.json \
>     --automl_path /home/yichao/MyDocuments/automl
2021-09-17 18:20:02.568124: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
Processing Image 5000 / 5000
loading annotations into memory...
Done (t=0.63s)
creating index...
index created!
Loading and preparing results...
Converting ndarray to lists...
(18146, 7)
0/18146
DONE (t=0.13s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=8.83s).
Accumulating evaluation results...
DONE (t=1.46s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.282
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.397
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.315
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.053
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.319
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.494
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.242
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.316
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.316
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.052
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.349
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.562

显存占用情况

Sat Sep 18 11:27:42 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.63.01    Driver Version: 470.63.01    CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0  On |                  N/A |
| 27%   43C    P0    28W /  75W |   1094MiB /  3903MiB |     36%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1465      G   /usr/lib/xorg/Xorg                305MiB |
|    0   N/A  N/A     19780      C   python                            785MiB |
+-----------------------------------------------------------------------------+

tensorRT INT8

cd /home/yichao/MyDocuments/TensorRT/samples/python/efficientdet

python eval_coco.py \
    --engine /home/yichao/Downloads/saved_model_trt_int8/engine.trt \
    --input /home/yichao/Downloads/COCO/val2017 \
    --annotations /home/yichao/Downloads/COCO/annotations/instances_val2017.json \
    --automl_path /home/yichao/MyDocuments/automl

(tensorRT-efficientdet) yichao@yichao:~/MyDocuments/TensorRT/samples/python/efficientdet$ time python eval_coco.py \
>     --engine /home/yichao/Downloads/saved_model_trt_int8/engine.trt \
>     --input /home/yichao/Downloads/COCO/val2017 \
>     --annotations /home/yichao/Downloads/COCO/annotations/instances_val2017.json \
>     --automl_path /home/yichao/MyDocuments/automl
2021-09-18 10:20:40.389173: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
Processing Image 5000 / 5000
loading annotations into memory...
Done (t=0.65s)
creating index...
index created!
Loading and preparing results...
Converting ndarray to lists...
(14482, 7)
0/14482
DONE (t=0.13s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=7.95s).
Accumulating evaluation results...
DONE (t=1.38s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.227
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.318
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.254
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.024
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.231
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.430
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.199
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.254
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.254
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.022
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.245
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.492

real	2m40.845s
user	1m45.329s
sys	0m7.975s

显存占用情况

Sat Sep 18 10:23:19 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.63.01    Driver Version: 470.63.01    CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0  On |                  N/A |
| 27%   42C    P0    18W /  75W |   1279MiB /  3903MiB |      3%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1465      G   /usr/lib/xorg/Xorg                305MiB |
|    0   N/A  N/A     16664      C   python                            972MiB |
+-----------------------------------------------------------------------------+

8. 比较原生的tensorflow和tensorRT

tensorRT FP32

cd /home/yichao/MyDocuments/TensorRT/samples/python/efficientdet

python compare_tf.py \
    --engine /home/yichao/Downloads/saved_model_trt_fp32/engine.trt \
    --saved_model /home/yichao/Downloads/saved_model \
    --input /home/yichao/Downloads/coco_calib \
    --output /home/yichao/Downloads/output_fp32

(tensorRT-efficientdet) yichao@yichao:~/MyDocuments/TensorRT/samples/python/efficientdet$ time python compare_tf.py \
>     --engine /home/yichao/Downloads/saved_model_trt_fp32/engine.trt \
>     --saved_model /home/yichao/Downloads/saved_model \
>     --input /home/yichao/Downloads/coco_calib \
>     --output /home/yichao/Downloads/output_fp32
2021-09-18 11:51:59.627768: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
2021-09-18 11:52:05.397166: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcuda.so.1
2021-09-18 11:52:05.397267: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-09-18 11:52:05.397505: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: NVIDIA GeForce GTX 1650 computeCapability: 7.5
coreClock: 1.68GHz coreCount: 14 deviceMemorySize: 3.81GiB deviceMemoryBandwidth: 119.24GiB/s
2021-09-18 11:52:05.397524: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
2021-09-18 11:52:05.397555: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11
2021-09-18 11:52:05.397577: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11
2021-09-18 11:52:05.466217: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcufft.so.10
2021-09-18 11:52:05.466545: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcurand.so.10
2021-09-18 11:52:05.466862: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcusolver.so.11'; dlerror: libcusolver.so.11: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: :/usr/local/cuda/lib64:/home/yichao/360Downloads/TensorRT-8.0.1.6/lib:/home/yichao/miniconda3/envs/tensorRT-yolov5/lib/libmkl_intel_lp64.so
2021-09-18 11:52:05.495741: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusparse.so.11
2021-09-18 11:52:05.495966: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8
2021-09-18 11:52:05.496021: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1766] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2021-09-18 11:52:05.803052: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-09-18 11:52:05.803585: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-09-18 11:52:05.803601: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264]      
2021-09-18 11:52:12.462233: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
2021-09-18 11:52:12.505942: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 2899885000 Hz
Processing 100 / 100 images (TensorFlow)
Processing 100 / 100 images (TensorRT)
Processing 100 / 100 images (Visualization)

real	0m52.613s
user	1m23.590s
sys	0m3.248s

显存占用情况

Sat Sep 18 11:52:37 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.63.01    Driver Version: 470.63.01    CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0  On |                  N/A |
| 33%   44C    P0    18W /  75W |   1146MiB /  3903MiB |      2%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1465      G   /usr/lib/xorg/Xorg                305MiB |
|    0   N/A  N/A     20720      C   python                            837MiB |
+-----------------------------------------------------------------------------+

tensorRT FP16

cd /home/yichao/MyDocuments/TensorRT/samples/python/efficientdet

python compare_tf.py \
    --engine /home/yichao/Downloads/saved_model_trt_fp16/engine.trt \
    --saved_model /home/yichao/Downloads/saved_model \
    --input /home/yichao/Downloads/coco_calib \
    --output /home/yichao/Downloads/output_fp16

(tensorRT-efficientdet) yichao@yichao:/media/yichao/蚁巢文件/YOYOFile/YOYOFile/PyProjects/TensorRT/samples/python/efficientdet$ time python compare_tf.py     --engine /media/yichao/蚁巢文件/YOYOFile/EfficientDet/saved_model_trt_fp16/engine.trt     --saved_model /media/yichao/蚁巢文件/YOYOFile/EfficientDet/saved_model     --input /media/yichao/蚁巢文件/YOYOFile/EfficientDet/coco_calib     --output /media/yichao/蚁巢文件/YOYOFile/EfficientDet/output_fp16
2021-10-22 16:23:31.638538: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
2021-10-22 16:23:32.897147: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcuda.so.1
2021-10-22 16:23:32.897287: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-10-22 16:23:32.897550: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: NVIDIA GeForce GTX 1650 computeCapability: 7.5
coreClock: 1.68GHz coreCount: 14 deviceMemorySize: 3.81GiB deviceMemoryBandwidth: 119.24GiB/s
2021-10-22 16:23:32.897571: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
2021-10-22 16:23:32.897604: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11
2021-10-22 16:23:32.897625: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11
2021-10-22 16:23:32.898508: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcufft.so.10
2021-10-22 16:23:32.898545: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcurand.so.10
2021-10-22 16:23:32.898692: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcusolver.so.11'; dlerror: libcusolver.so.11: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: :/usr/local/cuda/lib64:/usr/local/cuda-10.2/lib64:/home/yichao/MyDocuments/PyProjects/software/TensorRT-7.0.0.11/lib:/home/yichao/miniconda3/envs/tensorRT-yolov5/lib/libmkl_intel_lp64.so
2021-10-22 16:23:32.899198: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusparse.so.11
2021-10-22 16:23:32.899227: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8
2021-10-22 16:23:32.899236: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1766] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2021-10-22 16:23:33.134158: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-10-22 16:23:33.134560: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-10-22 16:23:33.134572: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264]      
2021-10-22 16:23:40.393064: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
2021-10-22 16:23:40.438255: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 2899885000 Hz
len(batch_images): ['/media/yichao/蚁巢文件/YOYOFile/EfficientDet/coco_calib/COCO_train2014_000000000009.jpg']
...
...
len(batch_images): ['/media/yichao/蚁巢文件/YOYOFile/EfficientDet/coco_calib/COCO_train2014_000000061328.jpg']
len(batch_images): ['/media/yichao/蚁巢文件/YOYOFile/EfficientDet/coco_calib/COCO_train2014_000000061399.jpg']
Processing 100 / 100 images (TensorFlow)
get_batch time: 15944


time_1: 15944
len(batch_images): ['/media/yichao/蚁巢文件/YOYOFile/EfficientDet/coco_calib/COCO_train2014_000000000009.jpg']
...
...
len(batch_images): ['/media/yichao/蚁巢文件/YOYOFile/EfficientDet/coco_calib/COCO_train2014_000000061328.jpg']
len(batch_images): ['/media/yichao/蚁巢文件/YOYOFile/EfficientDet/coco_calib/COCO_train2014_000000061399.jpg']
Processing 100 / 100 images (TensorRT)
get_batch time: 2519


time_2: 2519
Processing 100 / 100 images (Visualization)

real	0m35.517s
user	1m5.616s
sys	0m3.689s

数据一：

测试数据	分辨率	TensorFlow耗时/ms	TensorRT耗时/ms	加速比
COCO，100张	640x480	16658	2577	6.5
COCO，100张	640x480	15944	2519	6.3
COCO，100张	640x480	16982	2559	6.6
COCO，100张	640x480	16026	2533	6.3

数据二：

测试数据	分辨率	TensorFlow耗时/ms	TensorRT耗时/ms	加速比
COCO，100张	640x480	16089	5268	3
COCO，100张	640x480	11888	2509	4.7
COCO，100张	640x480	12012	3008	4
COCO，100张	640x480	11803	3183	3.7
COCO，100张	640x480	11776	3140	3.7

测试数据	分辨率	TensorFlow耗时/ms	TensorRT耗时/ms	加速比
person_horse，100张	1280x720	12891	3452	3.7
person_horse，100张	1280x720	14989	3480	4.3
person_horse，100张	1280x720	15565	3515	4.4
person_horse，100张	1280x720	12883	3456	3.7

注意： $加速比=\frac{TensorFlow耗时}{TensorRT耗时}$

显存占用情况

Sat Sep 18 11:33:21 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.63.01    Driver Version: 470.63.01    CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0  On |                  N/A |
| 30%   41C    P0    18W /  75W |   1094MiB /  3903MiB |      2%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1465      G   /usr/lib/xorg/Xorg                305MiB |
|    0   N/A  N/A     20023      C   python                            785MiB |
+-----------------------------------------------------------------------------+

tensorRT INT8

cd /home/yichao/MyDocuments/TensorRT/samples/python/efficientdet

python compare_tf.py \
    --engine /home/yichao/Downloads/saved_model_trt_int8/engine.trt \
    --saved_model /home/yichao/Downloads/saved_model \
    --input /home/yichao/Downloads/coco_calib \
    --output /home/yichao/Downloads/output_int8

(tensorRT-efficientdet) yichao@yichao:~/MyDocuments/TensorRT/samples/python/efficientdet$ time python compare_tf.py \
>     --engine /home/yichao/Downloads/saved_model_trt_int8/engine.trt \
>     --saved_model /home/yichao/Downloads/saved_model \
>     --input /home/yichao/Downloads/coco_calib \
>     --output /home/yichao/Downloads/output_int8
2021-09-18 09:25:28.721580: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
2021-09-18 09:25:34.855431: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcuda.so.1
2021-09-18 09:25:34.855675: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-09-18 09:25:34.856186: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: NVIDIA GeForce GTX 1650 computeCapability: 7.5
coreClock: 1.68GHz coreCount: 14 deviceMemorySize: 3.81GiB deviceMemoryBandwidth: 119.24GiB/s
2021-09-18 09:25:34.856231: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
2021-09-18 09:25:34.856312: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11
2021-09-18 09:25:34.856366: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11
2021-09-18 09:25:34.938908: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcufft.so.10
2021-09-18 09:25:34.939165: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcurand.so.10
2021-09-18 09:25:34.939584: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcusolver.so.11'; dlerror: libcusolver.so.11: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: :/usr/local/cuda/lib64:/home/yichao/360Downloads/TensorRT-8.0.1.6/lib:/home/yichao/miniconda3/envs/tensorRT-yolov5/lib/libmkl_intel_lp64.so
2021-09-18 09:25:34.974029: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusparse.so.11
2021-09-18 09:25:34.974272: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8
2021-09-18 09:25:34.974324: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1766] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2021-09-18 09:25:35.443757: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-09-18 09:25:35.445486: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-09-18 09:25:35.445558: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264]      
2021-09-18 09:25:42.378910: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
2021-09-18 09:25:42.434208: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 2899885000 Hz
Processing 100 / 100 images (TensorFlow)
Processing 100 / 100 images (TensorRT)
Processing 100 / 100 images (Visualization)

real	0m53.631s
user	1m19.881s
sys	0m3.897s

显存占用情况

Sat Sep 18 10:27:43 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.63.01    Driver Version: 470.63.01    CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0  On |                  N/A |
| 27%   40C    P0    18W /  75W |   1094MiB /  3903MiB |      4%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1465      G   /usr/lib/xorg/Xorg                305MiB |
|    0   N/A  N/A     16827      C   python                            785MiB |
+-----------------------------------------------------------------------------+

你可能感兴趣的:(深度学习,深度学习,tensorRT,EfficientDet)

机器学习与深度学习间关系与区别 ℒℴѵℯ心·动ꦿ໊ོ꫞ 人工智能学习深度学习 python
一、机器学习概述定义机器学习（MachineLearning,ML）是一种通过数据驱动的方法，利用统计学和计算算法来训练模型，使计算机能够从数据中学习并自动进行预测或决策。机器学习通过分析大量数据样本，识别其中的模式和规律，从而对新的数据进行判断。其核心在于通过训练过程，让模型不断优化和提升其预测准确性。主要类型1.监督学习（SupervisedLearning）监督学习是指在训练数据集中包含输入
将cmd中命令输出保存为txt文本文件落难Coder Windows cmd window
最近深度学习本地的训练中我们常常要在命令行中运行自己的代码，无可厚非，我们有必要保存我们的炼丹结果，但是复制命令行输出到txt是非常麻烦的，其实Windows下的命令行为我们提供了相应的操作。其基本的调用格式就是：运行指令>输出到的文件名称或者具体保存路径测试下，我打开cmd并且ping一下百度：pingwww.baidu.com>./data.txt看下相同目录下data.txt的输出：如果你再
推荐3家毕业AI论文可五分钟一键生成！文末附免费教程！小猪包333 写论文人工智能 AI写作深度学习计算机视觉
在当前的学术研究和写作领域，AI论文生成器已经成为许多研究人员和学生的重要工具。这些工具不仅能够帮助用户快速生成高质量的论文内容，还能进行内容优化、查重和排版等操作。以下是三款值得推荐的AI论文生成器：千笔-AIPassPaper、懒人论文以及AIPaperPass。千笔-AIPassPaper千笔-AIPassPaper是一款基于深度学习和自然语言处理技术的AI写作助手，旨在帮助用户快速生成高质
AI大模型的架构演进与最新发展季风泯灭的季节 AI大模型应用技术二人工智能架构
随着深度学习的发展，AI大模型（LargeLanguageModels,LLMs）在自然语言处理、计算机视觉等领域取得了革命性的进展。本文将详细探讨AI大模型的架构演进，包括从Transformer的提出到GPT、BERT、T5等模型的历史演变，并探讨这些模型的技术细节及其在现代人工智能中的核心作用。一、基础模型介绍：Transformer的核心原理Transformer架构的背景在Transfo
[实践应用] 深度学习之模型性能评估指标 YuanDaima2048 深度学习工具使用深度学习人工智能损失函数性能评估 pytorch python 机器学习
文章总览：YuanDaiMa2048博客文章总览深度学习之模型性能评估指标分类任务回归任务排序任务聚类任务生成任务其他介绍在机器学习和深度学习领域，评估模型性能是一项至关重要的任务。不同的学习任务需要不同的性能指标来衡量模型的有效性。以下是对一些常见任务及其相应的性能评估指标的详细解释和总结。分类任务分类任务是指模型需要将输入数据分配到预定义的类别或标签中。以下是分类任务中常用的性能指标：准确率(
[实践应用] 深度学习之优化器 YuanDaima2048 深度学习工具使用 pytorch 深度学习人工智能机器学习 python 优化器
文章总览：YuanDaiMa2048博客文章总览深度学习之优化器1.随机梯度下降（SGD）2.动量优化（Momentum）3.自适应梯度（Adagrad）4.自适应矩估计（Adam）5.RMSprop总结其他介绍在深度学习中，优化器用于更新模型的参数，以最小化损失函数。常见的优化函数有很多种，下面是几种主流的优化器及其特点、原理和PyTorch实现：1.随机梯度下降（SGD）原理:随机梯度下降通过
生成式地图制图 Bwywb_3 深度学习机器学习深度学习生成对抗网络
生成式地图制图（GenerativeCartography）是一种利用生成式算法和人工智能技术自动创建地图的技术。它结合了传统的地理信息系统（GIS）技术与现代生成模型（如深度学习、GANs等），能够根据输入的数据自动生成符合需求的地图。这种方法在城市规划、虚拟环境设计、游戏开发等多个领域具有应用前景。主要特点：自动化生成：通过算法和模型，系统能够根据输入的地理或空间数据自动生成地图，而无需人工逐
吴恩达深度学习笔记(30)-正则化的解释极客Array
正则化（Regularization）深度学习可能存在过拟合问题——高方差，有两个解决方法，一个是正则化，另一个是准备更多的数据，这是非常可靠的方法，但你可能无法时时刻刻准备足够多的训练数据或者获取更多数据的成本很高，但正则化通常有助于避免过拟合或减少你的网络误差。如果你怀疑神经网络过度拟合了数据，即存在高方差问题，那么最先想到的方法可能是正则化，另一个解决高方差的方法就是准备更多数据，这也是非常
个人学习笔记7-6：动手学深度学习pytorch版-李沐浪子L 深度学习深度学习笔记计算机视觉 python 人工智能神经网络 pytorch
#人工智能##深度学习##语义分割##计算机视觉##神经网络#计算机视觉13.11全卷积网络全卷积网络（fullyconvolutionalnetwork，FCN）采用卷积神经网络实现了从图像像素到像素类别的变换。引入l转置卷积（transposedconvolution）实现的，输出的类别预测与输入图像在像素级别上具有一一对应关系：通道维的输出即该位置对应像素的类别预测。13.11.1构造模型下
深度学习-点击率预估-研究论文2024-09-14速读 sp_fyf_2024 深度学习人工智能
深度学习-点击率预估-研究论文2024-09-14速读1.DeepTargetSessionInterestNetworkforClick-ThroughRatePredictionHZhong,JMa,XDuan,SGu,JYao-2024InternationalJointConferenceonNeuralNetworks,2024深度目标会话兴趣网络用于点击率预测摘要：这篇文章提出了一种新
损失函数与反向传播 Star_. PyTorch pytorch 深度学习 python
损失函数定义与作用损失函数(lossfunction)在深度学习领域是用来计算搭建模型预测的输出值和真实值之间的误差。1.损失函数越小越好2.计算实际输出与目标之间的差距3.为更新输出提供依据（反向传播)常见的损失函数回归常见的损失函数有：均方差（MeanSquaredError，MSE）、平均绝对误差（MeanAbsoluteErrorLoss，MAE）、HuberLoss是一种将MSE与MAE
【深度学习】训练过程中一个OOM的问题，太难查了 weixin_40293999 深度学习深度学习人工智能
现象：各位大佬又遇到过ubuntu的这个问题么？现象是在训练过程中，ssh上不去了，能ping通，没死机，但是ubunutu的pc侧的显示器，鼠标啥都不好用了。只能重启。问题原因：OOM了95G，尼玛！！！！pytorch爆内存了，然后journald假死了，在journald被watchdog干掉之后，系统就崩溃了。这种规模的爆内存一般，即使被oomkill了，也要卡半天的，确实会这样，能不能配
云服务业界动态简报-20180128 Captain7
一、青云青云QingCloud推出深度学习平台DeepLearningonQingCloud，包含了主流的深度学习框架及数据科学工具包，通过QingCloudAppCenter一键部署交付，可以让算法工程师和数据科学家快速构建深度学习开发环境，将更多的精力放在模型和算法调优。二、腾讯云1.腾讯云正式发布腾讯专有云TCE(TencentCloudEnterprise)矩阵，涵盖企业版、大数据版、AI
机器学习VS深度学习 nfgo 机器学习
机器学习（MachineLearning,ML）和深度学习（DeepLearning,DL）是人工智能（AI）的两个子领域，它们有许多相似之处，但在技术实现和应用范围上也有显著区别。下面从几个方面对两者进行区分：1.概念层面机器学习：是让计算机通过算法从数据中自动学习和改进的技术。它依赖于手动设计的特征和数学模型来进行学习，常用的模型有决策树、支持向量机、线性回归等。深度学习：是机器学习的一个子领
大数据毕业设计hadoop+spark+hive知识图谱租房数据分析可视化大屏租房推荐系统 58同城租房爬虫房源推荐系统房价预测系统计算机毕业设计机器学习深度学习人工智能 2401_84572577 程序员大数据 hadoop 人工智能
做了那么多年开发，自学了很多门编程语言，我很明白学习资源对于学一门新语言的重要性，这些年也收藏了不少的Python干货，对我来说这些东西确实已经用不到了，但对于准备自学Python的人来说，或许它就是一个宝藏，可以给你省去很多的时间和精力。别在网上瞎学了，我最近也做了一些资源的更新，只要你是我的粉丝，这期福利你都可拿走。我先来介绍一下这些东西怎么用，文末抱走。（1）Python所有方向的学习路线（
深度学习-13-小语言模型之SmolLM的使用皮皮冰燃深度学习深度学习
文章附录1SmolLM概述1.1SmolLM简介1.2下载模型2运行2.1在CPU/GPU/多GPU上运行模型2.2使用torch.bfloat162.3通过位和字节的量化版本3应用示例4问题及解决4.1attention_mask和pad_token_id报错4.2max_new_tokens=205参考附录1SmolLM概述1.1SmolLM简介SmolLM是一系列尖端小型语言模型，提供三种规
基于深度学习的农作物病害检测 SEU-WYL 深度学习dnn 深度学习人工智能
基于深度学习的农作物病害检测利用卷积神经网络（CNN）、生成对抗网络（GAN）、Transformer等深度学习技术，自动识别和分类农作物的病害，帮助农业工作者提高作物管理效率、减少损失。1.农作物病害检测的挑战病害种类繁多：农作物病害的类型多样，不同病害在同一作物上的表现差异很大，同时同一种病害在不同生长阶段的症状也可能不同。环境影响：天气、光照、湿度等外部环境因素会影响农作物的表现，使得病害检
基于深度学习的文本引导的图像编辑 SEU-WYL 深度学习dnn 深度学习人工智能
基于深度学习的文本引导的图像编辑（Text-GuidedImageEditing）是一种通过自然语言文本指令对图像进行编辑或修改的技术。它结合了图像生成和自然语言处理（NLP）的最新进展，使用户能够通过描述性文本对图像内容进行精确的调整和操控。1.文本引导的图像编辑的挑战文本和图像之间的对齐：如何将文本中的语义信息准确地映射到图像中的特定区域或元素是一个关键挑战。这涉及到多模态数据的对齐和理解。编
深度学习--对抗生成网络（GAN, Generative Adversarial Network） Ambition_LAO 深度学习生成对抗网络
对抗生成网络（GAN,GenerativeAdversarialNetwork）是一种深度学习模型，由IanGoodfellow等人在2014年提出。GAN主要用于生成数据，通过两个神经网络相互对抗，来生成以假乱真的新数据。以下是对GAN的详细阐述，包括其概念、作用、核心要点、实现过程、代码实现和适用场景。1.概念GAN由两个神经网络组成：生成器（Generator）和判别器（Discrimina
深度学习：怎么看pth文件的参数奥利给少年深度学习人工智能
.pth文件是PyTorch模型的权重文件，它通常包含了训练好的模型的参数。要查看或使用这个文件，你可以按照以下步骤操作：1.确保你有模型的定义你需要有创建这个.pth文件时所用的模型的代码。这意味着你需要有模型的类定义和架构。2.加载模型权重使用PyTorch的load_state_dict方法来加载权重。这里是如何操作的：importtorchimporttorch.nnasnn#定义模型结构
chatgpt赋能python：如何在Python中安装Keras库？ turensu ChatGpt python chatgpt keras 计算机
如何在Python中安装Keras库？Keras是一个简单易用的神经网络库，由FrançoisChollet编写。它在Python编程语言中实现了深度学习的功能，可以使您更轻松地构建和试验不同类型的神经网络。如果您是一名Python开发人员，肯定会想知道如何在您的Python项目中安装Keras库。在本文中，我们将向您展示如何安装和配置Keras库。步骤1：安装Python要使用Keras库，您需
如何理解深度学习的训练过程奋斗的草莓熊深度学习人工智能 python scikit-learn virtualenv numpy pandas
文章目录1.训练是干什么？2.预训练模型进行训练，主要更改的是预训练模型的什么东西？1.训练是干什么？以yolov5为例子，训练的目的是把一组输入猫狗图像放到神经网络中，得到一个输出模型，这个模型下次可以直接用来识别哪个是猫，哪个是狗2.预训练模型进行训练，主要更改的是预训练模型的什么东西？超参数（Hyperparameters）：这是模型结构中定义的参数，比如：卷积核大小（kernel_size
Keras深度学习框架入门及实战指南司莹嫣Maude
Keras深度学习框架入门及实战指南keraskeras-team/keras:是一个基于Python的深度学习库，它没有使用数据库。适合用于深度学习任务的开发和实现，特别是对于需要使用Python深度学习库的场景。特点是深度学习库、Python、无数据库。项目地址:https://gitcode.com/gh_mirrors/ke/keras一、项目介绍Keras简介Keras是一款高级神经网络
深度学习驱动的车牌识别：技术演进与未来挑战逼子歌深度学习车牌识别神经网络字符识别 YOLO 卷积神经网络
一、引言1.1研究背景在当今社会，智能交通系统的发展日益重要，而车牌识别作为其关键组成部分，发挥着至关重要的作用。车牌识别技术广泛应用于交通管理、停车场管理、安防监控等领域。在交通管理中，它可以用于车辆识别、交通违法监控和车流统计等，提高交通管理的效率和准确性。在停车场管理中，实现车辆的自动识别和收费，提升管理和服务水平。在安防监控领域，可用于追踪嫌疑人及犯罪行为。深度学习的出现为车牌识别带来了重
每天五分钟玩转深度学习PyTorch：模型参数优化器torch.optim 幻风_huanfeng 深度学习框架pytorch 深度学习 pytorch 人工智能神经网络机器学习优化算法
本文重点在机器学习或者深度学习中，我们需要通过修改参数使得损失函数最小化(或最大化)，优化算法就是一种调整模型参数更新的策略。在pytorch中定义了优化器optim，我们可以使用它调用封装好的优化算法，然后传递给它神经网络模型参数，就可以对模型进行优化。本文是学习第6步(优化器)，参考链接pytorch的学习路线随机梯度下降算法在深度学习和机器学习中，梯度下降算法是最常用的参数更新方法，它的公式
什么是AIGC？有哪些免费工具？ chent_某位 AIGC
AIGC（AIGeneratedContent），即“人工智能生成内容”，是指通过人工智能技术自动生成各种类型的数字内容。AIGC让机器能够根据输入的信息或数据生成符合人类需求的文本、图像、音频、视频等内容，极大提高了内容创作的效率。AIGC的背景与起源随着深度学习和自然语言处理技术的快速发展，人工智能已经不再局限于简单的任务，如分类、预测和数据分析，而是具备了生成内容的能力。生成式AI模型，如O
transformer架构(Transformer Architecture)原理与代码实战案例讲解 AI架构设计之禅大数据AI人工智能 Python入门实战计算科学神经计算深度学习神经网络大数据人工智能大型语言模型 AI AGI LLM Java Python 架构设计 Agent RPA
transformer架构(TransformerArchitecture)原理与代码实战案例讲解关键词：Transformer,自注意力机制,编码器-解码器,预训练,微调,NLP,机器翻译作者：禅与计算机程序设计艺术/ZenandtheArtofComputerProgramming1.背景介绍1.1问题的由来自然语言处理（NLP）领域的发展经历了从规则驱动到统计驱动再到深度学习驱动的三个阶段。
如何有效的学习AI大模型？ Python程序员罗宾学习人工智能语言模型自然语言处理架构
学习AI大模型是一个系统性的过程，涉及到多个学科的知识。以下是一些建议，帮助你更有效地学习AI大模型：基础知识储备：数学基础：学习线性代数、概率论、统计学和微积分等，这些是理解机器学习算法的数学基础。编程技能：掌握至少一种编程语言，如Python，因为大多数AI模型都是用Python实现的。理论学习：机器学习基础：了解监督学习、非监督学习、强化学习等基本概念。深度学习：学习神经网络的基本结构，如卷
【深度学习】【OnnxRuntime】【Python】模型转化、环境搭建以及模型部署的详细教程牙牙要健康深度学习 onnx onnxruntime 深度学习 python 人工智能
【深度学习】【OnnxRuntime】【Python】模型转化、环境搭建以及模型部署的详细教程提示:博主取舍了很多大佬的博文并亲测有效,分享笔记邀大家共同学习讨论文章目录【深度学习】【OnnxRuntime】【Python】模型转化、环境搭建以及模型部署的详细教程前言模型转换--pytorch转onnxWindows平台搭建依赖环境onnxruntime调用onnx模型ONNXRuntime推理核
基于深度学习的多模态信息检索 SEU-WYL 深度学习dnn 深度学习人工智能
基于深度学习的多模态信息检索（MultimodalInformationRetrieval,MMIR）是指利用深度学习技术，从包含多种模态（如文本、图像、视频、音频等）的数据集中检索出满足用户查询意图的相关信息。这种方法不仅可以处理单一模态的数据，还可以在多种模态之间建立关联，从而更准确地满足用户需求。1.多模态信息检索的挑战异构数据表示：多模态数据通常具有不同的特征和表示形式（如文本的词嵌入与图
JVM StackMapTable 属性的作用及理解 lijingyao8206 jvm 字节码 Class文件 StackMapTable
在Java 6版本之后JVM引入了栈图(Stack Map Table)概念。为了提高验证过程的效率，在字节码规范中添加了Stack Map Table属性，以下简称栈图，其方法的code属性中存储了局部变量和操作数的类型验证以及字节码的偏移量。也就是一个method需要且仅对应一个Stack Map Table。在Java 7版
回调函数调用方法百合不是茶 java
最近在看大神写的代码时,.发现其中使用了很多的回调 ,以前只是在学习的时候经常用到 ,现在写个笔记记录一下代码很简单: MainDemo :调用方法得到方法的返回结果
[时间机器]制造时间机器需要一些材料 comsci 制造
根据我的计算和推测,要完全实现制造一台时间机器,需要某些我们这个世界不存在的物质和材料... 甚至可以这样说,这种材料和物质,我们在反应堆中也无法获得......
开口埋怨不如闭口做事邓集海邓集海做人做事工作
“开口埋怨，不如闭口做事。”不是名人名言，而是一个普通父亲对儿子的训导。但是，因为这句训导，这位普通父亲却造就了一个名人儿子。这位普通父亲造就的名人儿子，叫张明正。　　　　张明正出身贫寒，读书时成绩差，常挨老师批评。高中毕业，张明正连普通大学的分数线都没上。高考成绩出来后，平时开口怨这怨那的张明正，不从自身找原因，而是不停地埋怨自己家庭条件不好、埋怨父母没有给他创造良好的学习环境。　　　　
jQuery插件开发全解析，类级别与对象级别开发 IT独行者 jquery 开发插件　函数
jQuery插件的开发包括两种：一种是类级别的插件开发，即给 jQuery添加新的全局函数，相当于给 jQuery类本身添加方法。 jQuery的全局函数就是属于 jQuery命名空间的函数，另一种是对象级别的插件开发，即给 jQuery对象添加方法。下面就两种函数的开发做详细的说明。 1 、类级别的插件开发类级别的插件开发最直接的理解就是给jQuer
Rome解析Rss 413277409 Rome解析Rss
import java.net.URL; import java.util.List; import org.junit.Test; import com.sun.syndication.feed.synd.SyndCategory; import com.sun.syndication.feed.synd.S
RSA加密解密无量加密解密 rsa
RSA加密解密代码代码有待整理 package com.tongbanjie.commons.util; import java.security.Key; import java.security.KeyFactory; import java.security.KeyPair; import java.security.KeyPairGenerat
linux 软件安装遇到的问题 aichenglong linux 遇到的问题 ftp
1 ftp配置中遇到的问题 500 OOPS: cannot change directory 出现该问题的原因:是SELinux安装机制的问题.只要disable SELinux就可以了修改方法:1 修改/etc/selinux/config 中SELINUX=disabled 2 source /etc
面试心得 alafqq 面试
最近面试了好几家公司。记录下；支付宝，面试我的人胖胖的，看着人挺好的；博彦外包的职位，面试失败；阿里金融，面试官人也挺和善，只不过我让他吐血了。。。由于印象比较深，记录下； 1，自我介绍 2，说下八种基本类型；（算上string。楼主才答了3种，哈哈，string其实不是基本类型，是引用类型） 3，什么是包装类，包装类的优点； 4，平时看过什么书？NND，什么书都没看过。。照样
java的多态性探讨百合不是茶 java
java的多态性是指main方法在调用属性的时候类可以对这一属性做出反应的情况 //package 1; class A{ public void test(){ System.out.println("A"); } } class D extends A{ public void test(){ S
网络编程基础篇之JavaScript-学习笔记 bijian1013 JavaScript
1.documentWrite <html> <head> <script language="JavaScript"> document.write("这是电脑网络学校"); document.close(); </script> </h
探索JUnit4扩展：深入Rule bijian1013 JUnit Rule 单元测试
本文将进一步探究Rule的应用，展示如何使用Rule来替代@BeforeClass，@AfterClass，@Before和@After的功能。在上一篇中提到，可以使用Rule替代现有的大部分Runner扩展，而且也不提倡对Runner中的withBefores()，withAfte
[CSS]CSS浮动十五条规则 bit1129 css
这些浮动规则，主要是参考CSS权威指南关于浮动规则的总结，然后添加一些简单的例子以验证和理解这些规则。 1. 所有的页面元素都可以浮动 2. 一个元素浮动后，会成为块级元素，比如<span>,a, strong等都会变成块级元素 3.一个元素左浮动，会向最近的块级父元素的左上角移动，直到浮动元素的左外边界碰到块级父元素的左内边界；如果这个块级父元素已经有浮动元素停靠了
【Kafka六】Kafka Producer和Consumer多Broker、多Partition场景 bit1129 partition
0.Kafka服务器配置 3个broker 1个topic，6个partition，副本因子是2 2个consumer，每个consumer三个线程并发读取 1. Producer package kafka.examples.multibrokers.producers; import java.util.Properties; import java.util.
zabbix_agentd.conf配置文件详解 ronin47 zabbix 配置文件
Aliaskey的别名，例如 Alias=ttlsa.userid:vfs.file.regexp[/etc/passwd,^ttlsa:.:([0-9]+),,,,\1]，或者ttlsa的用户ID。你可以使用key：vfs.file.regexp[/etc/passwd,^ttlsa:.: ([0-9]+),,,,\1]，也可以使用ttlsa.userid。备注: 别名不能重复，但是可以有多个
java--19.用矩阵求Fibonacci数列的第N项 bylijinnan fibonacci
参考了网上的思路，写了个Java版的： public class Fibonacci { final static int[] A={1,1,1,0}; public static void main(String[] args) { int n=7; for(int i=0;i<=n;i++){ int f=fibonac
Netty源码学习-LengthFieldBasedFrameDecoder bylijinnan java netty
先看看LengthFieldBasedFrameDecoder的官方API http://docs.jboss.org/netty/3.1/api/org/jboss/netty/handler/codec/frame/LengthFieldBasedFrameDecoder.html API举例说明了LengthFieldBasedFrameDecoder的解析机制，如下：实
AES加密解密 chicony 加密解密
AES加解密算法，使用Base64做转码以及辅助加密： package com.wintv.common; import javax.crypto.Cipher; import javax.crypto.spec.IvParameterSpec; import javax.crypto.spec.SecretKeySpec; import sun.misc.BASE64Decod
文件编码格式转换 ctrain 编码格式
package com.test; import java.io.File; import java.io.FileInputStream; import java.io.FileOutputStream; import java.io.IOException; import java.io.InputStream; import java.io.OutputStream;
mysql 在linux客户端插入数据中文乱码 daizj mysql 中文乱码
1、查看系统客户端，数据库，连接层的编码查看方法： http://daizj.iteye.com/blog/2174993 进入mysql，通过如下命令查看数据库编码方式： mysql> show variables like 'character_set_%'; +--------------------------+------
好代码是廉价的代码 dcj3sjt126com 程序员读书
长久以来我一直主张：好代码是廉价的代码。当我跟做开发的同事说出这话时，他们的第一反应是一种惊愕，然后是将近一个星期的嘲笑，把它当作一个笑话来讲。当他们走近看我的表情、知道我是认真的时，才收敛一点。当最初的惊愕消退后，他们会用一些这样的话来反驳： “好代码不廉价，好代码是采用经过数十年计算机科学研究和积累得出的最佳实践设计模式和方法论建立起来的精心制作的程序代码。” 我只
Android网络请求库——android-async-http dcj3sjt126com android
在iOS开发中有大名鼎鼎的ASIHttpRequest库，用来处理网络请求操作，今天要介绍的是一个在Android上同样强大的网络请求库android-async-http，目前非常火的应用Instagram和Pinterest的Android版就是用的这个网络请求库。这个网络请求库是基于Apache HttpClient库之上的一个异步网络请求处理库，网络处理均基于Android的非UI线程，通
ORACLE 复习笔记之SQL语句的优化 eksliang SQL优化 Oracle sql语句优化 SQL语句的优化
转载请出自出处：http://eksliang.iteye.com/blog/2097999 SQL语句的优化总结如下 sql语句的优化可以按照如下六个步骤进行：合理使用索引避免或者简化排序消除对大表的扫描避免复杂的通配符匹配调整子查询的性能 EXISTS和IN运算符下面我就按照上面这六个步骤分别进行总结：
浅析：Android 嵌套滑动机制（NestedScrolling） gg163 android 移动开发滑动机制嵌套
谷歌在发布安卓 Lollipop版本之后，为了更好的用户体验，Google为Android的滑动机制提供了NestedScrolling特性 NestedScrolling的特性可以体现在哪里呢？ 比如你使用了Toolbar，下面一个ScrollView，向上滚
使用hovertree菜单作为后台导航 hvt JavaScript jquery .net hovertree asp.net
hovertree是一个jquery菜单插件，官方网址：http://keleyi.com/jq/hovertree/ ，可以登录该网址体验效果。 0.1.3版本：http://keleyi.com/jq/hovertree/demo/demo.0.1.3.htm hovertree插件包含文件： http://keleyi.com/jq/hovertree/css
SVG 教程（二）矩形天梯梦 svg
SVG <rect> SVG Shapes SVG有一些预定义的形状元素，可被开发者使用和操作：矩形 <rect> 圆形 <circle> 椭圆 <ellipse> 线 <line> 折线 <polyline> 多边形 <polygon> 路径 <path>
一个简单的队列 luyulong java 数据结构队列
public class MyQueue { private long[] arr; private int front; private int end; // 有效数据的大小 private int elements; public MyQueue() { arr = new long[10]; elements = 0; front
基础数据结构和算法九：Binary Search Tree sunwinner Algorithm
A binary search tree (BST) is a binary tree where each node has a Comparable key (and an associated value) and satisfies the restriction that the key in any node is larger than the keys in all
项目出现的一些问题和体会 Steven-Walker DAO Web servlet
第一篇博客不知道要写点什么，就先来点近阶段的感悟吧。这几天学了servlet和数据库等知识，就参照老方的视频写了一个简单的增删改查的，完成了最简单的一些功能，使用了三层架构。 dao层完成的是对数据库具体的功能实现，service层调用了dao层的实现方法，具体对servlet提供支持。 &
高手问答：Java老A带你全面提升Java单兵作战能力！ ITeye管理员 java
本期特邀《Java特种兵》作者：谢宇，CSDN论坛ID: xieyuooo 针对JAVA问题给予大家解答，欢迎网友积极提问，与专家一起讨论! 作者简介：淘宝网资深Java工程师，CSDN超人气博主，人称“胖哥”。 CSDN博客地址： http://blog.csdn.net/xieyuooo 作者在进入大学前是一个不折不扣的计算机白痴，曾经被人笑话过不懂鼠标是什么，