Triton测试

Trtion可以部署在cloud端，通过GRPC调用，也可以部署在边缘端，通过CAPI调用。

1 启动docker

 这个以deepstream的triton镜像为例。

$ docker run --gpus all -itd --ipc=host --rm -v /tmp/.X11-unix:/tmp/.X11-unix  -v $HOME:/home   -e DISPLAY=$DISPLAY -w /opt/nvidia/deepstream/deepstream nvcr.io/nvidia/deepstream:6.1.1-triton

2 GRPC方式

2.1 启动tritonserver

下载server的代码到本地，git clone https://github.com/triton-inference-server/server.git

到docs目录下，下载模型，启动tritonserver.

$ cd /home/code/triton/server/docs/examples

$ ./fetch_models.sh

$ tritonserver --model-repository=model_repository

2.2 测试demo

从Releases · triton-inference-server/server · GitHub下载client sdk ，如： v2.25.0_ubuntu2004.clients.tar.gz，解压到本地，执行测试，

$ ../bin/image_client -m inception_graphdef -s INCEPTION -c 3 -b 2 qa/images/mug.jpg

返回结果如下：

Request 0, batch size 2
Image 'qa/images/mug.jpg':
    0.754047 (505) = COFFEE MUG
    0.157065 (969) = CUP
    0.002878 (968) = ESPRESSO
Image 'qa/images/mug.jpg':
    0.754047 (505) = COFFEE MUG
    0.157065 (969) = CUP
    0.002878 (968) = ESPRESSO

3 CAPI方式

3.1 编译simple

server目录的simple没有提供Makefile，按这个Makefile进行编译，缺少的东西需要添加。

cat Makefile
TARGET=simple
GCC=g++
GCC_PARMS+= -I./rdjson   -I /usr/local/cuda-$(CUDA_VER)/include
GCC_PARMS+=-I/opt/tritonserver/include -D TRITON_ENABLE_GPU=ON -D TRITON_MIN_COMPUTE_CAPABILITY=5.3

GCC_LIBS=  -L/usr/local/cuda-$(CUDA_VER)/lib64/ -L/opt/tritonserver/lib/
GCC_LIBS+=-lpthread -ltritonserver  -lcudart

all: $(TARGET)


%.o: %.cc
        $(GCC) $(GCC_PARMS) -c -g -o $@ $^

$(TARGET): $(TARGET).o
        $(GCC) $^ $(GCC_LIBS) -o $@

clean:
        rm -f $(TARGET).o $(TARGET)

.PHONY: all clean

3.2 测试simple

编译成功后，传入模型路径给simple, 如： ./simple -r ../../docs/examples/model_repository/，部分结果打印如下：

OUTPUT0 is stored in system memory
OUTPUT1 is stored in system memory
0 + 1 = 1
0 - 1 = -1
1 + 1 = 2
1 - 1 = 0
2 + 1 = 3
2 - 1 = 1
3 + 1 = 4
3 - 1 = 2
4 + 1 = 5
4 - 1 = 3
5 + 1 = 6
5 - 1 = 4
6 + 1 = 7
6 - 1 = 5
7 + 1 = 8
7 - 1 = 6
8 + 1 = 9
8 - 1 = 7
9 + 1 = 10
9 - 1 = 8
10 + 1 = 11
10 - 1 = 9
11 + 1 = 12
11 - 1 = 10
12 + 1 = 13
12 - 1 = 11
13 + 1 = 14
13 - 1 = 12
14 + 1 = 15
14 - 1 = 13
15 + 1 = 16
15 - 1 = 14