virtio-gpu

软件环境：ubuntu20.04 aosp
硬件环境：x86 PC
目标：在host ubuntu20.04启动支持图形(by virtio-gpu)的Ubuntu20.04或Android虚拟机

1 说明

• 本文搭建基于QEMU-KVM(或Crosvm-KVM)的虚拟化平台，启动Ubuntu20.04虚拟机(或Android虚拟机)，支持mesa-virgl 3D加速环境，并对2D、3D流程简单分析。

1.1 virtio-gpu概述

• 概述
  VIRTIO设备的行为与普通PCI设备一样。有一个配置空间、一些专用内存和中断。第二个非常重要的一点，VirtIO设备与用作FIFO 队列的ring buffer通信。该设备完全在QEMU中模拟，可以通过在来宾和主机之间共享公共页面来实现DMA传输。
• 通信队列
  在virtio-gpu 上，有 2 个VQ队列。一个专用于cursor，另一个专用于ctrl。要在队列中发送命令，流程这样的：

guest：在host上分配页面
guest：发送一个header和在环形缓冲区中的物理页面（guest POV）的pointers。
guest：发送中断
VMExit
host：QEMU 读取header and pointers。转换地址以匹配本地虚拟地址范围。
host：读取命令，执行
host：写回环形缓冲区
host：发送中断
guest：处理中断，读取环形缓冲区并处理应答

• 分类
  目前基于Virtio-gpu有3种实现，2D，3D（virgl，gfxstream），其中2D谷歌也称做swiftshader，qemu支持2D和virgl 3D；crosvm支持swiftshader，virgl，gfxstream
• Virgl
  VirGL 可以概括为一个简单的状态机，跟踪资源并将命令缓冲区转换为一系列 OpenGL 调用。它公开了两种命令： 2D 和 3D。
  2D命令。主要集中在资源管理上。我们可以通过创建 2D 资源在主机上分配内存。然后通过将此资源的内存区域链接到访客的物理页面来初始化 DMA 传输。为了简化客户机上应用程序之间的资源管理，VirGL 还添加了一个简单的上下文功能。资源创建是全局的，但要使用它们，您必须将它们附加到上下文中。
  3D 命令。这些与我们在Vulkan等API中可以找到的内容很接近。可以设置viewport, scissor state, 创建VBO和绘制，同时也支持着色器。首先需要将它们转换为 TGSI，一种类似汇编的表示。一旦到主机上，它们将被重新转换为GLSL并发送到OpenGL。

2 安装编译

2.1 依赖编译

• 依赖

#pip3 meson等依赖
sudo apt install python3-pip
sudo pip3 install meson
sudo apt install libegl-dev libglvnd-dev libgbm1 libgbm-dev mesa-utils llvm llvm-9-dev libpciaccess-dev libwayland-egl-backend-dev ninja-build libx11-dev libegl1-mesa-dev libdrm-dev cmake

• libepoxy. qemu用于打开open渲染库

#libepoxy库
git clone https://github.com/anholt/libepoxy.git
cd libepoxy
mkdir build && cd build && meson .. && meson install && cd ..

• virglrenderer. qemu用于转换galiinum中间TGSI指令转换为GLSL

#virglrenderer
git clone https://github.com/freedesktop/virglrenderer.git
cd virglrenderer
#将meson.build -> project -> buildtype修改为debug后，gdb调试可看见行号
mkdir build && cd build && meson .. && meson install && cd ..

• mesa-drm. mesa的依赖

#mesa-drm.
git clone https://github.com/freedesktop/mesa-drm.git
cd mesa-drm 
mkdir build && meson build/ && ninja -C build install 

• mesa. 用于host提供opengl实现

glxinfo |grep renderer
#如果显示renderer string使用 llvmpipe，说明是软件渲染，一般是ssh环境下
#如果renderer string使用Intel/SVGA3D，说明3D驱动安装正确,硬件渲染
#如果renderer string使用virgl，说明Guest virgl 3D加速OK

#安装meson等依赖
sudo apt install python3-pip
sudo pip3 install meson mako
sudo apt-get install -y libegl-dev libglvnd-dev libgbm1 libgbm-dev mesa-utils llvm-11 llvm-11-dev libpciaccess-dev libwayland-egl-backend-dev ninja-build libx11-dev libegl1-mesa-dev libdrm-dev cmake libelf-dev libbison-dev flex libxrandr-dev valgrind libunwind-dev wayland-scanner++ libwayland-bin libwayland-dev libxdamage-dev libxcb-glx0-dev libx11-xcb-dev libxcb-dri2-0-dev libxcb-dri3-dev libxcb-present-dev libxshmfence-dev libxxf86vm-dev libelf-dev libwayland-dev wayland-protocols libxdamage-dev libxdamage-dev libxcb-glx0-dev libxcb-shm0-dev libx11-xcb-dev libxcb-dri2-0-dev libxcb-dri3-dev libxcb-present-dev libxshmfence-dev libxxf86vm-dev libxrandr-dev glslang-tools  
#mesa源码包下载：https://archive.mesa3d.org//；官方文档：https://docs.mesa3d.org/install.html
git clone https://gitlab.freedesktop.org/mesa/mesa.git
或者
wget https://archive.mesa3d.org//mesa-21.2.6.tar.xz
xz -d mesa-21.2.6.tar.xz
tar xvf mesa-21.2.6.tar
cd mesa-21.2.6
meson builddir/ 
ninja -C builddir/
sudo ninja -C builddir/ install

2.2 编译qemu

• qemu. 编译qemu才能用于GDB调试。qemu 4.2.1能使能，6.1打不开virgl

git clone https://github.com/qemu/qemu.git
git checkout remotes/origin/stable-4.2   #4.2版本virgl使能成功
#git checkout remotes/origin/stable-6.1  #6.1版本virgl无法使能
sudo apt install libglib2.0-dev libpixman-1-dev libgtk-3-dev        #安装qemu依赖 
./configure --enable-kvm --enable-debug --enable-gtk --target-list=x86_64-softmmu --enable-virglrenderer --enable-opengl #开启gtk，virgl,opengl等
make -j 4
#make install #可选，更建议如下，把qemu文件拷贝出来运行
cp pc-bios/* /virgl-3d-ubuntu/qemu 
cp /virgl-3d-ubuntu/qemu 

3 创建虚拟机

3.1 创建VM

#制作Ubuntu镜像
qemu-img create -q -f qcow2 ubuntu.img 10G
#获取ubuntu iso
wget http://old-releases.ubuntu.com/releases/20.04/ubuntu-20.04-live-server-amd64.iso
#安装iso镜像到ubuntu.img
qemu-system-x86_64 --enable-kvm -m 1024 -smp 2  -boot d -hda ubuntu.img -cdrom ubuntu-20.04-live-server-amd64.iso
#运行vm. qemu 4.2.1报错; 5.3,6.1.1只能启动2D，virgl没打开
qemu-system-x86_64 --enable-kvm -m 1024 -smp 1 -serial stdio -device virtio-gpu -display gtk,gl=on ubuntu.img -D qemu_log.txt
#运行vm. qemu 4.2.1可以开启virgl; 6.1.1只能启动2D，virgl没打开
qemu-system-x86_64 --enable-kvm -m 1024 -smp 1 -serial stdio -vga virtio -display gtk,gl=on ubuntu.img -D qemu_log.txt

• 如下图为虚拟机drm日志显示virgl 3D加速已开始
  

3.2 VM查看

lspci #查看virtio-gpu设备00:02.0, Vendor ID：1af4, Device ID:1050(modern)

#如果进入命令行(可能安装ubuntu时没有选图形)，安装图形包，重启后可见VM ubuntu图形界面
sudo apt-get install ubuntu-desktop
lshw -c video
dmesg | grep drm #查看virtio-gpu是否支持3D加速
glxinfo | grep renderer #查看mesa renderer是否virgl
glmarks2

4 VIRTIO-GPU分析

4.1 概述

• virti-gpu 3D架构如下
  
• Guest Ioctl. 架构的最上层是Guest系统里的应用程序（如mesa3d，minigbm等通过libdrm调用），libdrm通过设备节点（例如：/dev/dri/renderDxxx）与内核驱动通信。DRM_VIRTIO_GPU驱动支持多种命令，下面列举了最常用到的几种请求命令：

命令	说明
VIRTGPU_GET_CAPS	获取GPU的能力信息
VIRTGPU_GETPARAM	获取GPU的特性信息
VIRTGPU_RESOURCE_CREATE	创建resource
VIRTGPU_RESOURCE_INFO	获取resource信息
VIRTGPU_MAP	映射资源到用户空间（零拷贝）
VIRTGPU_EXECBUFFER	向GPU发送命令
VIRTGPU_TRANSFER_FROM_HOST	从GPU中读取数据
VIRTGPU_TRANSFER_TO_HOST	向GPU写入数据

• vring. 作为前端驱动和后端设备通信的桥梁。前端virtio-gpu驱动使用virtio_gpu_queue_ctrl_buffer()和virtio_gpu_queue_cursor()通过vring与后端进行通信。这两个函数分别向GPU发送控制请求和光标请求。对应地，virtioio-gpu后端中virtio_gpu_ctrl_bh()和virtio_gpu_cursor_bh()分别处理这两个请求。

/* virtio-gpu device */
virtio_gpu_ctrl_bh()
|-> virtio_gpu_handle_ctrl()
    |-> virtio_gpu_process_cmdq()
        | /* 2D */
        |-> virtio_gpu_simple_process_cmd()
        | /* 3D */
        |-> virtio_gpu_virgl_process_cmd()

• virtio-gpu device. 作为消费者，virtio-gpu后端virtio_gpu_process_cmdq()从vring队列中读取请求数据。如果支持3D特性，它会调用virtio_gpu_virgl_process_cmd()来处理请求。否则，它会调用virtio_gpu_simple_process_cmd()。当3D特性开启时，前后端通信的VIRTIO-GPU协议支持命令如下：

命令	处理函数
VIRTIO_GPU_CMD_CTX_CREATE	virgl_cmd_context_create()
VIRTIO_GPU_CMD_CTX_DESTROY	virgl_cmd_context_destroy()
VIRTIO_GPU_CMD_RESOURCE_CREATE_2D	virgl_cmd_create_resource_2d()
VIRTIO_GPU_CMD_RESOURCE_CREATE_3D	virgl_cmd_create_resource_3d()
VIRTIO_GPU_CMD_SUBMIT_3D	virgl_cmd_submit_3d()
VIRTIO_GPU_CMD_TRANSFER_TO_HOST_2D	virgl_cmd_transfer_to_host_2d()
VIRTIO_GPU_CMD_TRANSFER_TO_HOST_3D	virgl_cmd_transfer_to_host_3d()
VIRTIO_GPU_CMD_TRANSFER_FROM_HOST_3D	virgl_cmd_transfer_from_host_3d()
VIRTIO_GPU_CMD_RESOURCE_ATTACH_BACKING	virgl_resource_attach_backing()
VIRTIO_GPU_CMD_RESOURCE_DETACH_BACKING	virgl_resource_detach_backing()
VIRTIO_GPU_CMD_SET_SCANOUT	virgl_cmd_set_scanout()
VIRTIO_GPU_CMD_RESOURCE_FLUSH	virgl_cmd_resource_flush()
VIRTIO_GPU_CMD_RESOURCE_UNREF	virgl_cmd_resource_unref()
VIRTIO_GPU_CMD_CTX_ATTACH_RESOURCE	virgl_cmd_ctx_attach_resource()
VIRTIO_GPU_CMD_CTX_DETACH_RESOURCE	virgl_cmd_ctx_detach_resource()
VIRTIO_GPU_CMD_GET_CAPSET_INFO	virgl_cmd_get_capset_info()
VIRTIO_GPU_CMD_GET_CAPSET	virgl_cmd_get_capset()
VIRTIO_GPU_CMD_GET_DISPLAY_INFO	virtio_gpu_get_display_info()
VIRTIO_GPU_CMD_GET_EDID	virtio_gpu_get_edid()

• Virglrenderer. virtio-gpu后端设备大部分的命令会被转发到virglrenderer，以VIRTIO_GPU_CMD_SUBMIT_3D命令为例，Guest应用可以通过发送VIRTGPU_EXECBUFFER命令直接请求virtio-gpu驱动，virtio-gpu驱动会以VIRTIO_GPU_CMD_SUBMIT_3D命令转发到Host virto-gpu设备，再将命令转发给virglrenderer，virglrenderer解析buf中virgl context cmd(简称CCMD)，针对不同CCMD进行3D解码处理。处理过程和CCMD列表如下：
  

VIRGL_CCMD	描述
VIRGL_CCMD_NOP = 0	vrend_decode_dummy
VIRGL_CCMD_CREATE_OBJECT = 1	vrend_decode_create_object
VIRGL_CCMD_BIND_OBJECT	vrend_decode_bind_object
VIRGL_CCMD_DESTROY_OBJECT	vrend_decode_destroy_object
VIRGL_CCMD_SET_VIEWPORT_STATE	vrend_decode_set_viewport_state
VIRGL_CCMD_SET_FRAMEBUFFER_STATE	vrend_decode_set_framebuffer_state
VIRGL_CCMD_SET_VERTEX_BUFFERS	vrend_decode_set_vertex_buffers
VIRGL_CCMD_CLEAR	vrend_decode_clear
VIRGL_CCMD_DRAW_VBO	vrend_decode_draw_vbo
VIRGL_CCMD_RESOURCE_INLINE_WRITE	vrend_decode_resource_inline_write
VIRGL_CCMD_SET_SAMPLER_VIEWS	vrend_decode_set_sampler_views
VIRGL_CCMD_SET_INDEX_BUFFER	vrend_decode_set_index_buffer
VIRGL_CCMD_SET_CONSTANT_BUFFER	vrend_decode_set_constant_buffer
VIRGL_CCMD_SET_STENCIL_REF	vrend_decode_set_stencil_ref
VIRGL_CCMD_SET_BLEND_COLOR	vrend_decode_set_blend_color
VIRGL_CCMD_SET_SCISSOR_STATE	vrend_decode_set_scissor_state
VIRGL_CCMD_BLIT	vrend_decode_blit
VIRGL_CCMD_RESOURCE_COPY_REGION	vrend_decode_resource_copy_region
VIRGL_CCMD_BIND_SAMPLER_STATES	vrend_decode_bind_sampler_states
VIRGL_CCMD_BEGIN_QUERY	vrend_decode_begin_query
VIRGL_CCMD_END_QUERY	vrend_decode_end_query
VIRGL_CCMD_GET_QUERY_RESULT	vrend_decode_get_query_result
VIRGL_CCMD_SET_POLYGON_STIPPLE	vrend_decode_set_polygon_stipple
VIRGL_CCMD_SET_CLIP_STATE	vrend_decode_set_clip_state
VIRGL_CCMD_SET_SAMPLE_MASK	vrend_decode_set_sample_mask
VIRGL_CCMD_SET_STREAMOUT_TARGETS	vrend_decode_set_streamout_targets
VIRGL_CCMD_SET_RENDER_CONDITION	vrend_decode_set_render_condition
VIRGL_CCMD_SET_UNIFORM_BUFFER	vrend_decode_set_uniform_buffer
VIRGL_CCMD_SET_SUB_CTX	vrend_decode_set_sub_ctx
VIRGL_CCMD_CREATE_SUB_CTX	vrend_decode_create_sub_ctx
VIRGL_CCMD_DESTROY_SUB_CTX	vrend_decode_destroy_sub_ctx
VIRGL_CCMD_BIND_SHADER	vrend_decode_bind_shader
VIRGL_CCMD_SET_TESS_STATE	vrend_decode_set_tess_state
VIRGL_CCMD_SET_MIN_SAMPLES	vrend_decode_set_min_samples
VIRGL_CCMD_SET_SHADER_BUFFERS	vrend_decode_set_shader_buffers
VIRGL_CCMD_SET_SHADER_IMAGES	vrend_decode_set_shader_images
VIRGL_CCMD_MEMORY_BARRIER	vrend_decode_memory_barrier
VIRGL_CCMD_LAUNCH_GRID	vrend_decode_launch_grid
VIRGL_CCMD_SET_FRAMEBUFFER_STATE_NO_ATTACH	vrend_decode_set_framebuffer_state_no_attach
VIRGL_CCMD_TEXTURE_BARRIER	vrend_decode_texture_barrier
VIRGL_CCMD_SET_ATOMIC_BUFFERS	vrend_decode_set_atomic_buffers
VIRGL_CCMD_SET_DEBUG_FLAGS	vrend_decode_set_debug_mask
VIRGL_CCMD_GET_QUERY_RESULT_QBO	vrend_decode_get_query_result_qbo
VIRGL_CCMD_TRANSFER3D	vrend_decode_transfer3d
VIRGL_CCMD_END_TRANSFERS	vrend_decode_dummy
VIRGL_CCMD_COPY_TRANSFER3D	vrend_decode_copy_transfer3d
VIRGL_CCMD_SET_TWEAKS	vrend_decode_set_tweaks
VIRGL_CCMD_CLEAR_TEXTURE	vrend_decode_clear_texture
VIRGL_CCMD_PIPE_RESOURCE_CREATE	vrend_decode_pipe_resource_create
VIRGL_CCMD_PIPE_RESOURCE_SET_TYPE	vrend_decode_pipe_resource_set_type
VIRGL_CCMD_GET_MEMORY_INFO	vrend_decode_get_memory_info
VIRGL_CCMD_SEND_STRING_MARKER	vrend_decode_send_string_marker
VIRGL_CCMD_LINK_SHADER	vrend_decode_link_shader
VIRGL_CCMD_CREATE_VIDEO_CODEC	vrend_decode_create_video_codec
VIRGL_CCMD_DESTROY_VIDEO_CODEC	vrend_decode_destroy_video_codec
VIRGL_CCMD_CREATE_VIDEO_BUFFER	vrend_decode_create_video_buffer
VIRGL_CCMD_DESTROY_VIDEO_BUFFER	vrend_decode_destroy_video_buffer
VIRGL_CCMD_BEGIN_FRAME	vrend_decode_begin_frame
VIRGL_CCMD_DECODE_MACROBLOCK	vrend_decode_dummy
VIRGL_CCMD_DECODE_BITSTREAM	vrend_decode_decode_bitstream
VIRGL_CCMD_ENCODE_BITSTREAM	vrend_decode_encode_bitstream
VIRGL_CCMD_END_FRAME	vrend_decode_end_frame
VIRGL_MAX_COMMANDS

4.2 VM前端

• 如下图为基于VirtioGPU的前后端调用关系。依次为Guest App => Guest mesa3D => Guest Virtio-GPU Driver => Host Qemu => Host Virglrenderer => Host GPU

• 前端调用框架。此处前端以基于mesa-drm的Android12为例

• 在普通 GUI 应用中，界面是由 Graphics Widgets （例如 Gtk+、Qt）布局，进而生成 Scene
  Graph（SG），通过遍历SG渲染在surface。最后，这个surface （或曰 buffer），提交给 Mutter
  /Wayland/SurfaceFlinger来进行合成。

• 这里展开两个细节，其一，遍历 SG 进行渲染时，通常是通过 Cario 或 Skia 等绘图工具：绘图工具常有多实现后端。其中 GPU 加速后端，常见基于 OpenGL ES。OpenGL ES 是个 API 的SPEC，基于开源的实现方案常由Mesa提供。相关 buffer 分配，最终落实在通过DRM 接口，分配GEM Buffer Object，例如minigbm提供gbm_bo_create()实现

• 调用关系. mesa3d/minigbm对libdrm通过drmIoctl调用，libdrm通过ioctl调用virtio-gpu驱动，virtio-gpu通过VIRTIO_GPU_CMD调用到qemu virtio-device. 涉及模块间调用接口如下：

mesa3d	minigbm	libdrm drmIoctl	virtio-gpu ioctl	virtio-gpu func	依赖说明
virgl_drm_resource_map	gbm_bo_map	DRM_IOCTL_VIRTGPU_MAP	VIRTGPU_MAP	virtio_gpu_map_ioctl	mmap dumb依赖gem
virgl_drm_winsys_submit_cmd	gbm_bo_create	DRM_IOCTL_VIRTGPU_EXECBUFFER	VIRTGPU_EXECBUFFER	virtio_gpu_execbuffer_ioctl	VIRTIO_GPU_CMD_SUBMIT_3D
virgl_drm_winsys_create	gbm_create_device	DRM_IOCTL_VIRTGPU_GETPARAM	VIRTGPU_GETPARAM	virtio_gpu_getparam_ioctl	仅copy配置
virgl_drm_winsys_resource_create	gbm_bo_create	DRM_IOCTL_VIRTGPU_RESOURCE_CREATE	VIRTGPU_RESOURCE_CREATE	virtio_gpu_resource_create_ioctl	VIRTIO_GPU_CMD_CTX_CREATE VIRTIO_GPU_CMD_RESOURCE_CREATE_3D
virgl_drm_winsys_resource_create_handle	gralloc0_perform gralloc0_lock_async_ycbcr	DRM_IOCTL_VIRTGPU_RESOURCE_INFO	VIRTGPU_RESOURCE_INFO	virtio_gpu_resource_info_ioctl	仅赋值配置
virgl_bo_transfer_get	drv_bo_invalidate	DRM_IOCTL_VIRTGPU_TRANSFER_FROM_HOST	VIRTGPU_TRANSFER_FROM_HOST	virtio_gpu_transfer_from_host_ioctl	VIRTIO_GPU_CMD_TRANSFER_FROM_HOST_3D
virgl_bo_transfer_put	gbm_bo_unmap	DRM_IOCTL_VIRTGPU_TRANSFER_TO_HOST	VIRTGPU_TRANSFER_TO_HOST	virtio_gpu_transfer_to_host_ioctl	VIRTIO_GPU_CMD_TRANSFER_TO_HOST_3D
virgl_drm_resource_wait	gbm_bo_unmap	DRM_IOCTL_VIRTGPU_WAIT	VIRTGPU_WAIT	virtio_gpu_wait_ioctl	wait rcu
virgl_drm_get_caps	gbm_create_device	DRM_IOCTL_VIRTGPU_GET_CAPS	VIRTGPU_GET_CAPS	virtio_gpu_get_caps_ioctl	VIRTIO_GPU_CMD_GET_CAPSET
virgl_drm_winsys_resource_create_blob	gbm_bo_create	DRM_IOCTL_VIRTGPU_RESOURCE_CREATE_BLOB	VIRTGPU_RESOURCE_CREATE_BLOB	virtio_gpu_resource_create_blob_ioctl	VIRTIO_GPU_CMD_RESOURCE_CREATE_BLOB
	gbm_bo_create	DRM_IOCTL_VIRTGPU_CONTEXT_INIT	VIRTGPU_CONTEXT_INIT	virtio_gpu_context_init_ioctl	VIRTIO_GPU_CMD_CTX_CREATE

4.2.1 minigbm

• 在现代的移动设备上，通常借由OpenGL ES来进行图形渲染。而OpenGL ES和Platform OS上的 Window System的交互，是通过EGL来隔开的，EGL主要实现了Buffer相关接口
• 在开源图形栈中EGL的实现：

对于 App：EGL基于Wayland（以及 libgbm）来实现接口
对于 Compositor：EGL基于DRM（以及 DRM 进一步封装，如 libgbm）来实现接口

• minigbm通过libdrm调用Virtio-gpu相关ioctl

virtgpu_cross_domain.c:122:     ret = drmIoctl(drv->fd, DRM_IOCTL_VIRTGPU_EXECBUFFER, &exec);
virtgpu_cross_domain.c:131:     ret = drmIoctl(drv->fd, DRM_IOCTL_VIRTGPU_WAIT, &wait_3d);
virtgpu_cross_domain.c:289:     ret = drmIoctl(drv->fd, DRM_IOCTL_VIRTGPU_GET_CAPS, &args);
virtgpu_cross_domain.c:312:     ret = drmIoctl(drv->fd, DRM_IOCTL_VIRTGPU_CONTEXT_INIT, &init);
virtgpu_cross_domain.c:333:     ret = drmIoctl(drv->fd, DRM_IOCTL_VIRTGPU_MAP, &map);
virtgpu_cross_domain.c:427:     ret = drmIoctl(bo->drv->fd, DRM_IOCTL_VIRTGPU_MAP, &gem_map);
virtgpu_virgl.c:498:    ret = drmIoctl(bo->drv->fd, DRM_IOCTL_VIRTGPU_RESOURCE_CREATE, &res_create);
virtgpu_virgl.c:722:    ret = drmIoctl(drv->fd, DRM_IOCTL_VIRTGPU_RESOURCE_CREATE_BLOB, &drm_rc_blob);
virtgpu_virgl.c:867:    ret = drmIoctl(bo->drv->fd, DRM_IOCTL_VIRTGPU_TRANSFER_FROM_HOST, &xfer);
virtgpu_virgl.c:879:    ret = drmIoctl(bo->drv->fd, DRM_IOCTL_VIRTGPU_WAIT, &waitcmd);
virtgpu_virgl.c:943:    ret = drmIoctl(bo->drv->fd, DRM_IOCTL_VIRTGPU_TRANSFER_TO_HOST, &xfer);
virtgpu_virgl.c:958:    ret = drmIoctl(bo->drv->fd, DRM_IOCTL_VIRTGPU_WAIT, &waitcmd);
virtgpu_virgl.c:1070:   ret = drmIoctl(bo->drv->fd, DRM_IOCTL_VIRTGPU_RESOURCE_INFO_CROS, &res_info);
virtgpu.c:49:           int ret = drmIoctl(drv->fd, DRM_IOCTL_VIRTGPU_GETPARAM, &get_param);

4.2.2 mesa3d

• mesa3d是图形驱动的开源实现，Virtio-GPU方案中依赖mesa-virgl实现将OpenGL指令转换为TGSI中间形式向内核提交。通过libdrm调用Virtio-gpu相关ioctl

src/gallium/winsys/virgl/drm/virgl_drm_winsys.c:201:   ret = drmIoctl(qdws->fd, DRM_IOCTL_VIRTGPU_RESOURCE_CREATE_BLOB, &drm_rc_blob);
src/gallium/winsys/virgl/drm/virgl_drm_winsys.c:257:   ret = drmIoctl(qdws->fd, DRM_IOCTL_VIRTGPU_RESOURCE_CREATE, &createcmd);
src/gallium/winsys/virgl/drm/virgl_drm_winsys.c:331:   return drmIoctl(vdws->fd, DRM_IOCTL_VIRTGPU_TRANSFER_TO_HOST, &tohostcmd);
src/gallium/winsys/virgl/drm/virgl_drm_winsys.c:360:   return drmIoctl(vdws->fd, DRM_IOCTL_VIRTGPU_TRANSFER_FROM_HOST, &fromhostcmd);
src/gallium/winsys/virgl/drm/virgl_drm_winsys.c:486:   if (drmIoctl(qdws->fd, DRM_IOCTL_VIRTGPU_RESOURCE_INFO, &info_arg)) {
src/gallium/winsys/virgl/drm/virgl_drm_winsys.c:565:   if (drmIoctl(qdws->fd, DRM_IOCTL_VIRTGPU_MAP, &mmap_arg))
src/gallium/winsys/virgl/drm/virgl_drm_winsys.c:591:   ret = drmIoctl(qdws->fd, DRM_IOCTL_VIRTGPU_WAIT, &waitcmd);
src/gallium/winsys/virgl/drm/virgl_drm_winsys.c:855:   ret = drmIoctl(qdws->fd, DRM_IOCTL_VIRTGPU_EXECBUFFER, &eb);
src/gallium/winsys/virgl/drm/virgl_drm_winsys.c:898:   ret = drmIoctl(vdws->fd, DRM_IOCTL_VIRTGPU_GET_CAPS, &args);
src/gallium/winsys/virgl/drm/virgl_drm_winsys.c:903:   ret = drmIoctl(vdws->fd, DRM_IOCTL_VIRTGPU_GET_CAPS, &args);
src/gallium/winsys/virgl/drm/virgl_drm_winsys.c:1060:  ret = drmIoctl(drmFD, DRM_IOCTL_VIRTGPU_GETPARAM, &getparam);

4.2.3 libdrm

• libdrm用于封装DRM驱动IOCTL，其中封装VIRTIO-GPU提供IOCTL为

DRM_IOCTL_VIRTGPU_MAP 
DRM_IOCTL_VIRTGPU_EXECBUFFER 
DRM_IOCTL_VIRTGPU_GETPARAM 
DRM_IOCTL_VIRTGPU_RESOURCE_CREATE                 
DRM_IOCTL_VIRTGPU_RESOURCE_INFO 
DRM_IOCTL_VIRTGPU_TRANSFER_FROM_HOST 
DRM_IOCTL_VIRTGPU_TRANSFER_TO_HOST 
DRM_IOCTL_VIRTGPU_WAIT                            
DRM_IOCTL_VIRTGPU_GET_CAPS 
DRM_IOCTL_VIRTGPU_RESOURCE_CREATE_BLOB                            
DRM_IOCTL_VIRTGPU_CONTEXT_INIT

4.2.4 virtio-gpu

• virtio-gpu前端框架如上图所示. 作为DRM驱动的一种实现（当然也可以有别的实现，如MTK，RK等），不仅实现自己的ioctl，还向Drm core注册如GEM,MODESET,ATOMIC等其他操作回调，用于支持内存管理，模式设置，提交等功能。
• virtio-gpu是一个支持MODESET/GEM/RENDER/ATOMIC的DRM驱动

static struct drm_driver driver = {
	.driver_features = DRIVER_MODESET | DRIVER_GEM | DRIVER_RENDER | DRIVER_ATOMIC,

 - DRIVER_GEM：该feature 告诉DRM Core当前驱动支持GEM操作，如buffer的分配和释放，
   以及GEM OPEN/FLINK/CLOSE等操作。自己需要提供.dumb_create回调，接口用于创建 gem object，并分配物理buffer
 - DRIVER_MODESET：该feature告诉DRM Core当前驱动支持modesetting操作。自己需要做modeset初始化，
   mode_config.funcs由virtio_gpu_mode_funcs实现，其中.fb_create回调接口用于创建framebuffer object，并绑定gem objects
 - DRIVER_ATOMIC：该feature告诉DRM Core当前驱动支持Atomic操作，其中.atomic_commit接口是atomic操作的主要入口函数

• virtio-gpu_probe. virtio-gpu驱动自身的初始化过程

virtio_gpu_probe
    |-> virtio_gpu_init
        |-> virtio_gpu_init_vq //初始化ctrlq和cursorq，设置callback
        |-> //virtio_gpu一系列成员初始化，与后端通信
        |-> virtio_gpu_modeset_init //modeset初始化
            |-> drmm_mode_config_init //drm mode配置结构体初始化
            |-> mode_config.funcs //modern版本drm mode callback，支持atomic_commit提交送显
            |-> mode_config.helper_private //modern版本helper callback，支持atomic_commit_tail
            |-> //设置mode最小/最大长宽，支持page_flip方式送显
            |-> vgdev_output_init //drm plane/crtc/connector/encoder/connector初始化
                |-> drm_universal_plane_init   //回调virtio_gpu_plane_funcs，支持plane ioctl
                |-> drm_crtc_init_with_planes  //回调virtio_gpu_crtc_funcs，支持crtc ioctl
                |-> drm_crtc_helper_add        //回调virtio_gpu_crtc_funcs，支持atomic ioctl
                |-> drm_connector_init        //回调virtio_gpu_connector_funcs，支持connector ioctl
                |-> drm_simple_encoder_init   //回调virtio_gpu_enc_helper_funcs，支持connector ioctl 
        |-> virtio_gpu_cmd_get_display_info //向后端请求display info
    |-> drm_dev_register //注册drm设备

• ioctl GEM调用. virtio-gpu向drm core注册GEM操作callback(注册到drm_driver成员)，实现buffer分配管理

funcs	drm_driver成员说明
.dumb_create	GEM操作.由DRM_IOCTL_MODE_CREATE_DUMB调用,创建cpu dumb buffer
.dumb_map_offset	GEM操作. 由DRM_IOCTL_MODE_MAP_DUMB调用,映射指定偏移的dumb buffer
.prime_handle_to_fd	GEM操作. 由drmPrimeHandleToFD调用
.prime_fd_to_handle	GEM操作.由drmPrimeFDToHandle调用
.gem_prime_mmap	GEM操作. drm_gem_dmabuf_mmap调用，映射dma_buf内存
.gem_prime_export	GEM操作. 实现一个dma_buf exporter驱动
.gem_prime_import	GEM操作.drm_gem_prime_fd_to_handle调用（该函数实际由virtio-gpu注册.prime_fd_to_handle），增加gem引用计数
.gem_prime_import_sg_table	GEM操作. drm_gem_prime_fd_to_handle调用，在virtio-gpu中未支持
.gem_create_object	GEM操作. 创建object并设置shmem funcs

• ioctl VIRTGPU调用. virtio-gpu驱动实现的自有.ioctl，为VIRTGPU_XX类型命令，libdrm封装为DRM_IOCTL_VIRTGPU_XX类型命令，常见命令为：

命令	说明
VIRTGPU_GET_CAPS	获取GPU的能力信息
VIRTGPU_GETPARAM	获取GPU的特性信息
VIRTGPU_RESOURCE_CREATE	创建resource
VIRTGPU_RESOURCE_INFO	获取resource信息
VIRTGPU_MAP	映射资源到用户空间（零拷贝）
VIRTGPU_EXECBUFFER	向GPU发送命令
VIRTGPU_TRANSFER_FROM_HOST	从GPU中读取数据
VIRTGPU_TRANSFER_TO_HOST	向GPU写入数据

• ioctl MODESET调用. virtio-gpu向drm core注册ModeSet操作回调，上层通过drmIoctl/drmMode调用，实现crtc，plane，connector，mode等设置。此为传统方式，更倾向于使用ATOMIC这种现代方式。下面是一些libdrm接口和virtio-gpu callback对应关系

libdrm	Virtio-GPU callback	说明
drmModeSetCrtc	virtio_gpu_crtc_funcs.set_config	设置crtc
drmModePageFlip	virtio_gpu_crtc_funcs.page_flip	传统方式送显
drmModeSetPlane	virtio_gpu_plane_funcs.update_plane	更新plane
drmModeGetConnector	virtio_gpu_connector_funcs.fill_modes
drmModeAddFB2	virtio_gpu_mode_funcs.fb_create	创建framebuffer

• ioctl ATOMIC调用. virtio-gpu向drm core注册ATOMIC操作回调，上层通过drmIoctl/drmMode调用，实现mode设置，Atomic意为：本次commit操作，要么成功，要么保持原来的状态不变。如android中drm hwcomposer，执行送显流程中，依次调用addProperty去设置crtc，plane的属性，所有的请求会放进atomic_req数组，最后执行drmModeAtomicCommit把所有请求参数配置到硬件并进行刷图操作

 /* 提交所有的请求参数配置到硬件并进行刷图操作,比pageflip刷图更先进 */
 drmModeAtomicCommit()
   | /* libdrm drmIoctl with DRM_IOCTL_MODE_ATOMIC */
   |-> drm_mode_atomic_ioctl
       |-> drm_atomic_commit
           |-> config->funcs->atomic_commit
           | /* 实际为virtio_gpu_mode_funcs注册的.atomic_commit回调 */
           |-> drm_atomic_helper_commit

• shmem funcs. gem shmem对象的callbacks，用于drm core gem模块调用

 static const struct drm_gem_object_funcs virtio_gpu_shmem_funcs = {
	.free = virtio_gpu_free_object,
	.open = virtio_gpu_gem_object_open,
	.close = virtio_gpu_gem_object_close,
	.print_info = drm_gem_shmem_print_info,
	.export = virtgpu_gem_prime_export,
	.pin = drm_gem_shmem_pin,
	.unpin = drm_gem_shmem_unpin,
	.get_sg_table = drm_gem_shmem_get_sg_table,
	.vmap = drm_gem_shmem_vmap,
	.vunmap = drm_gem_shmem_vunmap,
	.mmap = drm_gem_shmem_mmap,
};

• gem_prime_export. drm_driver成员.gem_prime_export,实现了一种基于virtio的dma_buf exporter driver，（当然也有别的实现，如android ion驱动就是一种dma_buf exporter driver实现），用于将dma-buf导出，别的驱动或用户态可以作为importer使dma-buf，接口依赖由virtio_dma_buf.ko提供，virtio_dma_buf.ko对基础dma_buf接口封装。

 #1. virtio_dma_buf_ops
 const struct virtio_dma_buf_ops virtgpu_dmabuf_ops =  {
	.ops = {
		.cache_sgt_mapping = true,
		.attach = virtio_dma_buf_attach,
		.detach = drm_gem_map_detach,
		.map_dma_buf = drm_gem_map_dma_buf,
		.unmap_dma_buf = drm_gem_unmap_dma_buf,
		.release = drm_gem_dmabuf_release,
		.mmap = drm_gem_dmabuf_mmap, //将dma-buf的物理内存直接映射到用户空间，内核中由kzalloc分配
		.vmap = drm_gem_dmabuf_vmap, //将dma-buf映射到内核空间，物理上可以不连续，虚拟地址空间连续
		.vunmap = drm_gem_dmabuf_vunmap,
	},
	.device_attach = drm_gem_map_attach,
	.get_uuid = virtgpu_virtio_get_uuid,
};
#2. 声明DMA_BUF
DEFINE_DMA_BUF_EXPORT_INFO
#3. 注册exp回调
virtio_dma_buf_export(&exp_info)

4.3 VIRTIO-GPU 2D后端

4.3.1 概述

• VIRTIO-GPU 2D后端有两种，一种是依赖pixman进行2D处理，另一种是virgl_renderer的2D处理，本节讲解第一种。
• 0 说明

Host： Ubuntu20.04 qemu6.1.1 
Guset： Ubuntu20.04
#使用该环境未能使能virgl加速，仅2D加速

4.3.1 后端流程

• 1 流程

virtio_gpu_class_init -> virtio_gpu_device_realize -> 虚拟机内核启动 -> virtio_gpu_get_config 
-> virtio_gpu_handle_ctrl -> virtio_gpu_simple_process_cmd -> pixman 2D处理

• 2 virtio_gpu_class_init. 设置多个回调

#Qemu启动时根据qom type_initialize调用virtio_gpu_class_init
virtio_gpu_class_init #注册回调，gpu操作和设备操作
    handle_ctrl:virtio_gpu_handle_ctrl #处理控制
    process_cmd:virtio_gpu_simple_process_cmd #处理2D VIRTIO_GPU_CMD
    update_cursor_data:virtio_gpu_update_cursor_data #处理鼠标数据
    realize:virtio_gpu_device_realize  #初始化ctrl/cursor vq，设置回调
    reset:virtio_gpu_reset 
    get_config:virtio_gpu_get_config  #拷贝一份后端的virtio-gpu配置
    set_config:virtio_gpu_set_config

• 3 virtio_gpu_device_realize. 初始化和设置VQ回调

qemu_init->qdev_realize->device_set_realize -> virtio_pci_realize -> virtio_device_realize
#初始化ctrl/curse vq
#设置ctrl_bh/cursor_bh回调，实际调用到上文注册的handle_ctrl
#调用栈

• 4 virtio_gpu_get_config. 获取设备virtio_config

#VM读取config触发陷出，memory_region_dispatch_read -> virtio_config_readl
#调用栈

#拷贝配置

• 5 virtio_gpu_handle_ctrl

#通过aio通知进行回调
virtqueue_pop #从VQ中命令出队，加入cmdq队列
virtio_gpu_process_cmdq #处理cmdq
process_cmd #处理单个cmd
virtio_gpu_simple_process_cmd #实际调用virtio_gpu_class_init注册的回调，处理2D VIRTIO_GPU_CMD
#调用栈

• 6 virtio_gpu_simple_process_cmd

#调用栈

4.3.3 VIRTIO_GPU_CMD介绍

• 2D命令介绍

命令	说明
VIRTIO_GPU_CMD_GET_DISPLAY_INFO	获取中断的显示信息，该信息由qemu的VirtIOGPU.req_state字段保存，qemu收到命令后直接从内存中取出信息返回给前端
VIRTIO_GPU_CMD_RESOURCE_CREATE_2D	创建一个图片资源，不填充像素
VIRTIO_GPU_CMD_RESOURCE_UNREF	删除一个图片资源
VIRTIO_GPU_CMD_RESOURCE_FLUSH	向终端传输图片像素数据，显示图像
VIRTIO_GPU_CMD_TRANSFER_TO_HOST_2D	将后端qemu维护的像素数据传递给成pixman库，从进行像素处理
VIRTIO_GPU_CMD_RESOURCE_ATTACH_BACKING	将前端传递的像素数据地址从virtio队列中解析出来，让qemu维护的iov缓存映射到此地址，前端通过这个命令让后端Qemu能记录到前端分配像素地 址。注意，这条命令的本质是映射，让Qemu的iov数据结构能够关联前端的像素地址

• 2D CMD处理流程

...//Vm内核启动
...
drm子系统virtio-gpu-pci加载
virtio_gpu_get_edid  #创建一个edid并返回给VM
  virtio_gpu_generate_edid
  virtio_gpu_ctrl_response  #cmd写回给VM
virtio_gpu_get_display_info  #填充display_info并返回给VM
  virtio_gpu_base_fill_display_info
  virtio_gpu_ctrl_response
virtio_gpu_resource_create_2d #Host端创建resource，cmd只有错赋值
virtio_gpu_resource_attach_backing #Host端，cmd只有错赋值
  virtio_gpu_create_mapping_iov #res进行dma_map操作
  virtio_gpu_ctrl_response_nodata #返回状态

...  #内核流程加载
virtio_gpu_set_scanout  #通过pixman为scanout创建surface，更新scanout的坐标长宽等
  virtio_gpu_update_scanout #将virtio_gpu_rect ss.r的id/x/y等更新到scanout
virtio_gpu_transfer_to_host_2d #结合pixman转换为host 2d，cmd只有错赋值
  iov_to_buf 
virtio_gpu_set_scanout
virtio_gpu_resource_flush  #结合pixman更新目标区域，cmd只有错赋值
  dpy_gfx_update
virtio_gpu_transfer_to_host_2d
virtio_gpu_set_scanout
virtio_gpu_resource_flush
virtio_gpu_transfer_to_host_2d
... #在这三个CMD循环往复
... #内核继续加载
virtio_gpu_resource_create_2d
virtio_gpu_resource_attach_backing
virtio_gpu_resource_create_2d
virtio_gpu_resource_attach_backing
virtio_gpu_transfer_to_host_2d
#此处显示出ubuntu启动图标

virtio_gpu_transfer_to_host_2d
virtio_gpu_set_scanout
virtio_gpu_resource_flush
virtio_gpu_transfer_to_host_2d
virtio_gpu_set_scanout #scanou_id=0
virtio_gpu_resource_flush
...
#显示登录界面，登录
virtio_gpu_resource_create_2d
virtio_gpu_resource_detach_backing
virtio_gpu_resource_unref
...

• 7 数据结构

virtio_gpu_get_display_info
(gdb) p display_info
$11 = {hdr = {type = 4353, flags = 0, fence_id = 0, ctx_id = 0, padding = 0}, pmodes = {{r = {x = 0, y = 0, width = 1024, height = 768}, enabled = 1, flags = 0}, {r = {x = 0, y = 0, width = 0,
height = 0}, enabled = 0, flags = 0} <repeats 15 times>}} 

virtio_gpu_set_scanout 
(gdb) p *r
$3 = {x = 0, y = 0, width = 1024, height = 768}
virtio_gpu_resource_create_2d
(gdb) p c2d
$5 = {hdr = {type = 257, flags = 0, fence_id = 0, ctx_id = 0, padding = 0}, resource_id = 7, format = 2, width = 1024, height = 768} 

4.4 VIRTIO-GPU 3D后端

4.4.1 概述

• 本节主要介绍virtio-gpu后端依赖virglrenderer的支持实现OpenGl 3D加速。

• 环境

Host：Ubuntu20.04 qemu-stable-4.2
Guest：Ubuntu20.04
启动参数：qemu-system-x86_64 --enable-kvm -m 1024 -smp 1 -serial stdio -vga virtio -display gtk,gl=on ubuntu.img

4.4.2 后端流程

• 流程. 下面为后端调用大致流程，并详细介绍

virtio_gpu_class_init 
-> virtio_gpu_device_realize #注册各种回调
-> 虚拟机GRUB启动 
-> virtio_gpu_get_config   #获取配置
-> virtio_gpu_handle_ctrl  #处理控制流
-> virtio_gpu_process_cmdq #检查cmdq队列,simple或者virgl处理
-> virtio_gpu_virgl_process_cmd #2D和3D处理
  -> 多种VIRTIO_GPU_CMD处理. 依赖libvirglrender 
  -> opengl调用到图形驱动

1. virtio_gpu_class_init. 初始化结构体和设置多个回调

#qemu type_init初始化调用由type_register_static注册的virtio_gpu_info结构
gl_unlock：virtio_gpu_gl_unblock
realize：virtio_gpu_device_realize #设置VQ和回调
reset：virtio_gpu_reset #退出清理
get_config：virtio_gpu_get_config
set_config：vdc->set_config = virtio_gpu_set_config
vmsd：vmstate_virtio_gpu
props：virtio_gpu_properties

调用栈：

2. virtio_gpu_device_realize. VirtioGPU设备实现：设置ctrl/cursor回调，VQ

  virtio_gpu_virgl_get_num_capsets #获取capset个数(2)赋值到virtio_config.num_capsets
  virtio_gpu_base_device_realize #设置回调为后面的ctrl_bh，cursor_bh
  virtio_get_queue #获取ctrl/cursor VQ
  g->ctrl_bh = qemu_bh_new(virtio_gpu_ctrl_bh, g) #ctrl回调为virtio_gpu_handle_ctrl
  g->cursor_bh = qemu_bh_new(virtio_gpu_cursor_bh, g) #cursor回调
  #初始化reslist/cmdq/fenceq链表

调用栈：

3. GRUB启动ubuntu

4. virtio_gpu_get_config. 获取设备的virtio_config

#拷贝一份virtio-device设置给前端，方式为前端read memory region
memcpy(config, &g->virtio_config, sizeof(g->virtio_config))
#配置为
 p g->virtio_config
 $7 = {events_read = 1, events_clear = 0, num_scanouts = 1, num_capsets = 2}

调用栈：

5. virtio_gpu_handle_ctrl. 处理ctrl流

#aio通知前面注册的ctrl_bh(实际为virtio_gpu_ctrl_bh)，处理ctrl流
virtio_gpu_virgl_init #CONFIG_VIRGL支持，初始化virgl
    virgl_renderer_init #libvirglrenderer实现
virtqueue_pop #CMD从VQ出队
virtio_gpu_process_cmdq #CMD处理实际为virtio_gpu_virgl_process_cmd
virtio_gpu_virgl_fence_poll #CONFIG_VIRGL支持，fence_pull再次处理CMD
    virgl_renderer_poll #libvirglrenderer实现
    virtio_gpu_process_cmdq #检查cmdq队列空了就跳过，有就处理

调用栈：

6. virtio_gpu_process_cmdq

#从cmdq中取cmd
#有cmd就调用virtio_gpu_virgl_process_cmd（VIRGL）或者virtio_gpu_simple_process_cmd（2D）处理

4.4.3 CMD处理

1. virtio_gpu_virgl_process_cmd. 作为qemu从VirtQueue取出cmd后的主处理函数，cmd结构为virtio_gpu_ctrl_command，如下图所示，比较重要是存放数据的elem和存放控制头的cmd_hdr.
   
2. 根据cmd->hdr.type使用不同方法处理cmd
3. qemu后端通过VIRTIO_GPU_FILL_CMD将cmd中out_sg[ivo_cnt].iov_base拷贝到指定结构体，组装为如下多种形式结构，用于不同CMD的处理。因此，cmd-from-vq实际只使用和保留了out_sg.iov_base地址指向的内容。下图中out_sg[ivo_cnt].iov_base内容即为ctrl head+qemu metadata
   
4. 后端数据处理流程

如上图所示为后端qemu+virglrenderer数据处理流程
qemu:
①qemu从VirtQueue取出cmd交由virtio_gpu_virgl_process_cmd()处理，如从cmd中解析出cmd_hdr.type为VIRTIO_GPU_CMD_SUBMIT_3D，交由virgl_cmd_submit_3d()处理
②virgl_cmd_submit_3d()首先拷贝cmd-from-VQ中elem.out_sg[0].iov_base，内含hdr+size+padding，其中size为out_sg[1].iov_base有效数据长度； 然后将(hdr.ctx_id, out_sg[1].iov_base，wordSize)提交给virglrenderer
virglrenderer：
③vrend_decode_ctx_submit_cmd()循环解析buf中VIRGL_CCMD（包含cmd+len+payload），根据decode_table[cmd]调用不同方法进行3D解码

4.4.4 VIRTIO_GPU_CMD介绍

3D命令介绍：(https://github.com/Keenuts/virtio-gpu-documentation/blob/master/src/virtio-gpu.md)

• VIRTIO_GPU支持CMD列表

命令	处理函数
VIRTIO_GPU_CMD_CTX_CREATE	virgl_cmd_context_create()
VIRTIO_GPU_CMD_CTX_DESTROY	virgl_cmd_context_destroy()
VIRTIO_GPU_CMD_RESOURCE_CREATE_2D	virgl_cmd_create_resource_2d()
VIRTIO_GPU_CMD_RESOURCE_CREATE_3D	virgl_cmd_create_resource_3d()
VIRTIO_GPU_CMD_SUBMIT_3D	virgl_cmd_submit_3d()
VIRTIO_GPU_CMD_TRANSFER_TO_HOST_2D	virgl_cmd_transfer_to_host_2d()
VIRTIO_GPU_CMD_TRANSFER_TO_HOST_3D	virgl_cmd_transfer_to_host_3d()
VIRTIO_GPU_CMD_TRANSFER_FROM_HOST_3D	virgl_cmd_transfer_from_host_3d()
VIRTIO_GPU_CMD_RESOURCE_ATTACH_BACKING	virgl_resource_attach_backing()
VIRTIO_GPU_CMD_RESOURCE_DETACH_BACKING	virgl_resource_detach_backing()
VIRTIO_GPU_CMD_SET_SCANOUT	virgl_cmd_set_scanout()
VIRTIO_GPU_CMD_RESOURCE_FLUSH	virgl_cmd_resource_flush()
VIRTIO_GPU_CMD_RESOURCE_UNREF	virgl_cmd_resource_unref()
VIRTIO_GPU_CMD_CTX_ATTACH_RESOURCE	virgl_cmd_ctx_attach_resource()
VIRTIO_GPU_CMD_CTX_DETACH_RESOURCE	virgl_cmd_ctx_detach_resource()
VIRTIO_GPU_CMD_GET_CAPSET_INFO	virgl_cmd_get_capset_info()
VIRTIO_GPU_CMD_GET_CAPSET	virgl_cmd_get_capset()
VIRTIO_GPU_CMD_GET_DISPLAY_INFO	virtio_gpu_get_display_info()
VIRTIO_GPU_CMD_GET_EDID	virtio_gpu_get_edid()

• VIRTIO_GPU_CMD_CTX_CREATE

virgl_cmd_context_create #3D创建context
    virgl_renderer_context_create #virgl创建
    (gdb) p cc
    $1 = {hdr = {type = 512, flags = 0, fence_id = 0, ctx_id = 1, padding = 0}, nlen = 13, padding = 0, debug_name = "systemd-udevd", '\000' <repeats 50 times>} 

• VIRTIO_GPU_CMD_CTX_ATTACH_RESOURCE

virgl_cmd_ctx_attach_resource #3D attach资源
    virgl_renderer_ctx_attach_resource
    p att_res
    $2 = {hdr = {type = 514, flags = 0, fence_id = 0, ctx_id = 1, padding = 0}, resource_id = 2, padding = 0}
    virgl_cmd_ctx_attach_resource()
    |
    |/* virglrenderer */
    |-> virgl_renderer_resource_attach_iov()
        |
        |/* 将resource数据拷贝到驱动中 */
        |-> vrend_renderer_resource_attach_iov()

• VIRTIO_GPU_CMD_RESOURCE_CREATE_3D

virgl_cmd_create_resource_3d #创建3D资源
    (gdb) p args
    $11 = {handle = 3, target = 2, format = 67, bind = 10, width = 1, height = 1, depth = 1, array_size = 1, last_level = 0, nr_samples = 0, flags = 0}
    virgl_renderer_resource_create 
        |/* libvirglrenderer处理 */
        |-> vrend_renderer_resource_create(args)
            |-> vrend_renderer_resource_copy_args(args, gr)
            |-> vrend_renderer_resource_allocate_texture(gr)
                |-> gr->target = tgsitargettogltarget()

• VIRTIO_GPU_CMD_SUBMIT_3D. Qemu传递(buffer, context ID, word count)给virglrenderer的3D解码器

virgl_cmd_submit_3d /* 将out_sg内容拷贝提交给virglrenderer，前后端传递的是out_sg指针，不传递数据 */
    | /* 将out_sg[0].iov_base的前sizeof(cs)bytes拷贝到cs, 该32字节实际为hdr+size+padding */
    |-> VIRTIO_GPU_FILL_CMD(cs); //
    |-> g_malloc /* 申请cs.size(4136)字节buf */
    /* cmd->elem.out_sg(已转为HVA),此处是取out_sg[1].iov_base中的内容，长度保存在cs.size */
    |-> iov_to_buf /* 将out_sg[1].iov_base地址cs.size(4136)字节拷贝到buf */
    /* (iov=0x555557019258, iov_cnt=2, offset=32, buf=0x5555587afde0, bytes=4136) */
    |-> virgl_renderer_submit_cmd(buf, cs.hdr.ctx_id, cs.size / 4) /* 将buf、ctx_id、字宽提交给virglrender */
       /* cs = {hdr = {type = 519, flags = 1, fence_id = 1, ctx_id = 2, padding = 0}, size = 4136} */
        | /* 从CTX_CREATE中找到注册的submit_cmd回调 */
        |-> vrend_decode_ctx_submit_cmd()
            | /* decode_table[VIRGL_CCMD_RESOURCE_INLINE_WRITE]找到CMD相应回调 */
            |-> vrend_decode_resource_inline_write()
                |-> vrend_transfer_inline_write()
                    |-> vrend_renderer_transfer_write_iov() /* 完成真正的写入操作 */
                        | /* 若storage_bits不是VREND_STORAGE_GL_BUFFER就malloc堆内存，将绘制数据从iov拷贝到buf再使用opengl绘制；
                        | /* storage_bits其他类型可以直接用iov向opengl提交 */
                        | /* 不同GL_TEXTURE target使用不同gl命令 */
                        |-> glDrawPixels/glTexSubImage3D/glTexSubImage2D

• VIRTIO_GPU_CMD_CTX_ATTACH_RESOURCE

virgl_cmd_ctx_attach_resource #3D attach资源
    virgl_renderer_ctx_attach_resource
    p att_res
    $2 = {hdr = {type = 514, flags = 0, fence_id = 0, ctx_id = 1, padding = 0}, resource_id = 2, padding = 0}
    virgl_cmd_ctx_attach_resource()

• 7 virtio_gpu_virgl_process_cmd:Ctrl CMD处理流程

... #内核启动加载
virgl_renderer_force_ctx_0 #？
virgl_cmd_get_capset_info #返回capset版本
    virgl_renderer_get_cap_set #获取max_version:2和max_size:688
    virtio_gpu_ctrl_response #返回给前端，virgl_process_cmd处理完毕
virtio_gpu_get_edid #获取edid
    virtio_gpu_generate_edid #由qemu生成，包含product，video，screen，display，color，blocks等
    virtio_gpu_ctrl_response #答复非前端
virtio_gpu_get_display_info #
virgl_cmd_context_create #3D创建context
    virgl_renderer_context_create #virgl创建
virgl_cmd_set_scanout
virgl_cmd_transfer_to_host_2d
virgl_cmd_set_scanout
virgl_cmd_resource_flush
...
virgl_cmd_ctx_attach_resource #3D attach资源
    virgl_renderer_ctx_attach_resource
virgl_cmd_create_resource_3d #创建3D资源
    virgl_renderer_resource_create 
virgl_cmd_submit_3d #3D提交，将out_sg内容拷贝提交给virgl，传递的是out_sg地址，不传递数据
    g_malloc #申请4136字节内存
    iov_to_buf #iov为cmd->elem.out_sg(已转为HVA)，将out_sg内容拷贝到buf
    virgl_renderer_submit_cmd #将buf提交给virgl

• 6 virtio_gpu_virgl_process_cmd:CMD命令统计

 #统计运行glxgear 5秒各个CMD运行在后端执行次数
 VIRTIO_GPU_CMD_SUBMIT_3D   命令字调用最频繁并有内存拷贝，其他命令字没有内存拷贝

4.4.4 Fence同步机制

• virtio_gpu_virgl_init. qemu在处理ctrl cmd前初始化virgl和fence_poll定时器

virtio_gpu_handle_ctrl
    |-> virtio_gpu_virgl_init
        /* 给virgl初始化函数传了virgl_renderer_callback */
        |-> virgl_renderer_init(g, 0, &virtio_gpu_3d_cbs)
        /* 创建ms级定时器，定时回调virtio_gpu_fence_poll */
        |-> g->fence_poll = timer_new_ms(QEMU_CLOCK_VIRTUAL, virtio_gpu_fence_poll, g);

• virtio_gpu_3d_cbs. qemu注册给virglrender的callback

  static struct virgl_renderer_callbacks virtio_gpu_3d_cbs = {
    .version             = 1,
    .write_fence         = virgl_write_fence,
    .create_gl_context   = virgl_create_context,
    .destroy_gl_context  = virgl_destroy_context,
    .make_current        = virgl_make_context_current,
};

• virgl_renderer_create_fence.qemu创建fence_id

virtio_gpu_virgl_process_cmd
 /* cmd->cmd_hdr.flags为1的CMD触发创建fence；
 目前发现VIRTIO_GPU_CMD_SUBMIT_3D和VIRTIO_GPU_CMD_RESOURCE_CREATE_3D */
 virgl_renderer_create_fence /* 将cmd->cmd_hdr.fence_id给下去即可 */
   |-> vrend_renderer_create_ctx0_fence
       |-> vrend_renderer_create_fence
           |-> malloc /* vrend_fence结构，将fence_id，ctx，flgs赋值 */
           |-> fence->glsyncobj = glFenceSync(GL_SYNC_GPU_COMMANDS_COMPLETE, 0);
           /* glFenceSync创建一个新的fence同步对象，将fence命令插入GL命令流并将其与该同步对象相关联，并返回与该同步对象相对应的非零名称 */
           /* 当fence命令满足了同步对象的指定条件时，GL将用信号通知同步对象，从而使所有在同步中阻塞的glWaitSync和glClientWaitSync命令解除阻塞 */
           |-> glFlush()

• virgl_write_fence. 通知给前端fence信息

//从fenceq检测fence_id是否大于fence；将小于该值的，发送response通知前端。
ctx0_fence_retire /* in virglrenderer */
    |-> QTAILQ_FOREACH_SAFE /* 编译fenceq */
    /* cmd->cmd_hdr.fence_id > fence的cmd通知前端*/
    |-> virtio_gpu_ctrl_response_nodata /* 将cmd_hdr拷贝回去，携带fence信息 */
    |-> g_free(cmd) 

• virtio_gpu_fence_poll. qemu发起检查retire fence并通知前端

 virtio_gpu_handle_ctrl
     |-> virtio_gpu_process_cmdq
     |-> virtio_gpu_virgl_fence_poll /* 每一次处理cmdq后，都要pull fence */
         |-> virgl_renderer_poll /* virglrenderer处理 */
             |-> vrend_renderer_poll /* 未使用use_async_fence_cb */
                 |-> vrend_renderer_check_fences
                     |-> vrend_renderer_force_ctx_0 /* 切换到ctx0 */
                     |-> do_wait(fence, /* can_block */ false))  /* glClientWaitSync等待fence */
                     /* 将fence加入到retired_fences链表，其不会被signal */
                     |-> vrend_renderer_check_queries /* atomic_store */
                     /* 对retired_fences链表进行write操作 */
                     |-> ctx0_fence_retire /* ctx->fence_retire回调 */
                     /* cbs->write_fence回调，实际为qemu初始化virgl时设置的virtio_gpu_3d_cbs */
                     |-> virgl_write_fence 
         |-> virtio_gpu_process_cmdq
         |-> timer_mod /* 将定时器延长10ms */

4.5 Virglrenderer

4.5.1 流程

1. virtio-gpu后端3D CMD都会依赖virglrenderer的接口
2. virglrenderer初始化. virtio_gpu_handle_ctrl -> virgl_renderer_init
3. Context初始化. VIRTIO_GPU_CMD_CTX_CREATE -> 初始化context

  a. vrend_decode_ctx_init_base #检查decode_table和注册3D指令回调
     ctx->destroy = vrend_decode_ctx_destroy;
     ctx->attach_resource = vrend_decode_ctx_attach_resource;
     ctx->detach_resource = vrend_decode_ctx_detach_resource;
     ctx->transfer_3d = vrend_decode_ctx_transfer_3d;
     ctx->get_blob = vrend_decode_ctx_get_blob;
     ctx->submit_cmd = vrend_decode_ctx_submit_cmd;
     ctx->get_fencing_fd = vrend_decode_ctx_get_fencing_fd;
     ctx->retire_fences = vrend_decode_ctx_retire_fences;
     ctx->submit_fence = vrend_decode_ctx_submit_fence;
  b. vrend_create_context #初始化grctx

4. 3D解码. virglrender从VIRTIO_GPU_CMD_SUBMIT_3D携带的buf中解析VIRGL_CCMD

 VIRTIO_GPU_CMD_SUBMIT_3D
    | /* 向virglrender提交(buf,ctx_id,word_size) */
    | /* virglrender从buf中解析VIRGL_CCMD */
    |-> vrend_decode_ctx_submit_cmd() /* buf包含多个virgl_cmd */
        |-> decode_table[VIRGL_CCMD] /* 针对不同VIRGL_CCMD回调 */

4.5.2 decode_table

1. decode_table[cmd]. virgl 3D解码表，根据buf中virgl cmd调用不同方法进行解码

/* virglrender从buf中解析VIRGL_CCMD */
vrend_decode_ctx_submit_cmd() /* vrend_decode.c：buf包含多个virgl_cmd */
    buf_size = 4136, sizeof(uint32_t)=4, buf_total=4136/4= 1034 bytes
    uint32_t len = *buf >> 16;  /* *buf意为buf[0]，前4字节内容；右移16位意为取高2字节为len */
    uint32_t cmd = *buf & 0xff; /* 与0xff取&，意为取*buf的前16个位，即buf的前2个字节为cmd */

// 如上图为buf: 起始地址0x5555587dd400，*buf为0x03ff002c，len为高位0x03ff，cmd为低位0x002c
// 第1个CCMD常为占位CMD，空实现
// 第2个CCMD起始为0x5555587dd400+0x1000 ((0x03ff+1)*4) = 0x5555587de400

while(buf_offset < buf_total) // buf前1024*4字节为头，后面每个CCMD长度为len+1（len为payload长度，单位4字节）
  decode_table[VIRGL_CCMD] /* 针对不同VIRGL_CCMD回调 */
    /* cmd=44 len=1023，buf_offset=1024*/
    vrend_decode_dummy /* CCMD0：VIRGL_CCMD_NOP 空实现 */
    /*  len=1 cmd=29 buf_offset=1026 */
    vrend_decode_create_sub_ctx  /* CCMD2：VIRGL_CCMD_CREATE_SUB_CTX */
       /* payload为ctx_sub_id */
       glGenFramebuffers/glBindFramebuffer
    vrend_decode_set_sub_ctx /* CCMD3：VIRGL_CCMD_SET_SUB_CTX */
    vrend_decode_set_tweaks  /* CCMD4：VIRGL_CCMD_SET_TWEAKS */
    vrend_decode_set_tweaks  /* CCMD5：VIRGL_CCMD_SET_TWEAKS */

2. buf格式说明.buf包含多个CCMD，每个CCMD包含cmd+len+payload，cmd+len占4个字节，payload占len*4字节，如下图为ccmd示意图：

4.5.3 VIRGL_CCMD介绍

• virgl_context_cmd是VIRTIO_GPU_CMD_SUBMIT_3D在virglrender环境中3D解码表，统称VIRGL_CCMD，每条命令基本都依赖一系列OpenGL接口实现
• virgl_context_cmd支持的命令列表：

VIRGL_CCMD	描述
VIRGL_CCMD_NOP = 0	vrend_decode_dummy
VIRGL_CCMD_CREATE_OBJECT = 1	vrend_decode_create_object
VIRGL_CCMD_BIND_OBJECT	vrend_decode_bind_object
VIRGL_CCMD_DESTROY_OBJECT	vrend_decode_destroy_object
VIRGL_CCMD_SET_VIEWPORT_STATE	vrend_decode_set_viewport_state
VIRGL_CCMD_SET_FRAMEBUFFER_STATE	vrend_decode_set_framebuffer_state
VIRGL_CCMD_SET_VERTEX_BUFFERS	vrend_decode_set_vertex_buffers
VIRGL_CCMD_CLEAR	vrend_decode_clear
VIRGL_CCMD_DRAW_VBO	vrend_decode_draw_vbo
VIRGL_CCMD_RESOURCE_INLINE_WRITE	vrend_decode_resource_inline_write
VIRGL_CCMD_SET_SAMPLER_VIEWS	vrend_decode_set_sampler_views
VIRGL_CCMD_SET_INDEX_BUFFER	vrend_decode_set_index_buffer
VIRGL_CCMD_SET_CONSTANT_BUFFER	vrend_decode_set_constant_buffer
VIRGL_CCMD_SET_STENCIL_REF	vrend_decode_set_stencil_ref
VIRGL_CCMD_SET_BLEND_COLOR	vrend_decode_set_blend_color
VIRGL_CCMD_SET_SCISSOR_STATE	vrend_decode_set_scissor_state
VIRGL_CCMD_BLIT	vrend_decode_blit
VIRGL_CCMD_RESOURCE_COPY_REGION	vrend_decode_resource_copy_region
VIRGL_CCMD_BIND_SAMPLER_STATES	vrend_decode_bind_sampler_states
VIRGL_CCMD_BEGIN_QUERY	vrend_decode_begin_query
VIRGL_CCMD_END_QUERY	vrend_decode_end_query
VIRGL_CCMD_GET_QUERY_RESULT	vrend_decode_get_query_result
VIRGL_CCMD_SET_POLYGON_STIPPLE	vrend_decode_set_polygon_stipple
VIRGL_CCMD_SET_CLIP_STATE	vrend_decode_set_clip_state
VIRGL_CCMD_SET_SAMPLE_MASK	vrend_decode_set_sample_mask
VIRGL_CCMD_SET_STREAMOUT_TARGETS	vrend_decode_set_streamout_targets
VIRGL_CCMD_SET_RENDER_CONDITION	vrend_decode_set_render_condition
VIRGL_CCMD_SET_UNIFORM_BUFFER	vrend_decode_set_uniform_buffer
VIRGL_CCMD_SET_SUB_CTX	vrend_decode_set_sub_ctx
VIRGL_CCMD_CREATE_SUB_CTX	vrend_decode_create_sub_ctx
VIRGL_CCMD_DESTROY_SUB_CTX	vrend_decode_destroy_sub_ctx
VIRGL_CCMD_BIND_SHADER	vrend_decode_bind_shader
VIRGL_CCMD_SET_TESS_STATE	vrend_decode_set_tess_state
VIRGL_CCMD_SET_MIN_SAMPLES	vrend_decode_set_min_samples
VIRGL_CCMD_SET_SHADER_BUFFERS	vrend_decode_set_shader_buffers
VIRGL_CCMD_SET_SHADER_IMAGES	vrend_decode_set_shader_images
VIRGL_CCMD_MEMORY_BARRIER	vrend_decode_memory_barrier
VIRGL_CCMD_LAUNCH_GRID	vrend_decode_launch_grid
VIRGL_CCMD_SET_FRAMEBUFFER_STATE_NO_ATTACH	vrend_decode_set_framebuffer_state_no_attach
VIRGL_CCMD_TEXTURE_BARRIER	vrend_decode_texture_barrier
VIRGL_CCMD_SET_ATOMIC_BUFFERS	vrend_decode_set_atomic_buffers
VIRGL_CCMD_SET_DEBUG_FLAGS	vrend_decode_set_debug_mask
VIRGL_CCMD_GET_QUERY_RESULT_QBO	vrend_decode_get_query_result_qbo
VIRGL_CCMD_TRANSFER3D	vrend_decode_transfer3d
VIRGL_CCMD_END_TRANSFERS	vrend_decode_dummy
VIRGL_CCMD_COPY_TRANSFER3D	vrend_decode_copy_transfer3d
VIRGL_CCMD_SET_TWEAKS	vrend_decode_set_tweaks
VIRGL_CCMD_CLEAR_TEXTURE	vrend_decode_clear_texture
VIRGL_CCMD_PIPE_RESOURCE_CREATE	vrend_decode_pipe_resource_create
VIRGL_CCMD_PIPE_RESOURCE_SET_TYPE	vrend_decode_pipe_resource_set_type
VIRGL_CCMD_GET_MEMORY_INFO	vrend_decode_get_memory_info
VIRGL_CCMD_SEND_STRING_MARKER	vrend_decode_send_string_marker
VIRGL_CCMD_LINK_SHADER	vrend_decode_link_shader
VIRGL_CCMD_CREATE_VIDEO_CODEC	vrend_decode_create_video_codec
VIRGL_CCMD_DESTROY_VIDEO_CODEC	vrend_decode_destroy_video_codec
VIRGL_CCMD_CREATE_VIDEO_BUFFER	vrend_decode_create_video_buffer
VIRGL_CCMD_DESTROY_VIDEO_BUFFER	vrend_decode_destroy_video_buffer
VIRGL_CCMD_BEGIN_FRAME	vrend_decode_begin_frame
VIRGL_CCMD_DECODE_MACROBLOCK	vrend_decode_dummy
VIRGL_CCMD_DECODE_BITSTREAM	vrend_decode_decode_bitstream
VIRGL_CCMD_ENCODE_BITSTREAM	vrend_decode_encode_bitstream
VIRGL_CCMD_END_FRAME	vrend_decode_end_frame
VIRGL_MAX_COMMANDS

5 DEBUG

5.1 GDB调试内核

• 1 编译内核

#安装依赖
sudo apt-get install  libncurses5-dev   openssl libssl-dev build-essential pkg-config libc6-dev bison flex libelf-dev zlibc minizip libidn11-dev libidn11 -y
uname -r 
#下载一个与当前ubuntu版本接近的内核源码
wget https://mirrors.edge.kernel.org/pub/linux/kernel/v5.x/linux-5.11.10.tar.gz
tar xvf linux-5.11.10.tar.gz

make mrproper
#可以使用xconfig配置，依赖qt5
sudo apt install qt5-default
make xconfig 

#或者拷贝原ubuntu配置
cp -v /boot/config-$(uname -r) .config
make localmodconfig
#全都默认回车

 - CONFIG_DEBUG_INFO=y在5.11默认已打开
 - 关闭KASLR
Processor type and features ---->
    [ ] Randomize the address of the kernel image (KASLR)

• .config中置空SIG_KEY引用的文件，参考如下：

#编译内核
make bzImage -j6
#编译模块
make modules -j6
sudo make modules_install 

• 2 制作启动文件. 用于更新ubuntu内核

#必须要/lib目录的modules，不然无法生成initrd
mkinitramfs /lib/modules/5.11.10/ -o ./modules-vm/initrd.img-5.11.10-generic
#将bzImage System.map initrd.img-5.11.10-generic拷贝到目标Linux的/boot目录
cp initrd.img-5.11.10-generic /boot/initrd.img-5.11.10-generic
cp arch/x86/boot/bzImage    /boot/vmlinuz-5.11.10-generic
cp ./System.map    /boot/System.map-5.11.10

• 3 更新grub. 检查grub.cfg是否更改

cd /boot/grub/ ; update-grub2

• 2 qemu debug kernel

#前提：重新编译内核，增加内核调试。还是没能成功给内核打断点
#此处使用qemu 4.2调试内核，查看内核virtio-gpu流程 
#qemu启动增加  -s -S参数
#gdb debug编译的内核
gdb vmlinux
(gdb) target remote localhost:1234
(gdb) b start_kernel

5.2 GDB调试qemu

• 编译qemu带上–enable-debug 参数, smp建议使用单核

gdb ./qemu-system-x86_64
(gdb) set args --enable-kvm -m 1024 -smp 1 -serial stdio -vga virtio -display gtk,gl=on ../ubuntu.img
(gdb) b virtio_gpu_class_init
(gdb) r

5.3 qemu trace events

• 0 在qemu中跟踪一些函数调用次数和调用结果，可以使用qemu trace events
• 1 编译qemu时默认–enable-trace-backends=log
• 2 将trace events写到/tmp/events文件

#events参考qemu trace-events文件
echo "virtio_gpu_cmd_get_display_info" >> /tmp/events
echo "virtio_gpu_cmd_res_create_3d" >> /tmp/events

• 3 启动qemu时增加-trace events=/tmp/events，-D输出qemu日志到文件

./qemu-system-x86_64-ori --enable-kvm -m 1024 -smp 1 -serial stdio -vga virtio -display gtk,gl=on ../../ubuntu.img -trace events=/tmp/events -D qemu.log

• 4 查看qemu.log记录trace事件

3742117@1669195454.235531:virtio_gpu_cmd_get_display_info
3742117@1669195469.487838:virtio_gpu_cmd_res_create_3d res 0x3, fmt 0x43, w 1, h 1, d 1
3742117@1669195469.550025:virtio_gpu_cmd_res_create_3d res 0x4, fmt 0x2, w 640, h 480, d 1

6 问题解决

• 1. 安装libepoxy时PolicyKit daemon: GDBus.Error:org.freedesktop.PolicyKit1.Error.Failed: No session for cookie

sudo apt install policykit-1-gnome
/usr/lib/policykit-1-gnome/polkit-gnome-authentication-agent-1 &

• 2. 编译mesa builddir/，缺少很多依赖

C shared or static library 'elf' not found --> sudo apt-get install libelf-dev
Dependency "wayland-scanner" not found --> sudo apt-get install libwayland-dev
Dependency "wayland-protocols" not found --> sudo apt-get install wayland-protocols
Dependency "xdamage" not found --> sudo apt-get install libxdamage-dev
Dependency "xcb-glx" not found --> sudo apt-get install libxcb-glx0-dev
Dependency "xcb-shm" not found --> sudo apt-get install libxcb-shm0-dev
Dependency "x11-xcb" not found --> sudo apt-get install libx11-xcb-dev
Dependency "xcb-dri2" not found --> sudo apt-get install libxcb-dri2-0-dev
Dependency "xcb-dri3" not found --> sudo apt-get install libxcb-dri3-dev
Dependency "xcb-present" not found --> sudo apt-get install libxcb-present-dev
Dependency "xshmfence" not found --> sudo apt-get install libxshmfence-dev
Dependency "xxf86vm" not found --> sudo apt-get install libxxf86vm-dev
Dependency "xrandr" not found --> sudo apt-get install libxrandr-dev

• 3. Couldn’t find current GLX or EGL

#vmvare player和native ubuntu遇到epoxy问题
qemu-system-x86_64: ../src/dispatch_common.c:863: epoxy_get_proc_address: Assertion `0 && "Couldn't find current GLX or EGL context.\n"' failed.

– 谷歌看起来没有很好的解决方法。应该是qemu版本问题，使用更新的

• 4. 搭建virtualbox环境

共享粘贴板：设备 -> 共享粘贴板 ->  双向
调整分辨率： Ctrl + C；设备 -> 安装增强功能 -> CD -> 安装软件 ->  重启 -> 视图 -> 自动调整窗口尺寸

qemu_gl_create_compile_shader: compile fragment error
0:2(10): error: GLSL ES 3.00 is not supported. Supported versions are: 1.10, 1.20, and 1.00 ES
 -- virtual box支持GLSL ES版本太低（1.0，而物理机3.2），跑不了virgl

8 参考

virtio-gpu原理
virtio-gpu官博
前后端