视频驱动V4L2子系统驱动架构
1 概述
Video4 for Linux 2是Linux内核中关于视频设备的内核驱动框架，为上层的访问底层的视频设备提供了统一的接口。凡是内核中的子系统都是有抽象硬件的差异，为上层提供统一的接口和提取出公共代码冗余等。V4L2支持三类设备：视频输入/输出设备、VBI设备和radio设备(其实还支持更多类型的设备，暂不讨论)，分别会在/dev目录下产生videoX、radioX和vbiX设备节点。
2 驱动框架1
图中芯片模块对应Soc的各个子模块，video_device结构体主要用来控制Soc的video模块，v4l2_device会包含多个v4l2_subdev, 每个v4l2_subdev用来控制各自的子模块，某些驱动不需要v4l2_subdev，依靠video模块就能实现功能。
3 驱动框架2
Linux系统中视频输入设备主要包括以下四个部分：
1. 字符设备驱动程序核心：v4l2本身就是一个字符设备，具有字符设备所有特性，暴露接口给用户空间。
2. v4l2驱动核心： 主要是构建一个内核中标准视频设备驱动的框架，为视频操作提供统一的接口函数。
3. 平台v4l2设备驱动：在v4l2框架下，根据平台自身的特性实现与平台相关的v4l2驱动部分，包括注册video_device和v4l2_dev。
4. 具体sensor驱动：主要上电、提供工作时钟、视频图像裁剪、流IO开关等，实现各种设备控制方法供上层调用 并注册v4l2_subdev。
4 V4L2框架
结构体v4l2_device、video_device、v4l2_subdev和v4l2_fh是搭建框架的主要元素。
从上图可看出，V4L2框架是一个标准的树形结构，v4l2_device充当了父设备，通过链表把所有注册到其下的子设备管理起来，这些设备可以GRABBER、VBI或RADIO。
v4l2_subdev是子设备，v4l2_subdev结构体包含了对设备操作的ops和ctrls，这部分代码和硬件相关，需要根据硬件实现，像摄像头设备需要实现控制上下电、读取ID、饱和度、对比度和视频数据流打开关闭的接口函数。
video_device用于创建子设备节点，把操作设备的接口暴露给用户空间。
v4l2_fh是每个子设备的文件句柄，在打开设备节点文件设置，方便 上层索引到v4l2_ctrl_handler, v4l2_ctrl_handler管理设备的ctrls，这些ctrls(摄像头设备)包括调节饱和度、对比度和白平衡等。
5 V4L2 core介绍
5.1 概述
v4l2驱动代码在drivers\media\v4l2-core文件夹下，可根据字面意思来理解基本的功能。videobuf是实现视频的内存分配，对于v4l2和v4l分别对应不同的文件，如videobuf-core和videobuf2-core, v4l2-dev、v4l2-device、v4l2-subdev分别对应video_device、v4l2_device、v4l2_subdev的实现，v4l2-ioctl实现ioctl等等。
video驱动代码在driver/media目录下，下面分多个子目录，其中platform目录存放的是不同Soc的驱动代码，对应video_device; 其他大多子目录如：i2c、mmc、usb、tuners、radio等对应subdev的实现。
v4l2驱动框架最重要的是理解ioctl， 另外v4l2驱动框架最主要的是各个ioctl实现的功能，这些实现方式需要在实际操作中多加理解，不是难点。
| 核心模块 | 由v4l2-dev.c实现，主要作用包括申请字符主设备号、注册class和提供video device注册注册等相关函数。 |
| v4l2框架 | 由v4l2-device.c、v4l2-subdev.c、v4l2-fh.c、v4l2-ctrls.c等文件实现，构建v4l2框架。 |
| videobuf管理 | 由videobuf2-core.c、videobuf2-dma-contig.c、videobuf2-memops.c、videobuf2-vmalloc.c、v4l2-mem2mem.c等文件实现，完成videobuffer的分配、管理和注销。 |
| ioctl框架 | 由v4l2-ioctl.c文件实现，构建v4l2_ioctl框架。 |
5.2 关键结构体说明
struct video_device 用来创建设备节点/dev/videoX
/*
* Newer version of video_device, handled by videodev2.c
* This version moves redundant code from video device code to
* the common handler
*/
/**
* struct video_device - Structure used to create and manage the V4L2 device
* nodes.
*
* @entity: &struct media_entity
* @intf_devnode: pointer to &struct media_intf_devnode
* @pipe: &struct media_pipeline
* @fops: pointer to &struct v4l2_file_operations for the video device
* @device_caps: device capabilities as used in v4l2_capabilities
* @dev: &struct device for the video device
* @cdev: character device
* @v4l2_dev: pointer to &struct v4l2_device parent
* @dev_parent: pointer to &struct device parent
* @ctrl_handler: Control handler associated with this device node.
* May be NULL.
* @queue: &struct vb2_queue associated with this device node. May be NULL.
* @prio: pointer to &struct v4l2_prio_state with device's Priority state.
* If NULL, then v4l2_dev->prio will be used.
* @name: video device name
* @vfl_type: V4L device type, as defined by &enum vfl_devnode_type
* @vfl_dir: V4L receiver, transmitter or m2m
* @minor: device node 'minor'. It is set to -1 if the registration failed
* @num: number of the video device node
* @flags: video device flags. Use bitops to set/clear/test flags.
* Contains a set of &enum v4l2_video_device_flags.
* @index: attribute to differentiate multiple indices on one physical device
* @fh_lock: Lock for all v4l2_fhs
* @fh_list: List of &struct v4l2_fh
* @dev_debug: Internal device debug flags, not for use by drivers
* @tvnorms: Supported tv norms
*
* @release: video device release() callback
* @ioctl_ops: pointer to &struct v4l2_ioctl_ops with ioctl callbacks
*
* @valid_ioctls: bitmap with the valid ioctls for this device
* @lock: pointer to &struct mutex serialization lock
*
* .. note::
* Only set @dev_parent if that can't be deduced from @v4l2_dev.
*/
struct video_device
{
#if defined(CONFIG_MEDIA_CONTROLLER)
struct media_entity entity;
struct media_intf_devnode *intf_devnode;
struct media_pipeline pipe;
#endif
//设备操作函数集
const struct v4l2_file_operations *fops;
u32 device_caps;
/* sysfs */
//v4l设备
struct device dev;
//字符设备
struct cdev *cdev;
struct v4l2_device *v4l2_dev;
struct device *dev_parent;
struct v4l2_ctrl_handler *ctrl_handler;
struct vb2_queue *queue;
struct v4l2_prio_state *prio;
/* device info */
char name[32];
enum vfl_devnode_type vfl_type;
enum vfl_devnode_direction vfl_dir;
int minor;
u16 num;
unsigned long flags;
//此属性用来区分一个物理设备上的多个索引
int index;
/* V4L2 file handles */
spinlock_t fh_lock;
struct list_head fh_list;
int dev_debug;
v4l2_std_id tvnorms;
/* callbacks */
void (*release)(struct video_device *vdev);
const struct v4l2_ioctl_ops *ioctl_ops;
DECLARE_BITMAP(valid_ioctls, BASE_VIDIOC_PRIVATE);
struct mutex *lock;
};
struct v4l2_device 用来描述一个v4l2设备实例
/**
* struct v4l2_device - main struct to for V4L2 device drivers
*
* @dev: pointer to struct device.
* @mdev: pointer to struct media_device, may be NULL.
* @subdevs: used to keep track of the registered subdevs
* @lock: lock this struct; can be used by the driver as well
* if this struct is embedded into a larger struct.
* @name: unique device name, by default the driver name + bus ID
* @notify: notify operation called by some sub-devices.
* @ctrl_handler: The control handler. May be %NULL.
* @prio: Device's priority state
* @ref: Keep track of the references to this struct.
* @release: Release function that is called when the ref count
* goes to 0.
*
* Each instance of a V4L2 device should create the v4l2_device struct,
* either stand-alone or embedded in a larger struct.
*
* It allows easy access to sub-devices (see v4l2-subdev.h) and provides
* basic V4L2 device-level support.
*
* .. note::
*
* #) @dev->driver_data points to this struct.
* #) @dev might be %NULL if there is no parent device
*/
struct v4l2_device {
//指向设备模型的指针
struct device *dev;
//指向一个媒体控制器的指针
struct media_device *mdev;
//管理子设备的双向链表，所有注册到子设备都需要加入到这个链表中
struct list_head subdevs;
//全局锁
spinlock_t lock;
//设备名称
char name[V4L2_DEVICE_NAME_SIZE];
//通知回调函数，通常用于设备传递事件，这些事件可以是自定义事件
void (*notify)(struct v4l2_subdev *sd,
unsigned int notification, void *arg);
//控制句柄
struct v4l2_ctrl_handler *ctrl_handler;
//设备优先级状态，一般有后台、交互、记录三种优先级，依次变高
struct v4l2_prio_state prio;
//本结构体的引用追踪
struct kref ref;
//设备释放函数
void (*release)(struct v4l2_device *v4l2_dev);
};
struct v4l2_subdev 用来描述一个v4l2的子设备实例
/**
* struct v4l2_subdev - describes a V4L2 sub-device
*
* @entity: pointer to &struct media_entity
* @list: List of sub-devices
* @owner: The owner is the same as the driver's &struct device owner.
* @owner_v4l2_dev: true if the &sd->owner matches the owner of @v4l2_dev->dev
* owner. Initialized by v4l2_device_register_subdev().
* @flags: subdev flags. Can be:
* %V4L2_SUBDEV_FL_IS_I2C - Set this flag if this subdev is a i2c device;
* %V4L2_SUBDEV_FL_IS_SPI - Set this flag if this subdev is a spi device;
* %V4L2_SUBDEV_FL_HAS_DEVNODE - Set this flag if this subdev needs a
* device node;
* %V4L2_SUBDEV_FL_HAS_EVENTS - Set this flag if this subdev generates
* events.
*
* @v4l2_dev: pointer to struct &v4l2_device
* @ops: pointer to struct &v4l2_subdev_ops
* @internal_ops: pointer to struct &v4l2_subdev_internal_ops.
* Never call these internal ops from within a driver!
* @ctrl_handler: The control handler of this subdev. May be NULL.
* @name: Name of the sub-device. Please notice that the name must be unique.
* @grp_id: can be used to group similar subdevs. Value is driver-specific
* @dev_priv: pointer to private data
* @host_priv: pointer to private data used by the device where the subdev
* is attached.
* @devnode: subdev device node
* @dev: pointer to the physical device, if any
* @fwnode: The fwnode_handle of the subdev, usually the same as
* either dev->of_node->fwnode or dev->fwnode (whichever is non-NULL).
* @async_list: Links this subdev to a global subdev_list or @notifier->done
* list.
* @asd: Pointer to respective &struct v4l2_async_subdev.
* @notifier: Pointer to the managing notifier.
* @subdev_notifier: A sub-device notifier implicitly registered for the sub-
* device using v4l2_device_register_sensor_subdev().
* @pdata: common part of subdevice platform data
*
* Each instance of a subdev driver should create this struct, either
* stand-alone or embedded in a larger struct.
*
* This structure should be initialized by v4l2_subdev_init() or one of
* its variants: v4l2_spi_subdev_init(), v4l2_i2c_subdev_init().
*/
struct v4l2_subdev {
#if defined(CONFIG_MEDIA_CONTROLLER)
//媒体控制器的实体
struct media_entity entity;
#endif
struct list_head list;
struct module *owner;
bool owner_v4l2_dev;
u32 flags;
//指向一个v4l2设备
struct v4l2_device *v4l2_dev;
//子设备的操作函数集
const struct v4l2_subdev_ops *ops;
//子设备内部操作函数集
const struct v4l2_subdev_internal_ops *internal_ops;
//控制句柄
struct v4l2_ctrl_handler *ctrl_handler;
//子设备名称
char name[V4L2_SUBDEV_NAME_SIZE];
//子设备所在的组标识
u32 grp_id;
//子设备私有数据指针，一般指向总线接口的客户端
void *dev_priv;
//子设备私有数据指针，一般指向总线接口的host端
void *host_priv;
//设备节点
struct video_device *devnode;
struct device *dev;
struct fwnode_handle *fwnode;
struct list_head async_list;
struct v4l2_async_subdev *asd;
struct v4l2_async_notifier *notifier;
struct v4l2_async_notifier *subdev_notifier;
struct v4l2_subdev_platform_data *pdata;
};
每个模块对应的实现函数
/**
* struct v4l2_subdev_ops - Subdev operations
*
* @core: pointer to &struct v4l2_subdev_core_ops. Can be %NULL
* @tuner: pointer to &struct v4l2_subdev_tuner_ops. Can be %NULL
* @audio: pointer to &struct v4l2_subdev_audio_ops. Can be %NULL
* @video: pointer to &struct v4l2_subdev_video_ops. Can be %NULL
* @vbi: pointer to &struct v4l2_subdev_vbi_ops. Can be %NULL
* @ir: pointer to &struct v4l2_subdev_ir_ops. Can be %NULL
* @sensor: pointer to &struct v4l2_subdev_sensor_ops. Can be %NULL
* @pad: pointer to &struct v4l2_subdev_pad_ops. Can be %NULL
*/
struct v4l2_subdev_ops {
const struct v4l2_subdev_core_ops *core;
const struct v4l2_subdev_tuner_ops *tuner;
const struct v4l2_subdev_audio_ops *audio;
const struct v4l2_subdev_video_ops *video;
const struct v4l2_subdev_vbi_ops *vbi;
const struct v4l2_subdev_ir_ops *ir;
const struct v4l2_subdev_sensor_ops *sensor;
const struct v4l2_subdev_pad_ops *pad;
};
struct v4l2_fh 用来跟踪文件句柄实例