论文阅读：Fast Optical Flow using Dense Inverse Search

1. 论文总述

最近开始接触光流方面的工作，在大概了解了LK光流法之后，我就开始看这篇论文了，然而并没有看懂，所以本文只能记录个大概，具体细节后续等我弄明白了，再来补充。

本文主要是实现快速的稠密光流计算，是在金字塔LK光流法的基础，提出了一个逆向搜索，加快了patch correspondences？具体这个逆向搜索的详细讲解是在论文的附录部分，主要是把本来在等号右边的光流delta_u和delta_v的位置移到了等号左边，即为逆向搜索，原理后续还要细看。

2021_09_14补充：

最近基本把DIS搞懂了，就来填一下以前挖的坑：
基于LK光流法的优化求解流程：在参考帧image1中选择一个patch1，然后在需要被对齐的帧中找到另一个patch2，patch1和patch2的方差最小（参考博客），这时候求得的dx dy即为想要的解（理论基础是根据泰勒一阶展开让其偏导等于0），但由于这种得到的dx dy只是近似解，所以经典的传统光流算法中都是迭代求解：即dx = dx0 + delta（dx），由上述的参考博客可知：每次迭代求解的时候，都需要计算被对齐image的海塞矩阵，这就导致计算量很大。
DIS的主要工作就是：把参考帧当做被对齐帧，相当于角色互换，这样就只用计算一次海塞矩阵和梯度图像，省了不少计算量和时间！！！ 具体可以看DIS的OpenCV源码

In this paper we focus on improving the speed of optical flow in general, non-domain-specific scenarios, while
remaining close to the state-of-the-art flow quality. We propose two novel components with low time complexity, one
using inverse search for fast patch correspondences, and one
based on multi-scale aggregation for fast dense flow estimation. Additionally, a fast variational refinement step further
improves the solution of our dense inverse search-based
method. Altogether, we obtain speed-ups of 1-2 orders of
magnitude over state-of-the-art methods at comparable flow
quality operating points (Fig. 1).
The run-times are in the
range of 10-600 Hz on 1024×436 resolution images by us-
ing a single CPU core on a common desktop PC,

2. 光流鲁棒性遇到的挑战

A robust optical flow algorithm should cope with discontinuities (outliers, occlusions, motion discontinuities),
appearance changes (illumination, chromacity, blur, deformations), and large displacements. Decades after the pioneering research of Horn and Schunck [27] and Lucas and
Kanade [35] we have solutions for the first two issues [9, 38] and recent endeavors lead to significant progress in handling large displacements [45, 13, 11, 31, 49, 28, 36, 41, 2, 51, 53, 54, 55, 18]. This came at the cost of high runtimes usually not acceptable in computationally constrained
scenarios such as real-time applications.

3. 保持精度的同时提高速度的一些方法

• feature descriptors
• The feature matching usually employs a (reciprocal)
  nearest neighbor operation
• An optimization problem is often at the core of the flow
  extraction methods
• Parallel computation
• Learning.Most of the optical flow methods exploit training images for parameter tuning. However, this is only a rough embedding of prior knowledge. Only recently methods were proposed that successfully learn specific models from such training material. Wulff et al. [54]
  assume that any optical flow field can be approximated by
  a decomposition over a small learned basis of flow fields.
  Fischer et al. [18] construct Convolutional Neural Networks
  (CNNs) capable to solve the optical flow estimation.
• Coarse-to-fine optimizations
• Branch and bound and Priority queues
• Dynamic Vision Sensors

4. Fast inverse search for correspondences

关于逆向搜索的具体解释：

逆向搜索得到稀疏光流场，输入：1）中的输出，im1,im2在当前层的图像I0,I1_ext(I1扩展了一点重复的边缘)

 目标：I0(x,y) = I1_ext(x+u,y+v)  ---得到光流(u,v)

 传统光流求解方案：迭代求解 ：I0(x,y) = I1_ext(x+u+delU,y+v+delV)    u += delU,  v+= delV，文章公式：

 逆向搜索算法： I0(x+delU,y+delV) = I1_ext(x+u,y+v)    u += delU,  v+= delV，文章中公式

优势：每次迭代之前只需要计算一次Jacobian and Hessian ，计算速度提升很多，精度相差不大

这里参考了博客：DIS 光流详解

5. Fast optical flow with multi-scale reasoning

6. Fast Variational refinement

6.1 DIS 速度

这是论文附录中所示的速度，测试OpenCV中实现的DIS，发现速度基本和论文中这个指标一致

7. 参数选择及3个结论

8. 与其他方法对比及4个结论

We have four observations: i) Nearest Neighbor search
has a low number of incorrect matches and precise correspondences, but is very prone to outliers. ii) DeepMatching
has a high percentage of erroneous correspondences (with
small errors), but are very good at large displacements. iii)
In contrast to this, our method (DIS w/o Densification) generally produces fewer correspondences with small errors,
but is strongly affected by outliers. This is due to the fact
that the implicit SSD (sum of squared differences) error
minimization is not invariant to changes in orientation, contrast, and deformations. iv) Averaging all patches in each
scale (DIS), taking into account their photometric error as
described in eq. (3), introduces robustness towards these
outliers. It also decreases the error for approximately correct matches. Furthermore, it enables reducing the number
of patches at coarser scales, leading to lower run-time.

9. 本方法的不足之处

In the supplementary material we show variants of Fig. 4,
and 5 where all preprocessing times are included. Furthermore, we provide flow error maps on Sintel, where typical
failure cases of our method at motion discontinuities, large
displacements, and frame boundaries are observable.

10. 高帧率的影响

Often, a simpler and faster algorithm, combined with a
higher temporal resolution in the data, can yield better accuracy than a more powerful algorithm, on lower temporal resolutions. This has been analysed in detail in [24]
for the task of visual odometry. As noted in [8, 6] this is
also the case for optical flow, where large displacements,due to low-frame rate or strong motions are significantly
more difficult to estimate than small displacements. In contrast to the recent focus on handling ever larger displacements [13, 55, 51, 49, 36], we want to analyse how decreasing the run-time while increasing the frame-rate affects our
algorithm.

参考文献

1. DIS 光流详解
2. 总结：光流–LK光流–基于金字塔分层的LK光流–中值流
3. 计算机视觉–光流法(optical flow)简介
4. INDEMIND带你玩转OpenCV 4.0（一）：DIS光流