论文笔记Monocular Depth Estimation through Virtual-world Supervision and Real-world SfM Self-Supervision

Problem Formulation

• MDE on CNN : $\Psi (\theta ;x)\to d$
• ${\theta }^{\ast }={\min }_{\theta }\mathcal{L}(\theta ;X^r,X^s.Y^s)$.

Contributions

• A CNN architecture training on virtual-world supervision and real-world SfM self supervision.
• Reduce domain discrepancied between supervised (virtual world) and semi-supervised (real world) data at the space of the extracted features (backbone bottleneck) by Gradient reveral layer(GRL).

Methods

• Assume two sources of Data 1. Real-world traffic $X^r=\{x_t^r{\}}_{t=1}^{N^r}$, $N^r$ is the num of real-world sequences. 2. Analogous sequences $X^s=\{x_t^s{\}}_{t=1}^{N^s}$, $N_r$ is the Num of frames from the Virtual-world .

MonoDEVSNet architecture:$\Psi (\theta ;x)$

• $\Psi (\theta ;x)$ has three blocks: Encoding block with ${\theta }^{enc}$, a multi-scale pyramidal block, ${\theta }^{pyr}$ and a decoding block with ${\theta }^{dec}$.
• The role of the multi-scale pyramid block is to adapt the bottleneck of the chosen encoder to the decoder.
• $\mathcal{L}$ relies on three different losses, ${\mathcal{L}}^{sf}(\theta ,{\mathcal{V}}^{sf};X^r),{\mathcal{L}}^{sp}(\theta ,X^s;Y^s),{\mathcal{L}}^{DA}({\theta }^{enc},{\mathcal{V}}^{DA};X^r,X^s)$.
• ${\mathcal{L}}^{sf}(\theta ,{\mathcal{V}}^{sf};X^r)$ Sfm self-supervised loss is almost like Mono2.
• ${\mathcal{L}}^{sp}(\theta ,X^s;Y^s)$ will discard pixels with $d_t^s(p)\ge d^{max}$.
  
• Domain adaptation loss ${\mathcal{L}}^{DA}({\theta }^{enc},{\mathcal{V}}^{DA};X^r,X^s)$
• Aim at learning the Depth features, so hope the feature couldn’t be distinguished whether from real(target domain) or virtual wold(source domain).
• In the Gradient-Reversal-Layer, the domain invariance of ${\theta }^{enc}$ is measured by a binary target/source domain-classifier CNN,D,of weights $\{{\theta }^{enc},{\mathcal{L}}^{DA}\}$.

• $D({\theta }^{enc},{\mathcal{V}}^{DA};X_t^r)$ outputs 1 if $x\in X^r$ and 0 if $x\in X^s$. This means that during forward passes of training, it acts as an identity function, while, during back-propagation, it reverses the gradient vector passing through it. Both the GRL and ${\mathcal{V}}^{DA}$ are required at training time, but not at testing time.

Unsupervised Domain Adaptation by Backpropagation[1]

• At training time, in order to obtain domain-invariant features, we seek the parameters ${\theta }_f$ of the feature mapping that maximize the loss of the domain classifier (by making the two feature distributions as similar as possible), while simultaneously seeking the parameters ${\theta }_d$ of the domain classifier that minimize the loss of the domain classifier. In addition, we seek to minimize the loss of the label predictor.
  

• Such reduction can be accomplished by introducing a special gradient reversal layer (GRL) defined as follows. The gradient reversal layer has no parameters associated with it (apart from the meta-parameter λ, which is not updated by backpropagation). During the forward propagation, GRL acts as an identity transform. During the backpropagation though, GRL takes the gradient from the subsequent level, multiplies it by −λ and passes it to the preceding layer.

Important Reference

[1] Y. Ganin and V. Lempitsky, “Unsupervised domain adaptation by backpropagation,” in Int. Conf. on Machine Learning (ICML), 2015.