[Paper note] FlowNet: Learning Optical Flow with Convolutional Networks

• paper
• code

Highlight

• First paper to use trained CNN for optical flow estimation
• Introduce novel correlation layer
• Refine network by upsampling

Model

• 
• FlowNetSimple: concatenate two consecutive images.
• FlowNetCorr: use correlation layer
• Correlation layer
  
  • Calculated between two feature maps
  • c(x1,x2)=∑o∈[−k,k]×[−k,k]<f1(x1+o),f2(x2+o)>
  • See model picture for an illustration
• Refinement
  
  • 
  • Concatenate the upsampled flow prediction and conv feature map

Experiment

• Datasets:
  
  • Middlebury
  • KITTI
  • Sintel
  • Flying Chairs (proposed, auto generated)
• Loss function: endpoint error – Euclidean distance between the predicted flow vector and GT.
• Conclusion
  
  • FlowNet performs a little worse than other OF algorithm, but obviously faster.
  • Network trained on Flying Chairs (auto generated) data has good generalization ability on natural scenes.