图像融合论文阅读：(DIF-Net)Unsupervised Deep Image Fusion With Structure Tensor Representations

@article{jung2020unsupervised,
title={Unsupervised deep image fusion with structure tensor representations},
author={Jung, Hyungjoo and Kim, Youngjung and Jang, Hyunsung and Ha, Namkoo and Sohn, Kwanghoon},
journal={IEEE Transactions on Image Processing},
volume={29},
pages={3845–3858},
year={2020},
publisher={IEEE}
}

---

论文级别：SCI AI
影响因子：10.6

[论文下载地址]

---

---

论文解读

早期2020年的一篇CNN-VIF论文。作者提出了一种基于CNN的【轻量级】【通用】图像融合网络DIF-Net
该网络使用CNN完成特征提取、特征融合、图像重建。
使用【多通道图像对比度】的【结构张量】作为损失函数。
该方法适用于任意维度的输入和输出。

一般来说，图像融合里的通用网络指的是可以实现多模态（红外-可见光，医学）、多聚焦、多曝光等图像融合的网络结构，并非某个特定图像融合任务专用的网络结构，具有广泛的适用性。

关键词

Image fusion, image contrast, structure tensor, convolutional neural network, and unsupervised learning.
图像融合，图像对比度，结构张量，卷积神经网络，无监督学习

核心思想

网络结构其实就是CNN+ ResBlock，融合使用Concat+卷积，重构和提取一样
主要有趣的地方在作者使用了对比度的结构张量作为损失函数。

扩展学习
[什么是图像融合？（一看就通，通俗易懂）]

相关背景知识

结构张量

令$I$为M通道图像，图像$I$像素（x，y）处的梯度可以用Jacobian矩阵表示：

$I_i$是图像$I$的第$i$个通道，$\nabla x$或者$\nabla y$是水平或者垂直方向的导数。
$v=[cos\theta ,sin\theta {]}^T$方向上的梯度由$J_{I^{x,y}}v$给出
假设使用欧几里得度量（也就是欧式距离），图像$I$上$v$方向的像素（x，y）处对比度，可以用$J_{I^{x,y}}v$计算：

2×2矩阵$Z_{I^{x,y}}=(J_{I^{x,y}}{)}^T{J}_{I^{x,y}}$是一个【结构张量（structure tensor）】：

结构张量是具有实值的对称矩阵，因此它具有两个实数且非负的特征值。结构张量的特征向量表示多通道图像对比度最大和最小的方向，相应的特征值表示变化率。

扩展学习
欧几里得度量

网络结构

作者提出的网络结构如下所示。目前看来清晰简单的结构，无需赘述，想要了解详细一点的同学可以去看原文。

损失函数

损失函数=强度项+结构项

数据集

• 训练：TNO

图像融合数据集链接
[图像融合常用数据集整理]

训练设置

实验

评价指标

• MI- FMI- X- SCD- H- P- M

扩展学习
[图像融合定量指标分析]

Baseline

• NSCT, CBF, GFF, GTF, ConvSR，VggML，FusionGAN，DenseFuse

✨✨✨扩展学习
✨✨✨强烈推荐必看博客[图像融合论文baseline及其网络模型]✨✨✨

实验结果

更多实验结果及分析可以查看原文：
[论文下载地址]

---

传送门

图像融合相关论文阅读笔记

[(MURF: Mutually Reinforcing Multi-Modal Image Registration and Fusion]
[(A Deep Learning Framework for Infrared and Visible Image Fusion Without Strict Registration]
[(APWNet)Real-time infrared and visible image fusion network using adaptive pixel weighting strategy]
[Dif-fusion: Towards high color fidelity in infrared and visible image fusion with diffusion models]
[Coconet: Coupled contrastive learning network with multi-level feature ensemble for multi-modality image fusion]
[LRRNet: A Novel Representation Learning Guided Fusion Network for Infrared and Visible Images]
[(DeFusion)Fusion from decomposition: A self-supervised decomposition approach for image fusion]
[ReCoNet: Recurrent Correction Network for Fast and Efficient Multi-modality Image Fusion]
[RFN-Nest: An end-to-end resid- ual fusion network for infrared and visible images]
[SwinFuse: A Residual Swin Transformer Fusion Network for Infrared and Visible Images]
[SwinFusion: Cross-domain Long-range Learning for General Image Fusion via Swin Transformer]
[(MFEIF)Learning a Deep Multi-Scale Feature Ensemble and an Edge-Attention Guidance for Image Fusion]
[DenseFuse: A fusion approach to infrared and visible images]
[DeepFuse: A Deep Unsupervised Approach for Exposure Fusion with Extreme Exposure Image Pair]
[GANMcC: A Generative Adversarial Network With Multiclassification Constraints for IVIF]
[DIDFuse: Deep Image Decomposition for Infrared and Visible Image Fusion]
[IFCNN: A general image fusion framework based on convolutional neural network]
[(PMGI) Rethinking the image fusion: A fast unified image fusion network based on proportional maintenance of gradient and intensity]
[SDNet: A Versatile Squeeze-and-Decomposition Network for Real-Time Image Fusion]
[DDcGAN: A Dual-Discriminator Conditional Generative Adversarial Network for Multi-Resolution Image Fusion]
[FusionGAN: A generative adversarial network for infrared and visible image fusion]
[PIAFusion: A progressive infrared and visible image fusion network based on illumination aw]
[CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion]
[U2Fusion: A Unified Unsupervised Image Fusion Network]
综述[Visible and Infrared Image Fusion Using Deep Learning]

图像融合论文baseline总结

[图像融合论文baseline及其网络模型]

其他论文

[3D目标检测综述：Multi-Modal 3D Object Detection in Autonomous Driving：A Survey]

其他总结

[CVPR2023、ICCV2023论文题目汇总及词频统计]

✨精品文章总结

✨[图像融合论文及代码整理最全大合集]
✨[图像融合常用数据集整理]

【如侵权请私信我删除】

如有疑问可联系：[email protected];
码字不易，【关注，收藏，点赞】一键三连是我持续更新的动力，祝各位早发paper，顺利毕业~