【论文合集】RGBD Semantic Segmentation

来源：GitHub - Yangzhangcst/RGBD-semantic-segmentation: A paper list of RGBD semantic segmentation (processing)

RGBD semantic segmentation

A paper list of RGBD semantic segmentation.

*Last updated: 2022/07/26

Update log

2020/May - update all of recent papers and make some diagram about history of RGBD semantic segmentation.
2020/July - update some recent papers (CVPR2020) of RGBD semantic segmentation.
2020/August - update some recent papers (ECCV2020) of RGBD semantic segmentation.
2020/October - update some recent papers (CVPR2020, WACV2020) of RGBD semantic segmentation.
2020/November - update some recent papers (ECCV2020, arXiv), the links of papers and codes for RGBD semantic segmentation.
2020/December - update some recent papers (PAMI, PRL, arXiv, ACCV) of RGBD semantic segmentation.
2021/February - update some recent papers (TMM, NeurIPS, arXiv) of RGBD semantic segmentation.
2021/April - update some recent papers (CVPR2021, ICRA2021, IEEE SPL, arXiv) of RGBD semantic segmentation.
2021/July - update some recent papers (CVPR2021, ICME2021, arXiv) of RGBD semantic segmentation.
2021/August - update some recent papers (IJCV, ICCV2021, IEEE SPL, arXiv) of RGBD semantic segmentation.
2022/January - update some recent papers (TITS, PR, IEEE SPL, arXiv) of RGBD semantic segmentation.
2022/March - update benchmark results on Cityscapes and ScanNet datasets.
2022/April - update some recent papers (CVPR, BMVC, IEEE TMM, arXiv) of RGBD semantic segmentation.
2022/May - update some recent papers of RGBD semantic segmentation.
2022/July - update some recent papers of RGBD semantic segmentation.

Datasets

The papers related to datasets used mainly in natural/color image segmentation are as follows.

• [NYUDv2] The NYU-Depth V2 dataset consists of 1449 RGB-D images showing interior scenes, which all labels are usually mapped to 40 classes. The standard training and test set contain 795 and 654 images, respectively.
• [SUN RGB-D] The SUN RGB-D dataset contains 10,335 RGBD images with semantic labels organized in 37 categories. The 5,285 images are used for training, and 5050 images are used for testing.
• [2D-3D-S] Stanford-2D-3D-Semantic dataset contains 70496 RGB and depth images as well as 2D annotation with 13 object categories. Areas 1, 2, 3, 4, and 6 are utilized as the training and Area 5 is used as the testing set.
• [Cityscapes] Cityscapes contains a diverse set of stereo video sequences recorded in street scenes from 50 different cities, with high quality pixel-level annotations of 5 000 frames in addition to a larger set of 20 000 weakly annotated frames.
• [ScanNet] ScanNet is an RGB-D video dataset containing 2.5 million views in more than 1500 scans, annotated with 3D camera poses, surface reconstructions, and instance-level semantic segmentations.

Metrics

The papers related to metrics used mainly in RGBD semantic segmentation are as follows.

• [PixAcc] Pixel accuracy
• [mAcc] Mean accuracy
• [mIoU] Mean intersection over union
• [f.w.IOU] Frequency weighted IOU

Performance tables

Speed is related to the hardware spec (e.g. CPU, GPU, RAM, etc), so it is hard to make an equal comparison. We select four indexes namely PixAcc, mAcc, mIoU, and f.w.IOU to make comparison. The closer the segmentation result is to the ground truth, the higher the above four indexes are.

NYUDv2

Method	PixAcc	mAcc	mIoU	f.w.IOU	Input	Ref. from	Published	Year
POR	59.1	28.4	29.1		RGBD		CVPR	2013
RGBD R-CNN	60.3	35.1	31.3	47(in LSD-GF)	RGBD		ECCV	2014
DeconvNet	69.9	56.4	42.7	56	RGB	LSD-GF	ICCV	2015
DeepLab	68.7	46.9	36.8	52.5	RGBD	STD2P	ICLR	2015
CRF-RNN	66.3	48.9	35.4	51	RGBD	STD2P	ICCV	2015
Multi-Scale CNN	65.6	45.1	34.1	51.4	RGB	LCSF-Deconv	ICCV	2015
FCN	65.4	46.1	34	49.5	RGBD	LCSF-Deconv	CVPR	2015
Mutex Constraints	63.8	31.5		48.5 (in LSD-GF)	RGBD		ICCV	2015
E2S2	58.1	52.9	31	44.2	RGBD	STD2P	ECCV	2016
BI-3000	58.9	39.3	27.7	43	RGBD	STD2P	ECCV	2016
BI-1000	57.7	37.8	27.1	41.9	RGBD	STD2P	ECCV	2016
LCSF-Deconv		47.3			RGBD		ECCV	2016
LSTM-CF		49.4			RGBD		ECCV	2016
CRF+RF+RFS	73.8				RGBD		PRL	2016
RDFNet-152	76	62.8	50.1		RGBD		ICCV	2017
SCN-ResNet152			49.6		RGBD		ICCV	2017
RDFNet-50	74.8	60.4	47.7		RGBD		ICCV	2017
CFN(RefineNet)			47.7		RGBD		ICCV	2017
RefineNet-152	73.6	58.9	46.5		RGB		CVPR	2017
LSD-GF	71.9	60.7	45.9	59.3	RGBD		CVPR	2017
3D-GNN		55.7	43.1		RGBD		ICCV	2017
DML-Res50			40.2		RGB		IJCAI	2017
STD2P	70.1	53.8	40.1	55.7	RGBD		CVPR	2017
PBR-CNN			33.2		RGB		ICCBS	2017
B-SegNet	68	45.8	32.4		RGB		BMVC	2017
FC-CRF	63.1	39	29.5	48.4	RGBD		TIP	2017
LCR	55.6	31.7	21.8	39.9	RGBD		ICIP	2017
SegNet	54.1	30.5	21	38.5	RGBD	LCR	TPAMI	2017
D-Refine-152	74.1	59.5	47		RGB		ICPR	2018
TRL-ResNet50	76.2	56.3	46.4		RGB		ECCV	2018
D-CNN		56.3	43.9		RGBD		ECCV	2018
RGBD-Geo	70.3	51.7	41.2	54.2	RGBD		MTA	2018
Context	70	53.6	40.6		RGB		TPAMI	2018
DeepLab-LFOV	70.3	49.6	39.4	54.7	RGBD	STD2P	TPAMI	2018
D-depth-reg	66.7	46.3	34.8	50.6	RGBD		PRL	2018
PU-Loop	72.1		44.5		RGB		CVPR	2018
C-DCNN	69	50.8	39.8		RGB		TNNLS	2018
GAD	84.8	68.7	59.6		RGB		CVPR	2019
CTS-IM	76.3		50.6		RGBD		ICIP	2019
PAP	76.2	62.5	50.4		RGB		CVPR	2019
KIL-ResNet101	75.1	58.4	50.2		RGB		ACPR	2019
2.5D-Conv	75.9		49.1		RGBD		ICIP	2019
ACNet			48.3		RGBD		ICIP	2019
3M2RNet	76	63	48		RGBD		SIC	2019
FDNet-16s	73.9	60.3	47.4		RGB		AAAI	2019
DMFNet	74.4	59.3	46.8		RGBD		IEEE Access	2019
MMAF-Net-152	72.2	59.2	44.8		RGBD		arXiv	2019
RTJ-AA			42		RGB		ICRA	2019
JTRL-ResNet50	81.3	60.0	50.3		RGB		TPAMI	2019
3DN-Conv	52.4		39.3		RGB		3DV	2019
SGNet	76.8	63.1	51		RGBD		TIP	2020
SCN-ResNet101			48.3		RGBD		TCYB	2020
RefineNet-Res152-Pool4	74.4	59.6	47.6		RGB		TPAMI	2020
TSNet	73.5	59.6	46.1		RGBD		IEEE IS	2020
PSD-ResNet50	77.0	58.6	51.0		RGB		CVPR	2020
Malleable 2.5D	76.9		50.9		RGBD		ECCV	2020
BCMFP+SA-Gate	77.9		52.4		RGBD		ECCV	2020
MTI-Net	75.3	62.9	49.0		RGB		ECCV	2020
VCD+RedNet		63.5	50.7		RGBD		CVPR	2020
VCD+ACNet		64.4	51.9		RGBD		CVPR	2020
SANet	75.9		50.7		RGB		arXiv	2020
Zig-Zag Net (ResNet152)	77.0	64.0	51.2		RGBD		TPAMI	2020
MCN-DRM		56.1	43.1		RGBD		ICNSC	2020
CANet	76.6	63.8	51.2		RGBD		ACCV	2020
CEN(ResNet152)	77.7	65.0	52.5		RGBD		NeurIPS	2020
ESANet			50.5		RGBD		ICRA	2021
LWM(ResNet152)	81.46	65.24	51.51		RGB		TMM	2021
GLPNet(ResNet101)	79.1	66.6	54.6		RGBD		arXiv	2021
ESOSD-Net(Xception-65)	73.3	64.7	45.0		RGB		arXiv	2021
NANet(ResNet101)	77.9		52.3		RGBD		IEEE SPL	2021
InverseForm	78.1		53.1		RGB		CVPR	2021
FSFNet	77.9		52.0		RGBD		ICME	2021
CSNet	77.5	63.6	51.5		RGBD		ISPRS JPRS	2021
ShapeConv	75.8	62.8	50.2	62.6	RGBD		ICCV	2021
CI-Net	72.7		42.6		RGB		arXiv	2021
RGBxD	76.7	63.5	51.1		RGBD		Neurocomput.	2021
TCD(ResNet101)	77.8		53.1		RGBD		IEEE SPL	2021
RAFNet-50	73.8	60.3	47.5		RGBD		Displays	2021
RTLNet	77.7		53.1		RGBD		IEEE SPL	2021
H3S-Fuse	78.3		53.5		RGB		BMVC	2021
EBANet	76.82		51.51		RGBD		ICCSIP	2021
CANet(ResNet101)	77.1	64.6	51.5		RGBD		PR	2022
ADSD(ResNet50)	77.5	65.3	52.5		RGBD		arXiv	2022
InvPT			53.56		RGB		arXiv	2022
PGDENet	78.1	66.7	53.7		RGBD		IEEE TMM	2022
CMX	80.1		56.9		RGBD		arXiv	2022
RFNet	80.1	64.7	53.5		RGBD		IEEE TETCI	2022
MTF	79.0	66.9	54.2		RGBD		CVPR	2022
FRNet	77.6	66.5	53.6		RGBD		IEEE JSTSP	2022
DRD	51.0		38.2		RGB		IEEE ICASSP	2022
SAMD	74.4	67.2	52.3	61.9	RGBD		Neurocomput.	2022
BFFNet-152			47.5		RGBD		IEEE ICSP	2022
MQTransformer			49.18		RGBD		arXiv	2022
GED	75.9	62.4	49.4		RGBD		MTA	2022
LDF	84.8	68.7	59.6		RGB		MTA	2022
PCGNet	77.6		52.1		RGBD		IEEE ICMEW	2022

SUN RGB-D

Method	PixAcc	mAcc	mIoU	f.w.IOU	Input	Ref. from	Published	Year
FCN	68.2	38.4	27.4		RGB	SegNet	CVPR	2015
DeconvNet	66.1	32.3	22.6		RGB	SegNet	ICCV	2015
IFCN	77.7	55.5	42.7		RGB		arXiv	2016
CFN(RefineNet)			48.1		RGBD		ICCV	2017
RDFNet-152	81.5	60.1	47.7		RGBD		ICCV	2017
RefineNet-Res152	80.6	58.5	45.9		RGB		CVPR	2017
3D-GNN		57	45.9		RGBD		ICCV	2017
DML-Res50			42.3		RGB		IJCAI	2017
HP-SPS	75.7	50.1	38		RGB		BMVC	2017
FuseNet	76.3	48.3	37.3		RGBD		ACCV	2017
LRN	72.5	46.8	33.1		RGB		arXiv	2017
SegNet	72.6	44.8	31.8		RGB	MMAF-Net-152	TPAMI	2017
B-SegNet	71.2	45.9	30.7		RGB		BMVC	2017
LSD-GF		58			RGBD		CVPR	2017
TRL-ResNet101	84.3	58.9	50.3		RGB		ECCV	2018
CCF-GMA	81.4	60.3	47.1		RGB		CVPR	2018
D-Refine-152	80.8	58.9	46.3		RGB		ICPR	2018
Context	78.4	53.4	42.3		RGB		TPAMI	2018
D-CNN		53.5	42		RGBD		ECCV	2018
G-FRNet-Res101	75.3	47.5	36.9		RGB		arXiv	2018
DeepLab-LFOV	71.9	42.2	32.1		RGB		TPAMI	2018
PU-Loop	80.3		45.1		RGB		CVPR	2018
C-DCNN	77.3	50	39.4		RGB		TNNLS	2018
GAD	85.5	74.9	54.5		RGB		CVPR	2019
KIL-ResNet101	84.8	58	52		RGB		ACPR	2019
PAP	83.8	58.4	50.5		RGB		CVPR	2019
3M2RNet	83.1	63.5	49.8		RGBD		SIC	2019
CTS	82.4		48.5		RGBD		ICIP	2019
2.5D-Conv	82.4		48.2		RGBD		ICIP	2019
ACNet			48.1		RGBD		ICIP	2019
MMAF-Net-152	81	58.2	47		RGBD		arXiv	2019
LCR-RGBD			42.4		RGBD		CVPRW	2019
EFCN-8s	76.9	53.5	40.7		RGB		TIP	2019
DSNet	75.6		32.1		RGB		ICASSP	2019
JTRL-ResNet101	84.8	59.1	50.8		RGB		TPAMI	2019
SCN-ResNet152			50.7		RGBD		TCYB	2020
SGNet	81.8	60.9	48.5		RGBD		TIP	2020
CGBNet	82.3	61.3	48.2		RGB		TIP	2020
CANet-ResNet101	81.9		47.7		RGB		arXiv	2020
RefineNet-Res152-Pool4	81.1	57.7	47		RGB		TPAMI	2020
PSD-ResNet50	84.0	57.3	50.6		RGB		CVPR	2020
BCMFP+SA-Gate	82.5		49.4		RGBD		ECCV	2020
QGN	82.4		45.4		RGBD		WACV	2020
VCD+RedNet		62.9	50.3		RGBD		CVPR	2020
VCD+ACNet		64.1	51.2		RGBD		CVPR	2020
SANet	82.3		51.5		RGB		arXiv	2020
Zig-Zag Net (ResNet152)	84.7	62.9	51.8		RGBD		TPAMI	2020
MCN-DRM		54.6	42.8		RGBD		ICNSC	2020
CANet	82.5	60.5	49.3		RGBD		ACCV	2020
CEN(ResNet152)	83.5	63.2	51.1		RGBD		NeurIPS	2020
AdapNet++			38.4		RGBD		IJCV	2020
ESANet			48.3		RGBD		ICRA	2021
LWM(ResNet152)	82.65	70.21	53.12		RGB		TMM	2021
GLPNet(ResNet101)	82.8	63.3	51.2		RGBD		arXiv	2021
NANet(ResNet101)	82.3		48.8		RGBD		IEEE SPL	2021
FSFNet	81.8		50.6		RGBD		ICME	2021
CSNet	82.0	63.1	52.8		RGBD		ISPRS JPRS	2021
ShapeConv(ResNet101)	82.0	58.5	47.6	71.2	RGBD		ICCV	2021
CI-Net	80.7		44.3		RGB		arXiv	2021
RGBxD	81.7	58.8	47.7		RGBD		Neurocomput.	2021
TCD(ResNet101)	83.1		49.5		RGBD		IEEE SPL	2021
RAFNet-50	81.3	59.4	47.2		RGBD		Displays	2021
GRBNet	81.3		45.7		RGBD		TITS	2021
RTLNet	81.3		45.7		RGBD		IEEE SPL	2021
CANet(ResNet101)	85.2		50.6		RGBD		PR	2022
ADSD(ResNet50)	81.8	62.1	49.6		RGBD		arXiv	2022
PGDENet	87.7	61.7	51.0		RGBD		IEEE TMM	2022
CMX	83.3		51.1		RGBD		IEEE TMM	2022
RFNet	87.3	59.0	50.7		RGBD		IEEE TETCI	2022
MTF	84.7	64.1	53.0		RGBD		CVPR	2022
FRNet	87.4	62.2	51.8		RGBD		IEEE JSTSP	2022
DRD	48.9		39.5		RGB		IEEE ICASSP	2022
SAMD		63.4			RGBD		Neurocomput.	2022
BFFNet-152	86.7		44.6		RGBD		IEEE ICSP	2022
LDF	85.5	68.3	47.5		RGB		MTA	2022
PCGNet	82.1		49.0		RGBD		IEEE ICMEW	2022

2D-3D-S

Method	PixAcc	mAcc	mIoU	f.w.IOU	Input	Ref. from	Published	Year
Deeplab	64.3	46.7	35.5	48.5	RGBD	MMAF-Net-152	ICLR	2015
D-CNN	65.4		35.9		RGBD	CMX	ECCV	2018
DeepLab-LFOV	88.0	42.2	69.8		RGB	PU-Loop	TPAMI	2018
D-CNN	65.4	55.5	39.5	49.9	RGBD		ECCV	2018
PU-Loop	91.0		76.5		RGB		CVPR	2018
MMAF-Net-152	76.5	62.3	52.9		RGBD		arXiv	2019
3M2RNet	79.8	75.2	63		RGBD		SIC	2019
ShapeConv	82.7		60.6		RGBD	CMX	ICCV	2021
CMX	82.6		62.1		RGBD		arXiv	2022

Cityscapes

Benchmark Suite – Cityscapes Dataset

ScanNet

Benchmark Results - ScanNet Benchmark (2D Semantic label benchmark)

Paper list

• [POR] Gupta, S., et al. (2013). Perceptual Organization and Recognition of Indoor Scenes from RGB-D Images. IEEE Conference on Computer Vision and Pattern Recognition: 564-571. [Paper] [Code]
• [RGBD R-CNN] Gupta, S., et al. (2014). Learning Rich Features from RGB-D Images for Object Detection and Segmentation. European Conference on Computer Vision: 345-360. [Paper] [Code]
• [FCN] Long, J., et al. (2015). Fully convolutional networks for semantic segmentation. IEEE Conference on Computer Vision and Pattern Recognition: 3431-3440. [Paper] [Code]
• [CRF-RNN] Zheng, S., et al. (2015). Conditional Random Fields as Recurrent Neural Networks. IEEE International Conference on Computer Vision: 1529-1537. [Paper] [Code]
• [Mutex Constraints] Deng, Z., et al. (2015). Semantic Segmentation of RGBD Images with Mutex Constraints. IEEE International Conference on Computer Vision: 1733-1741. [Paper] [Code]
• [DeepLab] Chen, L., et al. (2015). Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs. International Conference on Learning Representations. [Paper] [Code]
• [Multi-Scale CNN] Eigen, D. and R. Fergus (2015). Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-scale Convolutional Architecture. IEEE International Conference on Computer Vision: 2650-2658. [Paper] [Code]
• [DeconvNet] Noh, H., et al. (2015). Learning Deconvolution Network for Semantic Segmentation. International Conference on Computer Vision: 1520-1528. [Paper] [Code]
• [LSTM-CF] Li, Z., et al. (2016). LSTM-CF: Unifying Context Modeling and Fusion with LSTMs for RGB-D Scene Labeling. European Conference on Computer Vision: 541-557. [Paper] [Code]
• [LCSF-Deconv] Wang, J., et al. (2016). Learning Common and Specific Features for RGB-D Semantic Segmentation with Deconvolutional Networks. European Conference on Computer Vision: 664-679. [Paper] [Code]
• [BI] Gadde, R., et al. (2016). Superpixel Convolutional Networks using Bilateral Inceptions. European Conference on Computer Vision: 597-613. [Paper] [Code]
• [E2S2] Caesar, H., et al. (2016). Region-Based Semantic Segmentation with End-to-End Training. European Conference on Computer Vision: 381-397. [Paper] [Code]
• [IFCN] Shuai, B., et al. (2016). Improving Fully Convolution Network for Semantic Segmentation. arXiv:1611.08986. [Paper] [Code]
• [CRF+RF+RFS] Thøgersen, M., et al. (2016). Segmentation of RGB-D Indoor Scenes by Stacking Random Forests and Conditional Random Fields. Pattern Recognition Letters 80, 208-215. [Paper] [Code]
• [SegNet] Badrinarayanan, V., et al. (2017). SegNet: A Deep Convolutional EnCoder-Decoder Architecture for Image Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 39(12): 2481-2495. [Paper] [Code]
• [LSD-GF] Cheng, Y., et al. (2017). Locality-Sensitive Deconvolution Networks with Gated Fusion for RGB-D Indoor Semantic Segmentation. IEEE Conference on Computer Vision and Pattern Recognition: 1475-1483. [Paper] [Code]
• [LCR] Chu, J., et al. (2017). Learnable contextual regularization for semantic segmentation of indoor scene images. IEEE International Conference on Image Processing: 1267-1271. [Paper] [Code]
• [RefineNet] Lin, G., et al. (2017). RefineNet: Multi-path Refinement Networks for High-Resolution Semantic Segmentation. IEEE Conference on Computer Vision and Pattern Recognition : 5168-5177, [Paper] [Code1] [Code2]
• [FuseNet] Hazirbas, C., et al. (2017). FuseNet: Incorporating Depth into Semantic Segmentation via Fusion-Based CNN Architecture. Asian Conference on Computer Vision: 213-228. [Paper] [Code]
• [STD2P] He, Y., et al. (2017). STD2P: RGBD Semantic Segmentation Using Spatio-Temporal Data-Driven Pooling. IEEE Conference on Computer Vision and Pattern Recognition: 7158-7167. [Paper] [Code]
• [RDFNet] Lee, S., et al. (2017). RDFNet: RGB-D Multi-level Residual Feature Fusion for Indoor Semantic Segmentation. IEEE International Conference on Computer Vision: 4990-4999. [Paper] [Code]
• [CFN(RefineNet)] Lin, D., et al. (2017). Cascaded Feature Network for Semantic Segmentation of RGB-D Images. IEEE International Conference on Computer Vision: 1320-1328. [Paper] [Code]
• [3D-GNN] Qi, X., et al. (2017). 3D Graph Neural Networks for RGBD Semantic Segmentation. IEEE International Conference on Computer Vision: 5209-5218. [Paper] [Code1] [Code2]
• [DML-Res50] Shen, T., et al. (2017). Learning Multi-level Region Consistency with Dense Multi-label Networks for Semantic Segmentation. Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence: 2708-2714. [Paper] [Code]
• [PBR-CNN] Zhang, Y., et al. (2017). Physically-Based Rendering for Indoor Scene Understanding Using Convolutional Neural Networks. IEEE Conference on Computer Vision and Pattern Recognition: 5057-5065. [Paper] [Code]
• [FC-CRF] Liu, F., et al. (2017). Discriminative Training of Deep Fully Connected Continuous CRFs With Task-Specific Loss. IEEE Transactions on Image Processing 26(5), 2127-2136. [Paper] [Code]
• [HP-SPS] Park, H., et al. (2017). Superpixel-based semantic segmentation trained by statistical process control. British Machine Vision Conference. [Paper] [Code]
• [LRN] Islam, M. A., et al. (2017). Label Refinement Network for Coarse-to-Fine Semantic Segmentation. arXiv1703.00551. [Paper] [Code]
• [G-FRNet-Res101] Islam, M. A., et al. (2018). Gated Feedback Refinement Network for Coarse-to-Fine Dense Semantic Image Labeling. arXiv:1806.11266 [Paper] [Code]
• [CCF-GMA] Ding, H., et al. (2018). Context Contrasted Feature and Gated Multi-scale Aggregation for Scene Segmentation. IEEE Conference on Computer Vision and Pattern Recognition: 2393-2402. [Paper] [Code]
• [Context] Lin, G., et al. (2018). Exploring Context with Deep Structured Models for Semantic Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(6), 1352-1366. [Paper] [Code]
• [D-Refine-152] Chang, M., et al. (2018). Depth-assisted RefineNet for Indoor Semantic Segmentation. International Conference on Pattern Recognition: 1845-1850. [Paper] [Code]
• [D-depth-reg] Guo, Y. and T. Chen (2018). Semantic segmentation of RGBD images based on deep depth regression. Pattern Recognition Letters 109: 55-64. [Paper] [Code]
• [RGBD-Geo] Liu, H., et al. (2018). RGB-D joint modeling with scene geometric information for indoor semantic segmentation. Multimedia Tools and Applications 77(17): 22475-22488. [Paper] [Code]
• [D-CNN] Wang, W. and U. Neumann (2018). Depth-aware CNN for RGB-D Segmentation. European Conference on Computer Vision: 144-161. [Paper] Code
• [TRL-ResNet50/101] Zhang, Z., et al. (2018). Joint Task-Recursive Learning for Semantic Segmentation and Depth Estimation. European Conference on Computer Vision. [Paper] [Code]
• [DeepLab-LFOV] Chen, L., et al. (2018). DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(4), 834-848. [Paper] [Code]
• [PU-Loop] Kong, S. and C. Fowlkes (2018). Recurrent Scene Parsing with Perspective Understanding in the Loop. IEEE Conference on Computer Vision and Pattern Recognition: 956-965. [Paper] [Code]
• [PAD-Net] Xu, D., et al. (2018). PAD-Net: Multi-Tasks Guided Prediction-and-Distillation Network for Simultaneous Depth Estimation and Scene Parsing. IEEE Conference on Computer Vision and Pattern Recognition: 675-684. [Paper] [Code]
• [C-DCNN] Liu, J., et al. (2018) Collaborative Deconvolutional Neural Networks for Joint Depth Estimation and Semantic Segmentation. IEEE Transactions on Neural Networks and Learning Systems 29(11): 5655-5666. [Paper] [Code]
• [EFCN-8s] Shuai, B., et al. (2019). Toward Achieving Robust Low-Level and High-Level Scene Parsing. IEEE Transactions on Image Processing, 28(3), 1378-1390. [Paper] [Code]
• [3M2RNet] Fooladgar, F., and Kasaei, S. (2019). 3M2RNet: Multi-Modal Multi-Resolution Refinement Network for Semantic Segmentation. Science and Information Conference: 544-557. [Paper] [Code]
• [RFBNet] Deng, L., et al. (2019). RFBNet: Deep Multimodal Networks with Residual Fusion Blocks for RGB-D Semantic Segmentation. arXiv:1907.00135 [Paper] [Code]
• [MMAF-Net-152] Fooladgar, F. and S. Kasaei (2019). "Multi-Modal Attention-based Fusion Model for Semantic Segmentation of RGB-Depth Images." arXiv:1912.11691. [Paper] [Code]
• [LCR-RGBD] Giannone, G. and B. Chidlovskii (2019). Learning Common Representation from RGB and Depth Images. IEEE Conference on Computer Vision and Pattern Recognition Workshops. [Paper] [Code]
• [ACNet] Hu, X., et al. (2019). ACNET: Attention Based Network to Exploit Complementary Features for RGBD Semantic Segmentation. IEEE International Conference on Image Processing: 1440-1444. [Paper] [Code]
• [DSNet] Jiang, F., et al. (2019). DSNET: Accelerate Indoor Scene Semantic Segmentation. IEEE International Conference on Acoustics, Speech and Signal Processing: 3317-3321. [Paper] [Code]
• [GAD] Jiao, J., et al. (2019). Geometry-Aware Distillation for Indoor Semantic Segmentation*. IEEE Conference on Computer Vision and Pattern Recognition: 2864-2873. [Paper] [Code]
• [RTJ-AA] Nekrasov, V., et al. (2019). Real-Time Joint Semantic Segmentation and Depth Estimation Using Asymmetric Annotations. International Conference on Robotics and Automation: 7101-7107. [Paper] [Code]
• [CTS-IM] Xing, Y., et al. (2019). Coupling Two-Stream RGB-D Semantic Segmentation Network by Idempotent Mappings. IEEE International Conference on Image Processing: 1850-1854. [Paper] [Code]
• [2.5D-Conv] Xing, Y. J., et al. (2019). 2.5d Convolution for RGB-D Semantic Segmentation. IEEE International Conference on Image Processing: 1410-1414. [Paper] [Code]
• [DMFNet] Yuan, J., et al. (2019). DMFNet: Deep Multi-Modal Fusion Network for RGB-D Indoor Scene Segmentation. IEEE Access 7: 169350-169358. [Paper] [Code]
• [PAP] Zhang, Z., et al. (2019). Pattern-Affinitive Propagation Across Depth, Surface Normal and Semantic Segmentation. IEEE Conference on Computer Vision and Pattern Recognition: 4101-4110. [Paper] [Code]
• [KIL-ResNet101] Zhou, L., et al. (2019). KIL: Knowledge Interactiveness Learning for Joint Depth Estimation and Semantic Segmentation. Asian Conference on Pattern Recognition: 835-848. [Paper] [Code]
• [FDNet-16s] Zhen, M., et al. (2019). Learning Fully Dense Neural Networks for Image Semantic Segmentation. The Thirty-Third AAAI Conference on Artificial Intelligence: 9283-9290. [Paper] [Code]
• [JTRL-ResNet50/101] Zhang, Z., et al. (2019). Joint Task-Recursive Learning for RGB-D Scene Understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence. [Paper] [Code]
• [3DN-Conv] Chen, Y., et al. (2019). 3D Neighborhood Convolution: Learning Depth-Aware Features for RGB-D and RGB Semantic Segmentation. International Conference on 3D Vision. [Paper] [Code]
• [SGNet] Chen, L.-Z., et al. (2020). Spatial Information Guided Convolution for Real-Time RGBD Semantic Segmentation. IEEE Transactions on Image Processing. [Paper] [Code]
• [SCN-ResNet101] Lin, D., et al. (2020). SCN: Switchable Context Network for Semantic Segmentation of RGB-D Images. IEEE Transactions on Cybernetics 50(3): 1120-1131. [Paper] [Code]
• [RefineNet-Res152-Pool4] Lin, G., et al. (2020). RefineNet: Multi-Path Refinement Networks for Dense Prediction. IEEE Transactions on Pattern Analysis and Machine Intelligence 42(5): 1228-1242. [Paper] [Code]
• [CANet-ResNet101] Tang, Q., et al. (2020). Attention-guided Chained Context Aggregation for Semantic Segmentation. arXiv:2002.12041. [Paper] [Code]
• [CGBNet] Ding, H., et al. (2020). Semantic Segmentation with Context Encoding and Multi-Path Decoding. IEEE Transactions on Image Processing 29: 3520-3533. [Paper] [Code]
• [TSNet] Zhou, W., et al. (2020). TSNet: Three-stream Self-attention Network for RGB-D Indoor Semantic Segmentation. IEEE Intelligent Systems. [Paper] [Code]
• [PSD-ResNet50] Zhou, L., et al. (2020). Pattern-Structure Diffusion for Multi-Task Learning. IEEE Conference on Computer Vision and Pattern Recognition. [Paper] [Code]
• [Malleable 2.5D] Xing, Y., et al. (2020). Malleable 2.5D Convolution: Learning Receptive Fields along the Depth-axis for RGB-D Scene Parsing. European Conference on Computer Vision. [Paper] [Code]
• [BCMFP+SA-Gate] Chen X., et al. (2020). Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentation. European Conference on Computer Vision. [Paper] [Code]
• [MTI-Net] Vandenhende S., et al. (2020). MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learning. European Conference on Computer Vision. [Paper] [Code]
• [QGN] Kashyap C., et al. (2020). Quadtree Generating Networks: Efficient Hierarchical Scene Parsing with Sparse Convolutions. IEEE Winter Conference on Applications of Computer Vision. [Paper] [Code]
• [VCD+RedNet/ACNet] Xiong, Z.-T., et al. (2020). Variational Context-Deformable ConvNets for Indoor Scene Parsing. IEEE Conference on Computer Vision and Pattern Recognition. [Paper] [Code]
• [SANet] Yu, L., et al. (2020). Multi-layer Feature Aggregation for Deep Scene Parsing Models. arXiv:2011.02572. [Paper] [Code]
• [Zig-Zag Net] Lin, D., et al. (2020). Zig-Zag Network for Semantic Segmentation of RGB-D Images. IEEE Transactions on Pattern Analysis and Machine Intelligence 42(10): 2642-2655. [Paper] [Code]
• [MCN-DRM] Zheng, Z., et al. (2020). Multi-resolution Cascaded Network with Depth-similar Residual Module for Real-time Semantic Segmentation on RGB-D Images. IEEE International Conference on Networking, Sensing and Control (ICNSC). [Paper] [Code]
• [CANet] Zhou H., et al. (2020). RGB-D Co-attention Network for Semantic Segmentation. Asian Conference on Computer Vision. [Paper] [Code]
• [CEN] Wang, Y., et al. (2020). Deep Multimodal Fusion by Channel Exchanging. 34th Conference on Neural Information Processing Systems [Paper] [Code]
• [Z-ACN] Wu, Z., et al. (2020). Depth-Adapted CNN for RGB-D cameras. Asian Conference on Computer Vision. [Paper]
• [AdapNet++] Valada, A., et al. (2020). Self-Supervised Model Adaptation for Multimodal Semantic Segmentation. International Journal of Computer Vision [Paper] [Code]
• [ESANet] Seichter, D., et al. (2021). Efficient RGB-D Semantic Segmentation for Indoor Scene Analysis. IEEE International Conference on Robotics and Automation. [Paper] [Code]
• [LWM] Gu, Z., et al. (2021). Hard Pixel Mining for Depth Privileged Semantic Segmentation. IEEE Transactions on Multimedia. [Paper] [Code]
• [GLPNet] Chen, S., et al. (2021). Global-Local Propagation Network for RGB-D Semantic Segmentation. arXiv:2101.10801. [Paper] [Code]
• [ESOSD-Net] He, L., et al. (2021). SOSD-Net: Joint Semantic Object Segmentation and Depth Estimation from Monocular images. arXiv:2101.07422. [Paper] [Code]
• [NANet] Zhang, G., et al. (2021). Non-local Aggregation for RGB-D Semantic Segmentation. IEEE Signal Processing Letters. [Paper] [Code]
• [ARLoss] Cao, L., et al. (2021). Use square root affinity to regress labels in semantic segmentation. arXiv:2103.04990. [Paper] [Code]
• [InverseForm] Borse, S., et al. (2021). InverseForm: A Loss Function for Structured Boundary-Aware Segmentation. IEEE Conference on Computer Vision and Pattern Recognition. [Paper] [Code]
• [FSFNet] Su, Y., et al. (2021). Deep feature selection-and-fusion for RGB-D semantic segmentation. IEEE International Conference on Multimedia and Expo. [Paper] [Code]
• [3D-to-2D] Liu, Z., et al. (2021). 3D-to-2D Distillation for Indoor Scene Parsing. IEEE Conference on Computer Vision and Pattern Recognition. [Paper] [Code]
• [ATRC] Bruggemann, D., et al. (2021). Exploring Relational Context for Multi-Task Dense Prediction. International Conference on Computer Vision [Paper] [Code]
• [CSNet] Huan L., et al. (2021). Learning deep cross-scale feature propagation for indoor semantic segmentation. ISPRS Journal of Photogrammetry and Remote Sensing [Paper] [Code]
• [ShapeConv] Cao J., et al. (2021). ShapeConv: Shape-aware Convolutional Layer for Indoor RGB-D Semantic Segmentation. International Conference on Computer Vision [Paper] [Code]
• [CI-Net] Gao T., et al. (2021). CI-Net: Contextual Information for Joint Semantic Segmentation and Depth Estimation. arXiv:2107.13800 [Paper]
• [UMT] Du C., et al. (2021). Improving Multi-Modal Learning with Uni-Modal Teachers. arXiv:2106.11059 [Paper]
• [RGBxD] Cao J., et al. (2021). RGBxD: Learning depth-weighted RGB patches for RGB-D indoorsemantic segmentation. Neurocomputing [Paper]
• [TCD] Yue Y., et al. (2021). Two-Stage Cascaded Decoder for Semantic Segmentation of RGB-D Images. IEEE Signal Processing Letters [Paper]
• [RAFNet] Yan X., et al. (2021). RAFNet: RGB-D attention feature fusion network for indoor semantic segmentation. Displays. [Paper] [Code]
• [MAMD] RazzaghiYan P., et al. (2021). Modality adaptation in multimodal data. Expert Systems with Applications.[Paper] [Code]
• [FuseNet-3DEF] Terreran M., et al. (2022). Light deep learning models enriched with Entangled features for RGB-D semantic segmentation. Robotics and Autonomous Systems. [Paper] [Code]
• [GRBNet] Zhou W., et al. (2021). Gated-Residual Block for Semantic Segmentation Using RGB-D Data. IEEE Transactions on Intelligent Transportation Systems. [Paper] [Code]
• [RTLNet] Zhou W., et al. (2021). RTLNet: Recursive Triple-Path Learning Network for Scene Parsing of RGB-D Images. IEEE Signal Processing Letters. [Paper] [Code]
• [EBANet] Wang R., et al. (2021). EBANet: Efficient Boundary-Aware Network for RGB-D Semantic Segmentation. International Conference on Cognitive Systems and Signal Processing. [Paper]
• [CANet] Zhou H., et al. (2022). CANet: Co-attention network for RGB-D semantic segmentation. Pattern Recognition. [Paper] [Code]
• [ADSD] Zhang Y., et al. (2022). Attention-based Dual Supervised Decoder for RGBD Semantic Segmentation. arXiv:2201.01427 [Paper] [Code]
• [HS3] Borse S., et al. (2021). HS3: Learning with Proper Task Complexity in Hierarchically Supervised Semantic Segmentation. British Machine Vision Conference [Paper] [Code]
• [InvPT] Ye H., et al. (2022). Inverted Pyramid Multi-task Transformer for Dense Scene Understanding. arXiv:2203.07997 [Paper] [Code]
• [PGDENet] Zhou W., et al. (2022). PGDENet: Progressive Guided Fusion and Depth Enhancement Network for RGB-D Indoor Scene Parsing. IEEE Transactions on Multimedia [Paper] [Code]
• [CMX] Liu X., et al. (2022). CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers. arXiv:2203.04838 [Paper] [Code]
• [RFNet] Zhou W., et al. (2022). RFNet: Reverse Fusion Network With Attention Mechanism for RGB-D Indoor Scene Understanding. IEEE Transactions on Emerging Topics in Computational Intelligence [Paper] [Code]
• [MTF] Wang Y., et al. (2022). Multimodal Token Fusion for Vision Transformers. IEEE Conference on Computer Vision and Pattern Recognition. [Paper] [Code]
• [FRNet] Zhou W., et al. (2022). FRNet: Feature Reconstruction NetworkforRGB-DIndoor Scene Parsing. IEEE Journal of Selected Topics in Signal Processing. [Paper] [Code]
• [DRD] Fang T., et al. (2022). Depth Removal Distillation for RGB-D Semantic Segmentation. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). [Paper]
• [SAMD] Zhou F., et al. (2022). Scale-aware network with modality-awareness for RGB-D indoor semantic segmentation. Neurocomputing. [Paper]
• [BFFNet-152] He Y., et al. (2022). Bimodal Feature Propagation and Fusion for Realtime Semantic Segmentation on RGB-D Images. International Conference on Intelligent Computing and Signal Processing. [Paper]
• [MQTransformer] Xu Y., et al. (2022). Multi-Task Learning with Multi-Query Transformer for Dense Prediction. arXiv:2203.04838 [Paper]
• [GED] Zou W., et al. (2022). RGB‑D Gate‑guided edge distillation for indoor semantic segmentation. Multimedia Tools and Applications. [Paper]
• [LDF] Chen S., et al. (2022). Learning depth‑aware features for indoor scene understanding. Multimedia Tools and Applications. [Paper]
• [PCGNet] Liu H., et al. (2022). Pyramid-Context Guided Feature Fusion for RGB-D Semantic Segmentation. IEEE International Conference on Multimedia and Expo Workshops (ICMEW). [Paper]