国内外2D-to-3D方向的研究所以及文章

下面按照研究机构或者研究小组来分类，来了解国内外做2D-3D这个方向的一些情况。分别介绍如下：

1.         加拿大的CRC，即Communications Research Centre

研究中心的主页是http://www.crc.gc.ca/en/html/crc/home/home。这个研究中心Broadcast Technology下的Advanced Video Systems这个小组就是专门研究3D视频相关的技术。相关的网页是

http://www.crc.gc.ca/en/html/crc/home/3d/video_transfer/2d_3d_video_conversion。

AVS小组一共有4个研究方向：
 

• Video processing and coding (Team Leader: Dr. Liang Zhang)
• Video quality assessment and ATEL (Team Leader: Dr. Filippo Speranza)
• Human vision and 3DTV (Team Leader: Dr. James Tam)
• Multimedia communications (Team Leader: Dr. Omneya Issa)

做2D-3D这个方向主要有Liang Zhang，Carlos Vazquez，Wa James Tam。

这个小组做的2D-to-3D的转换基本是基于颜色的线索。这个研究中心发表文章的数据库：http://www1.crc.gc.ca/en/html/library/home/publications/pubdb/pubdb

与2D-3D相关的参考文章有：

[1]      W. J. Tam, L. Zhang, “Non-uniform smoothing of depth maps before image-based rendering”, Three-Dimensional TV, Video, and Display III, Philadelphia, PA, Oct. 26, 2004, Proceedings of SPIE, v. 5599, p. 173-183.

[2]      L. Zhang, W. J. Tam, D. Wang, “Stereoscopic image generation based on depth images”, ICIP 2004: International Conference on Image Processing, Singapore, Oct. 24-27, 2004 (CD-ROM p. 2993-2996).

[3]      L. Zhang, W. J. Tam, “Stereoscopic image generation based on depth images for 3D TV”, IEEE Transactions on Broadcasting, v. 51, n. 2, June 2005, p. 191-199.

[4]      W. J. Tam, L. Zhang, “3D-TV content generation: 2D-to-3D conversion”, IEEE International Conference on Multimedia and Expo (ICME), Toronto, ON, July 9-12, 2006. p.1869

[5]      C. Vazquez and W. J. Tam, “CRC-CSDM: 2D to 3D conversion using colour-based surrogate depth maps,” 3D Systems and Applications 3DSA 2010, Tokyo, Japan, May 19-21, 2010.

[6]      C. Vazquez and W. J. Tam, “A Non-conventional Approach to the Conversion of 2D Video and Film Content to Stereoscopic 3D,” 2010 SMPTE Annual Technical Conference & Expo, Hollywood, CA, USA, October 25-28, 2010.

[7]      L. Zhang, C. Vazquez, and S. Knorr, “3D-TV content creation: automatic 2D-to-3D conversion,” IEEE Trans. Broadcasting, vol. 56, no. 2, pp. 372-383, June 2011.

2.         清华大学，自动化系，戴琼海实验室

戴琼海是宽带网数字媒体技术实验室的主任，该实验室的主页是http://media.au.tsinghua.edu.cn/index.jsp。这个实验室的2D-3D方向主要是曹讯（Cao Xun）博士带领，2D-3D方向的主页是http://media.au.tsinghua.edu.cn/2Dto3D.jsp。

主要的研究方向有：

（1）半自动的2D-3D的转换

08年写的一篇文章[2]是描述了一种半自动的方法，深度传播算法采用KLT算法来跟踪人物轮廓，进而来传播深度，显然这种方法没有一般性，因为对于复杂的视频，采用KLT算法来实现跟踪本身就是很难的问题。09年写的一篇文章[3]，也是采用跟踪的方法来进行传播，不过采用的骨架跟踪的方法。先提取物体的骨架，然后进行跟踪，最后采用lazy snapping算法进行分割出物体。这种方法也存在类似的缺陷，不是每个视频中的帧都能分割成一个个有骨架的物体的，而且骨架的跟踪本身也不是很robust。11年Cao Xun才写了一篇比较综述的半自动系统的文章[5]发表在IEEE Trans. On Broadcasting上，该方法其实也不是很新的方法，但是至少是比较实用的，里面分割的方法是采用改进的Lazy Snapping算法实现多物体的分割，深度传播算法是采用SBF算法。到目前为止，这种算法是比较好的一种算法。

11年写的文章[7]是利用T形结的思路来手动分配图片的深度，并没有涉及到视频的深度传播问题，文章中说的传播是在一幅图片内的深度传播，并非视频帧之间的传播，帧内的深度传播也是比较重要的，可以大大减少manual cost。

文章[8]将2D-3D中的深度估计问题与遮挡和视觉关注模型结合起来，比较有新意。但是文章写的看不大懂，不知道怎么将遮挡和视觉关注结合的。从结果上来讲，效果一般，深度不是很准确。这种方法对于复杂的场景，也不太适合，因为复杂场景的视觉关注和遮挡分析都比较复杂。

（2）自动的2D-3D转换

08年写的文章[1]结合了运动和颜色的信息来实现自动的2D-3D转换，基本方法是还是采用运动分割的方法，不过运动分割采用的是光流得到MV图，然后利用彩色信息对其进行refine，得到运动分割的结果之后，也是根据一些常见的rules来分配深度，其中也需要人的采用，并不是说是完全自动的。

10年的文章[4]采用的是先分割后分配深度的方法，虽然是自动的方法，但是运动分割本身就是一个棘手的问题，所以不是很实用。

11年的文章[6]讲的是一些比较虚的问题，基本没讲什么实际的算法，只是一个介绍性的科普文章。

参考文章有：

[1]      Feng Xu, Guihua Er, Xudong Xie and Qionghai Dai, "2D-to-3D Conversion Based on Motion and Color Mergence", 3DTV Conference: The True Vision Capture, Transmission and Display of 3D Video, 2008.

[2]      Chenglei Wu, Guihua Er, Xudong Xie, Tao Li, Xun Cao, Qionghai Dai, "A Novel Method for Semi-automatic 2D to 3D Video Conversion", 3DTV Conference: The True Vision Capture, Transmission and Display of 3D Video, 2008.

[3]      Zheng Li, Xudong Xie, Xiaodong Liu, “An efficient 2D to 3D video conversion method based on skeleton line tracking”, 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video, 2009.

[4]      Youwei Yan, Feng Xu, Qionghai Dai, Xiaodong Liu. “A novel method for automatic 2D-to-3D video conversion”. 3DTV conference: The True Vision - Capture, Transmission and Display of 3D Video, 2010.

[5]      Xun Cao, Zheng Li and Qionghai Dai, " Semi-automatic 2D-to-3D conversion using disparity propagation" , IEEE Transactions on Broadcasting, Volume 57, June, 2011.

[6]      Xun Cao, Alan Bovik, Yao Wang and Qionghai Dai, " Converting 2D video to 3D: An Efficient Path to a 3D Experience" in IEEE Multimedia, Volume 18, Issue 4, pp.12-17, 2011.

[7]      Yan X, Yang Y, Er G H, Dai Q H. “Depth map generation for 2D-to-3D conversion based on limited user input and depth propagation”. in Proc. 3DTV Conference, Antalya, Turkey, 2011.

[8]      Jiahong Zhang, et al. “A Novel 2d-To-3d Scheme by Visual Attention and Occlusion Analysis”， in Proc. 3DTV Conference, Antalya, Turkey, 2011.

3.         国立台湾大学，Liang-Gee Chen实验室

Liang Gee Chen是DSPIC实验室的领导，01年就被选为IEEE Fellow，本硕博都在国立成功大学完成的。DSPIC实验室的主页是http://video.ee.ntu.edu.tw/。在这个实验室中，采用3D相关的人有蔡松芳 Sung-Fang Tsai博士（07级），李宗德 Chung-Te Li博士，叢培貴 Pei-Kuei Tsung博士，吳 謙 Chien Wu硕士，柯政遠 Chung-Yung Ko硕士。其中负责算法的是Chung-Te Li和Pei-Kuei Tsung，负责ic设计的是Sung-Fang Tsai。还有Chao-Chung Cheng，他是已经毕业的博士。

从发表的论文中可以看出，基本的算法都是Chao-Chung Cheng和Chung-Te Li负责的。

Yu-Lin Chang和Yi-Min Tsai这两个人可能是已经毕业了，他们也做过相关的算法工作。

所以排列下来，这个实验室3D方向应该是Yi-Min Tsai，Yu-Lin Chang，Chao-Chung Cheng（05级博士，已经毕业），Chung-Te Li（06级博士）。

相关的论文有：

[1]      Yi-Min Tsai and Yu-Lin Chang and Liang-Gee Chen, "Block-based Vanishing Line and Vanishing Point Detection for 3D Scene Reconstruction", in International Symposium on Intelligent Signal Processing and Communication Systems (ISPACS), 2006.

[2]      Yu-Lin Chang, Chih-Ying Fang, Li-Fu Ding, Shao-Yi Chen, Liang-Gee Chen, “Depth Map Generation for 2D-to-3D Conversion by Short-Term Motion Assisted Color Segmentation”, Multimedia and Expo, 2007 IEEE International Conference on, 2007 , Page(s): 1958 – 1961.

[3]      Yu-Lin Chang and Wei-Yin Chen and Jing-Ying Chang and Yi-Min Tsai and Chia-Lin Lee and Liang-Gee Chen, "Priority depth fusion for the 2D to 3D conversion system", in Three-Dimensional Image Capture and Applications, 2008, San Jose, CA, USA, published by SPIE, vol. 6805, pp. 680513-8.

[4]      Chao-Chung Cheng and Chung-Te Li and Po-Sen Huang and Tsung-Kai Lin and Yi-Min Tsai and Liang-Gee Chen, "A Block-based 2D-to-3D Conversion System with Bilateral Filter", in International Conference on Consumer Electronics (ICCE) , 2009, Las Vegas, NV, USA, published by IEEE.

[5]      Chao-Chung Cheng and Chung-Te Li and Liang-Gee Chen, "Video 2-D to 3-D conversion based on hybrid depth cueing", in Journal of the Society for Information Display , 2010.

[6]      Chao-Chung Cheng and Chung-Te Li and Liang-Gee Chen, " An Ultra-Low-Cost 2-D/3-D Video-Conversion System ", in SID 2010 International Symposium, Seminar, and Exhibition (SID), 2010, Seattle, WA, USA.

[7]      Chao-Chung Cheng and Chung-Te Li and Liang-Gee Chen, " A Novel 2D-to-3D Conversion System Using Edge Information ", in IEEE Transactions on Consumer Electronics, vol.56, no.3, pp.1739-1745, Aug. 2010.

[8]      Chao-Chung Cheng， Chung-Te Li， Liang-Gee Chen，” A 2D-to-3D conversion system using edge information”, Consumer Electronics (ICCE), 2010 Digest of Technical Papers International Conference on, 2010 , Page(s): 377 – 378.

[9]      Sung-Fang Tsai, Chao-Chung Cheng, Chung-Te Li, and Liang-Gee Chen, "A Real- Time 1080p 2D-to-3D Video Conversion System", in IEEE Transactions on Consumer Electronics, vol.57, no.2, pp.915-922, May 2011.

[10]  Sung-Fang Tsai, Chao-Chung Cheng, Chung-Te Li and Liang-Gee Chen, "A Real-Time 1080p 2D-to-3D Video Conversion System", in IEEE International Conference on Consumer Electronics (ICCE), Las Vegas, U.S.A, Jan. 2011.

[11]  Chien Wu, Chung-Te Li, Yen-Jei Lai, Chao-Chung Cheng and Liang-Gee Chen, "Disparity Remapping by Nonlinear Perceptual Discrimination", in International Conference on 3D systems and Applications (3DSA 2011),Seoul, Korea, June 2011.

[12]  Chung-Te Li, Yen-Jei Lai, Chien Wu, Chao-Chung Cheng and and Liang-Gee Chen, "A Quality Measurement Based on Object Formation for 3D Contents", in SID 2011 International Symposium, Seminar, and Exhibition (SID), Los Angeles, CA, USA, 2011.

[13]  Chung-Te Li, Yen-Chieh Lai, Chien Wu, “Perceptual multi-cues 2D-to-3D conversion system”, Visual Communications and Image Processing (VCIP), 2011 IEEE, 2011.

[14]  Chung-Te Li, Yen-Chieh Lai, Chien Wu, Sung-Fang Tsai and Liang-Gee Chen, "3D Image Correction by Hilbert Huang Decomposition", in IEEE International Conference on Consumer Electronics (ICCE), Las Vegas, U.S.A, Jan. 2012.

4.         浙江大学，CAD&CG实验室

主要是实验室由鲍虎军带领的computer vision group，group的主页是http://www.zjucvg.net/。这个group主要是由章国锋来负责的，他的主页是http://www.cad.zju.edu.cn/home/gfzhang/，他09年在浙大获得博士学位后就留校了，做了2年的博士后，现在是副教授。他主要是研究从单个视频恢复深度，采用的方法是SFM的方法，数学工具是bundle optimization，他做的结果在sfm邻域内还是不错的。主要的参考文章是：

[1]      Guofeng Zhang, Jiaya Jia, Tien-Tsin Wong and Hujun Bao, “Recovering Consistent Video Depth Maps via Bundle Optimization”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2008.

[2]      Guofeng Zhang, Jiaya Jia, Tien-Tsin Wong and Hujun Bao, “Consistent Depth Maps Recovery from a Video Sequence”, IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 31(6):974-988, 2009.

5.         飞利浦公司

主要是C. Varekamp负责算法。参考的文章有：

[1]      A. Redert,  R.-P. Berretty, C. Varekamp, O., Willemsen, J. Swillens, H. Driessen, “Philips 3D Solutions: From Content Creation to Visualization”, 3D Data Processing, Visualization, and Transmission, Third International Symposium on. pp. 429-431. 2006

[2]      C. Varekamp, B. Barenbrug, “Improved depth propagation for 2D to 3D video conversion using key-frames”, IETCVMP. 4th European Conference on Visual Media Production. pp. 1-7, Nov. 2007