2D-3D Fusion for Layer Dempositon of Urban Facades

    This is the first paper I have read to  fuse 2D-3D data for surface construction. The paper is accepted bt SIGGRAPH. The author is from Shenzhen institute, who is guide by Quanbao Chen ,one of the most influenceful researchers in Computer Graphics in China.

    In this paer,they use two kind of data sources: 2D photography and 3D LiDar scans.The two mode contains complementary information: the photograpy is of high resolusion and full of testure information but is view-dependent and lack of depth information.3D,on the other hand ,can provide depth information but always sparse,noisy and incomplete .

    So making full use of this two kinds of information is a good idea. First,They segmentate the image into polygonal regions with consistent depth,denoted as depth-layers. Afterwards,they  detect repetitive patterns on each layers. These patterns can be used for  removing outliers and recovering missing parts in  3D.

   The detial steps are belows:

         1.Depth-layer Decomposition

            (1) 2D-3D registration  To find the projection between the photograph and 3D. Fisrt rectify the iamge ,then mark two pairs of rectangles  on 2D and 3D  respctively, then  the roungh correspondence can be found, by which,we can project3D to 2D. This step is finished manually.

            (2)2D-3D segmentation use the 2D-3D fusion date to segmentate the image into segments. On 3D,use RANSANC to exact planar regions, use Manhattan prior to prune. detect 3D edges(insections of planes or in planes). Then Iverlay the  3D edges onto image.

             (3)Multi-label depth-layer generation A multi-label assign problem.

          2. 3D Geometry and Texture Fusion  This step is to enrich the 3D. 

               Detected repetitive paterns on each layer.Then for each set of the repetitives compress the coressponding floor into a slab.  And orthogonally project it onto a single frontal plane. Detec edges based on discontinuity density image. transfer the edges to floors and extend the edges to create local rectanglar fragments.Finally ue the depth infaomation to create an extrusion surface.( this sep is conatians a  push and pull peocess,fisrt  push the 3D into a plane the pull the 3D out)

 

    Advantages:    use combine 2D and  3D together,so can obtain more information for reconstruction.

   Disadvantages:  only work well on the facade with dominat plane or multi planes seprated with diffrent detph.