PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space（2017）

1 Introduction

点云的特点：

① collections of points in a Euclidean space

② invariant to permutations of its members

③ the distance metric defifines local neighborhoods that may exhibit different properties（距离度量定义不同属性邻域）

the density and other attributes of points may not be uniform across different locations — in 3D scanning the density variability can come from perspective effects, radial density variations, motion, etc.（不同位置的点的密度和其他属性可能不一致——在3D扫描中，密度的可变性可能来自透视效果、径向密度变化、运动等）

 

The basic idea of PointNet is to learn a spatial encoding of each point and then aggregate all individual point features to a global point cloud signature. By its design, PointNet does not capture local structure induced by the metric.

 

PointNet++（general idea）：

① We first partition the set of points into overlapping local regions by the distance metric of the underlying space.

② Similar to CNNs, we extract local features capturing fine geometric structures from small neighborhoods;

③ such local features are further grouped into larger units and processed to produce higher level features.

④ This process is repeated until we obtain the features of the whole point set.

 

two issues that need to be addressed： 

①  how to generate the partitioning of the point set 

② how to abstract sets of points or local features through a local feature learner  （PointNet）

 

解决第二个问题：

Choose local feature learner to be PointNet，As a basic building block, PointNet abstracts sets of local points or features into higher level representations. In this view, PointNet++ applies PointNet recursively on a nested partitioning of the input set.

 

解决第一个问题：

Each partition is defifined as a neighborhood ball in the underlying Euclidean space, whose parameters include centroid location and scale.The centroids are se lected by a farthest point sampling (FPS) algorithm.

PointNet++ leverages neighborhoods at multiple scales.

（FPS算法找质心，多尺度利用邻域）

 

 

2 Problem Statement

：离散度量空间

：点集

d：距离度量

M的密度非均匀

输入： ，和每个点的其他属性

学习：集合函数 f

f 可以是对X打标签的分类函数，也可以是为M中每个点打标签的分割函数

 

 

3 Method

 

3.1 Review of PointNet

Given an unordered point set  ，

define a set function f：  

maps a set of points to a vector：

γ and h are usually multi-layer perceptron (MLP) networks

 

 

 

3.2 Hierarchical Point Set Feature Learning

 

 

① The Sampling layer selects a set of points from input points, which defifines the centroids of local regions.

② Grouping  layer then constructs local region sets by finding “neighboring” points around the centroids.

③ PointNet  layer uses a mini-PointNet to encode local region patterns into feature vectors.

 

N × ( d + C )

N points with d-dim coordinates and C -dim point feature

 subsampled points with d-dim coordinates and new -dim feature vectors summarizing local context.

 

Sampling layer

① Given input points 

② use iterative farthest point sampling (FPS) to choose a subset of points ，

such that x i j is the most distant point (in metric distance) from the set { x i 1 , x i 2 , ..., x i j 1 } with regard to the rest points.

 

Grouping layer

input： a point set of size N × ( d + C )  and the coordinates of a set of centroids of size

output： groups of point sets of size   ，each group corresponds to a local region and K is the number of points in the neighborhood of centroid points（不同组之间K可能是不同的，但是接下来的PointNet层可以变为固定长度的区域特征向量）

 

和KNN确定邻域相比，球查询的邻域保证了固定的区域尺度

 

PointNet layer

input：

output：

 

首先进行坐标转换：，

is the coordinate of the centroid

 

 

3.3 Robust Feature Learning under Non-Uniform Sampling Density

 

hierarchical network + density adaptive PointNet layers =  PointNet++

 

 

Multi-scale grouping (MSG)

We train the network to learn an optimized strategy to combine the multi-scale features. This is done by randomly dropping out input points with a randomized probability for each instance, which we call random input dropout . Specififically, for each training point set, we choose a dropout ratio θ uniformly sampled from [0 , p ] where p ≤ 1 . For each point, we randomly drop a point with probability θ. In practice we set p = 0 . 95 to avoid generating empty point sets. In doing so we present the network with training sets of various sparsity (induced by θ) and varying uniformity (induced by randomness in dropout). During test, we keep all available points.(组合策略是要训练的)

 

Multi-resolution grouping (MRG)

features of a region at some level L i is a concatenation of two vectors. One vector (left in figure) is obtained by summarizing the features at each subregion from the lower level L i-1 using the set abstraction level. The other vector (right) is the feature that is obtained by directly processing all raw points in the local region using a single PointNet.

 

MRG计算量更高效

 

3.4 Point Feature Propagation for Set Segmentation

 

（default: p=2,k=3）

 

 

4 Experiments

4.3 Point Set Classifification in Non-Euclidean Metric Space

 

4.4 Feature Visualization

 

 

---

farthest point sampling (FPS) algorithm  选择质心

FPS的好处：Compared with random sampling, it has better coverage of the entire point set given the same number of centroids.

 

L. Luciano and A. B. Hamza. Deep learning with geodesic moments for 3d shape classifification. Pattern  Recognition Letters , 2017.

L. Yi, H. Su, X. Guo, and L. Guibas. Syncspeccnn: Synchronized spectral cnn for 3d shape segmentation. arXiv preprint arXiv:1612.00606 , 2016.

 

geodesic neighborhood

Euclidean neighborhood

 

扰动对象位置和点样本位置