Active Shape Model

ALL INFORMATION FOLLOWING IS SUMMARIZED FROM THE ACADEMICA PAPER FROM https://link.springer.com/chapter/10.1007/978-3-642-54851-2_1

---

1. labeling trainning set

Supposing we have labeled each of the $N$ images in the training set with $n$ landmarks. Now we get a landmark set to represent the shape. For the $i^{th}$ image, we denote the $j^{th}$ landmark coordinate point by $(x^{ij},y^{ij})$. And the $2n$ element vector describing the $n$ point of the $i^{th}$ image can be written as
$$X_i={\left[\begin{array}{c}{x}_{i0},y_{i0},x_{i1},y_{i1},\dots ,x_{in-1},y_{in-1}\end{array}\right]}^T$$
where $1\le i\le N$ . As a result, we generate $N$ such representative vectors from $N$ training images. Before carrying out statistical analysis on those vectors, we should ensure that the shapes represented are in the same coordinate frame.

2. aligning tranning set

As mentioned above, in order to study the statistical characteristics of the coordinates of the landmark points, it is important that the shapes represented should be in a common coordinate frame. To achieve this, all the shapes must be aligned to each other to remove variation that could affect the statistical analysis result. The aligning can be done by scaling, rotating and translating the shapes of the training set to meet the requirement. The aim of alignment is to minimize a weighted sum of squares of distances between equivalent landmarks on different images.

Procrustes analysis is the most popular method to align shapes into the same coordinate.

A weighted diagonal matrix W is used to give points that are more stable over the set more significance. Stable point is defined as having less movements respect to other points in a shape.

The following algorithm is an easy handling iterative approach to align the set of $N$ shapes to each other.

a. Align each shape to one of the shapes, for instance, the first one or the mean shape.

b. Calculate the mean shape from the aligned shape.

c. Normalize the pose of the current mean shape.

d. Realign the each shape with the normalized mean.

f. If converged, stop the process, or else go to step 2 and repeat.

The meaning of the normalization is

a. Scale the shape so that the distance between two points becomes a certain constant.

b. Rotate the shape so that the line joining two pre-specified landmarks is directed in a certain direction.

c. Translate the shape so that it becomes centered at a certain coordinate.

3. Statistical Model of Variation — PCA

After alignment, $N$ sets of vector $X_i$ representing the images of training set now contain the new coordinate. These vectors form a distribution from which we can generate new examples similar to the original ones. Also, we can determine clearly whether a new shape produced by the model during search procedure is allowable or acceptable.

$$b=P^{-1}(X-\bar{X})$$
where $b$ is derived from the eigenvalues. By changing the elements of $b$ , we can vary the shape.

But arbitrary $b$ could result in an unallowable shape. To avoid the problem, we can impose constrains on $b$ . By giving limits, we can control the changing of shape in order to generate plausible shapes. Since the variance of $b$ of the training set has a relationship with the eigenvalue $\lambda$ . The typical limits are
$$-3\sqrt{{\lambda }_i}\le b_i\le 3\sqrt{{\lambda }_i}$$
where ${\lambda }_i$ is the eigenvalue corresponding to the $i^{th}$ eigenvector.

4. Modeling Local Structure – Template Matching

It is not enough to obtain only the statistical characteristics of the position of landmarks. In order to find desired movement and make a good estimation of model position during the image search and classification procedure, besides shape information, a model containing gray-level information of the images in the training set should also be established. The core idea of the gray-level information modeling method is to collect pixels around each landmark and try to put the pixels’ gray information in a compact form so we can use it for image search.

For each landmark, we can sample $k$ pixels on either side of the landmark along a profile (the line passing through the landmark and perpendicular to the line connecting the landmark and its neighbors). Then we obtain a gray-level profile of $2k+1$ (include the landmark itself) length. We describe it by a vector $g$. To reduce the effect of global intensity changes, we do not use the actual vector $g$ but use the normalized derivative instead. It reflects the change of gray-level along the profile.

The gray-level profile of the $j^{th}$ landmark in the $i^{th}$ image is a vector of $2k+1$ element
$$g_{ij}={\left[\begin{array}{c}{g}_{ij0},g_{ij1},\dots ,g_{ij(2k)},g_{ij(2k+1)}\end{array}\right]}^T$$
Then, building statistical models (PCA) of the gray-level profile for all the landmarks in the training images.

5. Multi-resolution Active Shape Model

Based on the analysis above, the algorithm is implemented in a multi-resolution approach, which is involved in searching firstly in a coarse image for remote points with large jumps and refining the location in a series of finer resolution images by limiting the jump to only close points.

Each training image has its own pyramid.

During the training, we build statistical models of gray-level along profile through each landmark, at each level of the image pyramid.

During the search, we start our searching from the top level of the pyramid. In each image, the initial search position of the model is the search output of its upper level, until the lowest level is reached.

6. Image Search Using Active Shape Model

Now, based on the model, we will locate a new example of the object in an image. The idea is: first giving the model an initial position through some prior knowledge; second, we examine landmarks and their neighbors along gray-level profile to find better location of landmarks; third, we update the pose and shape parameters with suitable constraints and move the model to the new location and produce a plausible shape. The fact that the shapes are modeled so that they can only vary in a controllable way by constraining the weights of the principal components explains why such model is named Active Shape model or ASM.

We assume that an instance of an object is described as the mean shape obtained from the training set plus a weighted sum of the t principal components, with the possibility of this sum being scaled, rotated and translated.

6.1 Initial Estimate

We can express the estimate $X_i$ of the shape as a scaled, rotated and translated version of a shape $X_l$ .
$$X_i=M(s_i,{\theta }_i)[X_l]+t_i$$
Where $s^i$ , ${\theta }_i$ and $t_i$ is respectively scale, rotation and translation parameters. $t=[t_{xi},t_{yi},t_{xi},t_{yi},\dots ,t_{xi},t_{yi}{]}^T$ with a length of $2n$. $X_l$ can also be expressed as $X_l=\bar{X}+d{X}_l$, with $d{X}_l=P{b}_l$. $X_l$ is mean shape of the model.
$$X_i=M(s_i,{\theta }_i)[\bar{X}+P{b}_l]+t_i$$

6.2 Compute the Movements of Landmarks

Using a algorithm, the best fit position for the landmark $j$ can be found. For the shape $X_i$ , this process should be repeated to find a suggested new position for each landmark, so we get a position offset $d{X}_i$. Thus, we finally obtain an “ideal” shape $X_i^{\prime }=X_i+d{X}_i$ .

Generally, $X_i^{\prime }$ obtained through gray-level profile search is closer to the shape of the target object. Then we can adjust the pose parameters, namely scale, rotation and translation, to move the initial estimate as close as possible to the target object. But usually we do not update the $X_i$ to $X_i^{\prime }$ directly, because $X_i^{\prime }$ maybe doesn’t satisfy the shape constraints and generate an unallowable shape.

With this problem into consideration, the shape parameter $b$ should also be updated to make $X_i$ as close as possible to $X_i^{\prime }$ . Meanwhile, constraints should be imposed on $b$ to ensure that $X_i^{\prime }$ is a plausible shape.

$$X_i=M(s_i,{\theta }_i)[\bar{X}+P{b}_l]+t_i\stackrel{(1+ds),d\theta ,dt}{}{X}_i^{\prime }=X_i+d{X}_i$$

Compute $d{b}_l$ by solving the following equation

$$M(s_i(1+ds),{\theta }_i+d\theta )[\bar{X}+P(b_l+d{b}_l)]+t_i+dt=X_i+d{X}_i$$

Let $X_i=M(s_i,{\theta }_i)[\bar{X}+P{b}_l]+t_i$, we get

$$\begin{array}{r}M(s_i(1+ds),{\theta }_i+d\theta )[\bar{X}+P(b_l+d{b}_l)]+t_i+dt=M(s_i,{\theta }_i)[\bar{X}+P{b}_l]+t_i+d{X}_i\\ M(s_i(1+ds),{\theta }_i+d\theta )[\bar{X}+P(b_l+d{b}_l)]=M(s_i,{\theta }_i)[\bar{X}+P{b}_l]+d{X}_i-dt\end{array}$$

and since

$$M^{-1}(s,\theta )[\dots ]=M(s^{-1},-\theta )[\dots ]$$

we obtain

$$d{b}_l=P^{-1}(M((s_i(1+ds){)}^{-1},-(\theta +d\theta ))[M(s_i,{\theta }_i)[\bar{X}+P{b}_l]+d{X}_i-dt]-\bar{X})-b_l$$

Now, we have enough information to form a new shape estimate using the parameters above.

$$X_i^1=M(s_i(1+ds),{\theta }_i+d\theta )[X_l+Pd{b}_l]+t_i+dt$$

Then we start this procedure from $X_i^1$ to produce $X_i^2$, until there is no significant change of the shape.

The procedure of searching for a target object in a new image can be summarized as follows

1. Let $X_i=M(s_i,{\theta }_i)[X_l]+t_i$ be the initial estimate of the model in the new image, where $X_l=\bar{X}+P{b}_l$ , and $P$ contains the first $t$ principal component.

2. Use the gray-level profile of landmarks to search for the suggested movements of landmarks. By doing this, we can get a position offset $d{X}_i$ for each landmarks of the model. Then move the initial shape to a new plausible position $X_i+d{X}_i$.

3. Calculate the additional pose parameters $s,\theta ,t$.

4. Calculate the additional shape parameter $d{b}_l$ and notice that a suitable constraint (eg. $-3\sqrt{{\lambda }_i}<d{b}_l<3\sqrt{{\lambda }_i}$) should imposed on $b$ to avoid unallowable shapes appear.

5. Update the pose and shape parameters on $X_i$ to obtain a new shape $X_i^1$ close to the target. Then use $X_i^1$ as the estimate, repeat from step 2.

6. Stop the iteration until no significant change is found.