Non-linear Optimization

Least square method

1. Generalized pseudo-inverse

Simply Define :
$$Ax=b,A\in {\mathbb{R}}^{m\times n}$$
The solution can be generally defined as :
$$x=A^{+}b$$
Assume $rank(A)=r$, then deploy BC decomposition method:
$$A_{m\times n}=B_{m\times r}{C}_{r\times n}$$
The generalized pseudo-inverse of $A$ is:
$$A^{+}=C^T(C{C}^T{)}^{-1}(B^{T}B{)}^{-1}{B}^T$$

If $r=m$（row full rank)
$$A=EA$$

If $r=n$（column full rank)

$$A=AE$$
Then, we can deduce that if matrix $A$ is row full rank, the generalized pseudo-inverse of $A$ is：
$$A^{+}=A^T(A{A}^T{)}^{-1}$$
If $A$ is column full rank, $A^{+}$ is:
$$A^{+}=(A^{T}A{)}^{-1}{A}^T$$

2. Singular value decomposition (SVD)

Define:
$$A=U\Sigma V^T,A\in {\mathbb{R}}^{m\times n}$$

The least square problem is:
$$\min \Vert Ax-b{\Vert }_2^2$$
When $m>n=rank(A)$, it is overdetermined equations:
$$A=(U_1,U_2)(\begin{array}{c}{\Sigma }_1\\ 0\end{array})V^T$$
$U_1\in {\mathbb{R}}^{m\times r}$ ,$x$ can be described as:

$$x=V{\Sigma }_1^{-1}{U}_1^T\mathbf{b}$$

When $n>m=rank(A)$, then:
$$A=U^T(\begin{array}{cc}{\Sigma }_1& 0\end{array})(\begin{array}{c}{V}_1^T\\ V_2^T\end{array})$$
$V_1^T\in {\mathbb{R}}^{r\times n}$, thus:
$$x=V_1{\Sigma }_1^{-1}{U}^T\mathbf{b}$$

Reference:

1. 矩阵SVD分解