Fast Randomized Range Finder -- By structured random matrix

(Note: The following content is mainly excerpted from the reference [1] , [2] and [3] listed at the bottom.)

---

1 (Dense) Random projection method

To construct a subspace that captures the action of a matrix, that is, developing an algorithm for constructing low-rank approximation to the original matrix. The most common method is using the randomized projection method, as the following algorithm, named Randomized range finder [1].

---

Algorithm1: Randomized range finder
Input: $A\in R^{m\times n}$ and an integer $l$;
Output: An orthonormal matrix $Q\in R^{m\times l}$ whose range approximates the range of $A$
Steps:

1. Draw an $n\times l$ Gaussian random matrix $\Omega$;
2. Form the $m\times l$ matrix $Y=A\Omega$;
3. Construct an $m\times l$ matrix $Q$ whose columns form an orthonormal basis for the range of $Y$, e.g., using the QR factorization $Y=QR$.

---

Here, the entries of the random matrix $\Omega$ obey the Gaussian distribution, thus $\Omega$ is a dense matrix.

The bottleneck in the above algorithm is the computation of the matrix product $A\Omega$. When $\Omega$ is standard Gaussian, the cost of this multiplication is $O(mnl)$, the same as a rank-revealing QR algorithm.

The Key idea is to use a structured random matrix that allow us to compute the product in $O(mn\log (l))$ flops.

---

2 (Sparse) Random projection method

2.1 SRFT

[4] proposed a subsampled random Fourier transform (SRFT), which is perhaps the simplest example of a structured random matrix that meets the above goals.

Specifically, an SRFT is an $n\times l$ matrix of the following form
$$\Omega =\sqrt{\frac{n}{l}}DFR,\text{\hspace{1em}(1)}$$
where

• $D$ is an $n\times n$ diagonal matrix whose entries are independent random variables uniformly distributed on the complex unit circle;
• $F$ is the $n\times n$ unitary discrete Fourier transform (DFT), whose entries take the values
  $$f_{pq}=\frac{1}{\sqrt{n}}{e}^{\frac{-2\pi i(p-1)(q-1)}{n}}$$
  for $p,q=1,2,\cdots ,n$;
• $R$ is an $n\times l$ matrix that samples $l$ coordinates from $n$ uniformly at random; i.e., its $l$ columns are drawn randomly without replacement from the columns of the $n\times n$ identity matrix.

Then we can compute the sample matrix $Y=A\Omega$ using $O(mn\log (l))$ flops via a subsampled FFT [4]. Then form the basis $Q$ by orthonormalizing the columns of $Y$. See the following algorithm for details.

---

Algorithm2: Fast Randomized range finder
Input: $A\in R^{m\times n}$ and an integer $l$;
Output: An orthonormal matrix $Q\in R^{m\times l}$ whose range approximates the range of $A$
Steps:

1. Draw an $n\times l$ SRFT test matrix $\Omega$, as defined in (1);
2. Form the $m\times l$ matrix $Y=A\Omega$ using a (subsampled ) FFT;
3. Construct an $m\times l$ matrix $Q$ whose columns form an orthonormal basis for the range of $Y$, e.g., using the QR factorization $Y=QR$.

---

2.2 SEM

[2] proposed another sparse randomzied maritx named sparse embedding matrix (SEM).

Consider the random linear map $S=\Phi D$, where $S\in R^{k\times n}$, such that for $h:\{1,\cdots ,n\}\to \{1,\cdots ,k\}$, a random map such that for each $i\in \{1,\cdots ,n\}$, $h(i)=t^{\prime }$ for $t^{\prime }\in \{1,\cdots ,k\}$, with probability $1/t$, for a parameter $t\in N$ we have

1. $\Phi \in \{0,1{\}}^{k\times n}(k\le n)$ binary matrix with nonzero entries ${\Phi }_{h(i),i}=1$ and all the remaining entries equal to $0$. In other words, $\Phi$ is a matrix with a single $1$ in each column.
2. $D$ is an $n\times n$ random diagonal matrix where each diagonal entry is independently chosen to be either $+1$ or $-1$ with equal probabiltity.

A matrix $S$ that satisfies $1\&2$ is referred to as a sparse embedding matrix (SEM).

(Refer to Reference [2] for details.)

---

References:
[1] N.Halko, P.G.Martinsson, J.A.Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM Review, 2(53):217-288, 2011.
[2] Yariv Aizenbuda, Gil Shabatb, Amir Averbuchb. Randomized LU decomposition using sparse projections. Computers & Mathematics with Applications, 9(72): 2525-2534, 2016.
[3] 殷术亨. 矩阵 LU 分解及 Cholesky 分解的随机算法研究[D], 重庆大学， 2020.
[4] F. Woolfe, E. Liberty, V. Rokhlin, and M. Tygert. A fast randomized algorithm for the approximation of matrices, Appl. Comput. Harmon, Anal., 25: 335-366, 2008.