Convolutions and Fast Fourier Transform

The Problem

Given two vectors $a=(a_1,a_2,...,a_{n-1})$ and $b=(b_1,b_2,...,b_{n-1})$. The convolution of two vectors of length $n$ is a vector of length $2n-1$. 定义 convolution: $a\ast b$, where coordinate $k$ is equal to
$$\sum _{(i,j):i+j=k\text{ and }i,j<n}{a}_i{b}_i$$ In other words,
$$a\ast b=(a_0{b}_0,a_0{b}_1+a_1{b}_0,a_0{b}_2+a_1{b}_1+a_2{b}_0,...,a_{n-1}{b}_{n-1}+a_{n-1}{b}_{n-2},a_{n-1}{b}_{n-1})$$ 如何理解这个看似复杂的表达式？

不同于点乘和向量的和，要求两个向量的基底数量相同，可以对任意基底数的两个向量进行卷积。设若 $a=(a_0,a_1,...,a_{m-1}),b=(b_0,b_1,...,b_{n-1})$。那么 $a\ast b$ 是一个基底数量为 $m+n-1$ 的向量，其中 coordinate $k$ 等于
$$\sum _{(i,j):i+j=k,i<m,j<n}{a}_i{b}_j$$ We can picture this using the table of products $a_i{b}_j$ as before; the table is now rectangular, but we still compute coordinates by summing along the diagonals.

Example of Convolution

应用一：
Given two polynomials $A(x)=a_0+a_{1}x+a_2{x}^2+...a_{m-1}{x}^{m-1}$ and $B(x)=b_0+b_{1}x+b_2{x}^2+...+b_{n-1}{x}^{n-1}$, consider the polynomial $C(x)=A(x)B(x)$. In this polynomial $C(x)$, the coefficient on the $x^k$ term is equal to $$c_k=\sum _{(i,j):i+j=k}{a}_i{b}_j$$ In other words, the coefficient vector $c$ of $C(x)$ is the convolution of the coefficient vectors of $A(x)$ and $B(x)$.

应用二：signal processing - 卷积和傅立叶分析

应用三：Combining Histogram
Suppose we have the following two histograms:

• the annual income of all the men in the population
• the annual income of all the women.

We’d now like to produce a new histogram, showing for each $k$ the number of pairs $(M,W)$ for which man $M$ and woman $W$ have a combined income of $k$.

• $a=(a_0,a_1,...,a_{m-1})$ indicate that there are $a_i$ men with annual income equal to $i$.
• $b=(b_0,b_1,...,b_{m-1})$ indicate that there are $b_j$ men with annual income equal to $j$.

Computing the Convolution

Now we suppose $m=n$, for each $k$, we calculate the sum $$\sum _{(i,j):i+j=k}{a}_i{b}_j$$ This is $O(n^2)$ arithmetic operations.

Could one design an algorithm that bypasses the quadratic-size definition of convolution and computes it in some smarter way?

• Fast Fourier Transform (FFT) - computes the convolution of two vectors using $O(n\log n)$

Designing and Analyzing the Algorithm

Suppose we are given $a=(a_0,a_1,...,a_{n-1})$ and $b=(b_0,b_1,...,b_{n-1})$

• $A(x)=a_0+a_{1}x+a_2{x}^2+...+a_{n-1}{x}^{n-1}$
• $B(x)=a_0+b_{1}x+b_2{x}^2+...+b_{n-1}{x}^{n-1}$
• $C(x)=A(x)B(x)$
• $c=(c_0,c_1,...,c_{2n-2})$

Rather than multiplying $A$ and $B$ symbolically, we can treat them as functions of the variable $x$ and multiply them as follows:

1. Choose $2n$ values $x_1,x_2,...,x_{2n}$ and evaluate $A(x_j)$ and $B(x_j)$ for each of $j=1,2,...,2n$.
2. Compute $C(x_j)$ for each $j$ very easily: $C(x_j)=A(x_j)\cdot B(x_j)={\alpha }_j{\beta }_j$
3. Recover $C$ from its values on $x_1,x_2,...,x_{2n}$. Here we take advantage of a fundamental fact about polynomials: any polynomial of degree $d$ can be reconstructed from its values on any set of $d+1$ or more points. This is known as polynomial interpolation. Since $A$ and $B$ each have degree at most $n-1$, their product $C$ has degree at most $2n-2$, and so it can be reconstructed from the values $C(x_1)$, $C(x_2)$, . . . , $C(x_{2n})$ that we computed in step $(2)$.

第2步显然是 $O(n)$，但是第1、3步看起来不像是 $O(n)$，其中第1步明显是 $O(n^2)$。

Key Idea: find a set of $2n$ values $x_1$, $x_2$, . . . , $x_{2n}$ that are intimately related in some way, such that the work in evaluating $A$ and $B$ on all of them can be shared across different evaluations. - 满足这个的集合：complex roots of unity.

The Complex Roots of Unity

The $k^{th}$ roots of unity 定义：
对于一个正整数 $k$，$x^k=1$ 共有 $k$ 个不同的复数根, 即
$${\omega }_{j,k}=e^{2\pi ji/k}$$ for $j=0,1,2,...,k-1$

$k=8$ 时复数根在复平面上的分布是：

对于 $x_1,x_2,...,x_{2n}$ 选择为 the $(2n{)}^{th}$ roots of unity

The representation of a degree-$d$ polynomial $P$ by its values on the $(d+1{)}^{st}$ roots of unity is sometimes referred to as the discrete Fourier Transform of $P$.

A Recursive Procedure for Polynomial Evaluation

${\alpha }_j=A(x_j)=A({\omega }_{j,2n})$
${\beta }_j=B(x_j)=B({\omega }_{j,2n})$

How to break the evaluation of a polynomial into two equal-sized subproblems? - 分为偶数项和奇数项 (假设 $n$ 是2的倍数)
$$A_{even}(x)=a_0+a_{2}x+a_4{x}^2+...+a_{n-2}{x}^{(n-2)/2}$$ $$A_{odd}(x)=a_1+a_{3}x+a_5{x}^2+...+a_{n-1}{x}^{(n-2)/2}$$ 那么，
$$A(x)=A_{even}(x^2)+x{A}_{odd}(x^2)$$ Note:

1. $A_{even}$ and $A_{odd}$ have $(n-2)/2$ degree, which is half of the degree of $A$.
2. 若已知 $A_{even}(x^2)$ 和 $A_{odd}(x^2)$，$A(x)$ 的计算需要 constant time。

Thus, we can perform these evaluations in time $T(n/2)$ for each $A_{even}$ and $A_{odd}$, for a total time $2T(n/2)$