交叉相关与卷积 Cross-correlation v.s. convolution
卷积是一种广泛应用于信号处理、图像处理和其他工程/科学领域的技术。在深度学习中，一种模型结构卷积神经网络（CNN）就是以这种技术命名的。然而，深度学习中的卷积（Convolution）本质上是信号/图像处理中的互相关(Cross-correlation)。这两种操作之间有细微的区别，但不必深入细节。在信号/图像处理中，卷积被定义为两个函数的乘积在一个函数被反转和移位后的积分。另一方面，互相关被称为两个函数的滑动点积或滑动内积。互相关中的滤波器不反转。它直接通过函数f滑动。f和g之间的相交区域是互相关。
https://en.wikipedia.org/wiki/Convolution#/media/File:Comparison_convolution_correlation.svg 展示了相关性和互相关之间的区别。
在深度学习中，卷积中的滤波器是不可逆的。严格地说，这是相互关联的。我们基本上执行元素级的乘法和加法。但在深度学习中称之为卷积是一种惯例。这很好，因为过滤器的权重是在训练期间学习的。如果上述示例中的反向函数g是正确的函数，则经过训练后，学习的滤波器将看起来像反向函数g。因此，在训练前不需要像真正的卷积那样首先反转滤波器。
综上，从数学运算的角度上来看，互相关与卷积的区别为是否逆时针翻转核（Kernel or Filter）。互相关不翻转，卷积则将核逆时针翻转180°。在深度学习中，使用的卷积操作实际上就是互相关，这是一个习惯问题，不必深究。
其他相关的问题推导见此网站
tutorialspoint
Convolution
Convolution is a mathematical operation used to express the relation between input and output of an LTI system. It relates input, output and impulse response of an LTI system as
y ( t ) = x ( t ) ∗ h ( t ) y (t) = x(t) * h(t) y(t)=x(t)∗h(t)
Where y (t) = output of LTI
x (t) = input of LTI
h (t) = impulse response of LTI
There are two types of convolutions:
Continuous convolution
Discrete convolution
Continuous Convolution
continuous convolution
$ y(t) ,,= x (t) * h (t) $
= ∫ − ∞ ∞ x ( τ ) h ( t − τ ) d τ = \int_{-\infty}^{\infty} x(\tau) h (t-\tau)d\tau =∫−∞∞x(τ)h(t−τ)dτ
(or)
$= \int_{-\infty}^{\infty} x(t - \tau) h (\tau)d\tau $
Discrete Convolution
Discrete convolution
y ( n ) = x ( n ) ∗ h ( n ) y(n)\,\,= x (n) * h (n) y(n)=x(n)∗h(n)
$= \Sigma_{k = - \infty}^{\infty} x(k) h (n-k) $
(or)
$= \Sigma_{k = - \infty}^{\infty} x(n-k) h (k) $
By using convolution we can find zero state response of the system.
Deconvolution
Deconvolution is reverse process to convolution widely used in signal and image processing.
Properties of Convolution
Commutative Property
$ x_1 (t) * x_2 (t) = x_2 (t) * x_1 (t) $
Distributive Property
$ x_1 (t) * [x_2 (t) + x_3 (t) ] = [x_1 (t) * x_2 (t)] + [x_1 (t) * x_3 (t)]$
Associative Property
$x_1 (t) * [x_2 (t) * x_3 (t) ] = [x_1 (t) * x_2 (t)] * x_3 (t) $
Shifting Property
$ x_1 (t) * x_2 (t) = y (t) $
$ x_1 (t) * x_2 (t-t_0) = y (t-t_0) $
$ x_1 (t-t_0) * x_2 (t) = y (t-t_0) $
$ x_1 (t-t_0) * x_2 (t-t_1) = y (t-t_0-t_1) $
Convolution with Impulse
$ x_1 (t) * \delta (t) = x(t) $
$ x_1 (t) * \delta (t- t_0) = x(t-t_0) $
Convolution of Unit Steps
$ u (t) * u (t) = r(t) $
$ u (t-T_1) * u (t-T_2) = r(t-T_1-T_2) $
$ u (n) * u (n) = [n+1] u(n) $
Scaling Property
If $x (t) * h (t) = y (t) $
then x ( a t ) ∗ h ( a t ) = 1 ∣ a ∣ y ( a t ) x (a t) * h (a t) = {1 \over |a|} y (a t) x(at)∗h(at)=∣a∣1y(at)
Differentiation of Output
if y ( t ) = x ( t ) ∗ h ( t ) y (t) = x (t) * h (t) y(t)=x(t)∗h(t)
then $ {dy (t) \over dt} = {dx(t) \over dt} * h (t) $
or
$ {dy (t) \over dt} = x (t) * {dh(t) \over dt}$
Note:
Convolution of two causal sequences is causal.
Convolution of two anti causal sequences is anti causal.
Convolution of two unequal length rectangles results a trapezium.
Convolution of two equal length rectangles results a triangle.
A function convoluted itself is equal to integration of that function.
Example: You know that u ( t ) ∗ u ( t ) = r ( t ) u(t) * u(t) = r(t) u(t)∗u(t)=r(t)
According to above note, u ( t ) ∗ u ( t ) = ∫ u ( t ) d t = ∫ 1 d t = t = r ( t ) u(t) * u(t) = \int u(t)dt = \int 1dt = t = r(t) u(t)∗u(t)=∫u(t)dt=∫1dt=t=r(t)
Here, you get the result just by integrating u ( t ) u(t) u(t).
Limits of Convoluted Signal
If two signals are convoluted then the resulting convoluted signal has following range:
Sum of lower limits < t < sum of upper limits
Ex: find the range of convolution of signals given below
Limits of convoluted signal
Here, we have two rectangles of unequal length to convolute, which results a trapezium.
The range of convoluted signal is:
Sum of lower limits < t < sum of upper limits
$-1±2 < t < 2+2 $
$-3 < t < 4 $
Hence the result is trapezium with period 7.
Area of Convoluted Signal
The area under convoluted signal is given by A y = A x A h A_y = A_x A_h Ay=AxAh
Where Ax = area under input signal
Ah = area under impulse response
Ay = area under output signal
Proof: y ( t ) = ∫ − ∞ ∞ x ( τ ) h ( t − τ ) d τ y(t) = \int_{-\infty}^{\infty} x(\tau) h (t-\tau)d\tau y(t)=∫−∞∞x(τ)h(t−τ)dτ
Take integration on both sides
$ \int y(t)dt ,,, =\int \int_{-\infty}^{\infty}, x (\tau) h (t-\tau)d\tau dt $
$ =\int x (\tau) d\tau \int_{-\infty}^{\infty}, h (t-\tau) dt $
We know that area of any signal is the integration of that signal itself.
∴ A y = A x A h \therefore A_y = A_x\,A_h ∴Ay=AxAh
DC Component
DC component of any signal is given by
DC component = area of the signal period of the signal \text{DC component}={\text{area of the signal} \over \text{period of the signal}} DC component=period of the signalarea of the signal
Ex: what is the dc component of the resultant convoluted signal given below?
dc component of the resultant convoluted signal
Here area of x1(t) = length × breadth = 1 × 3 = 3
area of x2(t) = length × breadth = 1 × 4 = 4
area of convoluted signal = area of x1(t) × area of x2(t)
= 3 × 4 = 12
Duration of the convoluted signal = sum of lower limits < t < sum of upper limits
= -1 + -2 < t < 2+2
= -3 < t < 4
Period=7
∴ \therefore ∴ Dc component of the convoluted signal = area of the signal period of the signal \text{area of the signal} \over \text{period of the signal} period of the signalarea of the signal
Dc component = 12 7 {12 \over 7} 712
Discrete Convolution
Let us see how to calculate discrete convolution:
i. To calculate discrete linear convolution:
Convolute two sequences x[n] = {a,b,c} & h[n] = [e,f,g]
discrete linear convolution
Convoluted output = [ ea, eb+fa, ec+fb+ga, fc+gb, gc]
Note: if any two sequences have m, n number of samples respectively, then the resulting convoluted sequence will have [m+n-1] samples.
Example: Convolute two sequences x[n] = {1,2,3} & h[n] = {-1,2,2}
discrete linear convolution
Convoluted output y[n] = [ -1, -2+2, -3+4+2, 6+4, 6]
= [-1, 0, 3, 10, 6]
Here x[n] contains 3 samples and h[n] is also having 3 samples so the resulting sequence having 3+3-1 = 5 samples.
ii. To calculate periodic or circular convolution:
Periodic convolution is valid for discrete Fourier transform. To calculate periodic convolution all the samples must be real. Periodic or circular convolution is also called as fast convolution.
If two sequences of length m, n respectively are convoluted using circular convolution then resulting sequence having max [m,n] samples.
Ex: convolute two sequences x[n] = {1,2,3} & h[n] = {-1,2,2} using circular convolution
discrete linear convolution
Normal Convoluted output y[n] = [ -1, -2+2, -3+4+2, 6+4, 6].
= [-1, 0, 3, 10, 6]
Here x[n] contains 3 samples and h[n] also has 3 samples. Hence the resulting sequence obtained by circular convolution must have max[3,3]= 3 samples.
Now to get periodic convolution result, 1st 3 samples [as the period is 3] of normal convolution is same next two samples are added to 1st samples as shown below:
circular convolution result
∴ \therefore ∴ Circular convolution result y [ n ] = [ 9 6 3 ] y[n] = [9\quad 6\quad 3 ] y[n]=[963]
Correlation
Correlation is a measure of similarity between two signals. The general formula for correlation is
∫ − ∞ ∞ x 1 ( t ) x 2 ( t − τ ) d t \int_{-\infty}^{\infty} x_1 (t)x_2 (t-\tau) dt ∫−∞∞x1(t)x2(t−τ)dt
There are two types of correlation:
Auto correlation
Cros correlation
Auto Correlation Function
It is defined as correlation of a signal with itself. Auto correlation function is a measure of similarity between a signal & its time delayed version. It is represented with R( τ \tau τ).
Consider a signals x(t). The auto correlation function of x(t) with its time delayed version is given by
R 11 ( τ ) = R ( τ ) = ∫ − ∞ ∞ x ( t ) x ( t − τ ) d t [+ve shift] R_{11} (\tau) = R (\tau) = \int_{-\infty}^{\infty} x(t)x(t-\tau) dt \quad \quad \text{[+ve shift]} R11(τ)=R(τ)=∫−∞∞x(t)x(t−τ)dt[+ve shift]
= ∫ − ∞ ∞ x ( t ) x ( t + τ ) d t [-ve shift] \quad \quad \quad \quad \quad = \int_{-\infty}^{\infty} x(t)x(t + \tau) dt \quad \quad \text{[-ve shift]} =∫−∞∞x(t)x(t+τ)dt[-ve shift]
Where τ \tau τ = searching or scanning or delay parameter.
If the signal is complex then auto correlation function is given by
R 11 ( τ ) = R ( τ ) = ∫ − ∞ ∞ x ( t ) x ∗ ( t − τ ) d t [+ve shift] R_{11} (\tau) = R (\tau) = \int_{-\infty}^{\infty} x(t)x * (t-\tau) dt \quad \quad \text{[+ve shift]} R11(τ)=R(τ)=∫−∞∞x(t)x∗(t−τ)dt[+ve shift]
= ∫ − ∞ ∞ x ( t + τ ) x ∗ ( t ) d t [-ve shift] \quad \quad \quad \quad \quad = \int_{-\infty}^{\infty} x(t + \tau)x * (t) dt \quad \quad \text{[-ve shift]} =∫−∞∞x(t+τ)x∗(t)dt[-ve shift]
Properties of Auto-correlation Function of Energy Signal
Auto correlation exhibits conjugate symmetry i.e. R ( τ \tau τ) = R*(- τ \tau τ)
Auto correlation function of energy signal at origin i.e. at τ \tau τ=0 is equal to total energy of that signal, which is given as:
R (0) = E = $ \int_{-\infty}{\infty},|,x(t),|2,dt $
Auto correlation function $\infty {1 \over \tau} $,
Auto correlation function is maximum at τ \tau τ=0 i.e |R ( τ \tau τ) | ≤ R (0) ∀ τ \tau τ
Auto correlation function and energy spectral densities are Fourier transform pairs. i.e.
F . T [ R ( τ ) ] = Ψ ( ω ) F.T\,[ R (\tau) ] = \Psi(\omega) F.T[R(τ)]=Ψ(ω)
Ψ ( ω ) = ∫ − ∞ ∞ R ( τ ) e − j ω τ d τ \Psi(\omega) = \int_{-\infty}^{\infty} R (\tau) e^{-j\omega \tau} d \tau Ψ(ω)=∫−∞∞R(τ)e−jωτdτ
$ R (\tau) = x (\tau)* x(-\tau) $
Auto Correlation Function of Power Signals
The auto correlation function of periodic power signal with period T is given by
R ( τ ) = lim T → ∞ 1 T ∫ − T 2 T 2 x ( t ) x ∗ ( t − τ ) d t R (\tau) = \lim_{T \to \infty} {1\over T} \int_{{-T \over 2}}^{{T \over 2}}\, x(t) x* (t-\tau) dt R(τ)=T→∞limT1∫2−T2Tx(t)x∗(t−τ)dt
Properties
Auto correlation of power signal exhibits conjugate symmetry i.e. $ R (\tau) = R*(-\tau)$
Auto correlation function of power signal at τ = 0 \tau = 0 τ=0 (at origin)is equal to total power of that signal. i.e.
$R (0)= \rho $
Auto correlation function of power signal ∞ 1 τ \infty {1 \over \tau} ∞τ1,
Auto correlation function of power signal is maximum at τ \tau τ = 0 i.e.,
$ | R (\tau) | \leq R (0), \forall ,\tau$
Auto correlation function and power spectral densities are Fourier transform pairs. i.e.,
F . T [ R ( τ ) ] = s ( ω ) F.T[ R (\tau) ] = s(\omega) F.T[R(τ)]=s(ω)
s ( ω ) = ∫ − ∞ ∞ R ( τ ) e − j ω τ d τ s(\omega) = \int_{-\infty}^{\infty} R (\tau) e^{-j\omega \tau} d\tau s(ω)=∫−∞∞R(τ)e−jωτdτ
$R (\tau) = x (\tau)* x(-\tau) $
Density Spectrum
Let us see density spectrums:
Energy Density Spectrum
Energy density spectrum can be calculated using the formula:
E = ∫ − ∞ ∞ ∣ x ( f ) ∣ 2 d f E = \int_{-\infty}^{\infty} |\,x(f)\,|^2 df E=∫−∞∞∣x(f)∣2df
Power Density Spectrum
Power density spectrum can be calculated by using the formula:
P = Σ n = − ∞ ∞ ∣ C n ∣ 2 P = \Sigma_{n = -\infty}^{\infty}\, |\,C_n |^2 P=Σn=−∞∞∣Cn∣2
Cross Correlation Function
Cross correlation is the measure of similarity between two different signals.
Consider two signals x1(t) and x2(t). The cross correlation of these two signals R 12 ( τ ) R_{12}(\tau) R12(τ) is given by
R 12 ( τ ) = ∫ − ∞ ∞ x 1 ( t ) x 2 ( t − τ ) d t [+ve shift] R_{12} (\tau) = \int_{-\infty}^{\infty} x_1 (t)x_2 (t-\tau)\, dt \quad \quad \text{[+ve shift]} R12(τ)=∫−∞∞x1(t)x2(t−τ)dt[+ve shift]
= ∫ − ∞ ∞ x 1 ( t + τ ) x 2 ( t ) d t [-ve shift] \quad \quad = \int_{-\infty}^{\infty} x_1 (t+\tau)x_2 (t)\, dt \quad \quad \text{[-ve shift]} =∫−∞∞x1(t+τ)x2(t)dt[-ve shift]
If signals are complex then
R 12 ( τ ) = ∫ − ∞ ∞ x 1 ( t ) x 2 ∗ ( t − τ ) d t [+ve shift] R_{12} (\tau) = \int_{-\infty}^{\infty} x_1 (t)x_2^{*}(t-\tau)\, dt \quad \quad \text{[+ve shift]} R12(τ)=∫−∞∞x1(t)x2∗(t−τ)dt[+ve shift]
= ∫ − ∞ ∞ x 1 ( t + τ ) x 2 ∗ ( t ) d t [-ve shift] \quad \quad = \int_{-\infty}^{\infty} x_1 (t+\tau)x_2^{*} (t)\, dt \quad \quad \text{[-ve shift]} =∫−∞∞x1(t+τ)x2∗(t)dt[-ve shift]
R 21 ( τ ) = ∫ − ∞ ∞ x 2 ( t ) x 1 ∗ ( t − τ ) d t [+ve shift] R_{21} (\tau) = \int_{-\infty}^{\infty} x_2 (t)x_1^{*}(t-\tau)\, dt \quad \quad \text{[+ve shift]} R21(τ)=∫−∞∞x2(t)x1∗(t−τ)dt[+ve shift]
= ∫ − ∞ ∞ x 2 ( t + τ ) x 1 ∗ ( t ) d t [-ve shift] \quad \quad = \int_{-\infty}^{\infty} x_2 (t+\tau)x_1^{*} (t)\, dt \quad \quad \text{[-ve shift]} =∫−∞∞x2(t+τ)x1∗(t)dt[-ve shift]
Properties of Cross Correlation Function of Energy and Power Signals
Auto correlation exhibits conjugate symmetry i.e. R 12 ( τ ) = R 21 ∗ ( − τ ) R_{12} (\tau) = R^*_{21} (-\tau) R12(τ)=R21∗(−τ).
Cross correlation is not commutative like convolution i.e.
R 12 ( τ ) ≠ R 21 ( − τ ) R_{12} (\tau) \neq R_{21} (-\tau) R12(τ)=R21(−τ)
If R12(0) = 0 means, if $ \int_{-\infty}^{\infty} x_1 (t) x_2^* (t) dt = 0$, then the two signals are said to be orthogonal.
For power signal if $ \lim_{T \to \infty} {1\over T} \int_{{-T \over 2}}^{{T \over 2}}, x(t) x^* (t),dt $ then two signals are said to be orthogonal.
Cross correlation function corresponds to the multiplication of spectrums of one signal to the complex conjugate of spectrum of another signal. i.e.
R 12 ( τ ) ← → X 1 ( ω ) X 2 ∗ ( ω ) R_{12} (\tau) \leftarrow \rightarrow X_1(\omega) X_2^*(\omega) R12(τ)←→X1(ω)X2∗(ω)
This also called as correlation theorem.
Parseval’s Theorem
Parseval’s theorem for energy signals states that the total energy in a signal can be obtained by the spectrum of the signal as
$ E = {1\over 2 \pi} \int_{-\infty}^{\infty} |X(\omega)|^2 d\omega $
Note: If a signal has energy E then time scaled version of that signal x(at) has energy E/a.