就是说相较于之前的非线性激活结构,这里使用elementwise multiplication这种方式实现非线性激活。
z ( 1 ) = g ( x ; θ ( 1 ) ) z ( i + 1 ) = ( W ( i ) z ( i ) + b ( i ) ) ∘ g ( x ; θ ( i + 1 ) ) , i = 1 , … , k − 1 f ( x ) = W ( k ) z ( k ) + b ( k ) \begin{aligned} z^{(1)} & =g\left(x ; \theta^{(1)}\right) \\ z^{(i+1)} & =\left(W^{(i)} z^{(i)}+b^{(i)}\right) \circ g\left(x ; \theta^{(i+1)}\right), i=1, \ldots, k-1 \\ f(x) & =W^{(k)} z^{(k)}+b^{(k)} \end{aligned} z(1)z(i+1)f(x)=g(x;θ(1))=(W(i)z(i)+b(i))∘g(x;θ(i+1)),i=1,…,k−1=W(k)z(k)+b(k)
主要特点是他的输出可以看做一定数量sin函数的线性组合。
证明了一个定理:傅里叶网络的输出是一系列sin函数的线性组合。换句话说,FOURIERNET将其最终函数表示为传统傅里叶基的线性组合,例如"经典"随机傅里叶特征。
证明的主要思想是按照一下公式:
sin ( ω x + ϕ ) ∘ sin ( τ x + ψ ) = 1 2 cos ( ( ω − τ ) x + ϕ − ψ ) − 1 2 cos ( ( ω + τ ) x + ϕ + ψ ) \sin (\omega x+\phi) \circ \sin (\tau x+\psi)=\frac{1}{2} \cos ((\omega-\tau) x+\phi-\psi)-\frac{1}{2} \cos ((\omega+\tau) x+\phi+\psi) sin(ωx+ϕ)∘sin(τx+ψ)=21cos((ω−τ)x+ϕ−ψ)−21cos((ω+τ)x+ϕ+ψ)
该推论主要是阐述经过网络的每一层输出的线性组合,其系数是可以进行计算的。
α ˉ = { 1 2 k − 1 W i k , i k − 1 ( k − 1 ) … W i 3 , i 2 ( 2 ) W i 2 , i 1 ( 1 ) } ω ˉ = { s k ω i k ( k ) + … + s 2 ω i 2 ( 2 ) + ω i 1 ( 1 ) } ϕ ˉ = { s k ϕ i k ( k ) + … + s 2 ϕ i 2 ( 2 ) + ϕ i 1 ( 1 ) + π 2 ∑ i = 2 k s k } . \begin{aligned} & \bar{\alpha}=\left\{\frac{1}{2^{k-1}} W_{i_k, i_{k-1}}^{(k-1)} \ldots W_{i_3, i_2}^{(2)} W_{i_2, i_1}^{(1)}\right\} \\ & \bar{\omega}=\left\{s_k \omega_{i_k}^{(k)}+\ldots+s_2 \omega_{i_2}^{(2)}+\omega_{i_1}^{(1)}\right\} \\ & \bar{\phi}=\left\{s_k \phi_{i_k}^{(k)}+\ldots+s_2 \phi_{i_2}^{(2)}+\phi_{i_1}^{(1)}+\frac{\pi}{2} \sum_{i=2}^k s_k\right\} . \end{aligned} αˉ={2k−11Wik,ik−1(k−1)…Wi3,i2(2)Wi2,i1(1)}ωˉ={skωik(k)+…+s2ωi2(2)+ωi1(1)}ϕˉ={skϕik(k)+…+s2ϕi2(2)+ϕi1(1)+2πi=2∑ksk}.
这里的 s k s_k sk表示的是二进制符号+1与-1
就是可以看到,每一次相乘都会产生两个符号位正负的sin函数组合,这样通过改变乘性网络的深度,可以实现指数级的sin函数数量进行拟合。
相较于傅里叶特征的全局性,小波变换会有更多的局部特征进行表现。
使用的激活函数为
g j ( x ; θ ( i ) ) = exp ( − γ j ( i ) 2 ∥ x − μ j ( i ) ∥ 2 2 ) sin ( ω j ( i ) x + ϕ j ( i ) ) g_j\left(x ; \theta^{(i)}\right)=\exp \left(-\frac{\gamma_j^{(i)}}{2}\left\|x-\mu_j^{(i)}\right\|_2^2\right) \sin \left(\omega_j^{(i)} x+\phi_j^{(i)}\right) gj(x;θ(i))=exp(−2γj(i) x−μj(i) 22)sin(ωj(i)x+ϕj(i))
此时这里的优化参数 θ ( i ) \theta^{(i)} θ(i)为:
θ ( i ) = { γ 1 : d i ( i ) ∈ R , μ 1 : d i ( i ) ∈ R n , ω 1 : d i ( i ) ∈ R n , ϕ 1 : d i ( i ) ∈ R } \theta^{(i)}=\left\{\gamma_{1: d_i}^{(i)} \in \mathbb{R}, \mu_{1: d_i}^{(i)} \in \mathbb{R}^n, \omega_{1: d_i}^{(i)} \in \mathbb{R}^n, \phi_{1: d_i}^{(i)} \in \mathbb{R}\right\} θ(i)={γ1:di(i)∈R,μ1:di(i)∈Rn,ω1:di(i)∈Rn,ϕ1:di(i)∈R}
同样也对线性组合进行了证明:The output of a Gabor Network is given by a linear combination of Gabor bases
f j ( x ) = ∑ t = 1 T α ˉ t exp ( − 1 2 γ ˉ t ∥ x − μ ˉ t ∥ 2 ) sin ( ω ˉ t x + ϕ ˉ t ) + b ˉ , f_j(x)=\sum_{t=1}^T \bar{\alpha}_t \exp \left(-\frac{1}{2} \bar{\gamma}_t\left\|x-\bar{\mu}_t\right\|^2\right) \sin \left(\bar{\omega}_t x+\bar{\phi}_t\right)+\bar{b}, fj(x)=t=1∑Tαˉtexp(−21γˉt∥x−μˉt∥2)sin(ωˉtx+ϕˉt)+bˉ,
参数的初始化分布:
由于γ有效地充当了高斯的逆协方差项,因此**Gamma( α , β)**随机变量(高斯逆协方差的共轭先验)是该参数的合理选择。
我们也简单地选择每个μ ( i )在允许的输入空间x的范围内均匀分布
我们将每层的α项缩放1 / k,以在最后一层有效地控制这个参数
第一个是图像表示的实验,第二个是图像生成的实验。采用PSNR值进行比较判断。
这项任务的主要方法是给一个定向的点云,寻找一个函数实现 f : R 3 → R f: \mathbb{R}^3 \rightarrow \mathbb{R} f:R3→R的映射。这样函数的水平集 { x ∣ f ( x ) = 0 } \{x \mid f(x)=0\} {x∣f(x)=0}就是表现物体的形状
使用的损失函数与SIREN使用的损失函数相同:
L = ∑ x ∈ Ω λ 1 ∥ 1 − ∣ ∇ x N ( x ) ∣ ∥ + ∑ x ∈ Ω 0 λ 2 ∥ N ( x ) ∥ + λ 3 ( 1 − ⟨ ∇ x N ( x ) , n ( x ) ⟩ ) + ∑ x ∉ Ω 0 λ 4 exp ( − α ∣ N ( x ) ∣ ) L=\sum_{x \in \Omega} \lambda_1\left\|1-\left|\nabla_x N(x)\right|\right\|+\sum_{x \in \Omega_0} \lambda_2\|N(x)\|+\lambda_3\left(1-\left\langle\nabla_x N(x), n(x)\right\rangle\right)+\sum_{x \notin \Omega_0} \lambda_4 \exp (-\alpha|N(x)|) L=∑x∈Ωλ1∥1−∣∇xN(x)∣∥+∑x∈Ω0λ2∥N(x)∥+λ3(1−⟨∇xN(x),n(x)⟩)+∑x∈/Ω0λ4exp(−α∣N(x)∣)
这个文件主要定义MFN的基本结构与文章中提出的两个网络。
import torch
from torch import nn
import torch.nn.functional as F
import numpy as np
class MFNBase(nn.Module):
"""
Multiplicative filter network base class.
Expects the child class to define the 'filters' attribute, which should be
a nn.ModuleList of n_layers+1 filters with output equal to hidden_size.
"""
def __init__(
self, hidden_size, out_size, n_layers, weight_scale, bias=True, output_act=False
):
super().__init__()
self.linear = nn.ModuleList(
[nn.Linear(hidden_size, hidden_size, bias) for _ in range(n_layers)]
)
self.output_linear = nn.Linear(hidden_size, out_size)
self.output_act = output_act
for lin in self.linear:
lin.weight.data.uniform_(
-np.sqrt(weight_scale / hidden_size),
np.sqrt(weight_scale / hidden_size),
)
return
def forward(self, x):
# 对每一层先进行非线性激活,本篇文章使用的是Fourier与Gabor,首先对第一层进行滤波器相乘。这里使用的滤波器是一个列表形式的。
out = self.filters[0](x)
for i in range(1, len(self.filters)):
out = self.filters[i](x) * self.linear[i - 1](out)
out = self.output_linear(out)
if self.output_act:
out = torch.sin(out)
return out
class FourierLayer(nn.Module):
"""
Sine filter as used in FourierNet.
"""
def __init__(self, in_features, out_features, weight_scale):
super().__init__()
self.linear = nn.Linear(in_features, out_features)
self.linear.weight.data *= weight_scale # gamma
# 给线性层定义偏差,这里其实就是相位
self.linear.bias.data.uniform_(-np.pi, np.pi)
return
def forward(self, x):
return torch.sin(self.linear(x))
# 疑惑点1:这里的output_act是什么东西。
class FourierNet(MFNBase):
def __init__(
self,
in_size,
hidden_size,
out_size,
n_layers=3,
input_scale=256.0,
weight_scale=1.0,
bias=True,
output_act=False,
):
super().__init__(
hidden_size, out_size, n_layers, weight_scale, bias, output_act
)
self.filters = nn.ModuleList(
[
FourierLayer(in_size, hidden_size, input_scale / np.sqrt(n_layers + 1))
for _ in range(n_layers + 1)
]
)
class GaborLayer(nn.Module):
"""
Gabor-like filter as used in GaborNet.
"""
def __init__(self, in_features, out_features, weight_scale, alpha=1.0, beta=1.0):
super().__init__()
self.linear = nn.Linear(in_features, out_features)
self.mu = nn.Parameter(2 * torch.rand(out_features, in_features) - 1)
self.gamma = nn.Parameter(
torch.distributions.gamma.Gamma(alpha, beta).sample((out_features,))
)
self.linear.weight.data *= weight_scale * torch.sqrt(self.gamma[:, None])
self.linear.bias.data.uniform_(-np.pi, np.pi)
return
def forward(self, x):
D = (
(x ** 2).sum(-1)[..., None]
+ (self.mu ** 2).sum(-1)[None, :]
- 2 * x @ self.mu.T
)
return torch.sin(self.linear(x)) * torch.exp(-0.5 * D * self.gamma[None, :])
class GaborNet(MFNBase):
def __init__(
self,
in_size,
hidden_size,
out_size,
n_layers=3,
input_scale=256.0,
weight_scale=1.0,
alpha=6.0,
beta=1.0,
bias=True,
output_act=False,
):
super().__init__(
hidden_size, out_size, n_layers, weight_scale, bias, output_act
)
self.filters = nn.ModuleList(
[
GaborLayer(
in_size,
hidden_size,
input_scale / np.sqrt(n_layers + 1),
alpha / (n_layers + 1),
beta,
)
for _ in range(n_layers + 1)
]
)
参考链接:
- ModuleList — PyTorch 1.13 documentation
- 详解PyTorch中的ModuleList和Sequential - 知乎 (zhihu.com)
该方法主要是用于对网络中的参数进行均匀初始化,其中两个参数分别为分布的最小值与最大值。
这里是对每一个线性层的参数进行初始化,使其在 [ − w e i g h t s c a l e h i d d e n s i z e , w e i g h t s c a l e h i d d e n s i z e ] [-\sqrt{\frac{weight_scale}{hidden_size}}, \sqrt{\frac{weight_scale}{hidden_size}}] [−hiddensizeweightscale,hiddensizeweightscale]区间内。
参考连接:
- python - How do I initialize weights in PyTorch? - Stack Overflow
- deep-learning-v2-pytorch/weight_initialization_exercise.ipynb at master · udacity/deep-learning-v2-pytorch (github.com)