The process of obtaining an output from the network
Time-synchronous & order-synchronous sequence
aaabbbbbbccc => abc (probility 0.5)
aabbbbbbbccc => abc (probility 0.001)
cccddddddeee => cde (probility 0.4)
cccddddeeeee => cde (probility 0.4)
Option 2 is in fact a suboptimal decode that actually finds the most likely time-synchronous output sequence
The “merging” heuristics do not guarantee optimal order-synchronous sequences
Constrain that the first symbol in the decode must be the top left block
Compose a graph such that every path in the graph from source to sink represents a valid alignment
Main idea
Dynamically track the best path (and the score of the best path) from the source node to every node in the graph
D I V = ∑ t X e n t ( Y t , s y m b o l t b e s t p a t h ) = − ∑ t log Y ( t , s y m b o l t b e s t p a t h ) D I V=\sum_{t} X e n t\left(Y_{t}, s y m b o l_{t}^{b e s t p a t h}\right)=-\sum_{t} \log Y\left(t, s y m b o l_{t}^{b e s t p a t h}\right) DIV=t∑Xent(Yt,symboltbestpath)=−t∑logY(t,symboltbestpath)
Problem
Alternate view
Instead of only selecting the most likely alignment, use the statistical expectation over all possible alignments
D I V = E [ − ∑ t log Y ( t , s t ) ] D I V=E\left[-\sum_{t} \log Y\left(t, s_{t}\right)\right] DIV=E[−t∑logY(t,st)]
Use the entire distribution of alignments
This will mitigate the issue of suboptimal selection of alignment
Using the linearity of expectation
D I V = − ∑ t E [ log Y ( t , s t ) ] D I V=-\sum_{t} E\left[\log Y\left(t, s_{t}\right)\right] DIV=−t∑E[logY(t,st)]
D I V = − ∑ t ∑ S ∈ S 1 … S K P ( s t = S ∣ S , X ) log Y ( t , s t = S ) D I V=-\sum_{t} \sum_{S \in S_{1} \ldots S_{K}} P\left(s_{t}=S | \mathbf{S}, \mathbf{X}\right) \log Y\left(t, s_{t}=S\right) DIV=−t∑S∈S1…SK∑P(st=S∣S,X)logY(t,st=S)
P ( s t = S ∣ S , X ) P(s_{t}=S | \mathbf{S}, \mathbf{X}) P(st=S∣S,X) is the probability of seeing the specific symbol s s s at time t t t, given that the symbol sequence is an expansion of S = s 0 , . . . , S K − 1 S = s_0,...,S_{K-1} S=s0,...,SK−1 and given the input sequence X = X o , . . . , X N − 1 X = X_o,...,X_{N-1} X=Xo,...,XN−1
P ( s t = S r , S ∣ X ) P\left(s_{t}=S_{r}, \mathbf{S} | \mathbf{X}\right) P(st=Sr,S∣X) is the total probability of all valid paths in the graph for target sequence S S S that go through the symbol S r S_r Sr (the r t h r^{th} rth symbol in the sequence S 0 , . . . , S K − 1 S_0,...,S_{K-1} S0,...,SK−1 ) at time
P ( s t = S r , S ∣ X ) = P ( S 0 … S r , s t = S r ∣ X ) P ( s t + 1 ∈ succ ( S r ) , succ ( S r ) , … , S K − 1 ∣ X ) P\left(s_{t}=S_{r}, \mathbf{S} | \mathbf{X}\right)=P\left(S_{0} \ldots S_{r}, s_{t}=S_{r} | \mathbf{X}\right) P\left(s_{t+1} \in \operatorname{succ}\left(S_{r}\right), \operatorname{succ}\left(S_{r}\right), \ldots, S_{K-1} | \mathbf{X}\right) P(st=Sr,S∣X)=P(S0…Sr,st=Sr∣X)P(st+1∈succ(Sr),succ(Sr),…,SK−1∣X)
α ( t , l ) = ( α ( t − 1 , l ) + α ( t − 1 , l − 1 ) ) y t S ( l ) \alpha(t, l)=(\alpha(t-1, l)+\alpha(t-1, l-1)) y_{t}^{S(l)} α(t,l)=(α(t−1,l)+α(t−1,l−1))ytS(l)
log α ( t , l ) = log ( e log α ( t − 1 , l ) + e log α ( t − 1 , l − 1 ) ) + log y t S ( l ) \log \alpha(t, l)=\log \left(e^{\log \alpha(t-1, l)}+e^{\log \alpha(t-1, l-1)}\right)+\log y_{t}^{S(l)} logα(t,l)=log(elogα(t−1,l)+elogα(t−1,l−1))+logytS(l)
β ( t , r ) = P ( s t + 1 ∈ succ ( S r ) , succ ( S r ) , … , S K − 1 ∣ X ) \beta(t, r)=P\left(s_{t+1} \in \operatorname{succ}\left(S_{r}\right), \operatorname{succ}\left(S_{r}\right), \ldots, S_{K-1} | \mathbf{X}\right) β(t,r)=P(st+1∈succ(Sr),succ(Sr),…,SK−1∣X)
P ( s t = S r , S ∣ X ) = α ( t , r ) β ( t , r ) P\left(s_{t}=S_{r}, \mathbf{S} | \mathbf{X}\right)=\alpha(t, r) \beta(t, r) P(st=Sr,S∣X)=α(t,r)β(t,r)
γ ( t , r ) = P ( s t = S r ∣ S , X ) = P ( s t = S r , S ∣ X ) ∑ S r ′ P ( s t = S r ′ , S ∣ X ) = α ( t , r ) β ( t , r ) ∑ r ′ α ( t , r ′ ) β ( t , r ′ ) \gamma(t,r) = P\left(s_{t}=S_{r} | \mathbf{S}, \mathbf{X}\right)=\frac{P\left(s_{t}=S_{r}, \mathbf{S} | \mathbf{X}\right)}{\sum_{S_{r}^{\prime}} P\left(s_{t}=S_{r}^{\prime}, \mathbf{S} | \mathbf{X}\right)}=\frac{\alpha(t, r) \beta(t, r)}{\sum_{r^{\prime}} \alpha\left(t, r^{\prime}\right) \beta\left(t, r^{\prime}\right)} γ(t,r)=P(st=Sr∣S,X)=∑Sr′P(st=Sr′,S∣X)P(st=Sr,S∣X)=∑r′α(t,r′)β(t,r′)α(t,r)β(t,r)
γ ( t , r ) = 1 y t S ( r ) α ( t , r ) β ^ ( t , r ) ∑ r ′ 1 y t S ( r ) α ( t , r ) β ^ ( t , r ) \gamma(t, r)=\frac{\frac{1}{y_{t}^{S(r)}} \alpha(t, r) \hat{\beta}(t, r)}{\sum_{r^{\prime}} \frac{1}{y_{t}^{S(r)}} \alpha(t, r) \hat{\beta}(t, r)} γ(t,r)=∑r′ytS(r)1α(t,r)β^(t,r)ytS(r)1α(t,r)β^(t,r)
D I V = − ∑ t ∑ s ∈ S 0 … S K − 1 P ( s t = s ∣ S , X ) log Y ( t , s t = s ) D I V=-\sum_{t} \sum_{s \in S_{0} \ldots S_{K-1}} P\left(s_{t}=s | \mathbf{S}, \mathbf{X}\right) \log Y\left(t, s_{t}=s\right) DIV=−t∑s∈S0…SK−1∑P(st=s∣S,X)logY(t,st=s)
D I V = − ∑ t ∑ r γ ( t , r ) log y t S ( r ) D I V=-\sum_{t} \sum_{r} \gamma(t, r) \log y_{t}^{S(r)} DIV=−t∑r∑γ(t,r)logytS(r)
∇ Y t D I V = [ d D I V d y t S 0 d D I V d y t S 1 ⋯ d D I V d y t S L − 1 ] \nabla_{Y_{t}} D I V=\left[\begin{array}{llll} \frac{d D I V}{d y_{t}^{S_{0}}} & \frac{d D I V}{d y_{t}^{S_{1}}} & \cdots & \frac{d D I V}{d y_{t}^{S_{L-1}}} \end{array}\right] ∇YtDIV=[dytS0dDIVdytS1dDIV⋯dytSL−1dDIV]
Compare to Viterbi algorithm, components will be non-zero only for symbols that occur in the training instance
Compute derivative
d D I V d y t l = − ∑ r : S ( r ) = l d d y t S ( r ) γ ( t , r ) log y t S ( r ) \frac{d D I V}{d y_{t}^{l}}=-\sum_{r: S(r)=l} \frac{d}{d y_{t}^{S(r)}} \gamma(t, r) \log y_{t}^{S(r)} dytldDIV=−r:S(r)=l∑dytS(r)dγ(t,r)logytS(r)
d d y t S ( r ) γ ( t , r ) log y t S ( r ) = γ ( t , r ) y t S ( r ) + d γ ( t , r ) d y t S ( r ) log y t S ( r ) \frac{d}{d y_{t}^{S(r)}} \gamma(t, r) \log y_{t}^{S(r)}=\frac{\gamma(t, r)}{y_{t}^{S(r)}}+\frac{d \gamma(t, r)}{d y_{t}^{S(r)}} \log y_{t}^{S(r)} dytS(r)dγ(t,r)logytS(r)=ytS(r)γ(t,r)+dytS(r)dγ(t,r)logytS(r)
d D I V d y t l = − ∑ r : S ( r ) = l γ ( t , r ) y t S ( r ) \frac{d D I V}{d y_{t}^{l}}=-\sum_{r: S(r)=l} \frac{\gamma(t, r)}{y_{t}^{S(r)}} dytldDIV=−r:S(r)=l∑ytS(r)γ(t,r)
Setup the network
Initialize all parameters of the network
Foreach training instance
Backword pass
Perform the forward backward algorithm to compute α ( t , r ) \alpha(t,r) α(t,r) and β ( t , r ) \beta(t,r) β(t,r)
Compute derivative of divergence ∇ Y t D I V \nabla_{Y_{t}} D I V ∇YtDIV for each Y t Y_t Yt
∇ Y t D I V = [ d D I V d y t 0 d D I V d y t 1 ⋯ d D I V d y t L − 1 ] \nabla_{Y_{t}} D I V=\left[\begin{array}{llll}\frac{d D I V}{d y_{t}^{0}} & \frac{d D I V}{d y_{t}^{1}} & \cdots & \frac{d D I V}{d y_{t}^{L-1}}\end{array}\right] ∇YtDIV=[dyt0dDIVdyt1dDIV⋯dytL−1dDIV]
d D I V d y t l = − ∑ r : S ( r ) = l γ ( t , r ) y t S ( r ) \frac{d D I V}{d y_{t}^{l}}=-\sum_{r: S(r)=l} \frac{\gamma(t, r)}{y_{t}^{S(r)}} dytldDIV=−r:S(r)=l∑ytS(r)γ(t,r)
Aggregate derivatives over minibatch and update parameters