[Foundation of Machine Learning][Week 2][Guarantee of PLA] the Correctness Verification of PLA

Conditions:

1. For the data set D, there exists a $\displaystyle W_{f}$ which satisfies that for every n, $y_{n}W_{f}^{T}X_{n}>0$.
2. $\displaystyle W_{T+1}=W_{T}+y_{n(T)}X_{n(T)}$ AND $\displaystyle y_{n(T)}W_{T}^{T}X_{n(T)}<0$.

So, let us set $\rho = \min y_{n}W_{f}^{T}X_{n}>0$ AND $R^{2}= \max |X_{n}|^{2}$

As a result, we have some inferences.

 

Inferences:

1. $\displaystyle W_{f}^{T}W_{T+1}=W_{f}^{T}(W_{T}+y_{n(T)}X_{n(T)})=W_{f}^{T}W_{T}+y_{n(T)}W_{f}^{T}X_{n(T)} \ge W_{f}^{T}W_{T}+\rho$
2. $\displaystyle |W_{T+1}|^{2}=|W_{T}|^{2}+|y_{n(T)}X_{n(T)}|^{2}+2y_{n(T)}W_{T}^{T}X_{n(T)} \le |W_{T}|^{2}+|X_{n(T)}|^{2} \le |W_{T}|^{2} + R^{2}$

Thus, by using difference, we get $\displaystyle W_{f}^{T}W_{T+1} \ge (T+1)\rho$ AND $|W_{T+1}| \le \sqrt{T+1} R $

Then, we get $\displaystyle\frac{W_{f}^{T}W_{T+1}}{|W_{T+1}| |W_{f}|} \ge \sqrt{T+1} \frac{\rho}{R |W_{f}|}$