Keypoint-Guided Optimal Transport阅读笔记

假设已经直到源域和目标域之间的某几个点的对应关系

公式

${\Sigma }_m=\left\{\mathbf{p}\in {\mathbb{R}}_{+}^m|{\sum }_i{p}_i=1\right\}$
${\pi }_{i,:}$表示$\pi$的第$i$行
${\pi }_{:,j}$表示$\pi$的第$j$列
${⟨\mathbf{P},\mathbf{Q}⟩}_F=\mathrm{tr}\left({\mathbf{P}}^{\mathrm{T}}\mathbf{Q}\right)$

Background on Optimal Transport

Kantorovich Problem (KP)

考虑两组数据
源：$\mathbf{X}={\left\{x_i\right\}}_{i=1}^m$
目标：$\mathbf{Y}={\left\{y_j\right\}}_{j=1}^n$
分布$\mathbf{p}={\sum }_{i=1}^m{p}_i{\delta }_{x_i}$, $\mathbf{q}={\sum }_{j=1}^n{p}_j{\delta }_{y_j}$
$\mathbf{p}\in {\Sigma }_m,\mathbf{q}\in {\Sigma }_n$

代价矩阵：$\mathbf{C}=\left(C_{i,j}\right)\in {\mathbb{R}}^{m\times n}$
其中，$C_{i,j}=c\left(x_i,y_j\right)$
目标：将$\mathbf{p}$转为$\mathbf{q}$的最小代价
$$\underset{\pi \in \Pi (\mathbf{p},\mathbf{q})}{\min }{L}_{kp}(\pi )\triangleq ⟨\pi ,\mathbf{C}{⟩}_F,\text{ s.t. }\Pi (\mathbf{p},\mathbf{q})=\left\{\pi \in {\mathbb{R}}_{+}^{m\times n}\mid \pi 1_n=\mathbf{p},{\pi }^T{1}_m=\mathbf{q}\right\}$$

Partial OT model

考虑只将$\mathbf{p}$的部分转为$\mathbf{q}$所需要的代价
例如只传输$s$单位，其中$0\le s\le \min \left(\Vert \mathbf{p}{\Vert }_1,\Vert \mathbf{q}{\Vert }_1\right)$
$$\underset{\pi \in {\Pi }^s(\mathbf{p},\mathbf{q})}{\min }{L}_{kp}(\pi )\triangleq ⟨\pi ,\mathbf{C}{⟩}_F,\text{ s.t. }{\Pi }^s(\mathbf{p},\mathbf{q})=\left\{\pi \in {\mathbb{R}}_{+}^{m\times n}\mid \pi 1_n\le \mathbf{p},{\pi }^T{1}_m\le \mathbf{q},1_m^T\pi 1_n=s\right\}$$

注意这里$\mathbf{p}$和$\mathbf{q}$的量可以不一样

Gromov-Wasserstein (GW) model

$x_i$和$y_j$可能不在一个空间里，（比如图片和声音）
GW model使用的是域内的距离，源域${\mathbf{C}}^s=\left(C_{i,k}^s\right)\in {\mathbb{R}}^{m\times n}$,目标域${\mathbf{C}}^t=\left(C_{j,l}^t\right)\in {\mathbb{R}}^{n\times n}$
其中$C_{i,k}^s$表示$x_i$和$x_k$的距离
$C_{j,l}^t$表示$y_j$和$y_l$的距离

min ⁡ π ∈ Π ( p , q ) L g w ( π ) ≜ ∑ i , k = 1 m ∑ j , l = 1 n π i , j π k , l ∣ C i , k s − C j , l t ∣ 2 \min\limits_{\mathbf{\pi}\in \Pi\left(\mathbf{p},\mathbf{q}\right)} L_{gw}\left(\mathbf{\pi}\right) \triangleq \sum_{i,k=1}^{m}\sum_{j,l=1}^n \pi_{i,j}\pi_{k,l}\left|C_{i,k}^s - C_{j,l}^t\right|^2 π∈Π(p,q)min​Lgw​(π)≜i,k=1∑m​j,l=1∑n​πi,j​πk,l​ ​Ci,ks​−Cj,lt​ ​2

Keypoint-Guided Optimal Transport

Preservation of Matching of Keypoints in Transport

设关键点对$\mathcal{K}={\left\{\left(i_u,j_u\right)\right\}}_{u=1}^U$
$U$表示关键点对数量

用$\mathcal{I}={\left\{i_u\right\}}_{u=1}^U$和$\mathcal{J}={\left\{j_u\right\}}_{u=1}^U$表示源域和目标域的关键点的索引
例如$\mathcal{K}=\left\{\left(3,2\right),\left(6,5\right)\right\}$，那么$\mathcal{I}=\left\{3,6\right\},\mathcal{J}=\left\{2,5\right\}$

作者首先保证这几个关键点能匹配上

如果$\left(i,j\right)\in \mathcal{K}$，则最优传输方案$\tilde{\pi }$的第$i$行和第$j$列除了${\tilde{\pi }}_{i,j}$以外都为$0$
（因为$x_i$只能传输给$y_j$）
$$\tilde{\pi }=\mathbf{M}\odot \pi$$
定义
$$\Pi \left(\mathbf{p},\mathbf{q};\mathbf{M}\right)=\left\{\pi \in {\mathbb{R}}_{+}^{m\times n}|\left(\mathbf{M}\odot \pi \right)1_n=\mathbf{p},{\left(\mathbf{M}\odot \pi \right)}^T{1}_m=\mathbf{q}\right\}$$

其中
$$M_{i,j}=\{\begin{array}{ll}1,& \text{ if }(i,j)\in \mathcal{K},\\ 0,& \text{ if }i\in \mathcal{I}\text{ and }(i,j)\notin \mathcal{K},\\ 0,& \text{ if }j\in \mathcal{J}\text{ and }(i,j)\notin \mathcal{K},\\ 1,& \text{ otherwise }(\text{ i.e., }i\notin \mathcal{I}\text{ and }j\notin \mathcal{J})\end{array}$$
注意，这里是假设如果$\left(i,j\right)\in \mathbb{K}$,则$p_i=q_j$

$$M_{i,j}=\{\begin{array}{ll}1,& \text{ if }(i,j)\in \mathcal{K},\\ 0,& \text{ if }i\in \mathcal{I},p_i\le q_{\kappa (i)},\text{ and }(i,j)\notin \mathcal{K},\\ 0,& \text{ if }j\in \mathcal{J},p_{{\kappa }^{\prime }(j)}\ge q_j,\text{ and }(i,j)\notin \mathcal{K},\\ 1,& \text{ if }i\in \mathcal{I},p_i>q_{\kappa (i)},\text{ and }(i,j)\notin \mathcal{K},\\ 1,& \text{ if }j\in \mathcal{J},p_{{\kappa }^{\prime }(j)}<q_j,\text{ and }(i,j)\notin \mathcal{K},\\ 1,& \text{ otherwise }(\text{ i.e. },i\notin \mathcal{I},j\notin \mathcal{J}).\end{array}$$

其中$j=\kappa \left(i\right),i={\kappa }^{\prime }(j)$,若$\left(i,j\right)\in \mathcal{K}$

Modeling the Relation to Keypoints

作者提出保持每个点到每个关键点的关系

对于$x_k\in X$, 他到$x_{i_u}$的关系分数定义为
$$R_{k,i_u}^s=\frac{e^{-C_{k,i_u}^s/\tau }}{{\sum }_{u^{\prime }=1}^U{e}^{-C_{k,i_{u^{\prime }}}^s/\tau }},\forall i_u\in \mathcal{I}$$
其中$\tau =\rho {\max }_{i,k}\left\{C_{i,k}^s\right\}$
可以看出如果$x_k$到最近的关键点的分数最大

类似地
对于$y_l\in Y$, 他到$y_{j_u}$的关系分数定义为
$$R_{l,j_u}^t=\frac{e^{-C_{l,j_u}^t/{\tau }^{\prime }}}{{\sum }_{u^{\prime }=1}^U{e}^{-C_{l,j_{u^{\prime }}}^t/{\tau }^{\prime }}},\forall j_u\in \mathcal{J}$$
其中${\tau }^{\prime }=\rho {\max }_{j,l}\left\{C_{j,l}^t\right\}$

这里$\max$的目的是归一化，进而增加鲁棒性
$\rho$设置为$0.1$，是一个常见的温度系数

$R_k^s=\left(R_{k,i_1}^s,R_{k,i_2}^s,\cdots R_{k,i_U}^s\right)$
$R_l^t=\left(R_{l,j_1}^t,R_{l,j_2}^t,\cdots R_{l,j_U}^t\right)$

Realizing Keypoint Guidance by Relation Preservation

$$\mathbf{G}=\left(G_{k,l}\right)\in {\mathbb{R}}^{m\times n},G_{k,l}=d\left(R_k^s,R_l^t\right)$$
其中$d$衡量$R_k^s$和$R_l^t$的不相似度，如果$R_k^s$和$R_l^t$相似，那么$G_{k,l}$非常小
这里$d$设置为JS散度

KeyPoint-Guided model by ReLation preservation (KPG-RL) 定义为
$$\underset{\pi \in \Pi \left(\mathbf{p},\mathbf{q};\mathbf{M}\right)}{\min }{L}_{kpg}\left(\pi \right)\triangleq {⟨\mathbf{M}\odot \pi ,\mathbf{G}⟩}_F$$
根据定义
1.对应的关键点会被强制匹配
2.$\mathbf{G}$较小的地方，最优传输方案中对应的位置就会越大

$d\left(R_k^s,R_l^t\right)$基本取决于离得最近的关键点

求解：
求解可以用线性规划，也可以加入熵正则化用sinkhorn
$${\mathbf{u}}^{\left(l+1\right)}=\frac{\mathbf{p}}{\mathbf{K}{\mathbf{v}}^{\left(l\right)}},{\mathbf{v}}^{\left(l+1\right)}=\frac{\mathbf{q}}{{\mathbf{K}}^T{\mathbf{u}}^{\left(l+1\right)}}$$
其中$\mathbf{K}=\mathbf{M}\odot e^{-G/\epsilon }$

Imposing Keypoint Guidance in KP/GW

KPG-RL-KP
相当于加了个正则
$$\underset{\pi \in \Pi \left(\mathbf{p},\mathbf{q};\mathbf{M}\right)}{\min }\left\{\alpha L_{kp}\left(\mathbf{M}\odot \pi \right)+\left(1-\alpha \right)L_{kpg}\left(\pi \right)={⟨\mathbf{M}\odot \pi ,\alpha \mathbf{C}+\left(1-\alpha \right)\mathbf{G}⟩}_F\right\},\alpha \in \left(0,1\right)$$
依然可以用线性规划或者sinkhorn算法求解

KPG-RL-GW
$$\underset{\pi \in \Pi \left(\mathbf{p},\mathbf{q};\mathbf{M}\right)}{\min }\alpha L_{gw}\left(\mathbf{M}\odot \pi \right)+\left(1-\alpha \right)L_{kpg}\left(\pi \right),\alpha \in \left(0,1\right)$$
依然用Frank-Walfe算法求解

Extension to Partial OT

$$\underset{\pi \in {\Pi }^s(\mathbf{p},\mathbf{q};\mathbf{M})}{\min }\left\{L_{kpg}(\mathbf{M}\odot \pi )=⟨\mathbf{M}\odot \pi ,\mathbf{C}{⟩}_F\right\}$$
其中
$${\Pi }^s\left(\mathbf{p},\mathbf{q};\mathbf{M}\right)=\left\{\pi \in {\mathbb{R}}_{+}^{m\times n}|\left(\mathbf{M}\odot \pi \right)1_n\le \mathbf{p},{\left(\mathbf{M}\odot \pi \right)}^T{1}_m\le \mathbf{q},1_m^T\left(\mathbf{M}\odot \pi \right)1_n=s;{\left(\mathbf{M}\odot \pi \right)}_{i,:}{1}_n=p_i,\forall i\in \mathcal{I};{\left(\mathbf{M}\odot \pi \right)}_{:,j}{1}_m=p_i,\forall j\in \mathcal{J};\right\}$$

Chapel等人提出一种将partial ot转为ot的方法
增加dummy point，剩下的点传输给 dummy point

源域增加一个$\Vert \mathbf{q}{\Vert }_1-s$量的dummy point
目标域增加一个$\Vert \mathbf{p}{\Vert }_1-s$量的dummy point

令$\bar{\mathbf{p}}={\left({\mathbf{p}}^T,\Vert \mathbf{q}{\Vert }_1-s\right)}^T$
$\bar{\mathbf{q}}={\left({\mathbf{q}}^T,\Vert \mathbf{p}{\Vert }_1-s\right)}^T$
$$\bar{\mathbf{G}}=\left[\begin{array}{cc}\mathbf{G}& \xi 1_n\\ \xi 1_m^T& 2\xi +A\end{array}\right],\bar{\mathbf{M}}=\left[\begin{array}{cc}\mathbf{M}& \mathbf{a}\\ {\mathbf{b}}^T& 1\end{array}\right]$$
其中$A>0,\xi$是两个固定的标量，$a_i=0,\forall i\in \mathcal{I}$，否则$a_i=1$
$b_j=0,\forall j\in \mathcal{J}$，否则$b_j=1$

原问题转为
$$\underset{\bar{\pi }\in \Pi \left(\bar{\mathbf{p}},\bar{\mathbf{q}};\bar{\mathbf{M}}\right)}{\min }{⟨\bar{\mathbf{M}}\odot \bar{\pi },\bar{\mathbf{G}}⟩}_F$$

可以证明：假设$A>0,{\sum }_{i\in \mathcal{I}}{p}_i<s,{\sum }_{j\in \mathcal{I}}{q}_j<s$
则$\bar{\mathbf{M}}\odot {\bar{\pi }}^{\ast }$的左上$m\times n$矩阵就是$\mathbf{M}\odot {\pi }^{\ast }$

证明：
令$\ddot{\pi }={\bar{\pi }}_{1:m,1:n}^{\ast },t={\bar{\pi }}_{m+1,n+1}^{\ast }$
这个证明分为3步
1.证明$t=0$
2.证明$\ddot{\pi }\in {\Pi }^s\left(\mathbf{p},\mathbf{q};\mathbf{M}\right)$
3.证明$\mathbf{M}\odot \ddot{\pi }$是最优解

因为${\bar{\pi }}^{\ast }\in \Pi \left(\bar{\mathbf{p}},\bar{\mathbf{q}};\bar{\mathbf{M}}\right),t={\bar{\pi }}_{m+1,n+1}^{\ast }$,因此

$$\begin{array}{rl}{1}_{m+1}^T\left(\bar{\mathbf{M}}\odot {\bar{\pi }}^{\ast }\right)1_{n+1}& =\left[\begin{array}{cc}{1}_m^T& 1\end{array}\right]\left(\left[\begin{array}{cc}\mathbf{M}& \mathbf{a}\\ \mathbf{b}& 1\end{array}\right]\odot \left[\begin{array}{cc}\ddot{\pi }& {\bar{\pi }}_{1:m,n+1}^{\ast }\\ {\bar{\pi }}_{m+1,1:n}^{\ast }& {\bar{\pi }}_{m+1,n+1}^{\ast }\end{array}\right]\right)\left[\begin{array}{c}{1}_n\\ 1\end{array}\right]\\ & =1_m^T\left(\mathbf{M}\odot \ddot{\pi }\right)1_n+\sum _{i=1}^m{a}_i{\bar{\pi }}_{i,n+1}^{\ast }+\sum _{j=1}^n{b}_j{\pi }_{m+1,j}^{\ast }+t\end{array}$$

并且

$$1_{m+1}^T\left(\bar{\mathbf{M}}\odot {\bar{\pi }}^{\ast }\right)1_{n+1}=1_{m+1}^T\bar{\mathbf{p}}=\Vert \bar{\mathbf{p}}{\Vert }_1=\Vert \mathbf{p}{\Vert }_1+\Vert \mathbf{q}{\Vert }_1-s$$

$$\sum _{i=1}^m{a}_i{\bar{\pi }}_{i,n+1}^{\ast }+t=q_{n+1}=\Vert \mathbf{p}{\Vert }_1-s$$

$$\sum _{j=1}^n{b}_j{\bar{\pi }}_{m+1,j}^{\ast }+t=p_{n+1}=\Vert \mathbf{q}{\Vert }_1-s$$

因此

$$1_{m+1}^T\left(\bar{\mathbf{M}}\odot {\bar{\pi }}^{\ast }\right)1_{n+1}=1_m^T\left(\mathbf{M}\odot \ddot{\pi }\right)1_n+\Vert \mathbf{p}{\Vert }_1+\Vert \mathbf{q}{\Vert }_1-2s-t=\Vert \mathbf{p}{\Vert }_1+\Vert \mathbf{q}{\Vert }_1-s$$

进而

$$1_m^T\left(\mathbf{M}\odot \ddot{\pi }\right)1_n=s+t$$

第一步：证明$t=0$

$$\begin{array}{rl}⟨\bar{\mathbf{M}}\odot {\bar{\pi }}^{\ast },\mathbf{G}⟩& =\sum _{i=1}^m\sum _{j=1}^n{M}_{i,j}{\bar{\pi }}_{i,j}^{\ast }{G}_{i,j}+\xi \sum _{i=1}^m{a}_i{\bar{\pi }}_{i,n+1}^{\ast }\\ & +\xi \sum _{j=1}^n{b}_j{\bar{\pi }}_{m+1,j}^{\ast }+\left(2\xi +A\right){\bar{\pi }}_{m+1,n+1}^{\ast }\\ & =\sum _{i=1}^m\sum _{j=1}^n{M}_{i,j}{\bar{\pi }}_{i,j}^{\ast }{G}_{i,j}+\xi \left(\Vert \mathbf{p}{\Vert }_1+\Vert \mathbf{q}{\Vert }_1-2s-2t\right)+\left(2\xi +A\right)t\\ & =\sum _{i=1}^m\sum _{j=1}^n{M}_{i,j}{\bar{\pi }}_{i,j}^{\ast }{G}_{i,j}+\xi \left(\Vert \mathbf{p}{\Vert }_1+\Vert \mathbf{q}{\Vert }_1-2s\right)+At\end{array}$$

假设$t>0$, 那接下来就要构造一个可行解$\gamma$,满足${\gamma }_{m+1,n+1}=0$来形成矛盾

随机选择一个集合$S=\left\{\left(i,j\right)|{\bar{\pi }}_{i,j}^{\ast }>0,i\le m,j\le n,i\notin \mathcal{I},j\notin \mathcal{J}\right\}$
以及一对索引$\left(i_0,j_0\right)$,满足$S$中的约束，使得$\sum _{\left(i,j\right)\in S}{\bar{\pi }}_{i,j}^{\ast }\le t$以及$\sum _{\left(i,j\right)\in S}{\bar{\pi }}_{i,j}^{\ast }+{\pi }_{i_0,j_0}^{\ast }>t$

这样的$S$和$\left(i_0,j_0\right)$是一定存在的，因为

$$\begin{array}{rl}{1}_m^T\left(\mathbf{M}\odot \ddot{\pi }\right)1_n& =\sum _{i=1}^m\sum _{j=1}^n{M}_{i,j}{\bar{\pi }}_{i,j}^{\ast }\\ & =\sum _{i\in \mathcal{I},j}{M}_{i,j}{\bar{\pi }}_{i,j}^{\ast }+\sum _{i\notin \mathcal{I},j\in \mathcal{J}}{M}_{i,j}{\bar{\pi }}_{i,j}^{\ast }+\sum _{i\notin \mathcal{I},j\notin \mathcal{J}}{M}_{i,j}{\bar{\pi }}_{i,j}^{\ast }\\ & =\sum _{i\in \mathcal{I}}{p}_i+\sum _{i\notin \mathcal{I},j\notin \mathcal{J}}{\bar{\pi }}_{i,j}^{\ast }\end{array}$$

由$1_m^T\left(\mathbf{M}\odot \ddot{\pi }\right)1_n=s+t$以及$\sum _{i\in \mathcal{I}}{p}_i<s$, 有${\sum }_{i\notin \mathcal{I},j\notin \mathcal{J}}{\bar{\pi }}_{i,j}^{\ast }>t$

现在构造$\gamma$
$\forall \left(i,j\right)\in S$,令${\gamma }_{i,j}=0,{\gamma }_{i,n+1}={\bar{\pi }}_{i,n+1}^{\ast }+{\bar{\pi }}_{i,j}^{\ast },{\gamma }_{m+1,j}={\bar{\pi }}_{m+1,j}^{\ast }+{\bar{\pi }}_{i,j}^{\ast }$
${\gamma }_{i_0,j_0}={\bar{\pi }}_{i_0,j_0}^{\ast }-\left(t-\sum _{i\notin \mathcal{I},j\notin \mathcal{J}}{\bar{\pi }}_{i,j}^{\ast }\right),{\gamma }_{i_0,n+1}={\bar{\pi }}_{i_0,n+1}^{\ast }-\left(t-\sum _{i\notin \mathcal{I},j\notin \mathcal{J}}{\bar{\pi }}_{i,j}^{\ast }\right),{\gamma }_{m+1,j_0}={\bar{\pi }}_{m+1,j_0}^{\ast }-\left(t-\sum _{i\notin \mathcal{I},j\notin \mathcal{J}}{\bar{\pi }}_{i,j}^{\ast }\right)$
$\forall \left(i,j\right)\notin S,{\gamma }_{i,j}={\bar{\pi }}_{i,j}^{\ast },{\gamma }_{i,n+1}={\bar{\pi }}_{i,n+1}^{\ast },{\gamma }_{m+1,j}={\bar{\pi }}_{m+1,j}^{\ast }$
容易验证$\gamma \in \Pi \left(\bar{\mathbf{p}},\bar{\mathbf{q}};\bar{\mathbf{M}}\right)$
于是

$$⟨\bar{\mathbf{M}}\odot \gamma ,\mathbf{G}⟩=\sum _{i=1}^m\sum _{j=1}^n{M}_{i,j}{\gamma }_{i,j}{G}_{i,j}+\xi \left(\Vert \mathbf{p}{\Vert }_1+\Vert \mathbf{q}{\Vert }_1-2s\right)$$

利用$\bar{\mathbf{M}}\odot {\bar{\pi }}^{\ast }$的最优性，有

$$⟨\bar{\mathbf{M}}\odot \gamma ,\mathbf{G}⟩-⟨\bar{\mathbf{M}}\odot {\bar{\pi }}^{\ast },\mathbf{G}⟩=\sum _{i=1}^m\sum _{j=1}^n{M}_{i,j}\left({\gamma }_{i,j}-{\bar{\pi }}_{i,j}^{\ast }\right)G_{i,j}-At>0$$

由$\gamma$的定义，${\gamma }_{i,j}\le {\bar{\pi }}_{i,j}^{\ast }$
因此${\sum }_{i=1}^m{\sum }_{j=1}^n{M}_{i,j}\left({\gamma }_{i,j}-{\bar{\pi }}_{i,j}^{\ast }\right)G_{i,j}\le 0$, 进而矛盾
因此$t=0$

第二步：证明$\ddot{\pi }\in {\Pi }^s\left(\mathbf{p},\mathbf{q};\mathbf{M}\right)$

（1）${\bar{\pi }}^{\ast }\ge 0$，因此$\ddot{\pi }\ge 0$
（2）$\left(\bar{\mathbf{M}}\odot {\bar{\pi }}^{\ast }\right)1_{n+1}=\left[\begin{array}{cc}\mathbf{M}\odot \ddot{\pi }& \mathbf{a}\odot {\bar{\pi }}_{1:m,n+1}^{\ast }\\ \mathbf{b}\odot {\bar{\pi }}_{m+1,1:n}^{\ast }& 0\end{array}\right]\left[\begin{array}{c}{1}_n\\ 1\end{array}\right]=\left[\begin{array}{c}\mathbf{p}\\ \Vert \mathbf{q}{\Vert }_1-s\end{array}\right]$
因此$\left(\mathbf{M}\odot \ddot{\pi }\right)1_n+\mathbf{a}\odot {\bar{\pi }}_{1:m,n+1}^{\ast }=\mathbf{p}$,进而$\left(\mathbf{M}\odot \ddot{\pi }\right)1_n\le \mathbf{p}$
（3）与上面类似，${\left(\bar{\mathbf{M}}\odot {\bar{\pi }}^{\ast }\right)}^T{1}_{m+1}={\left({\mathbf{q}}^T,\Vert \mathbf{q}{\Vert }_1-s\right)}^T$,因此${\left(\mathbf{M}\odot \ddot{\pi }\right)}^T{1}_m\le \mathbf{q}$
（4）$1_m^T\left(\mathbf{M}\odot \ddot{\pi }\right)1_n=s+t=s$
（5）$\forall i\in \mathcal{I},{\left(\bar{\mathbf{M}}\odot {\bar{\pi }}^{\ast }\right)}_{i,:}{1}_{n+1}={\left(\mathbf{M}\odot \ddot{\pi }\right)}_{i,:}{1}_n+{\mathbf{a}}_i\odot {\bar{\pi }}_{i,n+1}^{\ast }=p_i$，又因为$a_i=0$,因此${\left(\mathbf{M}\odot \ddot{\pi }\right)}_{i,:}{1}_n=p_i$
（6）$\forall j\in \mathcal{J},1_{m+1}^T{\left(\bar{\mathbf{M}}\odot {\bar{\pi }}^{\ast }\right)}_{:,j}=1_m^T{\left(\mathbf{M}\odot \ddot{\pi }\right)}_{:,j}+{\mathbf{b}}_j\odot {\bar{\pi }}_{m+1,j}^{\ast }=q_j$，又因为$b_j=0$,因此$1_m^T{\left(\mathbf{M}\odot \ddot{\pi }\right)}_{:,j}=q_j$

因此$\ddot{\pi }\in {\Pi }^s\left(\mathbf{p},\mathbf{q};\mathbf{M}\right)$

第三步：证明$\mathbf{M}\odot \ddot{\pi }$是最优解

假设有一个更优解$\mathbf{M}\odot \gamma$,其中$\gamma \in {\Pi }^s\left(\mathbf{p},\mathbf{q};\mathbf{M}\right)$,

$$\sum _{i=1}^m\sum _{j=1}^n{M}_{i,j}{\gamma }_{i,j}{G}_{i,j}<\sum _{i=1}^m\sum _{j=1}^n{M}_{i,j}{\ddot{\pi }}_{i,j}{G}_{i,j}$$

构造$\bar{\gamma }$,其中
$i\le m,j\le n$时，${\bar{\gamma }}_{i,j}={\gamma }_{i,j}$
$\forall i\le m$，有${\bar{\gamma }}_{i,n+1}=p_i-\sum _{j=1}^n{\gamma }_{i,j}$
$\forall j\le n$，有${\bar{\gamma }}_{m+1,j}=q_j-\sum _{i=1}^n{\gamma }_{i,j}$
${\bar{\gamma }}_{m+1,n+1}=0$
容易验证，$\bar{\gamma }\in \Pi \left(\bar{\mathbf{p}},\bar{\mathbf{q}};\bar{\mathbf{M}}\right)$
同时

$$\begin{array}{rl}⟨\bar{M}\odot \bar{\gamma },\bar{G}⟩& =\sum _{i=1}^m\sum _{j=1}^n{M}_{i,j}{\gamma }_{i,j}{C}_{i,j}+\xi \left(\Vert \mathbf{p}{\Vert }_1+\Vert \mathbf{q}{\Vert }_1-2s\right)\\ & <\sum _{i=1}^m\sum _{j=1}^n{M}_{i,j}{\ddot{\pi }}_{i,j}{G}_{i,j}+\xi \left(\Vert \mathbf{p}{\Vert }_1+\Vert \mathbf{q}{\Vert }_1-2s\right)=⟨\bar{M}\odot {\bar{\pi }}^{\ast },\bar{G}⟩\end{array}$$

与${\bar{\pi }}^{\ast }$最优矛盾