数值计算笔记-线性方程组迭代解法
三种经典迭代法python例子
1. 矩阵极限： { A ( k ) } \{A^{(k)}\} {A(k)}为矩阵的序列，且 A = ( a i j ) ∈ R n × n A = (a_{ij}) \in R^{n \times n} A=(aij)∈Rn×n，若有
lim k → ∞ a i j ( k ) = a i j , ( i , j = 1 , ⋯ , n ) \lim_{k \rightarrow \infty} a_{ij}^{(k)} = a_{ij}, (i, j = 1, \cdots, n) k→∞limaij(k)=aij,(i,j=1,⋯,n)
则 { A ( k ) } \{A^{(k)}\} {A(k)}收敛于A，记为 lim k → ∞ A ( k ) = A \lim_{k \rightarrow \infty} A^{(k)} = A limk→∞A(k)=A
算子范数等价性： ∃ c 1 , c 2 > 0 \exist c_1, c_2 > 0 ∃c1,c2>0对 ∀ A ∈ R n × n \forall A \in R^{n \times n} ∀A∈Rn×n，和任意算子范数 ∣ ∣ ⋅ ∣ ∣ i ||\cdot||_i ∣∣⋅∣∣i，有
c 1 ∣ ∣ A ∣ ∣ ∞ ≤ ∣ ∣ A ∣ ∣ i ≤ c 2 ∣ ∣ A ∣ ∣ ∞ c_1 ||A||_\infty \leq ||A||_i \leq c_2 ||A||_\infty c1∣∣A∣∣∞≤∣∣A∣∣i≤c2∣∣A∣∣∞
范数与极限：其中 ∣ ∣ ⋅ ∣ ∣ ||\cdot|| ∣∣⋅∣∣表示任意一种算子范数
lim k → ∞ A ( k ) = A ⇔ lim k → ∞ ∣ ∣ A ( k ) − A ∣ ∣ = 0 \lim_{k \rightarrow \infty} A^{(k)} = A \Leftrightarrow \lim_{k \rightarrow \infty} ||A^{(k)} - A|| = 0 k→∞limA(k)=A⇔k→∞lim∣∣A(k)−A∣∣=0
极限与线性方程： lim k → ∞ A ( k ) = A \lim_{k \rightarrow \infty} A^{(k)} = A limk→∞A(k)=A充要条件： ∀ x ⃗ \forall \vec{x} ∀x有 lim k → ∞ A ( k ) x ⃗ = A x ⃗ \lim_{k \rightarrow \infty} A^{(k)} \vec{x} = A \vec{x} limk→∞A(k)x=Ax
2. 定义谱半径（ = 最大特征值）： A ∈ R n × n A\in R^{n \times n} A∈Rn×n特征值为 λ i , ( i = 1 , ⋯ , n ) \lambda_i, (i = 1, \cdots, n) λi,(i=1,⋯,n)，则 ρ ( A ) = max 1 < i ≤ n { λ i } \rho(A) = \max_{1 < i \leq n} \{\lambda_i\} ρ(A)=max1<i≤n{λi}成为谱半径
谱半径与算子范数：
（1） ρ ( A ) ≤ ∣ ∣ A ∣ ∣ \rho(A) \leq ||A|| ρ(A)≤∣∣A∣∣，即谱半径不大于任意算子范数
（2） ∣ ∣ A ∣ ∣ 2 = ρ ( A ) ||A||_2 = \rho(A) ∣∣A∣∣2=ρ(A)
谱半径与极限： B = ( b i j ) ∈ R n × n B = (b_{ij}) \in R^{n \times n} B=(bij)∈Rn×n，则 lim k → ∞ B n × n = 0 ⇔ ρ ( B ) < 1 \lim_{k \rightarrow \infty} B^{n \times n} = 0 \Leftrightarrow \rho(B) < 1 limk→∞Bn×n=0⇔ρ(B)<1
3. 迭代法基本形式：原方程 A x ⃗ = b ⃗ A\vec{x} = \vec{b} Ax=b，迭代形式 x ⃗ ( k + 1 ) = B x ⃗ ( k ) + f ⃗ \vec{x}^{(k + 1)} = B \vec{x}^{(k)} + \vec{f} x(k+1)=Bx(k)+f
算法：1阶定长迭代法
输入：初值 x ⃗ 0 \vec{x}_0 x0，B， f ⃗ \vec{f} f；输出： x ⃗ \vec{x} x
x ⃗ = x ⃗ 0 w h i l e ( ) x ⃗ = B x ⃗ + f ⃗ E n d \begin{aligned} &\vec{x} = \vec{x}_0\\ &while ()\\ & \quad\vec{x} = B\vec{x} + \vec{f}\\ &End \end{aligned} x=x0while()x=Bx+fEnd
4. 迭代法全局收敛充要条件：若迭代法 x ⃗ k + 1 = B x ⃗ k + f ⃗ \vec{x}_{k + 1} = B \vec{x}_k + \vec{f} xk+1=Bxk+f，中I - B非奇异。对 ∀ x ⃗ 0 \forall \vec{x}_0 ∀x0迭代法得到的序列 { x ⃗ k } \{\vec{x}_k\} {xk}收敛 ⇔ ρ ( B ) < 1 \Leftrightarrow \ \rho(B) < 1 ⇔ ρ(B)<1。在这种条件下 { x ⃗ k } \{\vec{x}_k\} {xk}的极限 x ⃗ ∗ \vec{x}^* x∗为 x ⃗ = B x ⃗ + f ⃗ \vec{x} = B \vec{x} + \vec{f} x=Bx+f唯一解
收敛条件：迭代形式 x ⃗ k + 1 = B x ⃗ k + f ⃗ \vec{x}_{k + 1} = B \vec{x}_k + \vec{f} xk+1=Bxk+f，若 ∀ ∣ ∣ B ∣ ∣ = q < 1 \forall ||B|| = q < 1 ∀∣∣B∣∣=q<1，则
（1）全局收敛
（2） ∣ ∣ x ⃗ k − x ⃗ ∗ ∣ ∣ ≤ q k ∣ ∣ x ⃗ 0 − x ⃗ ∗ ∣ ∣ ||\vec{x}_k - \vec{x}^*|| \leq q^k ||\vec{x}_0 - \vec{x}^*|| ∣∣xk−x∗∣∣≤qk∣∣x0−x∗∣∣
（3） ∣ ∣ x ⃗ k − x ⃗ ∗ ∣ ∣ ≤ q 1 − q ∣ ∣ x ⃗ k − x ⃗ k + 1 ∣ ∣ ||\vec{x}_k - \vec{x}^*|| \leq \frac{q}{1 - q} ||\vec{x}_k - \vec{x}_{k + 1}|| ∣∣xk−x∗∣∣≤1−qq∣∣xk−xk+1∣∣
（4） ∣ ∣ x ⃗ k − x ⃗ ∗ ∣ ∣ ≤ q k 1 − q ∣ ∣ x ⃗ 1 − x ⃗ 0 ∣ ∣ ||\vec{x}_k - \vec{x}^*|| \leq \frac{q^k}{1 - q} ||\vec{x}_1 - \vec{x}_0|| ∣∣xk−x∗∣∣≤1−qqk∣∣x1−x0∣∣
定义收敛阶：解序列为 x ⃗ k , x ⃗ k ∈ R n \vec{x}_k, \vec{x}_k \in R^n xk,xk∈Rn，收敛到向量 x ⃗ ∗ \vec{x}^* x∗，若
lim k → ∞ e ⃗ k + 1 e ⃗ k = c ≠ 0 e ⃗ k = x ⃗ k − x ⃗ ∗ \lim_{k \rightarrow \infty} \frac{\vec{e}_{k + 1}}{\vec{e}_k} = c \neq 0\\ \vec{e}_k = \vec{x}_k - \vec{x}^* k→∞limekek+1=c=0ek=xk−x∗
则迭代过程是p阶收敛的。1阶定常迭代法收敛速度： lim k → ∞ ∣ ∣ e ⃗ k + 1 ∣ ∣ ∣ ∣ e ⃗ k ∣ ∣ = ρ ( B ) \lim_{k \rightarrow \infty} \frac{||\vec{e}_{k + 1}||}{||\vec{e}_k||} = \rho(B) limk→∞∣∣ek∣∣∣∣ek+1∣∣=ρ(B)
定义收敛速度： R = − l o g 10 ρ ( B ) R = - log_{10} \rho(B) R=−log10ρ(B)
5. 雅可比迭代法
思路：第k次迭代得到的n维向量 x ⃗ k = ( x 1 ( k ) , ⋯ , x n ( k ) ) \vec{x}_k = (x_1^{(k)}, \cdots, x_n^{(k)}) xk=(x1(k),⋯,xn(k))。以 3 × 3 3 \times 3 3×3维矩阵A为例
{ a 11 x 1 + a 12 x 2 + a 13 x 3 = b 1 a 21 x 1 + a 22 x 2 + a 23 x 3 = b 2 a 31 x 1 + a 32 x 2 + a 33 x 3 = b 3 ⇒ { x 1 = − 1 a 11 ( a 12 x 2 + a 13 x 3 ) + b 1 a 11 x 2 = − 1 a 22 ( a 21 x 1 + a 13 x 2 ) + b 2 a 22 x 3 = − 1 a 33 ( a 31 x 1 + a 32 x 2 ) + b 3 a 33 \begin{cases} a_{11} x_1 + a_{12}x_2 + a_{13}x_3 = b_1\\ a_{21} x_1 + a_{22}x_2 + a_{23}x_3 = b_2\\ a_{31} x_1 + a_{32}x_2 + a_{33}x_3 = b_3 \end{cases} \Rightarrow \begin{cases} x_1^{} = -\frac{1}{a_{11}}(\qquad a_{12}x_2 + a_{13}x_3) + \frac{b_1}{a_{11}}\\ x_2 = -\frac{1}{a_{22}}(a_{21}x_1 \qquad + a_{13} x_2) + \frac{b_2}{a_{22}}\\ x_3 = -\frac{1}{a_{33}}(a_{31}x_1 + a_{32}x_2 \qquad) + \frac{b_3}{a_{33}} \end{cases} ⎩⎪⎨⎪⎧a11x1+a12x2+a13x3=b1a21x1+a22x2+a23x3=b2a31x1+a32x2+a33x3=b3⇒⎩⎪⎨⎪⎧x1=−a111(a12x2+a13x3)+a11b1x2=−a221(a21x1+a13x2)+a22b2x3=−a331(a31x1+a32x2)+a33b3
迭代公式：
{ A x ⃗ = b ⃗ A = D − L − U ⇒ x ⃗ k + 1 = D − 1 ( L + U ) x ⃗ k + D − 1 b ⃗ ⇒ { x ⃗ k + 1 = B x ⃗ k + f ⃗ B = D − 1 ( L + U ) f ⃗ = D − 1 b ⃗ \begin{aligned} &\begin{cases} A\vec{x} = \vec{b}\\ A = D - L - U \end{cases} \\ & \Rightarrow \vec{x}_{k + 1} = D^{-1}(L + U)\vec{x}_k + D^{-1}\vec{b}\\ & \Rightarrow \begin{cases} \vec{x}_{k + 1} = B\vec{x}_k + \vec{f}\\ B = D^{-1}(L + U)\\ \vec{f} = D^{-1}\vec{b} \end{cases} \end{aligned} {Ax=bA=D−L−U⇒xk+1=D−1(L+U)xk+D−1b⇒⎩⎪⎨⎪⎧xk+1=Bxk+fB=D−1(L+U)f=D−1b
其中D为A的对角线元素组成的对角矩阵；L、U为A下三角、上三角元素组成的元素，其对角元素为0
算法：输入：初值 x ⃗ \vec{x} x，A， b ⃗ \vec{b} b；输出： x ⃗ \vec{x} x
w h i l e ( 不 满 足 判 停 条 件 ) y ⃗ = x ⃗ f o r i = 1 → n x i = ( b i − ∑ j = 1 i − 1 a i j y j − ∑ j = i + 1 n a i j y j ) / a i i E n d E n d \begin{aligned} &while (不满足判停条件)\\ & \quad \vec{y} = \vec{x}\\ & \quad \quad for \ i = 1 \rightarrow n\\ & \quad \quad \quad x_i = (b_i - \sum_{j = 1}^{i - 1}{a_{ij} y_j} - \sum_{j = i + 1}^{n} a_{ij} y_{j}) / a_{ii}\\ & \quad \quad End\\ & End \end{aligned} while(不满足判停条件)y=xfor i=1→nxi=(bi−j=1∑i−1aijyj−j=i+1∑naijyj)/aiiEndEnd
6.高斯-塞尔德迭代法（G-S迭代法）
思路：
{ x 1 ( k + 1 ) = − 1 a 11 ( + a 12 x 2 ( k ) + a 13 x 3 ( k ) ) + b 1 a 11 x 2 ( k + 1 ) = − 1 a 22 ( a 21 x 1 ( k + 1 ) + a 23 x 3 ( k ) ) + b 2 a 22 x 3 ( k + 1 ) = − 1 a 33 ( a 31 x 1 ( k + 1 ) + a 32 x 2 ( k + 1 ) ) + b 3 a 3 3 \begin{cases} x_1^{(k+1)} = -\frac{1}{a_{11}}(\qquad + a_{12}x_2^{(k)} + a_{13}x_3^{(k)}) + \frac{b_1}{a_{11}}\\ x_2^{(k + 1)} = -\frac{1}{a_{22}}(a_{21} x_1^{(k + 1)} \qquad + a_{23}x_3^{(k)}) + \frac{b_2}{a_{22}}\\ x_3^{(k + 1)} = -\frac{1}{a_{33}}(a_{31}x_1^{(k + 1)} + a_{32}x_2^{(k + 1)} \qquad) + \frac{b_3}{a_33} \end{cases} ⎩⎪⎨⎪⎧x1(k+1)=−a111(+a12x2(k)+a13x3(k))+a11b1x2(k+1)=−a221(a21x1(k+1)+a23x3(k))+a22b2x3(k+1)=−a331(a31x1(k+1)+a32x2(k+1))+a33b3
迭代公式
{ A x ⃗ = b ⃗ A = D − L − U ⇒ x ⃗ k + 1 = D − 1 ( L x ⃗ k + 1 + U x ⃗ k ) + D − 1 b ⃗ ⇒ x ⃗ k + 1 = ( D − L ) − 1 U x ⃗ k + ( D − L ) − 1 b ⃗ ⇒ { x ⃗ k + 1 = B x ⃗ k + f ⃗ B = ( D − L ) − 1 U f ⃗ = ( D − L ) − 1 b ⃗ \begin{aligned} &\begin{cases} A\vec{x} = \vec{b}\\ A = D - L - U \end{cases} \\ & \Rightarrow \vec{x}_{k + 1} = D^{-1}(L\vec{x}_{k + 1} + U\vec{x}_k) + D^{-1}\vec{b}\\ & \Rightarrow \vec{x}_{k + 1} = (D - L)^{-1}U\vec{x}_k + (D - L)^{-1}\vec{b}\\ & \Rightarrow \begin{cases} \vec{x}_{k + 1} = B\vec{x}_k + \vec{f}\\ B = (D - L)^{-1}U\\ \vec{f} = (D - L)^{-1}\vec{b} \end{cases} \end{aligned} {Ax=bA=D−L−U⇒xk+1=D−1(Lxk+1+Uxk)+D−1b⇒xk+1=(D−L)−1Uxk+(D−L)−1b⇒⎩⎪⎨⎪⎧xk+1=Bxk+fB=(D−L)−1Uf=(D−L)−1b
算法：输入：初值 x ⃗ \vec{x} x，A， b ⃗ \vec{b} b；输出：近似解 x ⃗ \vec{x} x
w h i l e ( 不 满 足 判 停 条 件 ) f o r i = 1 → n x i = ( b i − ∑ j = 1 i − 1 a i j x j − ∑ j = i + 1 n a i j x j ) / a i i E n d E n d \begin{aligned} &while (不满足判停条件)\\ & \quad \quad for \ i = 1 \rightarrow n\\ & \quad \quad \quad x_i = (b_i - \sum_{j = 1}^{i - 1}{a_{ij} x_j} - \sum_{j = i + 1}^{n} a_{ij} x_{j}) / a_{ii}\\ & \quad \quad End\\ & End \end{aligned} while(不满足判停条件)for i=1→nxi=(bi−j=1∑i−1aijxj−j=i+1∑naijxj)/aiiEndEnd
7. 逐次超松弛迭代法（SOR迭代法）
思路：由G-S迭代法得到 x ⃗ k + 1 ′ \vec{x}_{k + 1}' xk+1′。则SOR迭代得到的向量为 x ⃗ k + 1 = ( 1 − w ) x ⃗ k + w x ⃗ k + 1 ′ \vec{x}_{k + 1} = (1 - w)\vec{x}_k + w\vec{x}_{k + 1}' xk+1=(1−w)xk+wxk+1′
迭代公式：
{ A x ⃗ = b ⃗ A = D − L − U ⇒ x ⃗ k + 1 = ( 1 − w ) x ⃗ k + w [ D − 1 ( L x ⃗ k + 1 + U x ⃗ k ) ] + D − 1 b ⃗ ⇒ x ⃗ k + 1 = ( D − w L ) − 1 ( ( 1 − w ) D + w U ) x ⃗ k + ( D − s L ) − 1 w b ⃗ ⇒ { x ⃗ k + 1 = B x ⃗ k + f ⃗ B = ( D − w L ) − 1 ( ( 1 − w ) D + w U ) f ⃗ = ( D − s L ) − 1 w b ⃗ \begin{aligned} &\begin{cases} A\vec{x} = \vec{b}\\ A = D - L - U \end{cases} \\ & \Rightarrow \vec{x}_{k + 1} = (1 - w)\vec{x}_k + w[D^{-1}(L\vec{x}_{k + 1} + U\vec{x}_k)] + D^{-1}\vec{b}\\ & \Rightarrow \vec{x}_{k + 1} = (D - wL)^{-1}((1-w)D + wU)\vec{x}_k + (D - sL)^{-1}w\vec{b}\\ & \Rightarrow \begin{cases} \vec{x}_{k + 1} = B\vec{x}_k + \vec{f}\\ B = (D - wL)^{-1}((1-w)D + wU)\\ \vec{f} = (D - sL)^{-1}w\vec{b} \end{cases} \end{aligned} {Ax=bA=D−L−U⇒xk+1=(1−w)xk+w[D−1(Lxk+1+Uxk)]+D−1b⇒xk+1=(D−wL)−1((1−w)D+wU)xk+(D−sL)−1wb⇒⎩⎪⎨⎪⎧xk+1=Bxk+fB=(D−wL)−1((1−w)D+wU)f=(D−sL)−1wb
算法：输入：初值 x ⃗ \vec{x} x，A， b ⃗ \vec{b} b，参数w；输出：近似解 x ⃗ \vec{x} x
w h i l e ( 不 满 足 判 停 条 件 ) f o r i = 1 → n x i = ( 1 − w ) x i + w ( b i − ∑ j = 1 i − 1 a i j x j − ∑ j = i + 1 n a i j x j ) / a i i E n d E n d \begin{aligned} &while (不满足判停条件)\\ & \quad \quad for \ i = 1 \rightarrow n\\ & \quad \quad \quad x_i = (1 - w)x_i + w(b_i - \sum_{j = 1}^{i - 1}{a_{ij} x_j} - \sum_{j = i + 1}^{n} a_{ij} x_{j}) / a_{ii}\\ & \quad \quad End\\ & End \end{aligned} while(不满足判停条件)for i=1→nxi=(1−w)xi+w(bi−j=1∑i−1aijxj−j=i+1∑naijxj)/aiiEndEnd
8. 雅可比迭代收敛的充要条件：已知对角元素 a i i > 0 ( i = 1 , ⋯ , n ) a_{ii} > 0 (i = 1, \cdots, n) aii>0(i=1,⋯,n)。 A x ⃗ = b ⃗ A\vec{x} = \vec{b} Ax=b迭代法收敛 ⇔ \Leftrightarrow ⇔A和2D-A正定，其中D为A对角线元素组成的矩阵
推论：设B为雅可比迭代中矩阵。若 ∣ ∣ B ∣ ∣ ∞ < 1 ||B||_\infty < 1 ∣∣B∣∣∞<1或者 ∣ ∣ B ∣ ∣ 1 < 1 ||B||_1 < 1 ∣∣B∣∣1<1，则G-S迭代收敛
9. 定义可约矩阵：设 A = ( a i j ) ∈ R n × n A = (a_{ij}) \in R^{n \times n} A=(aij)∈Rn×n，若有排列矩阵P（一系列初等矩阵的乘积）使
P T A P = [ A 11 A 12 O A 22 ] P^TAP = \left[ \begin{matrix} A_{11}& A_{12}\\ O& A_{22} \end{matrix} \right] PTAP=[A11OA12A22]
其中 A 11 , A 22 A_{11}, A_{22} A11,A22为阶数不等于0的方阵，则A为可约矩阵
对角占优性质：若A为严格对角占优矩阵，或不可约的弱对角占优矩阵，则A非奇异
10. G-S/SOR迭代法收敛条件
（1）若矩阵A严格对角占优，或不可约对角占优，则雅可比迭代法和G-S迭代法收敛
（2）条件同（1），松弛因子 0 < w ≤ 1 0 < w \leq 1 0<w≤1的SOR迭代法收敛
正定矩阵收敛性质
（1）若A对称正定，则G-S迭代法和 0 < w < 2 0 < w < 2 0<w<2的SOR迭代法收敛
（2）若SOR迭代法收敛，则 w ∈ ( 0 , 2 ) w \in (0, 2) w∈(0,2)
11. 最速下降法（= 梯度下降法）
推导：
（1）用变分原理将 A x ⃗ = b ⃗ A\vec{x} = \vec{b} Ax=b变为求 ϕ ( x ⃗ ) = 1 2 x ⃗ T A x ⃗ − b ⃗ T x ⃗ \phi(\vec{x}) = \frac{1}{2} \vec{x}^TA\vec{x} - \vec{b}^T\vec{x} ϕ(x)=21xTAx−bTx最小值点
（2）迭代方式： x ⃗ k + 1 = x ⃗ k + α k p ⃗ k \vec{x}_{k + 1} = \vec{x}_k + \alpha_k \vec{p}_k xk+1=xk+αkpk，其中 α k \alpha_k αk搜索步长， p ⃗ k \vec{p}_k pk搜索方向
（3）确定方向：下降速度最开的方向为负梯度方向，即 − ∇ ϕ ( x ⃗ ) = − ( ∂ ϕ ∂ x 1 , ⋯ , ∂ ϕ ∂ x n ) T -\nabla \phi(\vec{x}) = -(\frac{\partial \phi}{\partial x_1}, \cdots, \frac{\partial \phi}{\partial x_n})^T −∇ϕ(x)=−(∂x1∂ϕ,⋯,∂xn∂ϕ)T
（4）确定步长
α k = a r g min α 1 2 ( x ⃗ k + α p ⃗ k ) T A ( x ⃗ k + α p ⃗ k ) − b ⃗ T ( x ⃗ k + α p ⃗ k ) = a r g min α 1 2 α 2 p ⃗ k T A p ⃗ k − α r ⃗ k p ⃗ k + ϕ ( x ⃗ k ) ( r ⃗ k = b ⃗ − A x ⃗ ) ⇒ α k = r ⃗ k T p ⃗ k p ⃗ k A p ⃗ k \begin{aligned} \alpha_k &= arg \min_\alpha \frac{1}{2} (\vec{x}_k + \alpha \vec{p}_k)^T A (\vec{x}_k + \alpha \vec{p}_k) - \vec{b}^T (\vec{x}_k + \alpha \vec{p}_k)\\ &= arg \min_\alpha \frac{1}{2} \alpha^2 \vec{p}_k^TA\vec{p}_k - \alpha\vec{r}_k\vec{p}_k + \phi(\vec{x}_k) \qquad (\vec{r}_k = \vec{b} - A\vec{x})\\ &\Rightarrow \alpha_k = \frac{\vec{r}_k^T \vec{p}_k}{\vec{p}_k A\vec{p}_k} \end{aligned} αk=argαmin21(xk+αpk)TA(xk+αpk)−bT(xk+αpk)=argαmin21α2pkTApk−αrkpk+ϕ(xk)(rk=b−Ax)⇒αk=pkApkrkTpk
算法：输入：初值 x ⃗ \vec{x} x，A， b ⃗ \vec{b} b；输出：近似值解 x ⃗ \vec{x} x
r ⃗ = b ⃗ − A x ⃗ w h i l e 不 满 足 判 停 条 件 α = r ⃗ T r ⃗ / ( r ⃗ T A r ⃗ ) x ⃗ = x ⃗ + α r ⃗ r ⃗ = r ⃗ − α A r ⃗ E n d \begin{aligned} &\vec{r} = \vec{b} - A\vec{x}\\ &while 不满足判停条件\\ &\quad \alpha = \vec{r}^T\vec{r} / (\vec{r}^T A\vec{r})\\ &\quad \vec{x} = \vec{x} + \alpha \vec{r}\\ &\quad \vec{r} = \vec{r} - \alpha A\vec{r}\\ &End \end{aligned} r=b−Axwhile不满足判停条件α=rTr/(rTAr)x=x+αrr=r−αArEnd
12. 共轭梯度法
思路：第一步与最速下降法一样，之后每次迭代的搜索方向由2个向量线性组合生成，即 x ⃗ k + 1 = x ⃗ k + α ( ξ r ⃗ k + η p ⃗ k − 1 ) \vec{x}_{k + 1} = \vec{x}_k + \alpha (\xi \vec{r}_k + \eta \vec{p}_{k - 1}) xk+1=xk+α(ξrk+ηpk−1)
推导搜索方向：设 f ( ξ , η ) = ϕ ( x ⃗ k + ξ r ⃗ k + η p ⃗ k − 1 ) f(\xi, \eta) = \phi(\vec{x}_k + \xi \vec{r}_k + \eta \vec{p}_{k - 1}) f(ξ,η)=ϕ(xk+ξrk+ηpk−1)
令 ∂ f ∂ ξ = ∂ f ∂ η = 0 \frac{\partial f}{\partial \xi} = \frac{\partial f}{\partial \eta} = 0 ∂ξ∂f=∂η∂f=0，得到
{ ξ r ⃗ k T A x ⃗ r + η r ⃗ k T A p ⃗ k − 1 = r ⃗ T r ⃗ ξ r ⃗ k T A p ⃗ k − 1 + η p ⃗ k − 1 T A p ⃗ k − 1 = 0 ⇒ p ⃗ k = 1 ξ ( x ⃗ k + 1 − x ⃗ k ) = r ⃗ k + η ξ p ⃗ k − 1 = r ⃗ k + β p ⃗ k − 1 β k − 1 = η ξ = r ⃗ k T A p ⃗ k − 1 p ⃗ k − 1 T A p ⃗ k − 1 \begin{cases} \xi \vec{r}_k^T A \vec{x}_r + \eta \vec{r}_k^TA \vec{p}_{k - 1} = \vec{r}^T\vec{r}\\ \xi \vec{r}_k^T A \vec{p}_{k - 1} + \eta \vec{p}_{k - 1}^T A \vec{p}_{k - 1} = 0 \end{cases}\\ \Rightarrow \vec{p}_k = \frac{1}{\xi} (\vec{x}_{k + 1} - \vec{x}_k) = \vec{r}_k + \frac{\eta}{\xi} \vec{p}_{k - 1} = \vec{r}_{k} + \beta \vec{p}_{k - 1}\\ \beta_{k - 1} = \frac{\eta}{\xi} = \frac{\vec{r}^T_k A \vec{p}_{k - 1}}{\vec{p}_{k - 1}^T A \vec{p}_{k - 1}} {ξrkTAxr+ηrkTApk−1=rTrξrkTApk−1+ηpk−1TApk−1=0⇒pk=ξ1(xk+1−xk)=rk+ξηpk−1=rk+βpk−1βk−1=ξη=pk−1TApk−1rkTApk−1
步长推导不变，最终得到 x ⃗ k + 1 = x ⃗ k + α k p ⃗ k \vec{x}_{k + 1} = \vec{x}_k + \alpha_k \vec{p}_k xk+1=xk+αkpk
算法：输入：初值 x ⃗ 0 \vec{x}_0 x0，A， b ⃗ \vec{b} b；输出： x ⃗ k \vec{x}_k xk
r ⃗ 0 = b ⃗ − A x ⃗ 0 p ⃗ 0 = r ⃗ 0 k = 0 w h i l e 不 满 足 判 停 条 件 α k = r ⃗ k T p ⃗ k / ( p ⃗ k T A r ⃗ k ) x ⃗ k + 1 = x ⃗ k + α k p ⃗ k r ⃗ k + 1 = r ⃗ k − α k A p ⃗ k / / 下 面 计 算 下 一 次 的 步 长 和 方 向 β k = − r ⃗ k + 1 T A p ⃗ k / ( p ⃗ k T A p ⃗ k ) p ⃗ k + 1 = r ⃗ k + 1 + β k p ⃗ k k = k + 1 E n d \begin{aligned} &\vec{r}_0 = \vec{b} - A\vec{x}_0\\ &\vec{p}_0 = \vec{r}_0\\ &k = 0\\ &while 不满足判停条件\\ &\quad \alpha_k = \vec{r}^T_k\vec{p}_k / (\vec{p}^T_k A \vec{r}_k)\\ &\quad \vec{x}_{k + 1} = \vec{x}_k + \alpha_k \vec{p}_k\\ &\quad \vec{r}_{k + 1} = \vec{r}_k - \alpha_k A\vec{p}_k \qquad // 下面计算下一次的步长和方向\\ &\quad \beta_k = -\vec{r}_{k + 1}^T A \vec{p}_k / (\vec{p}_k^T A \vec{p}_k) \\ &\quad \vec{p}_{k + 1} = \vec{r}_{k + 1} + \beta_k \vec{p}_k\\ &\quad k = k + 1\\ &End \end{aligned} r0=b−Ax0p0=r0k=0while不满足判停条件αk=rkTpk/(pkTArk)xk+1=xk+αkpkrk+1=rk−αkApk//下面计算下一次的步长和方向βk=−rk+1TApk/(pkTApk)pk+1=rk+1+βkpkk=k+1End