pytorch 中矩阵乘法总结
导语：这是一篇关于Pytorch中各类乘法操作的总结和使用说明。
torch.dot()：Computes the dot product (inner product) of two tensors.
计算两个1-D 张量的点乘(内乘)。
torch.dot(torch.tensor([2, 3]), torch.tensor([2, 1]))
out: tensor(7)
torch.mm()
, 其中 , ， 输出 的维度是。该函数只用来计算两个2-D矩阵的矩阵乘法。
mat1 = torch.randn(2, 3)
mat2 = torch.randn(3, 3)
torch.mm(mat1, mat2)
out: tensor([[ 0.4851, 0.5037, -0.3633],
[-0.0760, -3.6705, 2.4784]])
torch.bmm()
，其中 ，, 输出 的维度是 。该函数的两个输入必须是3-D矩阵且第一维相同（batch size）。
input = torch.randn(10, 3, 4)
mat2 = torch.randn(10, 4, 5)
res = torch.bmm(input, mat2)
res.size()
out: torch.Size([10, 3, 5])
torch.matmul()
torch.matmul(input,other,out=None)→ Tensor
- input (Tensor) – the first tensor to be multiplied（参与乘法的第一个tensor）
- other (Tensor) – the second tensor to be multiplied（参与乘法的第二个tensor）
- out (Tensor, optional) – the output tensor.
这是两个tensors之间的矩阵乘法，具体怎么操作的，依据输入而定：
- 如果两个矩阵都是一维的，那么该函数的功能与torch.dot()一样，返回两个一维tensor的点乘结果。
tensor1 = torch.randn(4)
tensor2 = torch.randn(4)
torch.matmul(tensor1, tensor2).size()
out: torch.Size([])
- If both arguments are 2-dimensional, the matrix-matrix product is returned.如果两个矩阵都是二维的，那么该函数的功能与torch.mm()一样，返回两个二维矩阵的矩阵乘法。
tensor1 = torch.randn(3,4)
tensor2 = torch.randn(4,5)
torch.matmul(tensor1, tensor2).size()out: torch.Size([3, 5])
- 如果第一个tensor是一维的而第二个tensor是二维的，那么会在第一个tensor的维度上预填充一个1，然后执行二维矩阵乘法，最后将预填充的维度1去掉，
tensor1 = torch.randn(4)
tensor2 = torch.randn(4,5)
torch.matmul(tensor1, tensor2).size()14×45=1*5→5
out: torch.Size([5])
- 如果第一个tensor是二维或者二维以上的，而第二个tensor是一维的，那么将执行矩阵-向量乘法。
tensor1 = torch.randn(4,5)
tensor2 = torch.randn(5)
torch.matmul(tensor1, tensor2).size()
out: torch.Size([4])tensor1 = torch.randn(10, 3, 4)
tensor2 = torch.randn(4)
torch.matmul(tensor1, tensor2).size()
out: torch.Size([10, 3])
- If both arguments are at least 1-dimensional and at least one argument is N-dimensional (where N > 2), then a batched matrix multiply is returned. If the first argument is 1-dimensional, a 1 is prepended to its dimension for the purpose of the batched matrix multiply and removed after. If the second argument is 1-dimensional, a 1 is appended to its dimension for the purpose of the batched matrix multiple and removed after. The non-matrix (i.e. batch) dimensions are broadcasted (and thus must be broadcastable). For example, if
inputis a (j \times 1 \times n \times m)(j×1×n×m) tensor andotheris a (k \times m \times p)(k×m×p) tensor,outwill be an (j \times k \times n \times p)(j×k×n×p) tensor. - 如果两个tensor都至少是一维的，而且至少一个tensor的维度大于2，那么会发生批处理矩阵乘法。
tensor1 = torch.randn(10, 3, 4)
tensor2 = torch.randn(10, 4, 5)
torch.matmul(tensor1, tensor2).size()
out: torch.Size([10, 3, 5])-
- 如果第一个tensor是一维的，则会为它预填充一个1的维度，执行完批处理乘法以后再删除；
tensor1 = torch.randn(6)
tensor2 = torch.randn(2,6,3)
print(torch.matmul(tensor1, tensor2).size())out: torch.Size([2, 3])
-
- 如果第二个tensor是一维的，则会为它预填充一个1的维度，执行完批处理乘法以后再删除；
tensor1 = torch.randn(10, 3, 4)
tensor2 = torch.randn(4)
torch.matmul(tensor1, tensor2).size()
out: torch.Size([10, 3])tensor1 = torch.randn(10, 3, 4)
tensor2 = torch.randn(4,2)
print(torch.matmul(tensor1, tensor2).size())
out: torch.Size([10, 3, 2])
-
- 非矩阵维度（例如batch维度，我理解的矩阵维度应该是维度的最后两个，比如（a,b,c,d）中的c和d）会广播，例如输入是（j,1,n,m）和（k,m,p）,那么输出为（j,k,n,p）
tensor1 = torch.randn(2,5,3)
tensor2 = torch.randn(1,3,4)
print(torch.matmul(tensor1, tensor2).size())
out: torch.Size([2, 5, 4])tensor1 = torch.randn(2,1,3,4)
tensor2 = torch.randn(5,4,2)
print(torch.matmul(tensor1, tensor2).size())
out: torch.Size([2, 5, 3, 2])
torch.mul()
torch.mul(a, b)是矩阵a和b对应位相乘，a和b的维度必须相等，所以只要保证a和b的shape是broadcastable就可以。
a = torch.randn(3)
print(a)
print(torch.mul(a, 100))
out: tensor([0.1287, 0.7827, 0.2674])
tensor([12.8663, 78.2685, 26.7358])
a = torch.randn(4, 1)
print(a)
b = torch.randn(1, 4)
print(b)
print(torch.mul(a, b))
out:
tensor([[ 0.6419],
[ 2.2872],
[-1.1190],
[-0.5565]])
tensor([[ 0.5292, 1.0285, -2.9172, 0.8998]])
tensor([[ 0.3397, 0.6601, -1.8724, 0.5775],
[ 1.2104, 2.3524, -6.6723, 2.0581],
[-0.5922, -1.1508, 3.2643, -1.0069],
[-0.2945, -0.5724, 1.6235, -0.5008]])
torch.einsum()
torch.einsum()：爱因斯坦求和约定是一个非常神奇的函数，号称可以满足你的一切需求！功能强大，具体的学习内容可以参考：
论智：einsum满足你一切需要：深度学习中的爱因斯坦求和约定
下面列举一些具体的例子[1]
- 矩阵转置
a=torch.arange(6).reshape(2,3)
>>>tensor([[0, 1, 2],
[3, 4, 5]])
torch.einsum(‘ij->ji’,[a])
>>>tensor([[0, 3],
[1, 4],
[2, 5]])- 求和
a=torch.arange(6).reshape(2,3)
>>>tensor([[0, 1, 2],
[3, 4, 5]])
torch.einsum(‘ij->’,[a])
>>>tensor(15)- 列求和(行维度不变，列维度消失)
a=torch.arange(6).reshape(2,3)
>>>tensor([[0, 1, 2],
[3, 4, 5]])
torch.einsum(‘ij->j’,[a])
>>>tensor([ 3., 5., 7.])- 行求和(列维度不变，行维度消失)
a=torch.arange(6).reshape(2,3)
>>>tensor([[0, 1, 2],
[3, 4, 5]])
torch.einsum(‘ij->i’, [a])
>>>tensor([ 3., 12.])- 矩阵-向量相乘
a=torch.arange(6).reshape(2,3)
b=a[0]
>>>tensor([[0, 1, 2],
[3, 4, 5]])
torch.einsum(‘ik,k->i’,[a,b])
>>>tensor([ 5., 14.])- 矩阵-矩阵乘法
a=torch.arange(6).reshape(2,3)
b=torch.arange(15).reshape(3,5)
>>>tensor([[0, 1, 2],
[3, 4, 5]])
>>>tensor([[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14]])
torch.einsum(‘ik,kj->ij’,[a,b])
>>>tensor([[ 25, 28, 31, 34, 37],
[ 70, 82, 94, 106, 118]])
- 点积
a=torch.arange(6).reshape(2,3)
b=torch.arange(6,12).reshape(2,3)
>>>tensor([[0, 1, 2],
[3, 4, 5]])>>>tensor([[ 6, 7, 8],
[ 9, 10, 11]])
torch.einsum(‘ij,ij->’,[a,b])
>>>tensor(145.)
- 外积
a=torch.arange(3)
b=torch.arange(3,7)
>>>tensor([0, 1, 2])
>>>tensor([3, 4, 5, 6])
torch.einsum(‘i,j->ij’,[a,b])
>>>tensor([[ 0., 0., 0., 0.],
[ 3., 4., 5., 6.],
[ 6., 8., 10., 12.]])- batch矩阵乘
a=torch.randn(3,2,5)
b=torch.randn(3,5,3)
>>>tensor([[[-1.4131e+00, 3.8372e-02, 1.2436e+00, 5.4757e-01, 2.9478e-01],
[ 1.3314e+00, 4.4003e-01, 2.3410e-01, -5.3948e-01, -9.9714e-01]], [[-4.6552e-01, 5.4318e-01, 2.1284e+00, 9.5029e-01, -8.2193e-01],
[ 7.0617e-01, 9.8252e-01, -1.4406e+00, 1.0071e+00, 5.9477e-01]],
[[-1.0482e+00, 4.7110e-02, 1.0014e+00, -6.0593e-01, -3.2076e-01],
[ 6.6210e-01, 3.7603e-01, 1.0198e+00, 4.6591e-01, -7.0637e-04]]])
>>>tensor([[[-2.1797e-01, 3.1329e-04, 4.3139e-01],
[-1.0621e+00, -6.0904e-01, -4.6225e-01],
[ 8.5050e-01, -5.8867e-01, 4.8824e-01],
[ 2.8561e-01, 2.6806e-01, 2.0534e+00],
[-5.5719e-01, -3.3391e-01, 8.4069e-03]],
[[ 5.2877e-01, 1.4361e+00, -6.4232e-01],
[ 1.0813e+00, 8.5241e-01, -1.1759e+00],
[ 4.9389e-01, -1.7523e-01, -9.5224e-01],
[-1.3484e+00, -5.4685e-01, 8.5539e-01],
[ 3.7036e-01, 3.4368e-01, -4.9617e-01]],
[[-2.1564e+00, 3.0861e-01, 3.4261e-01],
[-2.3679e+00, -2.5035e-01, 1.8104e-02],
[ 1.1075e+00, 7.2465e-01, -2.0981e-01],
[-6.5387e-01, -1.3914e-01, 1.5205e+00],
[-1.6561e+00, -3.5294e-01, 1.9589e+00]]])
torch.einsum(‘ijk,ikl->ijl’,[a,b])
>>>tensor([[[ 1.3170, -0.7075, 1.1067],
[-0.1569, -0.2170, -0.6309]],
[[-0.1935, -1.3806, -1.1458],
[-0.4135, 1.7577, 0.3293]],
[[ 4.1854, 0.5879, -2.1180],
[-1.4922, 0.7846, 0.7267]]])- 多张量计算
A = torch.randn(3,5,4)
l = torch.randn(2,5)
r = torch.randn(2,4)
torch.einsum(‘bn,anm,bm->ba’, l, A, r) # compare torch.nn.functional.bilinear
tensor([[-0.3430, -5.2405, 0.4494],
[ 0.3311, 5.5201, -3.0356]])
参考
- ^einsum：爱因斯坦求和约定 https://www.cnblogs.com/mengnan/p/10319701.html