在上一篇中我们介绍了 mpi4py 中的规约发散操作方法，下面我们将介绍全发散操作。
对组内通信子上的全发散操作，组内所有进程都会执行散发操作，并从所有进程接收数据。
对组间通信子上的全发散操作，假定相关联的组为 group A 和 group B，group A 中的所有进程都会向 group B的各个进程散发消息，同时 group B 的各个进程也会向 group A 的各个进程散发消息。要求 group A 中进程 i 发送缓冲区的第 j 个位置，必须与 group B 中进程 j 接收缓冲区的第 i 个位置匹配；反之亦然。从 group A 到 group B 两个方向上传输的数据可以不对称，甚至在一个方向上传输的数据可以为 0。
方法接口
mpi4py 中的全发散操作的方法（MPI.Comm 类的方法）接口为：
alltoall(self, sendobj)
Alltoall(self, sendbuf, recvbuf)
Alltoallv(self, sendbuf, recvbuf)
Alltoallw(self, sendbuf, recvbuf)
以小写字母开头的 alltoall 可以全发散一系列可被 pickle 系列化的 Python 对象 sendobj。注意：对组内通信子上的该操作，系列长度是组内的进程数 size，而对组间通信子上的该操作，系列长度应是另一组的进程数目，以保证散发的消息可被顺利接收。
以大写字母开头的 Alltoall 只能全发散具有单段缓冲区接口的类数组对象，如 numpy 数组。参数 sendbuf/recvbuf 可以是一个长度为2或3的 list 或 tuple，类似于 [data, MPI.DOUBLE]，或者 [data, count, MPI.DOUBLE]，以指明发送/接收数据缓冲区，数据计数以及数据类型。当 count 省略时会利用 data 的字节长度和数据类型计算出对应的 count。对 numpy 数组，其计数和数据类型可以自动推断出来，因此可以直接以 data 作为参数传给 sendbuf/recvbuf。
Alltoall 只能散发长度相同的数据量，而 Alltoallv 则可散发不同长度的数据量，为此需要设置发散缓冲区的偏移索引和发送数量数组，以及接收缓冲区的偏移索引和接收数量数组。具体来说，sendbuf/recvbuf 要设置成类似于 [data, count, displ, MPI.DOUBLE], 其中 count 和 displ 都是一个整数系列，count 指明各个进程发送/接收的数据个数，displ 指明各个进程的数据段在发送/接收数据缓冲区中的起始偏离。因为要求进程 i 发送缓冲区的第 j 个位置必须与进程 j 接收缓冲区的第 i 个位置匹配，反之亦然，所以发送缓冲区的计数排列成的矩阵与接收缓冲区的计数排列成的矩阵应该互为转置矩阵。
Alltoallw 是对 Alltoallv 的进一步扩展，不仅可以发送不同长度的数据，还可以发送不同类型的数据，为此，它的参数 send_buf/recv_buf 需要分别指定发送/接收数据缓冲区，通信数据量数组，偏移信息数组和数据类型数组。注意：与 Alltoallv 不同的是，它的发送/接收数据偏移以字节为单位指定。因为要求进程 i 发送缓冲区的第 j 个位置必须与进程 j 接收缓冲区的第 i 个位置匹配，反之亦然，所以发送缓冲区的计数排列成的矩阵与接收缓冲区的计数排列成的矩阵应该互为转置矩阵，同时发送缓冲区的数据类型排列成的矩阵与接收缓冲区的数据类型排列成的矩阵应该互为转置矩阵。
对组内通信子对象的 Alltoall，Alltoallv 和 Alltoallw，可以将其 sendbuf 参数设置成 MPI.IN_PLACE，此时 recvbuf 将既作为发送缓冲区，又作为接收缓冲区。因为要求进程 i 发送缓冲区的第 j 个位置必须与进程 j 接收缓冲区的第 i 个位置匹配，反之亦然，所以缓冲区的计数排列成的矩阵以及数据类型排列成的矩阵都应该是对称的。
例程
下面给出全发散操作的使用例程。
# alltoall.py
"""
Demonstrates the usage of alltoall, Alltoall, Alltoallv, Alltoallw.
Run this with 4 processes like:
$ mpiexec -n 4 python alltoall.py
"""
import numpy as np
from mpi4py import MPI
comm = MPI.COMM_WORLD
rank = comm.Get_rank()
# ------------------------------------------------------------------------------
# alltoall
send_obj = [1.2, 'xy', {'a': 1}, (2,)]
recv_obj = comm.alltoall(send_obj)
print 'alltoall: rank %d has %s' % (rank, recv_obj)
# ------------------------------------------------------------------------------
# Alltoall
send_buf = np.arange(4, dtype='i')
recv_buf = np.empty(4, dtype='i')
comm.Alltoall(send_buf, recv_buf)
print 'Alltoall: rank %d has %s' % (rank, recv_buf)
# ------------------------------------------------------------------------------
# Alltoall with MPI.IN_PLACE
recv_buf = np.arange(4, dtype='i')
comm.Alltoall(MPI.IN_PLACE, recv_buf)
print 'Alltoall with MPI.IN_PLACE: rank %d has %s' % (rank, recv_buf)
# ------------------------------------------------------------------------------
# Alltoallv
send_buf = np.arange(8, dtype='i')
recv_buf = np.empty(8, dtype='i')
if rank == 0:
send_cnt = [2, 3, 1, 2]
send_dpl = [0, 2, 5, 6]
recv_cnt = [2, 3, 2, 1]
recv_dpl = [0, 2, 5, 7]
elif rank == 1:
send_cnt = [3, 2, 2, 1]
send_dpl = [0, 3, 5, 7]
recv_cnt = [3, 2, 1, 2]
recv_dpl = [0, 3, 5, 6]
elif rank == 2:
send_cnt = [2, 1, 3, 2]
send_dpl = [0, 2, 3, 6]
recv_cnt = [1, 2, 3, 2]
recv_dpl = [0, 1, 3, 6]
else:
send_cnt = [1, 2, 2, 3]
send_dpl = [0, 1, 3, 5]
recv_cnt = [2, 1, 2, 3]
recv_dpl = [0, 2, 3, 5]
# the matrix of recv_cnt should be the transpose of the matrix of send_cnt
# [[2, 3, 1, 2], [[2, 3, 2, 1],
# A = [3, 2, 2, 1], B = A.T = [3, 2, 1, 2],
# [2, 1, 3, 2], [1, 2, 3, 2],
# [1, 2, 2, 3]] [2, 1, 2, 3]]
comm.Alltoallv([send_buf, send_cnt, send_dpl, MPI.INT], [recv_buf, recv_cnt, recv_dpl, MPI.INT])
print 'Alltoallv: rank %d has %s' % (rank, recv_buf)
# ------------------------------------------------------------------------------
# Alltoallv with MPI.IN_PLACE
recv_buf = np.arange(8, dtype='i')
if rank == 0:
cnt = [2, 3, 1, 2]
dpl = [0, 2, 5, 6]
elif rank == 1:
cnt = [3, 1, 2, 2]
dpl = [0, 3, 4, 6]
elif rank == 2:
cnt = [1, 2, 2, 3]
dpl = [0, 1, 3, 5]
else:
cnt = [2, 2, 3, 1]
dpl = [0, 2, 4, 6]
# with MPI.IN_PLACE, the maxtrix of cnt should be symmetric
# [[2, 3, 1, 2],
# A = A.T = [3, 1, 2, 2],
# [1, 2, 2, 3],
# [2, 2, 3, 1]]
comm.Alltoallv(MPI.IN_PLACE, [recv_buf, cnt, dpl, MPI.INT])
print 'Alltoallv with MPI.IN_PLACE: rank %d has %s' % (rank, recv_buf)
# ------------------------------------------------------------------------------
# Alltoallw example 1
send_buf = bytearray(b"abcd")
recv_buf = bytearray(4)
cnt = [1, 1, 1, 1]
dpl = [0, 1, 2, 3]
typ = [MPI.CHAR, MPI.CHAR, MPI.CHAR, MPI.CHAR]
comm.Alltoallw([send_buf, cnt, dpl, typ], [recv_buf, cnt, dpl, typ])
# or
# comm.Alltoallw([send_buf, (cnt, dpl), typ], [recv_buf, (cnt, dpl), typ])
print 'Alltoallw1: rank %d has %s' % (rank, recv_buf)
# Alltoallw example 2
# create a send buffer as a numpy structured array with elements with different types
send_buf = np.array([('a', 1, 1.5, 3.2)], dtype=[('a', 'c'), ('b', 'i4'), ('c', 'f4'), ('d', 'f8')])
send_cnt = [1, 1, 1, 1]
send_dpl = [0, 1, 5, 9]
send_typ = [MPI.CHAR, MPI.INT, MPI.FLOAT, MPI.DOUBLE]
if rank == 0:
recv_buf = np.empty(4, dtype='c')
recv_cnt = [1, 1, 1, 1]
recv_dpl = [0, 1, 2, 3]
recv_typ = [MPI.CHAR, MPI.CHAR, MPI.CHAR, MPI.CHAR]
elif rank == 1:
recv_buf = np.empty(4, dtype='i4')
recv_cnt = [1, 1, 1, 1]
recv_dpl = [0, 4, 8, 12]
recv_typ = [MPI.INT, MPI.INT, MPI.INT, MPI.INT]
if rank == 2:
recv_buf = np.empty(4, dtype='f4')
recv_cnt = [1, 1, 1, 1]
recv_dpl = [0, 4, 8, 12]
recv_typ = [MPI.FLOAT, MPI.FLOAT, MPI.FLOAT, MPI.FLOAT]
elif rank == 3:
recv_buf = np.empty(4, dtype='f8')
recv_cnt = [1, 1, 1, 1]
recv_dpl = [0, 8, 16, 24]
recv_typ = [MPI.DOUBLE, MPI.DOUBLE, MPI.DOUBLE, MPI.DOUBLE]
comm.Alltoallw([send_buf, send_cnt, send_dpl, send_typ], [recv_buf, recv_cnt, recv_dpl, recv_typ])
print 'Alltoallw2: rank %d has %s' % (rank, recv_buf)
# ------------------------------------------------------------------------------
# Alltoallw with MPI.IN_PLACE
if rank == 0:
recv_buf = np.array([('a', 1, 1.5, 3.2)], dtype=[('a', 'c'), ('b', 'i4'), ('c', 'f4'), ('d', 'f8')])
recv_cnt = [1, 1, 1, 1]
recv_dpl = [0, 1, 5, 9]
recv_typ = [MPI.CHAR, MPI.INT, MPI.FLOAT, MPI.DOUBLE]
if rank == 1:
recv_buf = np.array([(2, 2.5, 4.2, 'b')], dtype=[('b', 'i4'), ('c', 'f4'), ('d', 'f8'), ('a', 'c')])
recv_cnt = [1, 1, 1, 1]
recv_dpl = [0, 4, 8, 16]
recv_typ = [MPI.INT, MPI.FLOAT, MPI.DOUBLE, MPI.CHAR]
if rank == 2:
recv_buf = np.array([(3.5, 5.2, 'c', 3)], dtype=[('c', 'f4'), ('d', 'f8'), ('a', 'c'), ('b', 'i4')])
recv_cnt = [1, 1, 1, 1]
recv_dpl = [0, 4, 12, 13]
recv_typ = [MPI.FLOAT, MPI.DOUBLE, MPI.CHAR, MPI.INT]
if rank == 3:
recv_buf = np.array([(6.2, 'd', 4, 4.5)], dtype=[('d', 'f8'), ('a', 'c'), ('b', 'i4'), ('c', 'f4')])
recv_cnt = [1, 1, 1, 1]
recv_dpl = [0, 8, 9, 13]
recv_typ = [MPI.DOUBLE, MPI.CHAR, MPI.INT, MPI.FLOAT]
# with MPI.IN_PLACE, both the maxtrix of cnt and typ should be symmetric
# [[1, 1, 1, 1], [[c, i, f, d],
# A = A.T = [1, 1, 1, 1], B = B.T = [i, f, d, c],
# [1, 1, 1, 1], [f, d, c, i],
# [1, 1, 1, 1]] [d, c, i, f]]
comm.Alltoallw(MPI.IN_PLACE, [recv_buf, recv_cnt, recv_dpl, recv_typ])
print 'Alltoallw with MPI.IN_PLACE: rank %d has %s' % (rank, recv_buf)
运行结果如下：
$ mpiexec -n 4 python alltoall.py
alltoall: rank 0 has [1.2, 1.2, 1.2, 1.2]
alltoall: rank 2 has [{'a': 1}, {'a': 1}, {'a': 1}, {'a': 1}]
alltoall: rank 3 has [(2,), (2,), (2,), (2,)]
alltoall: rank 1 has ['xy', 'xy', 'xy', 'xy']
Alltoall: rank 0 has [0 0 0 0]
Alltoall: rank 2 has [2 2 2 2]
Alltoall: rank 1 has [1 1 1 1]
Alltoall: rank 3 has [3 3 3 3]
Alltoall with MPI.IN_PLACE: rank 0 has [0 0 0 0]
Alltoall with MPI.IN_PLACE: rank 2 has [2 2 2 2]
Alltoall with MPI.IN_PLACE: rank 1 has [1 1 1 1]
Alltoall with MPI.IN_PLACE: rank 3 has [3 3 3 3]
Alltoallv: rank 2 has [5 5 6 3 4 5 3 4]
Alltoallv: rank 0 has [0 1 0 1 2 0 1 0]
Alltoallv: rank 1 has [2 3 4 3 4 2 1 2]
Alltoallv: rank 3 has [6 7 7 6 7 5 6 7]
Alltoallv with MPI.IN_PLACE: rank 2 has [5 4 5 3 4 4 5 6]
Alltoallv with MPI.IN_PLACE: rank 0 has [0 1 0 1 2 0 0 1]
Alltoallv with MPI.IN_PLACE: rank 1 has [2 3 4 3 1 2 2 3]
Alltoallv with MPI.IN_PLACE: rank 3 has [6 7 6 7 5 6 7 7]
Alltoallw1: rank 0 has aaaa
Alltoallw1: rank 1 has bbbb
Alltoallw1: rank 2 has cccc
Alltoallw1: rank 3 has dddd
Alltoallw2: rank 0 has ['a' 'a' 'a' 'a']
Alltoallw2: rank 2 has [ 1.5 1.5 1.5 1.5]
Alltoallw2: rank 1 has [1 1 1 1]
Alltoallw2: rank 3 has [ 3.2 3.2 3.2 3.2]
Alltoallw with MPI.IN_PLACE: rank 1 has [(1, 2.5, 5.2, 'd')]
Alltoallw with MPI.IN_PLACE: rank 2 has [( 1.5, 4.2, 'c', 4)]
Alltoallw with MPI.IN_PLACE: rank 0 has [('a', 2, 3.5, 6.2)]
Alltoallw with MPI.IN_PLACE: rank 3 has [( 3.2, 'b', 3, 4.5)]
以上我们介绍了 mpi4py 中的全发散操作方法，下一篇中我们将介绍扫描操作。