torch.backends.cudnn.deterministic = True torch.backends.cudnn.benchmark = False manual_seed控制程序的随机性

cuda 是NVIDIA 针对自家的CPU设计的并行计算的框架，

cuDNN是is a GPU-accelerated library of primitives for deep neural networks。是专门针对深度神经网络打造的加速库。

cuDNN Archive | NVIDIA Developerhttps://developer.nvidia.com/rdp/cudnn-archivetorch.backends.cudnn.benchmark = False

Reproducibility — PyTorch 1.11.0 documentationhttps://pytorch.org/docs/stable/notes/randomness.html

The cuDNN library, used by CUDA convolution operations, can be a source of nondeterminism across multiple executions of an application. When a cuDNN convolution is called with a new set of size parameters, an optional feature can run multiple convolution algorithms, benchmarking them to find the fastest one. Then, the fastest algorithm will be used consistently during the rest of the process for the corresponding set of size parameters. Due to benchmarking noise and different hardware, the benchmark may select different algorithms on subsequent runs, even on the same machine.

Disabling the benchmarking feature with torch.backends.cudnn.benchmark = False causes cuDNN to deterministically select an algorithm, possibly at the cost of reduced performance.

However, if you do not need reproducibility across multiple executions of your application, then performance might improve if the benchmarking feature is enabled with torch.backends.cudnn.benchmark = True.

大概意思是：torch.backends.cudnn.benchmark = True，在训练的时候，一个好的特征可能会经过好多不同的convolution algorithms的测试，选出来最快的，剩下的过程都会采用这个算法。

如果torch.backends.cudnn.benchmark = False，那么模型的表现可能不是最优的，但reproducibility是比较强的。

当算法固定的时候，但算法本身可能是nondeterministic的。

如果torch.use_deterministic_algorithms(True)，但有些算法本身是nondeterminitic的，所以就可能报错。有些算法是有deterministic的implementation的，使用时不会报错。

对比之下，torch.backends.cudnn.deterministic = True 只会针对这一个算法进行控制，而torch.use_deterministic_algorithms(True) 会对其他的操作产生影响(which will make other PyTorch operations behave deterministically, too)

同时，其他提升程序的reproducibility的方法，考虑seed的使用：