【Huggingface】DataCollatorForSeq2Seq中的pad_to_multiple_of选项

【Huggingface】DataCollatorForSeq2Seq中的pad_to_multiple_of选项

官方的解释是

pad_to_multiple_of (int, optional):
If set will pad the sequence to a multiple of the provided value.
This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability >=7.5 (Volta).

代码长这样

# DataCollatorForSeq2Seq实现中相关的部分
if self.pad_to_multiple_of is not None:
    max_label_length = (
       (max_label_length + self.pad_to_multiple_of - 1)
       // self.pad_to_multiple_of
       * self.pad_to_multiple_of
    )
features = self.tokenizer.pad(
        features,
        padding=self.padding,
        max_length=self.max_length,
        pad_to_multiple_of=self.pad_to_multiple_of,
        return_tensors=return_tensors,
)

也就是说当pad_to_multiple被设置成None以外的内容时，通常是8或者16，max_label_length会被调节成设置值的整数倍，这样做的目的是方便N卡更高效利用计算能力。