模型特征蒸馏:Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillation

 https://github.com/SwinTransformer/Feature-Distillation.

在Feature Whitening中使用不带参数的LayerNorm:

nn.LayerNorm(embed_dim, elementwise_affine=False)

在Projector中使用1x1卷积进行特征对齐：VIT中使用

self.decoder = nn.Sequential(
    nn.Conv1d(
        in_channels=self.encoder.num_features,
        out_channels=embed_dim, kernel_size=1),
)