Transformer-MM-Explainability

two modalities are separated by the [SEP] token，the numbers in each attention module represent the Eq. number.

E${}_h$ is the mean，$\nabla$A :=$\frac{\partial y_t}{\partial A}$for $y_t$ which is the model’s output.$\odot$ is the Hadamard product，remove the negative contributions before averaging.

aggregated self-attention matrix R${}^{qq}$，previous layers’ mixture of context is embodied by R${}^{qk}$.

感想

作者的实验在coco和ImageNet验证集上做的，不好follow