论文阅读 CLIPScore: A Reference-free Evaluation Metric for Image Captioning

论文阅读 CLIPScore: A Reference-free Evaluation Metric for Image Captioning

Problem： 前人指标需要参考文本的问题
Solution： 采用CLIP来解决需要参考的问题
Contribution: 提出一种不需要参考文本的评价指标；提出一个reference增强指标RefCLIPScore；verify that CLIP-S is sensitive to adversarially constructed image captions, where one noun-phrase has been swapped for a plausible (but incorrect) distractor; construct a corpus of images that have never been posted publicly online to verify that CLIP-S is able to reconstruct human judgments on never-before-seen images.

Related work

Reference-only image caption evaluation

• BLEU-4：which measures a version of precision between a candidate and the references
• ROUGE-L:which mea- sures a version of recall
• METEOR:which computes a word-level alignment
• CIDEr:which com- bines n-gram tf-idf weighting and stemming
• SPICE：which applies a semantic parser to a set of references, and com- putes similarity using the predicted scene graph
• BERT-S++:Yi et al. (2020) give a method for re-weighting BERTScore (Zhang et al., 2020) specifically tuned to the image caption generation domain

指标介绍相关文章：
https://blog.csdn.net/sophicchen/article/details/114978410

Reference+image caption evaluation：incorporate image-text grounding features in addition to references

• TIGEr (Jiang et al., 2019) uses a pretrained SCAN model (Lee et al., 2018)
• ViLBERTScore-F (Lee et al., 2020) uses a pre- trained ViLBERT model (Lu et al., 2019) that is also fine-tuned on 12 downstream vision and lan- guage tasks (Lu et al., 2020)

Self-retrieval for image captioning

Prior works have proposed incorporating a self-retrieval loss into caption generation, with the intuition that good captions should be able to uniquely identify their images with high accuracy (Dai and Lin, 2017; Luo et al., 2018; Liu et al., 2018); monitoring this type of loss can provide insight into how distinctive the captions are according to the model itself.

Reference-free evaluation

For image captioning, a version of VIFIDEL (Madhyastha et al., 2019) was proposed for reference-free eval- uation; however, VIFIDEL, computed based on a list of detected objects in the image from a fixed ob- ject vocabulary, generally produces less correlation with human ratings vs. reference-based metrics.

CLIPScore

Evaluating Caption Generations with CLIP

输入： image和caption
计算： 将输入的image和caption放入到特征提取器（CLIP）中获取embedding，计算两者的cosine距离

RefCLIPScore