论文阅读 CLIPScore: A Reference-free Evaluation Metric for Image Captioning

论文阅读 CLIPScore: A Reference-free Evaluation Metric for Image Captioning

Problem: 前人指标需要参考文本的问题
Solution: 采用CLIP来解决需要参考的问题
Contribution: 提出一种不需要参考文本的评价指标;提出一个reference增强指标RefCLIPScore;verify that CLIP-S is sensitive to adversarially constructed image captions, where one noun-phrase has been swapped for a plausible (but incorrect) distractor; construct a corpus of images that have never been posted publicly online to verify that CLIP-S is able to reconstruct human judgments on never-before-seen images.

Related work

Reference-only image caption evaluation

  • BLEU-4:which measures a version of precision between a candidate and the references
  • ROUGE-L:which mea- sures a version of recall
  • METEOR:which computes a word-level alignment
  • CIDEr:which com- bines n-gram tf-idf weighting and stemming
  • SPICE:which applies a semantic parser to a set of references, and com- putes similarity using the predicted scene graph
  • BERT-S++:Yi et al. (2020) give a method for re-weighting BERTScore (Zhang et al., 2020) specifically tuned to the image caption generation domain

指标介绍相关文章:
https://blog.csdn.net/sophicchen/article/details/114978410

Reference+image caption evaluation:incorporate image-text grounding features in addition to references

  • TIGEr (Jiang et al., 2019) uses a pretrained SCAN model (Lee et al., 2018)
  • ViLBERTScore-F (Lee et al., 2020) uses a pre- trained ViLBERT model (Lu et al., 2019) that is also fine-tuned on 12 downstream vision and lan- guage tasks (Lu et al., 2020)

Self-retrieval for image captioning

Prior works have proposed incorporating a self-retrieval loss into caption generation, with the intuition that good captions should be able to uniquely identify their images with high accuracy (Dai and Lin, 2017; Luo et al., 2018; Liu et al., 2018); monitoring this type of loss can provide insight into how distinctive the captions are according to the model itself.

Reference-free evaluation

For image captioning, a version of VIFIDEL (Madhyastha et al., 2019) was proposed for reference-free eval- uation; however, VIFIDEL, computed based on a list of detected objects in the image from a fixed ob- ject vocabulary, generally produces less correlation with human ratings vs. reference-based metrics.

CLIPScore

Evaluating Caption Generations with CLIP

输入: image和caption
计算: 将输入的image和caption放入到特征提取器(CLIP)中获取embedding,计算两者的cosine距离
论文阅读 CLIPScore: A Reference-free Evaluation Metric for Image Captioning_第1张图片

RefCLIPScore

论文阅读 CLIPScore: A Reference-free Evaluation Metric for Image Captioning_第2张图片

你可能感兴趣的:(论文阅读,论文阅读,多模态)