Tasks
Visual Description Generation
Image Description Generation
Standard Image Description Generation
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第1张图片](http://img.e-com-net.com/image/info8/ebe3154ad0e24b54946c06d2f8488a7f.jpg)
Dense Image Description Generation:旨在局部目标处生成描述
Image Paragraph Generation:生成段落
Spoken Language Image Description Generation:变写为说
Stylistic Image Description Generation:添加语言风格,例如幽默,
Unseen Objects Image Description Generation:
Diverse Image Description Generation:
Controllable Image Description Generation: control and select the objects in an image to generate descriptions.
Video Description Generation
Global Video Description Generation:
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第2张图片](http://img.e-com-net.com/image/info8/7609dc00a04e47da8b62ce97c043ca75.jpg)
Dense Video Description Generation: 类似与Dense Image Description Generation
Movie Description Generation: movie clips are used as input
Visual Storytelling
Image Storytelling:
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第3张图片](http://img.e-com-net.com/image/info8/bc6925a661ba4682a4b851a4d4b5ae37.jpg)
Video Storytelling:
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第4张图片](http://img.e-com-net.com/image/info8/3d1887f587fa4024a5dae3520f6ffdab.jpg)
Visual Question Answering
Image Question Answering
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第5张图片](http://img.e-com-net.com/image/info8/77c6cbde6b384b9fbc2da7d1acbb9d72.jpg)
Video Question Answering
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第6张图片](http://img.e-com-net.com/image/info8/0b9935b0ee75450f8e247318ad452eed.jpg)
Visual Dialog
Image Dialog
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第7张图片](http://img.e-com-net.com/image/info8/fe5f87d587d4451fbed2e2c483192d96.jpg)
Video Dialog
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第8张图片](http://img.e-com-net.com/image/info8/f757ab09b1ed4f219a1d65fdba890cae.jpg)
Visual Reasoning
Image Reasoning
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第9张图片](http://img.e-com-net.com/image/info8/bcf0d3dcd71049d7a081285cd9640e14.jpg)
Video Reasoning
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第10张图片](http://img.e-com-net.com/image/info8/34135908144c4880b48ed78681e3ad22.jpg)
Video Referring Expression
Image Referring Expression
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第11张图片](http://img.e-com-net.com/image/info8/108ba9fd18c14512963ce780e5d76ea4.jpg)
Video Referring Expression
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第12张图片](http://img.e-com-net.com/image/info8/c362cb578d47477e9fd562021243766d.jpg)
Visual Entailment
Image Entailment
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第13张图片](http://img.e-com-net.com/image/info8/8881fa03d7c84dcd9fc4fca3097a34e3.jpg)
Language-to-Vision Generation
Language-to-Image Generation
Sentence-level Language-to-Image Generation
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第14张图片](http://img.e-com-net.com/image/info8/3b1cf0bfa3e043868fa4fa03bc17165c.jpg)
Image Manipulation(图像编辑):生通过本文来引导图像的编辑, 同时保持其他文本不相关的区域,另一种方法是交互式的修改图像内容,还有一种是通过对话修改。
Fine-grain Image Generation(细粒度的图像生成):
Sequential Image Generation(序列图像生成):给定一段文字(多个句子),生成一系列的图像,就像故事的可视化,与image storytelling相反。
Language-to-Video Generation
需要更强的条件生成器,因为需要考虑时间维度
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第15张图片](http://img.e-com-net.com/image/info8/bbd53f449c6d4972b94880cbe00c14b9.jpg)
Vision-and-Language Navigation
Image and Language Navigation
Multimodal Machine Translation
Machine Translation with Image:将描述一副图像的源语言句子翻译成目标语言。
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第16张图片](http://img.e-com-net.com/image/info8/2054b7ff5b5241d1825a683c0dc66344.jpg)
Multisource MMT:不同点:多种语言同时描述一副图像
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第17张图片](http://img.e-com-net.com/image/info8/8167d7d4323c4b3baddef4575e0a0d72.jpg)
Machine Translation with Video
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第18张图片](http://img.e-com-net.com/image/info8/cde0719f9b61415b8ca56280deb91d9c.jpg)
Dataset
Image Description Generation
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第19张图片](http://img.e-com-net.com/image/info8/2559e72cedfa40e0bfc1ab81aa8e26b8.jpg)
- Flickr30K:https://shannon.cs.illinois.edu/DenotationGraph
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第20张图片](http://img.e-com-net.com/image/info8/289988bfaef2486e95fb6682ccd5ff71.jpg)
- Flickr30K-Entities:http://bryanplummer.com/Flickr30kEntities/
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第21张图片](http://img.e-com-net.com/image/info8/5c81f4dfe9014db88e1751e17e1fb55e.jpg)
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第22张图片](http://img.e-com-net.com/image/info8/0767715601694848a29ef7475a4d062b.jpg)
- MSCOCO:http://cocodataset.org/#home
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第23张图片](http://img.e-com-net.com/image/info8/efbe90f7f8474e318b97d397c98158d1.jpg)
- MSCOCO-Entities:https://github.com/aimagelab/show-control-and-tell
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第24张图片](http://img.e-com-net.com/image/info8/0abc3ea706a9479f9c8848bfa8e4f4fc.jpg)
- STAIR(Japanese captions):http://captions.stair.center
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第25张图片](http://img.e-com-net.com/image/info8/4c7a9471798d4c61a64b3be98ae4c858.jpg)
- Multi30K-CLID(German captions)
https://www.statmt.org/wmt16/multimodal-task.html
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第26张图片](http://img.e-com-net.com/image/info8/ff00270055574d769918a93b2be959a6.jpg)
https://www.statmt.org/wmt17/multimodal-task.html
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第27张图片](http://img.e-com-net.com/image/info8/7f8aba384b24485480a527618881e3ab.jpg)
http://www.statmt.org/wmt18/multimodal-task.html
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第28张图片](http://img.e-com-net.com/image/info8/df63f2daa7dd497cb827434d63547b8c.jpg)
- Conceptual Captions(large scale dataset):https://ai.google.com/research/ConceptualCaptions/download
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第29张图片](http://img.e-com-net.com/image/info8/8c9b140ad85a44f68ccb4154d1a485cd.jpg)
Video Description Generation
- Microsoft Video Description (MSVD,contain Chinese, English, German etc):http://www.cs.utexas.edu/users/ml/clamp/videoDescription
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第30张图片](http://img.e-com-net.com/image/info8/a88d82db2bc243509dfa022e5d30710e.jpg)
- MPII Cooking(consists of 65 different cooking activities):https://www.mpi-inf.mpg.de/departments/computer-vision-and-machine-learning/research/human-activity-recognition/mpii-cooking-activities-dataset
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第31张图片](http://img.e-com-net.com/image/info8/329b9bd60ae2487a86f7e8300ed96895.jpg)
- YouCook:http://web.eecs.umich.edu/~jjcorso/r/youcook
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第32张图片](http://img.e-com-net.com/image/info8/5e371d277d104caea39cc51d731211f5.jpg)
- YouCook II:http://youcook2.eecs.umich.edu
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第33张图片](http://img.e-com-net.com/image/info8/07791db4daf548d9a2edaaf72e2940ca.jpg)
- Textually Annotated Cooking Scenes (TACoS):http://www.coli.uni-saarland.de/projects/smile/page.php?id=tacos
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第34张图片](http://img.e-com-net.com/image/info8/4815dcee19c34b24849ea4b81a76e5f7.jpg)
- TACoS-MultiLevel:https://www.mpi-inf.mpg.de/departments/computer-vision-and-machine-learning/research/vision-and-language/tacos-multi-level-corpus
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第35张图片](http://img.e-com-net.com/image/info8/60aa419ffd2e42f1be54fcf281146ba9.jpg)
- MPII Movie Description (MPII-MD):https://www.mpi-inf.mpg.de/departments/computer-vision-and-machine-learning/research/vision-and-language/mpii-movie-description-dataset
- Montreal Video Annotation (M-VAD):https://mila.quebec/en/publications/public-datasets/m-vad
- MSR Video to Text (MSR-VTT):http://ms-multimedia-challenge.com/2017/dataset
- Videos Titles in the Wild (VTW):http://aliensunmin.github.io/project/video-language/index.html#VTW
- ActivityNet Captions (ANetCap):http://activity-net.org/challenges/2017/captioning.html
- ActivityNet Entities (ANetEntities):https://github.com/facebookresearch/ActivityNet-Entities
Image Storytelling
- New York City Storytelling (NYC-Storytelling):数据集被分成0.8,0.1,0.1,https://github.com/cesc-park/CRCN
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第36张图片](http://img.e-com-net.com/image/info8/b53008c61fa94a599d0513ba1cfd42e1.jpg)
- Disneyland Storytelling:数据集被分成0.8,0.1,0.1
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第37张图片](http://img.e-com-net.com/image/info8/382705a2e47f421d9b68ec979a72c70c.jpg)
- SIND:大规模数据集,数据集被分成0.8,0.1,0.1
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第38张图片](http://img.e-com-net.com/image/info8/04e03ddf78fc4d8dba5f2ff429fef4df.jpg)
- VIST:是SIND第二个版本,http://visionandlanguage.net/VIST
Video Storytelling
- VideoStory
- VideoStory-NUS
两个数据集都不是开源的
Image Question Answering
- VQA v1.0:answers也是open-ended,要么是少数单词,要么从多个给定答案中选择一个,https://visualqa.org
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第39张图片](http://img.e-com-net.com/image/info8/df57a3920eca464eb271fd1e2533fc44.jpg)
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第40张图片](http://img.e-com-net.com/image/info8/7b0f8b5871f34698803fc2ef9a37c188.jpg)
- OK-VQA:https://okvqa.allenai.org
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第41张图片](http://img.e-com-net.com/image/info8/25a9e93de53b4a23a7e1bbe2bc2b2756.jpg)
- KVQA:http://malllabiisc.github.io/resources/kvqa
Video Question Answering
- MovieQA:http://movieqa.cs.toronto.edu/home
- TVQA:http://tvqa.cs.unc.edu
- TVQA+:http://tvqa.cs.unc.edu/download_tvqa_plus.html
Image Dialog
- VisDial:https://visualdialog.org/data
- CLEVR-Dialog:https://github.com/satwikkottur/clevr-dialog
Video Dialog
- The Scene-Aware Dialog (AVSD):https://video-dialog.com
Image Reasoning
- Compositional Language and Elementary Visual Reasoning (CLEVR):https://cs.stanford.edu/people/jcjohns/clevr
- CLEVR-CoGenT:https://cs.stanford.edu/people/jcjohns/clevr
- GQA:https://cs.stanford.edu/people/dorarad/gqa
- Relational and Analogical Visual rEasoNing (RAVEN):http://wellyzhang.github.io/project/raven.html
Video Reasoning
- COG:https://github.com/google/cog#datasets
Image Referring Expression
Real Images
- RefCOCO:http://tamaraberg.com/referitgame
- RefCOCO+,
- RefCOCOg:https://github.com/lichengunc/refer
- RefClef:http://tamaraberg.com/referitgame
- GuessWhat:https://guesswhat.ai
Synthetic Images
- CLEVR-Ref+:https://cs.jhu.edu/~cxliu/2019/clevr-ref+
Video Referring Expression
- Cityscapes:https://www.cityscapes-dataset.com
- ORGaze:https://people.ee.ethz.ch/~arunv/ORGaze.html
Image Entailment
- V-SNLI
- SNLI-VE:https://github.com/necla-ml/SNLI-VE
Image Generation
- Oxford-102:http://www.robots.ox.ac.uk/~vgg/data/flowers/102
- Caltech-UCSD Birds (CUB):http://www.vision.caltech.edu/visipedia/CUB-200-2011.html
- MSCOCO-Gen:
Video Generation
没有开源的数据集
Image-and-Language Navigation
- Room-2-Room (R2R):https://bringmeaspoon.org
- ASKNAV:https://github.com/debadeepta/vnla
- TOUCHDOWN:https://github.com/lil-lab/touchdown
Machine Translation with Image
- Multi30K-MMT:https://www.statmt.org/wmt18/multimodal-task.html
Machine Translation with Video
- VATEX:http://vatex.org/main/index.html
Miscellaneous
不是为了特定任务而设计的,间接有助于上述任务。
Visual Genome
- Visual Genome: comprehend interactions and relationships between objects observed in an image, https://visualgenome.org
- How2:https://srvk.github.io/how2-dataset
- Berkeley Deep Drive eXplanation (BDD-X):au-tonomous driving, https://github.com/JinkyuKimUCB/BDD-X-dataset
Representation
Vision
Image Representation
- global feature representation: 常使用AlexNet, VGG, GoogLeNet, Inception-v3, Residual Nets (ResNet) and DenseNets学习全局特征,然而,一些语言和视觉结合任务不适合使用预训练的特征。
- local feature representation: R-CNN等
Video Representation
Language
常使用RNN, LSTM, BiLSTM, GRU, BiGRU, Transformer
Vision and Language
Visual Storytelling
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第42张图片](http://img.e-com-net.com/image/info8/94781096983a482eb65854554a102103.jpg)
![](http://img.e-com-net.com/image/info8/31cacb6ebe954196bd37f0f2aa3920e3.jpg)
Visual Dialog
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第43张图片](http://img.e-com-net.com/image/info8/dc55619cbeba47a6992de69332757c8e.jpg)
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第44张图片](http://img.e-com-net.com/image/info8/db020a700da74279af082aa3d49cffff.jpg)
Visual Reasoning
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第45张图片](http://img.e-com-net.com/image/info8/01450f77f1ed4800be0dc98937ab4377.jpg)
Visual Referring Expression
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第46张图片](http://img.e-com-net.com/image/info8/686fd94ba05242f7b4f62e61a010c60c.jpg)
![](http://img.e-com-net.com/image/info8/ec8a112444494258b619deb5f80f68e9.jpg)
Visual Entailment
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第47张图片](http://img.e-com-net.com/image/info8/0a6f7a1a9ad14143b82ea4a35bc825da.jpg)
Language-to-Vision Generation
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第48张图片](http://img.e-com-net.com/image/info8/f631f142caf54fa0a780f144339728e8.jpg)
![](http://img.e-com-net.com/image/info8/4977c865ebb3464f84b04032d7be6e66.jpg)
Vision-and-Language Navigation
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第49张图片](http://img.e-com-net.com/image/info8/692021faebaa4939ae6d6aa784da56b9.jpg)
Multimodal Machine Translation
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第50张图片](http://img.e-com-net.com/image/info8/898e75f29267415599348915fbfbf2c5.jpg)
![](http://img.e-com-net.com/image/info8/febc26cab6034ebfb19bdcf6862e2229.jpg)
Evaluation Measures
Common Measures
Language Metrics
机器生成的文本和参考文本的单词重叠程度
- Bilingual Evaluation Understudy (BLEU):是为机器翻译提出的,以比较机器生成的输出与人类的ground truth, 常用于Visual Caption Generation, Visual Storytelling, Video Dialog and Multimodal Machine Translation, https://leimao.github.io/blog/BLEU-Score/, https://en.wikipedia.org/wiki/BLEU
- Metric for Evaluation of Translation with Explicit Ordering (METEOR): 常用于Visual Caption Generation, Visual Storytelling, Video Dialog and Multimodal Machine Translation, https://en.wikipedia.org/wiki/METEOR
- Recall Oriented Understudy for Gisting Evaluation (ROUGE): 常用于Visual Caption Generation, http://www.ccs.neu.edu/home/vip/teach/DMcourse/5_topicmodel_summ/notes_slides/What-is-ROUGE.pdf
- Consensus based Image Description Evaluation (CIDEr): 常用于image caption generation evaluation, Video Caption Generation, Visual Storytelling and Video Dialog
- Semantic Propositional Image Captioning Evaluation (SPICE):
Retriev al Metrics
- Recall@k (R@k)
- Median Rank (MedRank)
- Mean Reciprocal Rank (MRR)
- Mean Rank (Mean)
- Normalized Discounted Cumulative Gain (NDCG)
Task-specific Metrics
Image Reasoning
- Querying attribute (QA)
- Compare Attribute (CA)
- Compare Numbers (CN)
- Count
-
Exist
Video Reasoning
- Pointing
- Yes/No
- Conditional (Condit)
- Attribute-related (Atts)
Language-to-Vision Generation
- Inception Score (IS)
- Fréchet Inception distance (FID)
- R-precision
Vision-and-Language Na vigation
- Path Length (PL)
- Navigation Error (NE)
- Success Rate (SR)
- Oracle Success Rate (OSR)
- Success Path Length (SPL)
Human Evaluation
State-of-the-Art Results
Visual Storytelling Results
Image Storytelling
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第51张图片](http://img.e-com-net.com/image/info8/aa5d1646975142c8b758e416c8c1cd6f.jpg)
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第52张图片](http://img.e-com-net.com/image/info8/b34dca2349294a9b903eea3dea0a1209.jpg)
Video Storytelling
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第53张图片](http://img.e-com-net.com/image/info8/d60969c8ba5b4b9eb93e8fd6637e9d02.jpg)
Visual Dialog Results
Image Dialog
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第54张图片](http://img.e-com-net.com/image/info8/2016c6aaf9de4f28b1078b7a113edabd.jpg)
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第55张图片](http://img.e-com-net.com/image/info8/7651a973c8794a049884250ad9cc51d5.jpg)
Video Dialog
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第56张图片](http://img.e-com-net.com/image/info8/7168ae511f4b45a38ea468e776242b4a.jpg)
Visual Reasoning Results
Image Reasoning
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第57张图片](http://img.e-com-net.com/image/info8/9eea82d6d6de4d67bf6dbfc72b64d4e9.jpg)
Video Reasoning
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第58张图片](http://img.e-com-net.com/image/info8/ed7b20935b424ef3a087d950e1a13ae8.jpg)
Image Referring Expression
Image Referring Expression
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第59张图片](http://img.e-com-net.com/image/info8/ad3be33b1e8147d69af968e4a2f53601.jpg)
Video Referring Expression
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第60张图片](http://img.e-com-net.com/image/info8/9fbf9f37580844aaab5870591d4618a4.jpg)
Visual Entailment Results
Image Entailment
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第61张图片](http://img.e-com-net.com/image/info8/df89c28f51a34fa4b584f224c4b7aa5d.jpg)
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第62张图片](http://img.e-com-net.com/image/info8/c9eb38dd8f3e4cccb654cc888e4249e4.jpg)
Language-to-Vision Generation Results
Language-to-Image Generation
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第63张图片](http://img.e-com-net.com/image/info8/6616788ed5b143ad9c2e4a143fcd38f7.jpg)
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第64张图片](http://img.e-com-net.com/image/info8/6c51253c39db45659a385a3e24f72375.jpg)
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第65张图片](http://img.e-com-net.com/image/info8/2c29f2d8064d4561a8b5569f583488ff.jpg)
Language-to-Video Generation
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第66张图片](http://img.e-com-net.com/image/info8/22a7beef43a14bb89f6d0ab106aee931.jpg)
Vision-and-Language Navigation Results
Image-and-Language Navigation
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第67张图片](http://img.e-com-net.com/image/info8/374d09586b0f469d95c4f6ce833a7ef9.jpg)
Multimodal Machine Translation Results
Machine Translation with Image
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第68张图片](http://img.e-com-net.com/image/info8/25a1da5b230745a5b2f785804d723cfe.jpg)
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第69张图片](http://img.e-com-net.com/image/info8/e820bc6a7b314926b9e494c7ebf827e1.jpg)
Machine Translation with Video
![Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods_第70张图片](http://img.e-com-net.com/image/info8/2d7132af84cf411784ae471897c1f11d.jpg)
Future Directions
- Leveraging External Knowledge
- Addressing Large-scale Data Limitations
- Combining Multiple Tasks
- Novel Neural Architectures for Representation: transformer
- Image vs Video:需要更多的关注video与language的结合
- Automatic Evaluation Measures: