Original title: KNVQA: A Benchmark for evaluation knowledge-based VQA
Authors: Sirui Cheng, Siyu Zhang, Jiayi Wu, Muchen Lan
In the world of advanced vision and language models, there’s been incredible progress, but there are some hurdles yet to overcome. These large models are amazing at understanding images and text, but they often struggle with two big issues: making up details and getting facts right. Evaluating these models has been a challenge too; existing methods focus on language understanding but miss out on assessing how well they combine both vision and language. That’s where KNVQA steps in—a new evaluation method designed specifically for knowledge-based tasks. It’s not just a test; it’s a whole dataset created with human input to check how accurate these models are in answering questions based on knowledge. By doing this, they’re not just evaluating these models comprehensively but also pinpointing where improvements can be made. Plus, this new evaluation method promises to be cost-effective, private, and easy to reproduce. It’s like shining a light on the strengths and weaknesses of these cutting-edge models.
Original article: https://arxiv.org/abs/2311.12639