Authors: Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, Tong Sun

Published on: June 29, 2023

Impact Score: 8.62

Arxiv code: Arxiv:2306.17107

Summary

• What is new: Enhancing visual instruction tuning with text-rich images for better understanding of textual details within images.
• Why this is important: Visual instruction-tuned models struggle with comprehending textual details within images.
• What the research proposes: Using OCR tools on text-rich images and prompting GPT-4 with recognized texts to generate conversations, improving comprehension.
• Results: Up to a 20% accuracy improvement on text-based VQA datasets and 91.42% on ScienceQA.

Technical Details

Technological frameworks used: GPT-4-based instruction-following evaluation, OCR tools

Models used: LLaVAR, LLaVA

Data used: 422K text-rich images from the LAION dataset, 16K GPT-4 generated conversations

Potential Impact

AI development firms, visual data analysis companies, educational tech companies, and social media platforms could benefit from the insights of this paper.

Want to implement this idea in a business?

We have generated a startup concept here: VisuaLingo AI.