Authors: Muhammad Maaz, Hanoona Rasheed, Abdelrahman Shaker, Salman Khan, Hisham Cholakal, Rao M. Anwer, Tim Baldwin, Michael Felsberg, Fahad S. Khan

Published on: February 22, 2024

Impact Score: 7.8

Arxiv code: Arxiv:2402.14818

Summary

• What is new: Introduction of a Large Multilingual Multimodal Model, \textsc{Palo}, with visual reasoning capabilities in 10 major languages, and the creation of the first multilingual multimodal benchmark.
• Why this is important: The lack of inclusivity in Vision-Language Models (VLMs) due to language limitations and the underrepresentation of languages such as Hindi, Arabic, Bengali, and Urdu.
• What the research proposes: A semi-automated translation approach using a fine-tuned Large Language Model to adapt multimodal instruction datasets to 10 major languages with minimal manual effort and training models across three scales to ensure scalability and generalization.
• Results: Substantial improvements in performance across multiple languages, especially in those that are underrepresented, compared to strong baselines.

Technical Details

Technological frameworks used: Large Multilingual Multimodal Model (\textsc{Palo})

Models used: Large Language Models for semi-automated translation and adaptation

Data used: Multimodal instruction dataset

Potential Impact

Companies in the technology, AI development, and multilingual application sectors could benefit from or be disrupted by the insights and capabilities of \textsc{Palo}.

Want to implement this idea in a business?

We have generated a startup concept here: PolyVisualize.