Authors: Gilles Baechler, Srinivas Sunkara, Maria Wang, Fedir Zubach, Hassan Mansoor, Vincent Etter, Victor Cărbune, Jason Lin, Jindong Chen, Abhanshu Sharma

Published on: February 07, 2024

Impact Score: 8.22

Arxiv code: Arxiv:2402.04615

Summary

• What is new: Introduction of ScreenAI, a model that outperforms existing models in UI and infographics understanding by incorporating a mixed dataset and a novel screen annotation task.
• Why this is important: The need for better understanding of screen user interfaces and infographics for improved human-machine interaction.
• What the research proposes: ScreenAI utilizes the PaLI architecture with a pix2struct patching strategy, combined with a unique dataset mix and a novel screen annotation task.
• Results: ScreenAI achieves state-of-the-art results on several UI- and infographics-based tasks and best-in-class performance on others, with only 5B parameters.

Technical Details

Technological frameworks used: PaLI architecture, pix2struct patching strategy

Models used: ScreenAI

Data used: Mixed datasets including a novel screen annotation task

Potential Impact

Design and development software companies, web development platforms, visual communication tools

Want to implement this idea in a business?

We have generated a startup concept here: Visulify.