Authors: Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Parul Chopra, Allie Del Giorno, Gustavo de Rosa, Matthew Dixon, Ronen Eldan, Dan Iter, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Jamie Huynh, Mojan Javaheripi, Xin Jin, Piero Kauffmann, Nikos Karampatziakis, Dongwoo Kim, Mahoud Khademi, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Chen Liang, Weishung Liu, Eric Lin, Zeqi Lin, Piyush Madan, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzant

Published on: April 22, 2024

Impact Score: 7.6

Arxiv code: Arxiv:2404.14219

Summary

• What is new: The introduction of phi-3-mini, a compact language model with 3.8 billion parameters, achieving high performance metrics comparable to larger models, and the novel dataset composed of filtered web and synthetic data.
• Why this is important: Existing large language models require substantial compute power for deployment, making them impractical for use on devices with limited resources.
• What the research proposes: Developing a smaller, efficient language model, phi-3-mini, that can run on a phone while maintaining competitive performance.
• Results: Phi-3-mini achieves 69% on MMLU and 8.38 on MT-bench, showcasing its ability to rival larger models like Mixtral 8x7B and GPT-3.5.

Technical Details

Technological frameworks used: nan

Models used: phi-3-mini, with extensions to phi-3-small and phi-3-medium for parameter scaling.

Data used: 3.3 trillion tokens from a new dataset combining heavily filtered web data with synthetic data.

Potential Impact

This innovation could impact markets requiring mobile deployment of AI, such as mobile computing, educational apps, and mobile-based customer service solutions.

Want to implement this idea in a business?

We have generated a startup concept here: MobiLingo AI.