Authors: Youhe Jiang, Ran Yan, Xiaozhe Yao, Yang Zhou, Beidi Chen, Binhang Yuan

Published on: November 20, 2023

Impact Score: 8.22

Arxiv code: Arxiv:2311.11514

Summary

• What is new: HexGen introduces a unique distributed inference engine supporting asymmetric partition of computations over tensor and pipeline parallelism in heterogeneous networks.
• Why this is important: The high inference costs associated with centralized data centers for large-scale foundation AI models.
• What the research proposes: HexGen allows flexible, distributed generative inference deployment across diverse GPUs, optimizing for low latency and high request rates.
• Results: HexGen achieves up to 2.3x lower latency or tolerates up to 4x more request rates compared to a homogeneous baseline within the same budget.

Technical Details

Technological frameworks used: HexGen

Models used: Llama-2 (70B)

Data used: nan

Potential Impact

Cloud computing services, AI service providers, companies with large-scale data center operations.

Want to implement this idea in a business?

We have generated a startup concept here: HexaServe.