Authors: Yuxi Wei, Zi Wang, Yifan Lu, Chenxin Xu, Changxing Liu, Hao Zhao, Siheng Chen, Yanfeng Wang

Published on: February 08, 2024

Impact Score: 8.22

Arxiv code: Arxiv:2402.05746

Summary

• What is new: Introduction of ChatSim, the first system enabling editable photo-realistic 3D driving scene simulations via natural language commands with external digital assets integration.
• Why this is important: Existing scene simulation approaches have limitations in user interaction efficiency, multi-camera photo-realistic rendering, and external digital assets integration.
• What the research proposes: ChatSim leverages a large language model for flexible command editing, employs a multi-camera neural radiance field method for photo-realism, and uses a novel lighting estimation method for scene-consistent rendering of digital assets.
• Results: Successful handling of complex language commands and generation of corresponding photo-realistic scene videos demonstrated on the Waymo Open Dataset.

Technical Details

Technological frameworks used: Large language model (LLM) agent collaboration framework, multi-camera neural radiance field, multi-camera lighting estimation method

Models used: LLM for natural language processing

Data used: Waymo Open Dataset

Potential Impact

Autonomous driving simulation software providers, automotive companies, and digital asset creation tools for 3D modeling and simulation

Want to implement this idea in a business?

We have generated a startup concept here: SceneCraft.