Authors: Pengfei Yu, Heng Ji

Published on: May 29, 2023

Impact Score: 8.05

Arxiv code: Arxiv:2305.18582

Summary

• What is new: A novel pipeline approach utilizing a self-prompting-based question-answer generation and associative distillation methods to update LLMs with current information.
• Why this is important: LLMs struggle to provide current information due to outdated pre-training data, and existing update methods lack generalizability and require structured data.
• What the research proposes: A new task formulation for updating information in LLMs using unstructured updating corpus, alongside a pipeline approach for self-prompting and associative distillation.
• Results: Significant improvement in factual consistency score by up to 0.16 and effective mitigation of forgetting with only 2.3% of the training tokens.

Technical Details

Technological frameworks used: Self-prompting-based question-answer generation, associative distillation

Models used: Large Language Models (LLMs)

Data used: Datasets from news articles (March-April 2023), Natural Questions benchmark

Potential Impact

Content generation and verification industries, edtech companies, news aggregation platforms, and AI-driven market analysis firms could benefit.

Want to implement this idea in a business?

We have generated a startup concept here: InfoSync AI.