Authors: Yifan Zhu, Chengyang Luo, Tang Qian, Lu Chen, Yunjun Gao, Baihua Zheng

Published on: October 07, 2024

Impact Score: 8.0

Arxiv code: Arxiv:2410.05091

Summary

• What is new: DIMS introduces a three-stage heterogeneous partitioning method, three-stage indexing structure, and a cost-based optimization model for efficient distributed similarity search.
• Why this is important: As data grows, existing single-machine similarity search methods struggle with efficiency and scalability. Current distributed methods also face challenges such as inefficient local data management and unbalanced workloads.
• What the research proposes: The DIMS framework, which includes a three-stage partitioning strategy for balanced workloads, a three-stage indexing structure for efficient object management, and concurrent search techniques with filtering and validation for efficient distribution.
• Results: Experiments show that DIMS significantly outperforms current distributed similarity search approaches in terms of efficiency and scalability.

Technical Details

Technological frameworks used: Distributed Index for similarity search in Metric Spaces (DIMS)

Models used: Three-stage indexing structure, cost-based optimization model

Data used: Various datasets applicable to real-world applications such as multimedia retrieval and personalized recommendation.

Potential Impact

Data analytics, multimedia retrieval, personalized recommendation services, trajectory analytics, and companies handling large-scale data such as tech giants and cloud service providers.

Want to implement this idea in a business?

We have generated a startup concept here: SearchSci.