Data Partitioning and Storage Strategies for Artificial Intelligence and Machine Learning Applications: A Review of Techniques

This paper presents a review of data partitioning and storage strategies critical for optimizing the performance and scalability of Artificial Intelligence (AI) and Machine Learning (ML) applications. As the volume and complexity of data continue to grow, effective data management becomes increasingly essential for ensuring the efficiency and accuracy of AI/ML models. The review categorizes and evaluates various data partitioning techniques, including hash-based, range-based and graph-based algorithms, highlighting their strengths, weaknesses, and suitability for different types of datasets and applications. It also explores the data processing challenges with AI/ML applications, data access patterns for AI/ML algorithms; and implementation techniques and evaluation metrics. By identifying key factors that influence the choice of partitioning and storage strategies, this review provides valuable insights for researchers and practitioners seeking to enhance the performance of AI/ML systems. The review also serves as a foundation for future research and development, guiding the design of more efficient and scalable data management solutions in artificial intelligence.