A Methodology for Efficient Data Processing in AI Applications Using Adaptive Sampling and Automated Testing

As data volumes continue to surge, the resources required to accelerate data processing for model training remain a challenge, leading to increased costs and extended processing times. This paper presents a novel method that combines Adaptive Sampling with an automated, self-adaptive comprehensive test suite module to address these challenges. This approach maintains model accuracy and ensures coverage of essential data required for business use cases. Experiments conducted on diverse datasets, some as large as several hundred terabytes, demonstrate that this method can reduce processing times by up to 75%. This achievement is realized by efficiently identifying and processing only representative samples. These results demonstrate the promising potential for improving data processing efficiency in model training for Artificial Intelligence (AI) applications across diverse sectors.