Enhanced YOLOv8-Based Instance Segmentation for Stacked Object Detection in Industrial Automation

Instance segmentation is essential for improving robotic perception and operational efficiency, especially in complex industrial settings. As automation progresses, accurate object detection and segmentation are key to enabling robotic arms to perform precise grasping and manipulation. However, real-world environments often include irregularly stacked objects with occlusion, scale variation, and cluttered backgrounds, making segmentation highly challenging. To address these issues, this paper presents an enhanced instance segmentation algorithm based on YOLOv8n, tailored for stacked object scenarios. The method integrates Adaptive Spatial Feature Fusion (ASF) to reduce computational cost and enhance multi-scale feature extraction by effectively connecting the backbone and neck. Additionally, a Deformable Convolution-based Residual (DWR) network structure (C2f-DWR) is adopted to improve spatial adaptability and robustness against occlusion and geometric distortion. Experimental results on a custom dataset show a 3.1% improvement in Mask mAP over standard YOLOv8, along with better precision, recall, and segmentation accuracy, contributing to more reliable robotic vision in industrial automation.