Hybrid Optical and Hyperspectral Sensing System for Multimodal Segmentation of Solid Waste

Sensing technologies are widely used in the solid waste automation for ensuring the waste detection at a conveyor belt and its consequent separation. However, the state-of-the-art sensing methods face the problem of continuous growth of waste volume, as well as its diversity, which implies severe performance requirements. In this article, we report on the intelligent sensing system based on the networked optical and hyperspectral cameras combined with the widely-used machine learning methods to solve the problem of object segmentation. The goal of this research work is to amalgamate different data modalities to improve the accuracy of waste identification in a real-life scenario. This research is centered on a multimodal object segmentation framework. It encompasses the collection of a dataset with distinct training and testing partitions, addressing segmentation challenges and label removal from the optical data. Additionally, the study tackles the segmentation issue for hyperspectral data where the hyperspectral models utilize optical predictions. The proposed approach facilitates the precise classification of various waste objects. The dataset is comprised of 12 object categories, including the assorted plastics, films, textiles, tetrapacks, and cellulosic materials. The proposed solution is validated on an industrial-scale conveyor belt operating at the speed of 3 m/s enabling the real-life conditions. The experimental results demonstrate an F1-score of 70% for optical model and Intersection over Union metric of 92% for the multimodal pipeline.