Multi-class multi-object detection and tracking system in 3 dimensions using YOLOv5

It is of great significance to detect and track multiple objects belonging to multiple classes in 3 dimensions and the same can be applied to various research domains. One such application pertains to multi-fish detection and tracking The problem is of significance to marine researchers who want to study the behaviour of fish in a constrained environment i.e. in an aquarium, using different sensors. For this study, a data set consisting of different fishes in an aquarium was constructed and Microsoft Kinect was used to capture the movements in 3 dimensions. The data set was cleaned, aligned, and calibration was done to make it suitable for object detection and tracking In this paper, YOLOv5s model was improvised, hereafter referred as YOLOv5-With Attention Mechanism (YOLOv5-WAM), and an object detection and tracking pipeline was designed to detect and track underwater objects (fishes in our work) belonging to multiple classes in 3D. YOLOv5-WAM can be adapted for use on devices having low compute availability, making it suitable for real time object detection and tracking using low cost experimental setup. YOLOv5-WAM showed about 1% improvement in mAP for our custom dataset while for Fish4Knowkdge and COCO dataset, the improvements were to the tune of 7% when compared to YOLOv5s. The results and analysis of the approach are presented in this paper.