A Comprehensive Design of Hybrid Residual (2+1)-Dimensional CNN and Dense Networks With Multi-Modal Sensor for Fish Appetite Detection

Fish is one of the most demanding protein sources in the food industry. However, the increasing demand must be followed by increasing production efficiency. One of the problems in fish production efficiency is an ineffective feeding method. In this paper, we address the problem of fish feeders using artificial intelligence in an aquarium. We propose fish appetite detection using multi-modal sensors resulting in the data for our AI system. The main aim is to improve the efficiency of fish feeding using a multi-modal sensor system. Our AI system consists of Residual (
$2+1$
)-Dimensional Convolutional Neural Networks (R(
$2+1$
)D-CNN) and dense networks to process video and accelerometer data. The video data is split into 20 frames and processed by an R(
$2+1$
)D-CNN. Furthermore, the accelerometer data is used to train several networks such as 1-dimensional Convolutional Neural Networks (CNN), Gated Recurrent Networks (GRU), and Dense Networks or Artificial Neural Networks (ANN). The system is implemented in an aquarium with two sensors i.e., a webcam camera and an accelerometer and a main board processing using Raspberry-Pi 4. Experimental results show that the proposed system outperforms other methods with validation accuracy up to 99.09% for the Zeromean dense model and up to 99.39% for the Filtered dense model. The results also show that the multi-modal sensor system increases the accuracy significantly without a significant increase in the number of parameters. The work is useful for automation and efficiency in aquaculture.