Multisensor Data Fusion and Time Series to Image Encoding for Hardness Recognition

Robots must possess advanced sensing and recognition skills to effectively manipulate objects in realistic environments. This article presents a unique approach for detecting hardness using a glove sensor. The proposed strategy involves fusing data from many sensors and using time series for image encoding to achieve the desired objective. This technique was established and subsequently refined with a particular objective in focus. The integration of data acquired from five distinct force sensors allows for a thorough comprehension of the characteristics of the material. Simultaneously, using the Gramian angular summation field (GASF), the Gramian angular difference field (GADF), and Markov transition field (MTF) representations facilitates the conversion of time-series data into visual images. This conversion enables the transformation of dynamic measurements into organized formats, enabling the application of 15 deep convolutional neural network (DCNN) models for analysis. Based on the findings of the empirical investigation, it seems that the GADF image encoding method exhibited a remarkably elevated average recognition rate (ARR) total when used in conjunction with DCNN. The conclusions above are derived from the empirical evidence from the conducted investigation. Furthermore, its combination with Vgg-19 yielded a notable ARR of 78.17%.