Synergy of CNN with Random Forest Based Hybrid Architecture to Estimate the Quality of Coherent Optical Communication

This paper introduces the adoption of a hybrid approach combining Convolution Neural Network (CNN) with a Random Forest regressor to estimate the Error Vector Magnitude (EVM) parameter precisely. This EVM variable plays a crucial role in monitoring the signal quality to determine the performance of coherent optical communications. We created a hybrid regression framework to regress the EVM values from complicated signal constellation images that are noisy signals while received. The proposed hybrid deep learning and machine learning framework leverages intricate features of constellation images extracted with the CNN model to train the robust machine learning algorithm to regress appropriately even when the number of constellation image samples is less. This characteristic is identified by testing the model with different versions of the dataset with a varied number of samples. To identify the appropriate candidate variant of the hybrid framework, we did a comparative study of CNN synergized with different machine learning models and concluded by exemplifying the results of the best hybrid variant. Amongst other variants, comparatively, the CNN model synergized with the Random forest algorithm achieves very low loss metrics while learning to regress the EVM variable with Mean Squared Error metric of 0.010, Mean Absolute Error metric of 0.022, R2 Score of 1 and Explained variance of 1.