CTRNN-Transformer Adding Continuous Time Neural Models to Transformers

This paper presents two novel transformer models, the Closed Form Continuous Time Transformer (CFC-T) and the Liquid Time Constant Transformer (LTC-T), advancing the R-Transformer model by incorporating closed form continuous time recurrent neural networks and liquid time constant networks instead of traditional positional encoding. These models are designed to address long-term dependency challenges and improve sequential computation efficiency, shortcomings often encountered in Recurrent Neural Networks (RNNs). Our evaluation across various continuous time datasets, focusing on applications in audio processing, demonstrate that both CFC-T and LTC-T outperform the R-Transformer, with the LTC-T consistently showing superior results. This performance is quantified by notable improvements in the processing of continuous time data like audio analysis and complex sequential tasks, establishing these models as promising alternatives to existing RNN-based approaches. The findings underscore the potential of integrating continuous-time networks into trans former architectures, suggesting applications in areas such as natural language processing and time series analysis.