MAdapter: A Multimodal Adapter for Liquid State Machines configures the Input Layer for the same Reservoir to enable Vision and Speech Classification

The human cortex is capable of multi-modal processing. Then, assuming that the biological model of the brain cortex is captured by Liquid State Machines (LSMs), it should be endowed with a certain universality and efficiency that enables the same reservoir to extract features from different modalities like image, video and speech. However, researchers working with single applications have tuned LSMs specifically for their target datasets and modalities - possibly to account for the vast difference in spiking statistics produced by each unique dataset. In this paper, we explore a strategy to define the input layer hyperparameters for the same Reservoir to exploit the generality and efficiency of a Reservoir Network towards different modalities of data. We show that a strategy to set the input layer hyperparameters such that the Reservoir activity (average spike rate) is tuned to the best performance for one modality/dataset works for other modalities as long as the input layer hyperparameters are tuned to ensure similar levels of input stimulus to the reservoir. We also propose to use time partitions of the output spike trains coming from the Reservoir to improve the classification performance for time-varying datasets. We attained test classification accuracy of 86.74% on Fashion-MNIST, 95.86% on Neuromorphic-MNIST and 86.2% on the full TI-46 speech classification dataset using input parameters transferred from MNIST optimization on the same 1000-neuron Reservoir. These are equivalent or superior to state of the art in LSM classification performance. Using time partitions further pushes up the test classification accuracy of Neuromorphic-MNIST to 98% and of TI-46 speech classification dataset to 92% which approaches Artificial Neural Network (ANN) and Convolutional Neural Network (CNN) levels of performance respectively. This work not only simplifies the task of identifying optimal parameters for the functioning of an LSM across multiple input modalities, but also unlocks the generalizing power of using the same Reservoir network for multiple tasks