Extended Features Based Random Vector Functional Link Network for Classification Problem

Random vector functional link (RVFL) network has been successfully employed in diverse domains such as computer vision and machine learning, due to its universal approximation capability. Recently, the shallow RVFL architecture has been extended to deep architectures. In deep architectures, multiple hidden layers are stacked for extracting informative features from the original feature space. Therefore, having rich features, deep models are very successful compared to shallow models. In this article, we propose an extended feature RVFL (efRVFL) model that is trained over extended feature space generated analytically from the original feature space. The proposed efRVFL model has three types of features, i.e., original features, supervised randomized (newly generated) features, and unsupervised randomized features, in its feature matrix. The proposed efRVFL model with additional features has capability to capture nonlinear hidden relationships within the dataset. The proposed efRVFL model is an unstable classifier, and thus, its performance can be improved further via ensemble learning. Ensemble models are stable and accurate and have better generalization performance than single models. Therefore, we also propose an ensemble of extended feature RVFL (en-efRVFL) model. Each base model of en-efRVFL is trained over different feature spaces so that more accurate and diverse base models can be generated. The outcome of the base models is integrated via average voting scheme. Empirical evaluation over 46 UCI classification datasets demonstrates that the proposed efRVFL and en-efRVFL models have better performance than RVFL and other given deep models. Furthermore, the experimental results over 12 sparse datasets show that the proposed en-efRVFL model has a winning performance among several deep feedforward neural networks (FNNs).

I. Introduction

The artificial neural network (ANN), such as single hidden layer feedforward neural (SLFN) networks, has been successfully employed for classification and regression problems [1], [2] due to their universal approximation capability [3], [4]. One classical learning technique to train SLFN is backpropagation (BP) algorithm [5], which usually suffers from slow convergence, local minima problem, and sensitive to learning rate. To alleviate these problems, randomization-based algorithms, such as Schmidt et al. [6] network, random vector functional link (RVFL) neural network (NN) [7], extreme learning machine (ELM) [8], and radial basis function network (RBFN) [9], [10], have been proposed. Pao et al. [7] and Pao and Takefuji [11] proposed the RVFL network, which has simple architecture and efficient performance. The training process of RVFL model has two stages. In the first stage, all the weights and biases from the input layer to the hidden layer are generated randomly within a given domain [12] and are fixed throughout the training phase. In the second stage, the output parameters are determined analytically via a closed-form solution [13], [14]. The RVFL network is fast in training and has good generalization performance. Thus, it has been employed in several areas, such as epileptic seizure classification [15], time series forecasting [16], daily crude oil price forecasting [17], and nonlinear system identification [18].