By Topic

Neural Networks, IEEE Transactions on

Issue 6 • Date June 2010

Filter Results

Displaying Results 1 - 21 of 21
  • Table of contents

    Page(s): C1
    Save to Project icon | Request Permissions | PDF file iconPDF (34 KB)  
    Freely Available from IEEE
  • IEEE Transactions on Neural Networks publication information

    Page(s): C2
    Save to Project icon | Request Permissions | PDF file iconPDF (38 KB)  
    Freely Available from IEEE
  • File Access Prediction Using Neural Networks

    Page(s): 869 - 882
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (1070 KB) |  | HTML iconHTML  

    One of the most vexing issues in design of a high-speed computer is the wide gap of access times between the memory and the disk. To solve this problem, static file access predictors have been used. In this paper, we propose dynamic file access predictors using neural networks to significantly improve upon the accuracy, success-per-reference, and effective-success-rate-per-reference by using neural-network-based file access predictor with proper tuning. In particular, we verified that the incorrect prediction has been reduced from 53.11% to 43.63% for the proposed neural network prediction method with a standard configuration than the recent popularity (RP) method. With manual tuning for each trace, we are able to improve upon the misprediction rate and effective-success-rate-per-reference using a standard configuration. Simulations on distributed file system (DFS) traces reveal that exact fit radial basis function (RBF) gives better prediction in high end system whereas multilayer perceptron (MLP) trained with Levenberg-Marquardt (LM) backpropagation outperforms in system having good computational capability. Probabilistic and competitive predictors are the most suitable for work stations having limited resources to deal with and the former predictor is more efficient than the latter for servers having maximum system calls. Finally, we conclude that MLP with LM backpropagation algorithm has better success rate of file prediction than those of simple perceptron, last successor, stable successor, and best k out of m predictors. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Sparse Approximation Through Boosting for Learning Large Scale Kernel Machines

    Page(s): 883 - 894
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (764 KB) |  | HTML iconHTML  

    Recently, sparse approximation has become a preferred method for learning large scale kernel machines. This technique attempts to represent the solution with only a subset of original data points also known as basis vectors, which are usually chosen one by one with a forward selection procedure based on some selection criteria. The computational complexity of several resultant algorithms scales as O(NM2) in time and O(NM) in memory, where N is the number of training points and M is the number of basis vectors as well as the steps of forward selection. For some large scale data sets, to obtain a better solution, we are sometimes required to include more basis vectors, which means that M is not trivial in this situation. However, the limited computational resource (e.g., memory) prevents us from including too many vectors. To handle this dilemma, we propose to add an ensemble of basis vectors instead of only one at each forward step. The proposed method, closely related to gradient boosting, could decrease the required number M of forward steps significantly and thus a large fraction of computational cost is saved. Numerical experiments on three large scale regression tasks and a classification problem demonstrate the effectiveness of the proposed approach. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Self-Sustained Irregular Activity in 2-D Small-World Networks of Excitatory and Inhibitory Neurons

    Page(s): 895 - 905
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (1214 KB) |  | HTML iconHTML  

    In this paper, we study the self-sustained irregular firing activity in 2-D small-world (SW) neural networks consisting of both excitatory and inhibitory neurons by computational modeling. For a proper proportion of unidirectional shortcuts, the stable self-sustained activity with irregular firing states indeed occurs in the considered network. By varying the shortcut density while keeping other system parameters fixed, different levels of irregular firing states, from weakly irregular to Poisson-like and burst firing states, are obtained in 2-D SW neural networks. It is also observed that this activity is sensitive to small perturbations, which might provide a possible mechanism for producing chaos. On the other hand, we find that several other system parameters, such as the network size and refractory period, have significant impact on this activity. Further simulation results show that the 2-D SW neural network can sustain such long-lasting firing behavior by using a smaller number of connections than the random neural network. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Monotone and Partially Monotone Neural Networks

    Page(s): 906 - 917
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (718 KB) |  | HTML iconHTML  

    In many classification and prediction problems it is known that the response variable depends on certain explanatory variables. Monotone neural networks can be used as powerful tools to build monotone models with better accuracy and lower variance compared to ordinary nonmonotone models. Monotonicity is usually obtained by putting constraints on the parameters of the network. In this paper, we will clarify some of the theoretical results on monotone neural networks with positive weights, issues that are sometimes misunderstood in the neural network literature. Furthermore, we will generalize some of the results obtained by Sill for the so-called min-max networks to the case of partially monotone problems. The method is illustrated in practical case studies. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • A New One-Layer Neural Network for Linear and Quadratic Programming

    Page(s): 918 - 929
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (619 KB) |  | HTML iconHTML  

    In this paper, we present a new neural network for solving linear and quadratic programming problems in real time by introducing some new vectors. The proposed neural network is stable in the sense of Lyapunov and can converge to an exact optimal solution of the original problem when the objective function is convex on the set defined by equality constraints. Compared with existing one-layer neural networks for quadratic programming problems, the proposed neural network has the least neurons and requires weak stability conditions. The validity and transient behavior of the proposed neural network are demonstrated by some simulation results. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Improved Computation for Levenberg–Marquardt Training

    Page(s): 930 - 937
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (645 KB) |  | HTML iconHTML  

    The improved computation presented in this paper is aimed to optimize the neural networks learning process using Levenberg-Marquardt (LM) algorithm. Quasi-Hessian matrix and gradient vector are computed directly, without Jacobian matrix multiplication and storage. The memory limitation problem for LM training is solved. Considering the symmetry of quasi-Hessian matrix, only elements in its upper/lower triangular array need to be calculated. Therefore, training speed is improved significantly, not only because of the smaller array stored in memory, but also the reduced operations in quasi-Hessian matrix calculation. The improved memory and time efficiencies are especially true for large sized patterns training. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Convergence and Objective Functions of Some Fault/Noise-Injection-Based Online Learning Algorithms for RBF Networks

    Page(s): 938 - 947
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (405 KB) |  | HTML iconHTML  

    In the last two decades, many online fault/noise injection algorithms have been developed to attain a fault tolerant neural network. However, not much theoretical works related to their convergence and objective functions have been reported. This paper studies six common fault/noise-injection-based online learning algorithms for radial basis function (RBF) networks, namely 1) injecting additive input noise, 2) injecting additive/multiplicative weight noise, 3) injecting multiplicative node noise, 4) injecting multiweight fault (random disconnection of weights), 5) injecting multinode fault during training, and 6) weight decay with injecting multinode fault. Based on the Gladyshev theorem, we show that the convergence of these six online algorithms is almost sure. Moreover, their true objective functions being minimized are derived. For injecting additive input noise during training, the objective function is identical to that of the Tikhonov regularizer approach. For injecting additive/multiplicative weight noise during training, the objective function is the simple mean square training error. Thus, injecting additive/multiplicative weight noise during training cannot improve the fault tolerance of an RBF network. Similar to injective additive input noise, the objective functions of other fault/noise-injection-based online algorithms contain a mean square error term and a specialized regularization term. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Adaptive FIR Neural Model for Centroid Learning in Self-Organizing Maps

    Page(s): 948 - 960
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (2746 KB) |  | HTML iconHTML  

    In this paper, a training method for the formation of topology preserving maps is introduced. The proposed approach presents a sequential formulation of the self-organizing map (SOM), which is based on a new model of the neuron, or processing unit. Each neuron acts as a finite impulse response (FIR) system, and the coefficients of the filters are adaptively estimated during the sequential learning process, in order to minimize a distortion measure of the map. The proposed FIR-SOM model deals with static distributions and it computes an ordered set of centroids. Additionally, the FIR-SOM estimates the learning dynamic of each prototype using an adaptive FIR model. A noteworthy result is that the optimized coefficients of the FIR processes tend to represent a moving average filter, regardless of the underlying input distribution. The convergence of the resulting model is analyzed numerically and shows good properties with respect to the classic SOM and other unsupervised neural models. Finally, the optimal FIR coefficients are shown to be useful for visualizing the cluster densities. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Realization of the Conscience Mechanism in CMOS Implementation of Winner-Takes-All Self-Organizing Neural Networks

    Page(s): 961 - 971
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (1409 KB) |  | HTML iconHTML  

    This paper presents a complementary metal-oxide-semiconductor (CMOS) implementation of a conscience mechanism used to improve the effectiveness of learning in the winner-takes-all (WTA) artificial neural networks (ANNs) realized at the transistor level. This mechanism makes it possible to eliminate the effect of the so-called ??dead neurons,?? which do not take part in the learning phase competition. These neurons usually have a detrimental effect on the network performance, increasing the quantization error. The proposed mechanism comes as part of the analog implementation of the WTA neural networks (NNs) designed for applications to ultralow power portable diagnostic devices for online analysis of ECG biomedical signals. The study presents Matlab simulations of the network's model, discusses postlayout circuit level simulations and includes results of measurement completed for the physical realization of the circuit. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Novel Maximum-Margin Training Algorithms for Supervised Neural Networks

    Page(s): 972 - 984
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (952 KB) |  | HTML iconHTML  

    This paper proposes three novel training methods, two of them based on the backpropagation approach and a third one based on information theory for multilayer perceptron (MLP) binary classifiers. Both backpropagation methods are based on the maximal-margin (MM) principle. The first one, based on the gradient descent with adaptive learning rate algorithm (GDX) and named maximum-margin GDX (MMGDX), directly increases the margin of the MLP output-layer hyperplane. The proposed method jointly optimizes both MLP layers in a single process, backpropagating the gradient of an MM-based objective function, through the output and hidden layers, in order to create a hidden-layer space that enables a higher margin for the output-layer hyperplane, avoiding the testing of many arbitrary kernels, as occurs in case of support vector machine (SVM) training. The proposed MM-based objective function aims to stretch out the margin to its limit. An objective function based on Lp-norm is also proposed in order to take into account the idea of support vectors, however, overcoming the complexity involved in solving a constrained optimization problem, usually in SVM training. In fact, all the training methods proposed in this paper have time and space complexities O(N) while usual SVM training methods have time complexity O(N 3) and space complexity O(N 2) , where N is the training-data-set size. The second approach, named minimization of interclass interference (MICI), has an objective function inspired on the Fisher discriminant analysis. Such algorithm aims to create an MLP hidden output where the patterns have a desirable statistical distribution. In both training methods, the maximum area under ROC curve (AUC) is applied as stop criterion. The third approach offers a robust training framework able to take the best of each proposed training method. The main idea is to compose a neural model by using neurons ex- - tracted from three other neural networks, each one previously trained by MICI, MMGDX, and Levenberg-Marquard (LM), respectively. The resulting neural network was named assembled neural network (ASNN). Benchmark data sets of real-world problems have been used in experiments that enable a comparison with other state-of-the-art classifiers. The results provide evidence of the effectiveness of our methods regarding accuracy, AUC, and balanced error rate. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • A BCM Theory of Meta-Plasticity for Online Self-Reorganizing Fuzzy-Associative Learning

    Page(s): 985 - 1003
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (2600 KB) |  | HTML iconHTML  

    Self-organizing neurofuzzy approaches have matured in their online learning of fuzzy-associative structures under time-invariant conditions. To maximize their operative value for online reasoning, these self-sustaining mechanisms must also be able to reorganize fuzzy-associative knowledge in real-time dynamic environments. Hence, it is critical to recognize that they would require self-reorganizational skills to rebuild fluid associative structures when their existing organizations fail to respond well to changing circumstances. In this light, while Hebbian theory (Hebb, 1949) is the basic computational framework for associative learning, it is less attractive for time-variant online learning because it suffers from stability limitations that impedes unlearning. Instead, this paper adopts the Bienenstock-Cooper-Munro (BCM) theory of neurological learning via meta-plasticity principles (Bienenstock et al., 1982) that provides for both online associative and dissociative learning. For almost three decades, BCM theory has been shown to effectively brace physiological evidence of synaptic potentiation (association) and depression (dissociation) into a sound mathematical framework for computational learning. This paper proposes an interpretation of the BCM theory of meta-plasticity for an online self-reorganizing fuzzy-associative learning system to realize online-reasoning capabilities. Experimental findings are twofold: 1) the analysis using S&P-500 stock index illustrated that the self-reorganizing approach could follow the trajectory shifts in the time-variant S&P-500 index for about 60 years, and 2) the benchmark profiles showed that the fuzzy-associative approach yielded comparable results with other fuzzy-precision models with similar online objectives. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • The Infinite Hidden Markov Random Field Model

    Page(s): 1004 - 1014
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (933 KB) |  | HTML iconHTML  

    Hidden Markov random field (HMRF) models are widely used for image segmentation, as they appear naturally in problems where a spatially constrained clustering scheme is asked for. A major limitation of HMRF models concerns the automatic selection of the proper number of their states, i.e., the number of region clusters derived by the image segmentation procedure. Existing methods, including likelihood- or entropy-based criteria, and reversible Markov chain Monte Carlo methods, usually tend to yield noisy model size estimates while imposing heavy computational requirements. Recently, Dirichlet process (DP, infinite) mixture models have emerged in the cornerstone of nonparametric Bayesian statistics as promising candidates for clustering applications where the number of clusters is unknown a priori; infinite mixture models based on the original DP or spatially constrained variants of it have been applied in unsupervised image segmentation applications showing promising results. Under this motivation, to resolve the aforementioned issues of HMRF models, in this paper, we introduce a nonparametric Bayesian formulation for the HMRF model, the infinite HMRF model, formulated on the basis of a joint Dirichlet process mixture (DPM) and Markov random field (MRF) construction. We derive an efficient variational Bayesian inference algorithm for the proposed model, and we experimentally demonstrate its advantages over competing methodologies. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Inference From Aging Information

    Page(s): 1015 - 1020
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (232 KB) |  | HTML iconHTML  

    For many learning tasks the duration of the data collection can be greater than the time scale for changes of the underlying data distribution. The question we ask is how to include the information that data are aging. Ad hoc methods to achieve this include the use of validity windows that prevent the learning machine from making inferences based on old data. This introduces the problem of how to define the size of validity windows. In this brief, a new adaptive Bayesian inspired algorithm is presented for learning drifting concepts. It uses the analogy of validity windows in an adaptive Bayesian way to incorporate changes in the data distribution over time. We apply a theoretical approach based on information geometry to the classification problem and measure its performance in simulations. The uncertainty about the appropriate size of the memory windows is dealt with in a Bayesian manner by integrating over the distribution of the adaptive window size. Thus, the posterior distribution of the weights may develop algebraic tails. The learning algorithm results from tracking the mean and variance of the posterior distribution of the weights. It was found that the algebraic tails of this posterior distribution give the learning algorithm the ability to cope with an evolving environment by permitting the escape from local traps. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Discriminant Analysis for Fast Multiclass Data Classification Through Regularized Kernel Function Approximation

    Page(s): 1020 - 1029
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (1058 KB) |  | HTML iconHTML  

    In this brief we have proposed the multiclass data classification by computationally inexpensive discriminant analysis through vector-valued regularized kernel function approximation (VVRKFA). VVRKFA being an extension of fast regularized kernel function approximation (FRKFA), provides the vector-valued response at single step. The VVRKFA finds a linear operator and a bias vector by using a reduced kernel that maps a pattern from feature space into the low dimensional label space. The classification of patterns is carried out in this low dimensional label subspace. A test pattern is classified depending on its proximity to class centroids. The effectiveness of the proposed method is experimentally verified and compared with multiclass support vector machine (SVM) on several benchmark data sets as well as on gene microarray data for multi-category cancer classification. The results indicate the significant improvement in both training and testing time compared to that of multiclass SVM with comparable testing accuracy principally in large data sets. Experiments in this brief also serve as comparison of performance of VVRKFA with stratified random sampling and sub-sampling. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • 2011 International Joint Conference on Neural Networks

    Page(s): 1030
    Save to Project icon | Request Permissions | PDF file iconPDF (870 KB)  
    Freely Available from IEEE
  • 2010 IEEE World Congress on Computational Intelligence

    Page(s): 1031
    Save to Project icon | Request Permissions | PDF file iconPDF (2434 KB)  
    Freely Available from IEEE
  • Scitopia.org [advertisement]

    Page(s): 1032
    Save to Project icon | Request Permissions | PDF file iconPDF (269 KB)  
    Freely Available from IEEE
  • IEEE Computational Intelligence Society Information

    Page(s): C3
    Save to Project icon | Request Permissions | PDF file iconPDF (37 KB)  
    Freely Available from IEEE
  • IEEE Transactions on Neural Networks Information for authors

    Page(s): C4
    Save to Project icon | Request Permissions | PDF file iconPDF (39 KB)  
    Freely Available from IEEE

Aims & Scope

IEEE Transactions on Neural Networks is devoted to the science and technology of neural networks, which disclose significant technical knowledge, exploratory developments, and applications of neural networks from biology to software to hardware.

 

This Transactions ceased production in 2011. The current retitled publication is IEEE Transactions on Neural Networks and Learning Systems.

Full Aims & Scope