System Notification:
We are currently experiencing intermittent issues impacting performance. We apologize for the inconvenience.
By Topic

Neural Networks, IEEE Transactions on

Issue 8 • Date Aug. 2008

Filter Results

Displaying Results 1 - 21 of 21
  • Table of contents

    Page(s): C1 - C4
    Save to Project icon | Request Permissions | PDF file iconPDF (37 KB)  
    Freely Available from IEEE
  • IEEE Transactions on Neural Networks publication information

    Page(s): C2
    Save to Project icon | Request Permissions | PDF file iconPDF (36 KB)  
    Freely Available from IEEE
  • Dynamics of Learning in Multilayer Perceptrons Near Singularities

    Page(s): 1313 - 1328
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (2066 KB) |  | HTML iconHTML  

    The dynamical behavior of learning is known to be very slow for the multilayer perceptron, being often trapped in the "plateau." It has been recently understood that this is due to the singularity in the parameter space of perceptrons, in which trajectories of learning are drawn. The space is Riemannian from the point of view of information geometry and contains singular regions where the Riemannian metric or the Fisher information matrix degenerates. This paper analyzes the dynamics of learning in a neighborhood of the singular regions when the true teacher machine lies at the singularity. We give explicit asymptotic analytical solutions (trajectories) both for the standard gradient (SGD) and natural gradient (NGD) methods. It is clearly shown, in the case of the SGD method, that the plateau phenomenon appears in a neighborhood of the critical regions, where the dynamical behavior is extremely slow. The analysis of the NGD method is much more difficult, because the inverse of the Fisher information matrix diverges. We conquer the difficulty by introducing the "blow-down" technique used in algebraic geometry. The NGD method works efficiently, and the state converges directly to the true parameters very quickly while it staggers in the case of the SGD method. The analytical results are compared with computer simulations, showing good agreement. The effects of singularities on learning are thus qualitatively clarified for both standard and NGD methods. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Robust State Estimation for Uncertain Neural Networks With Time-Varying Delay

    Page(s): 1329 - 1339
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (786 KB) |  | HTML iconHTML  

    The robust state estimation problem for a class of uncertain neural networks with time-varying delay is studied in this paper. The parameter uncertainties are assumed to be norm bounded. Based on a new bounding technique, a sufficient condition is presented to guarantee the existence of the desired state estimator for the uncertain delayed neural networks. The criterion is dependent on the size of the time-varying delay and on the size of the time derivative of the time-varying delay. It is shown that the design of the robust state estimator for such neural networks can be achieved by solving a linear matrix inequality (LMI), which can be easily facilitated by using some standard numerical packages. Finally, two simulation examples are given to demonstrate the effectiveness of the developed approach. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • A Novel Recurrent Neural Network for Solving Nonlinear Optimization Problems With Inequality Constraints

    Page(s): 1340 - 1353
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (708 KB) |  | HTML iconHTML  

    This paper presents a novel recurrent neural network for solving nonlinear optimization problems with inequality constraints. Under the condition that the Hessian matrix of the associated Lagrangian function is positive semidefinite, it is shown that the proposed neural network is stable at a Karush-Kuhn-Tucker point in the sense of Lyapunov and its output trajectory is globally convergent to a minimum solution. Compared with variety of the existing projection neural networks, including their extensions and modification, for solving such nonlinearly constrained optimization problems, it is shown that the proposed neural network can solve constrained convex optimization problems and a class of constrained nonconvex optimization problems and there is no restriction on the initial point. Simulation results show the effectiveness of the proposed neural network in solving nonlinearly constrained optimization problems. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Individual Stable Space: An Approach to Face Recognition Under Uncontrolled Conditions

    Page(s): 1354 - 1368
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (1745 KB) |  | HTML iconHTML  

    There usually exist many kinds of variations in face images taken under uncontrolled conditions, such as changes of pose, illumination, expression, etc. Most previous works on face recognition (FR) focus on particular variations and usually assume the absence of others. Instead of such a ldquodivide and conquerrdquo strategy, this paper attempts to directly address face recognition under uncontrolled conditions. The key is the individual stable space (ISS), which only expresses personal characteristics. A neural network named ISNN is proposed to map a raw face image into the ISS. After that, three ISS-based algorithms are designed for FR under uncontrolled conditions. There are no restrictions for the images fed into these algorithms. Moreover, unlike many other FR techniques, they do not require any extra training information, such as the view angle. These advantages make them practical to implement under uncontrolled conditions. The proposed algorithms are tested on three large face databases with vast variations and achieve superior performance compared with other 12 existing FR techniques. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Reinforcement-Learning-Based Dual-Control Methodology for Complex Nonlinear Discrete-Time Systems With Application to Spark Engine EGR Operation

    Page(s): 1369 - 1388
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (1579 KB)  

    A novel reinforcement-learning-based dual-control methodology adaptive neural network (NN) controller is developed to deliver a desired tracking performance for a class of complex feedback nonlinear discrete-time systems, which consists of a second-order nonlinear discrete-time system in nonstrict feedback form and an affine nonlinear discrete-time system, in the presence of bounded and unknown disturbances. For example, the exhaust gas recirculation (EGR) operation of a spark ignition (SI) engine is modeled by using such a complex nonlinear discrete-time system. A dual-controller approach is undertaken where primary adaptive critic NN controller is designed for the nonstrict feedback nonlinear discrete-time system whereas the secondary one for the affine nonlinear discrete-time system but the controllers together offer the desired performance. The primary adaptive critic NN controller includes an NN observer for estimating the states and output, an NN critic, and two action NNs for generating virtual control and actual control inputs for the nonstrict feedback nonlinear discrete-time system, whereas an additional critic NN and an action NN are included for the affine nonlinear discrete-time system by assuming the state availability. All NN weights adapt online towards minimization of a certain performance index, utilizing gradient-descent-based rule. Using Lyapunov theory, the uniformly ultimate boundedness (UUB) of the closed-loop tracking error, weight estimates, and observer estimates are shown. The adaptive critic NN controller performance is evaluated on an SI engine operating with high EGR levels where the controller objective is to reduce cyclic dispersion in heat release while minimizing fuel intake. Simulation and experimental results indicate that engine out emissions drop significantly at 20% EGR due to reduction in dispersion in heat release thus verifying the dual-control approach. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • A New Approach to Knowledge-Based Design of Recurrent Neural Networks

    Page(s): 1389 - 1401
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (463 KB) |  | HTML iconHTML  

    A major drawback of artificial neural networks (ANNs) is their black-box character. This is especially true for recurrent neural networks (RNNs) because of their intricate feedback connections. In particular, given a problem and some initial information concerning its solution, it is not at all obvious how to design an RNN that is suitable for solving this problem. In this paper, we consider a fuzzy rule base with a special structure, referred to as the fuzzy all-permutations rule base (FARB). Inferring the FARB yields an input-output (IO) mapping that is mathematically equivalent to that of an RNN. We use this equivalence to develop two new knowledge-based design methods for RNNs. The first method, referred to as the direct approach, is based on stating the desired functioning of the RNN in terms of several sets of symbolic rules, each one corresponding to a subnetwork. Each set is then transformed into a suitable FARB. The second method is based on first using the direct approach to design a library of simple modules, such as counters or comparators, and realize them using RNNs. Once designed, the correctness of each RNN can be verified. Then, the initial design problem is solved by using these basic modules as building blocks. This yields a modular and systematic approach for knowledge-based design of RNNs. We demonstrate the efficiency of these approaches by designing RNNs that recognize both regular and nonregular formal languages. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • A Neural Model for Compensation of Sensory Abnormalities in Autism Through Feedback From a Measure of Global Perception

    Page(s): 1402 - 1414
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (395 KB) |  | HTML iconHTML  

    Sensory abnormalities and weak central coherence (WCC), a processing bias for features and local information, are important characteristics associated with autism. This paper introduces a self-organizing map (SOM)-based computational model of sensory abnormalities in autism, and of a feedback system to compensate for them. Feedback relies on a measure of balance of coverage over four (sensory) domains. Different methods to compute this measure are discussed, as is the flexibility to configure the system using different control mechanisms. Statistically significant improvements throughout training are demonstrated for compensation of a simple (i.e., monotonically decreasing) hypersensitivity in one of the domains. Fine-tuning control parameters can lead to further gains, but a standard setup results in good performance. Significant improvements are also shown for complex hypersensitivities (i.e., increasing and decreasing through time) in two domains. Although naturally best suited to compensate hypersensitivities-stimuli filtering may mitigate neuron migration to a hypersensitive domain-the system is also shown to perform effectively when compensating hyposensitivities. With poor coverage balance in the model akin to poor global perception, WCC would be consistent with inadequate feedback, resulting in uncontrolled hyper- and/or hyposensitivities characteristic of autism, as seen in the topologies of the resulting SOMs. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • The Q -Norm Complexity Measure and the Minimum Gradient Method: A Novel Approach to the Machine Learning Structural Risk Minimization Problem

    Page(s): 1415 - 1430
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (762 KB) |  | HTML iconHTML  

    This paper presents a novel approach for dealing with the structural risk minimization (SRM) applied to a general setting of the machine learning problem. The formulation is based on the fundamental concept that supervised learning is a bi-objective optimization problem in which two conflicting objectives should be minimized. The objectives are related to the empirical training error and the machine complexity. In this paper, one general Q-norm method to compute the machine complexity is presented, and, as a particular practical case, the minimum gradient method (MGM) is derived relying on the definition of the fat-shattering dimension. A practical mechanism for parallel layer perceptron (PLP) network training, involving only quasi-convex functions, is generated using the aforementioned definitions. Experimental results on 15 different benchmarks are presented, which show the potential of the proposed ideas. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Configuration of Continuous Piecewise-Linear Neural Networks

    Page(s): 1431 - 1445
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (919 KB) |  | HTML iconHTML  

    The problem of constructing a general continuous piecewise-linear neural network is considered in this paper. It is shown that every projection domain of an arbitrary continuous piecewise-linear function can be partitioned into convex polyhedra by using difference functions of its local linear functions. Based on these convex polyhedra, a group of continuous piecewise-linear basis functions are formulated. It is proven that a linear combination of these basis functions plus a constant, which we call a standard continuous piecewise-linear neural network, can represent all continuous piecewise-linear functions. In addition, the proposed standard continuous piecewise-linear neural network is applied to solve some function approximation problems. A number of numerical experiments are presented to illustrate that the standard continuous piecewise-linear neural network can be a promising tool for function approximation. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Training Hard-Margin Support Vector Machines Using Greedy Stagewise Algorithm

    Page(s): 1446 - 1455
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (696 KB)  

    Hard-margin support vector machines (HM-SVMs) suffer from getting overfitting in the presence of noise. Soft-margin SVMs deal with this problem by introducing a regularization term and obtain a state-of-the-art performance. However, this disposal leads to a relatively high computational cost. In this paper, an alternative method, greedy stagewise algorithm for SVMs, named GS-SVMs, is presented to cope with the overfitting of HM-SVMs without employing the regularization term. The most attractive property of GS-SVMs is that its computational complexity in the worst case only scales quadratically with the size of training samples. Experiments on the large data sets with up to 400 000 training samples demonstrate that GS-SVMs can be faster than LIBSVM 2.83 without sacrificing the accuracy. Finally, we employ statistical learning theory to analyze the empirical results, which shows that the success of GS-SVMs lies in that its early stopping rule can act as an implicit regularization term. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Deterministic Learning for Maximum-Likelihood Estimation Through Neural Networks

    Page(s): 1456 - 1467
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (574 KB) |  | HTML iconHTML  

    In this paper, a general method for the numerical solution of maximum-likelihood estimation (MLE) problems is presented; it adopts the deterministic learning (DL) approach to find close approximations to ML estimator functions for the unknown parameters of any given density. The method relies on the choice of a proper neural network and on the deterministic generation of samples of observations of the likelihood function, thus avoiding the problem of generating samples with the unknown density. Under mild assumptions, consistency and convergence with favorable rates to the true ML estimator function can be proved. Simulation results are provided to show the good behavior of the algorithm compared to the corresponding exact solutions. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Visualization of Tree-Structured Data Through Generative Topographic Mapping

    Page(s): 1468 - 1493
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (3406 KB)  

    In this paper, we present a probabilistic generative approach for constructing topographic maps of tree-structured data. Our model defines a low-dimensional manifold of local noise models, namely, (hidden) Markov tree models, induced by a smooth mapping from low-dimensional latent space. We contrast our approach with that of topographic map formation using recursive neural-based techniques, namely, the self-organizing map for structured data (SOMSD) (Hagenbuchner et al., 2003). The probabilistic nature of our model brings a number of benefits: (1) naturally defined cost function that drives the model optimization; (2) principled model comparison and testing for overfitting; (3) a potential for transparent interpretation of the map by inspecting the underlying local noise models; (4) natural accommodation of alternative local noise models implicitly expressing different notions of structured data similarity. Furthermore, in contrast with the recursive neural-based approaches, the smooth nature of the mapping from the latent space to the local model space allows for calculation of magnification factors-a useful tool for the detection of data clusters. We demonstrate our approach on three data sets: a toy data set, an artificially generated data set, and on a data set of images represented as quadtrees. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Comments on "The Extreme Learning Machine

    Page(s): 1494 - 1495
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (65 KB)  

    This comment letter points out that the essence of the ldquoextreme learning machine (ELM)rdquo recently appeared has been proposed earlier by Broomhead and Lowe and Pao , and discussed by other authors. Hence, it is not necessary to introduce a new name "ELM". View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Reply to “Comments on “The Extreme Learning Machine””

    Page(s): 1495 - 1496
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (79 KB) |  | HTML iconHTML  

    In this reply, we refer to Wang and Wan's comments on our research publication. We found that the comment letter contains some inaccurate statements. The comment letter contains some contradictions as well. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Comments on "Adaptive Neural Control for a Class of Nonlinearly Parametric Time-Delay Systems

    Page(s): 1496 - 1498
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (115 KB)  

    In this comment, we point out an error in [1], which will show that the main result of the paper cannot be generalized for nonlinearly parametric time-delay systems considered in [1]. In [1], the authors considered the problem of the adaptive control for a class of nonlinearly parametric time-delay systems and applied successfully the proposed approach to second-order nonlinear time-delay systems with the term x1(t - tau)x2(t - tau) used as simulation examples in Section VI. However, we show in this comment that the control approach presented in [1] cannot be generalized although it is realizable for nonlinear second-order time-delay systems with the term x1(t - tau ) x2 (t - tau). In addition, we illustrate this error by one detail example. For simplicity, all the symbols in this comment are the same as those in [1]. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Reply to “Comments on “Adaptive Neural Control for a Class of Nonlinearly Parametric Time-Delay Systems””

    Page(s): 1498
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (54 KB)  

    For original paper see D. W. C. Ho et al., ibid., vol.16, no.3, p.625-35, (2005). For original paper see S. J. Yoo et al., ibid., vol.19, no.8, p.1496-8, (2008). This paper presents the reply to "Comments on ldquoAdaptive neural control for a class of nonlinearly parametric time-delay systemsrdquordquo. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Complexity Explained (Erdi, P.; 2008) [Book review]

    Page(s): 1499
    Save to Project icon | Request Permissions | PDF file iconPDF (27 KB)  
    Freely Available from IEEE
  • Special issue on computing with words

    Page(s): 1500
    Save to Project icon | Request Permissions | PDF file iconPDF (123 KB)  
    Freely Available from IEEE
  • IEEE Computational Intelligence Society Information

    Page(s): C3
    Save to Project icon | Request Permissions | PDF file iconPDF (34 KB)  
    Freely Available from IEEE

Aims & Scope

IEEE Transactions on Neural Networks is devoted to the science and technology of neural networks, which disclose significant technical knowledge, exploratory developments, and applications of neural networks from biology to software to hardware.

 

This Transactions ceased production in 2011. The current retitled publication is IEEE Transactions on Neural Networks and Learning Systems.

Full Aims & Scope