By Topic

Neural Networks, IEEE Transactions on

Issue 1 • Date Jan 1995

Filter Results

Displaying Results 1 - 25 of 33
  • Neighborhood sequential and random training techniques for CMAC

    Page(s): 196 - 202
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (676 KB)  

    An adaptive control algorithm based on Albus' CMAC (Cerebellar Model Articulation Controller) was studied with emphasis on how to train CMAC systems. Two training techniques-neighborhood sequential training and random training, have been devised. These techniques were used to generate mathematical functions, and both methods successfully circumvented the training interference resulting from CMAC's inherent generalization property. In the neighborhood sequential training method, a strategy was devised to utilize the discrete, finite state nature of the CMAC's address space for selecting points in the input space which would train CMAC systems in the most rapid manner possible. The random training method was found to converge on the training function with the greatest precision, although it requires longer training periods than the neighborhood sequential training method to achieve a desired performance level View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Asymptotic level density in topological feature maps

    Page(s): 230 - 236
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (608 KB)  

    The Kohonen algorithm entails a topology conserving mapping of an input pattern space X⊂Rn characterized by an a priori probability distribution P(x), x∈X, onto a discrete lattice of neurons r with virtual positions wr∈X. Extending results obtained by Ritter (1991) the authors show in the one-dimensional case for an arbitrary monotonously decreasing neighborhood function h(|r-r'|) that the point density D(Wr) of the virtual net is a polynomial function of the probability density P(x) with D(wr)~Pα(wr). Here the distortion exponent is given by α=(1+12R)/3(1+6R) and is determined by the normalized second moment R of the neighborhood function. A Gaussian neighborhood interaction is discussed and the analytical results are checked by means of computer simulations View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Canonical piecewise-linear networks

    Page(s): 43 - 50
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (852 KB)  

    In this paper, mapping networks will be considered from the viewpoint of the piecewise-linear (PWL) approximation. The so-called canonical representation plays a kernel role in the PWL representation theory. While this theory has been researched intensively in the contents of mathematics and circuit simulations, little has been seen in the research area about the theoretical aspect of neural networks. This paper modifies this theory and applies it as a mathematical support for mapping networks. The main modification is a “higher-level” generalization of the canonical representation with proofs of its availability in the set of PWL functions. The modified theory will first be used to study the canonical PWL feature of the popular multilayer perceptron-like (MLPL) networks. Second, it will be seen that the generalized canonical representation is itself suitable for a network implementation, which is called the standard canonical PWL network. More generally, the family of (generalized) canonical PWL networks is defined as those which may take the canonical PWL representation as a mathematical model. This family is large and practically meaningful. The standard canonical PWL networks may be taken as representatives in the family. The modification of the PWL representation theory as well as the introduction of this theory in the theoretical study of mapping networks, which provide a new concept of mapping networks, i.e., the canonical PWL network family, may be regarded as the main contributions of the paper View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Analog optimization with Wong's stochastic neural network

    Page(s): 258 - 260
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (196 KB)  

    We describe E. Wong's stochastic neural network (1989) and show that it can be used, in principle, to perform analog optimization. The optimization dynamics are analogous to those of simulated annealing. To show this, we use the theory developed in Holley and Stroock (1988) for the continuous-time simulated annealing process View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Diagonal recurrent neural networks for dynamic systems control

    Page(s): 144 - 156
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (880 KB)  

    A new neural paradigm called diagonal recurrent neural network (DRNN) is presented. The architecture of DRNN is a modified model of the fully connected recurrent neural network with one hidden layer, and the hidden layer comprises self-recurrent neurons. Two DRNN's are utilized in a control system, one as an identifier called diagonal recurrent neuroidentifier (DRNI) and the other as a controller called diagonal recurrent neurocontroller (DRNC). A controlled plant is identified by the DRNI, which then provides the sensitivity information of the plant to the DRNC. A generalized dynamic backpropagation algorithm (DBP) is developed and used to train both DRNC and DRNI. Due to the recurrence, the DRNN can capture the dynamic behavior of a system. To guarantee convergence and for faster learning, an approach that uses adaptive learning rates is developed by introducing a Lyapunov function. Convergence theorems for the adaptive backpropagation algorithms are developed for both DRNI and DRNC. The proposed DRNN paradigm is applied to numerical problems and the simulation results are included View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Existence and uniqueness results for neural network approximations

    Page(s): 2 - 13
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (916 KB)  

    Some approximation theoretic questions concerning a certain class of neural networks are considered. The networks considered are single input, single output, single hidden layer, feedforward neural networks with continuous sigmoidal activation functions, no input weights but with hidden layer thresholds and output layer weights. Specifically, questions of existence and uniqueness of best approximations on a closed interval of the real line under mean-square and uniform approximation error measures are studied. A by-product of this study is a reparametrization of the class of networks considered in terms of rational functions of a single variable. This rational reparametrization is used to apply the theory of Pade approximation to the class of networks considered. In addition, a question related to the number of local minima arising in gradient algorithms for learning is examined View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Gradient descent learning algorithm overview: a general dynamical systems perspective

    Page(s): 182 - 195
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (1368 KB)  

    Gives a unified treatment of gradient descent learning algorithms for neural networks using a general framework of dynamical systems. This general approach organizes and simplifies all the known algorithms and results which have been originally derived for different problems (fixed point/trajectory learning), for different models (discrete/continuous), for different architectures (forward/recurrent), and using different techniques (backpropagation, variational calculus, adjoint methods, etc.). The general approach can also be applied to derive new algorithms. The author then briefly examines some of the complexity issues and limitations intrinsic to gradient descent learning. Throughout the paper, the author focuses on the problem of trajectory learning View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Adaptive learning method in self-organizing map for edge preserving vector quantization

    Page(s): 278 - 280
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (384 KB)  

    The Kohonen's self-organizing map algorithm for vector quantization of images is modified to reduce the edge degradation in the coded image. The learning procedure is performed by adaptive learning rates that are determined according to the image block activity. The simulation result of 4×4 vector quantization for 512×512 image coding demonstrates the feasibility of the proposed method View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • The transversal imager: a photonic neurochip with programmable synaptic weights

    Page(s): 248 - 251
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (364 KB)  

    A photonic neural processor implemented in NMOS/CCD integrated circuit technology is described. The processor performs outer-product processing utilizing optical input of the synaptic weights and electrical input of the state vector, or, visa versa. The performance of the 32-neuron, 1024-synapse processor is presented View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Optimal adaptive k-means algorithm with dynamic adjustment of learning rate

    Page(s): 157 - 169
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (1088 KB)  

    Adaptive k-means clustering algorithms have been used in several artificial neural network architectures, such as radial basis function networks or feature-map classifiers, for a competitive partitioning of the input domain. This paper presents an enhancement of the traditional k-means algorithm. It approximates an optimal clustering solution with an efficient adaptive learning rate, which renders it usable even in situations where the statistics of the problem task varies slowly with time. This modification Is based on the optimality criterion for the k-means partition stating that: all the regions in an optimal k-means partition have the same variations if the number of regions in the partition is large and the underlying distribution for generating input patterns is smooth. The goal of equalizing these variations is introduced in the competitive function that assigns each new pattern vector to the “appropriate” region. To evaluate the optimal k-means algorithm, the authors first compare it to other k-means variants on several simple tutorial examples, then the authors evaluate it on a practical application: vector quantization of image data View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Learning capability assessment and feature space optimization for higher-order neural networks

    Page(s): 267 - 272
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (484 KB)  

    A technique for evaluating the learning capability and optimizing the feature space of a class of higher-order neural networks is presented. It is shown that supervised learning can be posed as an optimization problem in which inequality constraints are used to code the information contained in the training patterns and to specify the degree of accuracy expected from the neural network. The approach establishes: (a) whether the structure of the network can effectively learn the training patterns and, if it can, a connectivity which corresponds to satisfactorily learning; (b) those features which can be suppressed from the definition of the feature space without deteriorating performance; and (c) if the structure is not appropriate for learning the training patterns, the minimum set of patterns which cannot be learned. The technique is tested with two examples and results are discussed View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • An accelerated learning algorithm for multilayer perceptrons: optimization layer by layer

    Page(s): 31 - 42
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (1016 KB)  

    Multilayer perceptrons are successfully used in an increasing number of nonlinear signal processing applications. The backpropagation learning algorithm, or variations hereof, is the standard method applied to the nonlinear optimization problem of adjusting the weights in the network in order to minimize a given cost function. However, backpropagation as a steepest descent approach is too slow for many applications. In this paper a new learning procedure is presented which is based on a linearization of the nonlinear processing elements and the optimization of the multilayer perceptron layer by layer. In order to limit the introduced linearization error a penalty term is added to the cost function. The new learning algorithm is applied to the problem of nonlinear prediction of chaotic time series. The proposed algorithm yields results in both accuracy and convergence rates which are orders of magnitude superior compared to conventional backpropagation learning View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Improving the performance of Kanerva's associate memory

    Page(s): 125 - 130
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (504 KB)  

    A parallel associative memory first proposed by Kanerva (1988) is discussed. The major appeal of this memory is its ability to be trained very rapidly. A discrepancy between Kanerva's theoretical calculation of capacity and the actual capacity is demonstrated experimentally and a corrected theory is offered. A modified method of reading from memory is suggested which results in a capacity nearly the same as that originally predicted by Kanerva. The capacity of the memory is then analyzed for a different method of writing to memory. This method increases the capacity of the memory by an order of magnitude. A further modification is suggested which increases the learning rate of this method View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Fast neural net simulation with a DSP processor array

    Page(s): 203 - 213
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (1044 KB)  

    This paper describes the implementation of a fast neural net simulator on a novel parallel distributed-memory computer. A 60-processor system, named MUSIC (multiprocessor system with intelligent communication), is operational and runs the backpropagation algorithm at a speed of 330 million connection updates per second (continuous weight update) using 32-b floating-point precision. This is equal to 1.4 Gflops sustained performance. The complete system with 3.8 Gflops peak performance consumes less than 800 W of electrical power and fits into a 19-in rack. While reaching the speed of modern supercomputers, MUSIC still can be used as a personal desktop computer at a researcher's own disposal. In neural net simulation, this gives a computing performance to a single user which was unthinkable before. The system's real-time interfaces make it especially useful for embedded applications View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Back-propagation network and its configuration for blood vessel detection in angiograms

    Page(s): 64 - 72
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (808 KB)  

    A neural-network classifier for detecting vascular structures in angiograms was developed. The classifier consisted of a multilayer feedforward network window in which the center pixel was classified using gray-scale information within the window. The network was trained by using the backpropagation algorithm with the momentum term. Based on this image segmentation problem, the effect of changing network configuration on the classification performance was also characterized. Factors including topology, rate parameters, training sample set, and initial weights were systematically analyzed. The training set consisted of 75 selected points from a 256×256 digitized cineangiogram. While different network topologies showed no significant effect on performance, both the learning process and the classification performance were sensitive to the rate parameters. In a comparative study, the network demonstrated its superiority in classification performance. It was also shown that the trained neural-network classifier was equivalent to a generalized matched filter with a nonlinear decision tree View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Locally excitatory globally inhibitory oscillator networks

    Page(s): 283 - 286
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (412 KB)  

    A novel class of locally excitatory, globally inhibitory oscillator networks (LEGION) is proposed and investigated. The model of each oscillator corresponds to a standard relaxation oscillator with two time scales. In the network, an oscillator jumping up to its active phase rapidly recruits the oscillators stimulated by the same pattern, while preventing other oscillators from jumping up. Computer simulations demonstrate that the network rapidly achieves both synchronization within blocks of oscillators that are stimulated by connected regions and desynchronization between different blocks. This model lays a physical foundation for the oscillatory correlation theory of feature binding and may provide an effective computational framework for scene segmentation and figure/ground segregation in real time View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Process modeling with the regression network

    Page(s): 78 - 93
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (1536 KB)  

    A new connectionist network topology called the regression network is proposed. The structural and underlying mathematical features of the regression network are investigated. Emphasis is placed on the intricacies of the optimization process for the regression network and some measures to alleviate these difficulties of optimization are proposed and investigated. The ability of the regression network algorithm to perform either nonparametric or parametric optimization, as well as a combination of both, is also highlighted. It is further shown how the regression network can be used to model systems which are poorly understood on the basis of sparse data. A semi-empirical regression network model is developed for a metallurgical processing operation (a hydrocyclone classifier) by building mechanistic knowledge into the connectionist structure of the regression network model. Poorly understood aspects of the process are provided for by use of nonparametric regions within the structure of the semi-empirical connectionist model. The performance of the regression network model is compared to the corresponding generalization performance results obtained by some other nonparametric regression techniques View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Random noise effects in pulse-mode digital multilayer neural networks

    Page(s): 220 - 229
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (804 KB)  

    A pulse-mode digital multilayer neural network (DMNN) based on stochastic computing techniques is implemented with simple logic gates as basic computing elements. The pulse-mode signal representation and the use of simple logic gates for neural operations lead to a massively parallel yet compact and flexible network architecture, well suited for VLSI implementation. Algebraic neural operations are replaced by stochastic processes using pseudorandom pulse sequences. The distributions of the results from the stochastic processes are approximated using the hypergeometric distribution. Synaptic weights and neuron states are represented as probabilities and estimated as average pulse occurrence rates in corresponding pulse sequences. A statistical model of the noise (error) is developed to estimate the relative accuracy associated with stochastic computing in terms of mean and variance. Computational differences are then explained by comparison to deterministic neural computations. DMNN feedforward architectures are modeled in VHDL using character recognition problems as testbeds. Computational accuracy is analyzed, and the results of the statistical model are compared with the actual simulation results. Experiments show that the calculations performed in the DMNN are more accurate than those anticipated when Bernoulli sequences are assumed, as is common in the literature. Furthermore, the statistical model successfully predicts the accuracy of the operations performed in the DMNN View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • High speed paper currency recognition by neural networks

    Page(s): 73 - 77
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (480 KB)  

    In this paper a new technique is proposed to improve the recognition ability and the transaction speed to classify the Japanese and US paper currency. Two types of data sets, time series data and Fourier power spectra, are used in this study. In both cases, they are directly used as inputs to the neural network. Furthermore, we also refer a new evaluation method of recognition ability. Meanwhile, a technique is proposed to reduce the input scale of the neural network without preventing the growth of recognition. This technique uses only a subset of the original data set which is obtained using random masks. The recognition ability of using large data set and a reduced data set are discussed. In addition to that the results of using a reduced data set of the Fourier power spectra and the time series data are compared View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • The geometrical learning of binary neural networks

    Page(s): 237 - 247
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (900 KB)  

    In this paper, the learning algorithm called expand-and-truncate learning (ETL) is proposed to train multilayer binary neural networks (BNN) with guaranteed convergence for any binary-to-binary mapping. The most significant contribution of this paper is the development of a learning algorithm for three-layer BNN which guarantees the convergence, automatically determining a required number of neurons in the hidden layer. Furthermore, the learning speed of the proposed ETL algorithm is much faster than that of backpropagation learning algorithm in a binary field. Neurons in the proposed BNN employ a hard-limiter activation function, with only integer weights and integer thresholds. Therefore, this will greatly facilitate actual hardware implementation of the proposed BNN using currently available digital VLSI technology View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Robust principal component analysis by self-organizing rules based on statistical physics approach

    Page(s): 131 - 143
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (1184 KB)  

    This paper applies statistical physics to the problem of robust principal component analysis (PCA). The commonly used PCA learning rules are first related to energy functions. These functions are generalized by adding a binary decision field with a given prior distribution so that outliers in the data are dealt with explicitly in order to make PCA robust. Each of the generalized energy functions is then used to define a Gibbs distribution from which a marginal distribution is obtained by summing over the binary decision field. The marginal distribution defines an effective energy function, from which self-organizing rules have been developed for robust PCA. Under the presence of outliers, both the standard PCA methods and the existing self-organizing PCA rules studied in the literature of neural networks perform quite poorly. By contrast, the robust rules proposed here resist outliers well and perform excellently for fulfilling various PCA-like tasks such as obtaining the first principal component vector, the first k principal component vectors, and directly finding the subspace spanned by the first k vector principal component vectors without solving for each vector individually. Comparative experiments have been made, and the results show that the authors' robust rules improve the performances of the existing PCA algorithms significantly when outliers are present View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • A neural network filter to detect small targets in high clutter backgrounds

    Page(s): 252 - 257
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (528 KB)  

    The detection of objects in high-resolution aerial imagery has proven to be a difficult task. In the authors' application, the amount of image clutter is extremely high. Under these conditions, detection based on low-level image cues tends to perform poorly. Neural network techniques have been proposed in object detection applications due to proven robust performance characteristics. A neural network filter was designed and trained to detect targets in thermal infrared images. The feature extraction stage was eliminated and raw gray levels were utilized as input to the network. Two fundamentally different approaches were used to design the training sets. In the first approach, actual image data were utilized for training. In the second case, a model-based approach was adopted to design the training set vectors. The training set consisted of object and background data. The neuron transfer function was modified to improve network convergence and speed and the backpropagation training algorithm was used to train the network. The neural network filter was tested extensively on real image data. Receiver operating characteristic (ROC) curves were determined in each case. The detection and false alarm rates were excellent for the neural network filters. Their overall performance was much superior to that of the size-matched contrast-box filter, especially in the images with higher amounts of visual clutter View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • A single-iteration threshold Hamming network

    Page(s): 261 - 266
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (388 KB)  

    We analyze in detail the performance of a Hamming network classifying inputs that are distorted versions of one of its m stored memory patterns, each being a binary vector of length n. It is shown that the activation function of the memory neurons in the original Hamming network may be replaced by a simple threshold function. By judiciously determining the threshold value, the “winner-take-all” subnet of the Hamming network (known to be the essential factor determining the time complexity of the network's computation) may be altogether discarded. For m growing exponentially in n, the resulting threshold Hamming network correctly classifies the input pattern in a single iteration, with probability approaching 1 View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • A general mean-based iterative winner-take-all neural network

    Page(s): 14 - 24
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (864 KB)  

    In this paper, a new iterative winner-take-all (WTA) neural network is developed and analyzed. The proposed WTA neural net with one-layer structure is established under the concept of the statistical mean. For three typical distributions of initial activations, the convergence behaviors of the existing and the proposed WTA neural nets are evaluated by theoretical analyses and Monte Carlo simulations. We found that the suggested WTA neural network on average requires fewer than Log2M iterations to complete a WTA process for the three distributed inputs, where M is the number of competitors. Furthermore, the fault tolerances of the iterative WTA nets are analyzed and simulated. From the view points of convergence speed, hardware complexity, and robustness to the errors, the proposed WTA is suitable for various applications View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Limitations of neural networks for solving traveling salesman problems

    Page(s): 280 - 282
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (300 KB)  

    Feedback neural networks enjoy considerable popularity as a means of approximately solving combinatorial optimization problems. It is now well established how to map problems onto networks so that invalid solutions are never found. It is not as clear how the networks' solutions compare in terms of quality with those obtained using other optimization techniques; such issues are addressed in this paper. A linearized analysis of annealed network dynamics allows a prototypical network solution to be identified in a pertinent eigenvector basis. It is possible to predict the likely quality of this solution by examining optimal solutions in the same basis. Applying this methodology to traveling salesman problems, it appears that neural networks are well suited to the solution of Euclidean but not random problems; this is confirmed by extensive experiments. The failure of a network to adequately solve even 10-city problems is highly significant View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.

Aims & Scope

IEEE Transactions on Neural Networks is devoted to the science and technology of neural networks, which disclose significant technical knowledge, exploratory developments, and applications of neural networks from biology to software to hardware.

 

This Transactions ceased production in 2011. The current retitled publication is IEEE Transactions on Neural Networks and Learning Systems.

Full Aims & Scope