By Topic

Neural Networks, IEEE Transactions on

Issue 6 • Date Nov. 1992

Filter Results

Displaying Results 1 - 21 of 21
  • Comments on "Bayes statistical behavior and valid generalization of pattern classifying neural networks" [with reply]

    Page(s): 1026 - 1027
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (170 KB)  

    In the above-titled paper (ibid., vol.2, p.471-475, July 1991), the authors claim that neural network classifiers duplicate the decision rule created by the empirical Bayes rule. The commenter states that this statement is, in fact, not generally true and points out an error in the proof. The commenter also shows that the related true statement about the relation between neural and Bayes classifiers is of no practical use. In reply, the authors show that the example that the commenter uses to disprove the equation is misleading. They refute the commenter's second point as well.<> View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Comments on "Dynamic programming approach to optimal weight selection in multilayer neural networks" [with reply]

    Page(s): 1028 - 1029
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (95 KB)  

    The commenter claims that in the above-titled paper (ibid., vol.2, p.465-467, July 1991), which presents an efficient algorithm using dynamic programming to find weights which load a set of examples into a feedforward neural network with minimal error, a contradiction lies buried in the paper's notation. In reply, the author maintains that the comments are due to some misunderstandings about the implementation of dynamic-programming-based algorithms and clarifies the work.<> View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Wavelet networks

    Page(s): 889 - 898
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (800 KB)  

    A wavelet network concept, which is based on wavelet transform theory, is proposed as an alternative to feedforward neural networks for approximating arbitrary nonlinear functions. The basic idea is to replace the neurons by `wavelons', i.e., computing units obtained by cascading an affine transform and a multidimensional wavelet. Then these affine transforms and the synaptic weights must be identified from possibly noise corrupted input/output data. An algorithm of backpropagation type is proposed for wavelet network training, and experimental results are reported View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Massively parallel architectures for large scale neural network simulations

    Page(s): 876 - 888
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (1000 KB)  

    A toroidal lattice architecture (TLA) and a planar lattice architecture (PLA) are proposed as massively parallel neurocomputer architectures for large-scale simulations. The performance of these architectures is almost proportional to the number of node processors, and they adopt the most efficient two-dimensional processor connections for WSI implementation. They also give a solution to the connectivity problem, the performance degradation caused by the data transmission bottleneck, and the load balancing problem for efficient parallel processing in large-scale neural network simulations. The general neuron model is defined. Implementation of the TLA with transputers is described. A Hopfield neural network and a multilayer perceptron have been implemented and applied to the traveling salesman problem and to identity mapping, respectively. Proof that the performance increases almost in proportion to the number of node processors is given View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Avoiding false local minima by proper initialization of connections

    Page(s): 899 - 905
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (688 KB)  

    The training of neural net classifiers is often hampered by the occurrence of local minima, which results in the attainment of inferior classification performance. It has been shown that the occurrence of local minima in the criterion function is often related to specific patterns of defects in the classifier. In particular, three main causes for local minima were identified. Such an understanding of the physical correlates of local minima suggests sensible ways of choosing the weights from which the training process is initiated. A method of initialization is introduced and shown to decrease the possibility of local minima occurring on various test problems View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Feedback stabilization using two-hidden-layer nets

    Page(s): 981 - 990
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (880 KB)  

    The representational capabilities of one-hidden-layer and two-hidden-layer nets consisting of feedforward interconnections of linear threshold units are compared. It is remarked that for certain problems two hidden layers are required, contrary to what might be in principle expected from the known approximation theorems. The differences are not based on numerical accuracy or number of units needed, nor on capabilities for feature extraction, but rather on a much more basic classification into direct and inverse problems. The former correspond to the approximation of continuous functions, while the latter are concerned with approximating one-sided inverses of continuous functions, and are often encountered in the context of inverse kinematics determination or in control questions. A general result is given showing that nonlinear control systems can be stabilized using two hidden layers, but not, in general, using just one View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Can backpropagation error surface not have local minima

    Page(s): 1019 - 1021
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (288 KB)  

    It is shown theoretically that for an arbitrary T-element training set with t(tT) different inputs, the backpropagation error surface does not have suboptimal local minima if the network is capable of exactly implementing an arbitrary training set consisting of t different patterns. As a special case, the error surface of a backpropagation network with one hidden layer and t-1 hidden units has no local minima, if the network is trained by an arbitrary T-element set with t different inputs View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • A parallel network for visual cognition

    Page(s): 906 - 922
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (1564 KB)  

    The authors describe a parallel dynamical system designed to integrate model-based and data-driven approaches to image recognition in a neural network, and study one component of the system in detail. That component is the translation-invariant network of probabilistic cellular automata (PCA), which combines feature-detector outputs and collectively performs enhancement and recognition functions. Recognition is a novel application of the PCA. Given a model of the target object, conditions on the PCA weights are obtained which must be satisfied for object enhancement and noise rejection to occur, and engineered weights are constructed. For further refinement of the weights, a training algorithm derived from optimal control theory is proposed. System operation is illustrated with examples derived from visual, infrared, and laser-radar imagery View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Improving generalization performance using double backpropagation

    Page(s): 991 - 997
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (552 KB)  

    In order to generalize from a training set to a test set, it is desirable that small changes in the input space of a pattern do not change the output components. This can be done by forcing this behavior as part of the training algorithm. This is done in double backpropagation by forming an energy function that is the sum of the normal energy term found in backpropagation and an additional term that is a function of the Jacobian. Significant improvement is shown with different architectures and different test sets, especially with architectures that had previously been shown to have very good performance when trained using backpropagation. It is shown that double backpropagation, as compared to backpropagation, creates weights that are smaller, thereby causing the output of the neurons to spend more time in the linear region View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Digital neural emulators using tree accumulation and communication structures

    Page(s): 934 - 950
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (1328 KB)  

    Three digital artificial neural network processors suitable for the emulation of fully interconnected neural networks are proposed. The processors use N2 multipliers and an arrangement of tree structures that provide the communication and accumulation function either individually or in a combined manner using communicating adder trees. The performance for the emulation of an N-neuron network for all processors is achieved in 2log2N+C time units, where C is a constant equal to the multiplication, neuron activation, and internal fixed delays. The feasibility and characteristics of the proposed configurations to emulate single and/or multiple neural networks simultaneously are discussed, and a comparison with recently proposed neurocomputer architectures is reported View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • A note on the complexity of reliability in neural networks

    Page(s): 998 - 1002
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (368 KB)  

    It is shown that in a standard discrete neural network model with small fan-in, tolerance to random malicious faults can be achieved with a log-linear increase in the number of neurons and a constant factor increase in parallel time, provided fan-in can increase arbitrarily. A similar result is obtained for a nonstandard but closely related model with no restriction on fan-in View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Guaranteed convergence in a class of Hopfield networks

    Page(s): 951 - 961
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (956 KB)  

    A class of symmetric Hopfield networks with nonpositive synapses and zero threshold is analyzed in detail. It is shown that all stationary points have a one-to-one correspondence with the minimal vertex covers of certain undirected graphs, that the sequential Hopfield algorithm as applied to this class of networks converges in at most 2n steps (n being the number of neurons), and that the parallel Hopfield algorithm either converges in one step or enters a two-cycle in one step. The necessary and sufficient condition on the initial iterate for the parallel algorithm to converge in one step are given. A modified parallel algorithm which is guaranteed to converge in [3n/2] steps ([x] being the integer part of x) for an n-neuron network of this particular class is also given. By way of application, it is shown that this class naturally solves the vertex cover problem. Simulations confirm that the solution provided by this method is better than those provided by other known methods View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Characteristics of Hebbian-type associative memories having faulty interconnections

    Page(s): 969 - 980
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (892 KB)  

    The performance of Hebbian-type associative memories (HAMs) in the presence of faulty (open- and short-circuit) synaptic interconnections is examined and equations for predicting network reliability are developed. The results show that a network with open-circuit interconnection faults has a higher probability of direct convergence than a network with short-circuit interconnection faults when the fraction of failed interconnections is small and the short-circuit signal is large. The results are extended to the case where network attraction radius is considered. Under certain assumptions, it is found that the expected numbers of neurons with b, b-1, b-2,. . .,1 input error bits in their state update are equal. Because of the capability of error correction, an asynchronous HAM is also found to have a higher probability of direct convergence than a synchronous HAM. Using these results, network reliability and generalization capability can be estimated when both the interconnection faults and the number of error bits in the probe vectors are taken into account View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Second-order neural nets for constrained optimization

    Page(s): 1021 - 1024
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (312 KB)  

    Analog neural nets for constrained optimization are proposed as an analogue of Newton's algorithm in numerical analysis. The neural model is globally stable and can converge to the constrained stationary points. Nonlinear neurons are introduced into the net, making it possible to solve optimization problems where the variables take discrete values, i.e., combinatorial optimization View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Weighted learning of bidirectional associative memories by global minimization

    Page(s): 1010 - 1018
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (628 KB)  

    A weighted learning algorithm for bidirectional associative memories (BAMs) by means of global minimization, where each desired pattern is weighted, is described. According to the cost function that measures the goodness of the BAM, the learning algorithm is formulated as a global minimization problem and solved by a gradient descent rule. The learning approach guarantees not only that each desired pattern is stored as a stable state, but also that the basin of attraction is constructed as large as possible around each desired pattern. The existence of the weights, the asymptotic stability of each desired pattern and its basin of attraction, and the convergence of the proposed learning algorithm are investigated in an analytic way. A large number of computer experiments are reported to demonstrate the efficiency of the learning rule View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Classification trees with neural network feature extraction

    Page(s): 923 - 933
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (916 KB)  

    The ideal use of small multilayer nets at the decision nodes of a binary classification tree to extract nonlinear features is proposed. The nets are trained and the tree is grown using a gradient-type learning algorithm in the multiclass case. The method improves on standard classification tree design methods in that it generally produces trees with lower error rates and fewer nodes. It also reduces the problems associated with training large unstructured nets and transfers the problem of selecting the size of the net to the simpler problem of finding a tree of the right size. An efficient tree pruning algorithm is proposed for this purpose. Trees constructed with the method and the CART method are compared on a waveform recognition problem and a handwritten character recognition problem. The approach demonstrates significant decrease in error rate and tree size. It also yields comparable error rates and shorter training times than a large multilayer net trained with backpropagation on the same problems View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Handwritten digit recognition by neural networks with single-layer training

    Page(s): 962 - 968
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (672 KB)  

    It is shown that neural network classifiers with single-layer training can be applied efficiently to complex real-world classification problems such as the recognition of handwritten digits. The STEPNET procedure, which decomposes the problem into simpler subproblems which can be solved by linear separators, is introduced. Provided appropriate data representations and learning rules are used, performance comparable to that obtained by more complex networks can be achieved. Results from two different databases are presented: an European database comprising 8700 isolated digits and a zip code database from the US Postal Service comprising 9000 segmented digits. A hardware implementation of the classifier is briefly described View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Neural subnet design by direct polynomial mapping

    Page(s): 1024 - 1026
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (272 KB)  

    In a recent paper by M. Chen and M. Maury (1990), it was shown that multilayer perceptron neural networks can be used to form products of any number of inputs, thereby constructively proving universal approximation. This result is extended, and a method for the analysis and synthesis of single-input, single-output neural subnetworks is described. Given training samples of a function to be approximated, a feedforward neural network is designed which implements a polynomial approximation of the function with arbitrary accuracy. For comparison, example subnets are designed by classical backpropagation training and by mapping. The examples illustrate that the mapped subnets avoid local minima which backpropagation-trained subnets get trapped in and that the mapping approach is much faster View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • A two-phase optimization neural network

    Page(s): 1003 - 1009
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (532 KB)  

    A novel two-phase neural network that is suitable for solving a large class of constrained or unconstrained optimization problem is presented. For both types of problems with solutions lying in the interior of the feasible regions, the phase-one structure of the network alone is sufficient. When the solutions of constrained problems are on the boundary of the feasible regions, the proposed two-phase network is capable of achieving the exact solutions, in contrast to existing optimization neural networks which can obtain only approximate solutions. Furthermore, the network automatically provides the corresponding Lagrange multiplier associated with each constraint. Thus, for linear programming, the network solves both the primal problems and their dual problems simultaneously View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Enhanced training algorithms, and integrated training/architecture selection for multilayer perceptron networks

    Page(s): 864 - 875
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (948 KB)  

    The standard backpropagation-based multilayer perceptron training algorithm suffers from a slow asymptotic convergence rate. Sophisticated nonlinear least-squares and quasi-Newton optimization techniques are used to construct enhanced multilayer perceptron training algorithms, which are then compared to the backpropagation algorithm in the context of several example problems. In addition, an integrated approach to training and architecture selection that uses the described enhanced algorithms is presented, and its effectiveness illustrated in the context of synthetic and actual pattern recognition problems View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Gaussian networks for direct adaptive control

    Page(s): 837 - 863
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (2436 KB)  

    A direct adaptive tracking control architecture is proposed and evaluated for a class of continuous-time nonlinear dynamic systems for which an explicit linear parameterization of the uncertainty in the dynamics is either unknown or impossible. The architecture uses a network of Gaussian radial basis functions to adaptively compensate for the plant nonlinearities. Under mild assumptions about the degree of smoothness exhibit by the nonlinear functions, the algorithm is proven to be globally stable, with tracking errors converging to a neighborhood of zero. A constructive procedure is detailed, which directly translates the assumed smoothness properties of the nonlinearities involved into a specification of the network required to represent the plant to a chosen degree of accuracy. A stable weight adjustment mechanism is determined using Lyapunov theory. The network construction and performance of the resulting controller are illustrated through simulations with example systems View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.

Aims & Scope

IEEE Transactions on Neural Networks is devoted to the science and technology of neural networks, which disclose significant technical knowledge, exploratory developments, and applications of neural networks from biology to software to hardware.

 

This Transactions ceased production in 2011. The current retitled publication is IEEE Transactions on Neural Networks and Learning Systems.

Full Aims & Scope