By Topic

Neural Networks, IEEE Transactions on

Issue 4 • Date July 2003

Filter Results

Displaying Results 1 - 24 of 24
  • Full text access may be available. Click article title to sign in or learn about subscription options.
  • Book reviews

    Page(s): 968 - 970
    Save to Project icon | Request Permissions | PDF file iconPDF (162 KB)  
    Freely Available from IEEE
  • Competition and cooperation in neuronal processing

    Page(s): 860 - 868
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (529 KB)  

    A new type of model neuron is introduced as a building block of an associative memory. The neuron, which has a number of receptor zones, processes both the amplitude and the frequency of input signals, associating a small number of features encoded by those signals. Using this two-parameter input in our model compared to the one-dimensional inputs of conventional model neurons (e.g., the McCulloch Pitts neuron) offers an increased memory capacity. In our model, there is a competition among inputs in each zone with a subsequent cooperation of the winners to specify the output. The associative memory consists of a network of such neurons. A state-space model is used to define the neurodynamics. We explore properties of the neuron and the network and demonstrate its favorable capacity and recall capabilities. Finally, the network is used in an application designed to find trademarks that sound alike. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • A Kohonen-like decomposition method for the Euclidean traveling salesman problem-KNIES_DECOMPOSE

    Page(s): 869 - 890
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (1723 KB) |  | HTML iconHTML  

    In addition to the classical heuristic algorithms of operations research, there have also been several approaches based on artificial neural networks for solving the traveling salesman problem. Their efficiency, however, decreases as the problem size (number of cities) increases. A technique to reduce the complexity of a large-scale traveling salesman problem (TSP) instance is to decompose or partition it into smaller subproblems. We introduce an all-neural decomposition heuristic that is based on a recent self-organizing map called KNIES, which has been successfully implemented for solving both the Euclidean traveling salesman problem and the Euclidean Hamiltonian path problem. Our solution for the Euclidean TSP proceeds by solving the Euclidean HPP for the subproblems, and then patching these solutions together. No such all-neural solution has ever been reported. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • A mixture model for population codes of Gabor filters

    Page(s): 794 - 803
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (575 KB)  

    Population coding is a coding scheme which is ubiquitous in neural systems, and is also of more general use in coding stimuli, for example in vision problems. A population of responses to a stimulus can be used to represent not only the value of some variable in the environment, but a full probability distribution for that variable. The information is held in a distributed and encoded form, which may in some situations be more robust to noise and failures than conventional representations. Gabor filters are a popular choice for detecting edges in the visual field for several reasons. They are easily tuned for a variety of edge widths and orientations, and are considered a close model of the edge filters in the human visual system. In this paper, we consider population codes of Gabor filters with different orientations. A probabilistic model of Gabor filter responses is presented. Based on the analytically derived orientation tuning function and a parametric mixture model of the filter responses in the presence of local edge structure with single or multiple orientations a probability density function (pdf) of the local orientation in any point (x, y) can be extracted through a parameter estimation procedure. The resulting pdf of the local contour orientation captures not only angular information at edges, corners or T-junctions but also describes the certainty of the measurement which can be characterized in terms of the entropy of the individual mixture components. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • LDA/SVM driven nearest neighbor classification

    Page(s): 940 - 942
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (239 KB) |  | HTML iconHTML  

    Nearest neighbor (NN) classification relies on the assumption that class conditional probabilities are locally constant. This assumption becomes false in high dimensions with finite samples due to the curse of dimensionality. The NN rule introduces severe bias under these conditions. We propose a locally adaptive neighborhood morphing classification method to try to minimize bias. We use local support vector machine learning to estimate an effective metric for producing neighborhoods that are elongated along less discriminant feature dimensions and constricted along most discriminant ones. As a result, the class conditional probabilities can be expected to be approximately constant in the modified neighborhoods, whereby better classification performance can be achieved. The efficacy of our method is validated and compared against other competing techniques using a number of datasets. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Improving procedures for evaluation of connectionist context-free language predictors

    Page(s): 963 - 966
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (304 KB) |  | HTML iconHTML  

    This letter shows how seemingly minor differences in training and evaluation procedures used in recent studies of recurrent neural networks as context free language predictors can lead to significant differences in apparent network performance. We suggest standard evaluation procedures whose use would facilitate better reproductibility and comparability. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Neural-network construction and selection in nonlinear modeling

    Page(s): 804 - 819
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (776 KB)  

    We study how statistical tools which are commonly used independently can advantageously be exploited together in order to improve neural network estimation and selection in nonlinear static modeling. The tools we consider are the analysis of the numerical conditioning of the neural network candidates, statistical hypothesis tests, and cross validation. We present and analyze each of these tools in order to justify at what stage of a construction and selection procedure they can be most useful. On the basis of this analysis, we then propose a novel and systematic construction and selection procedure for neural modeling. We finally illustrate its efficiency through large-scale simulations experiments and real-world modeling problems. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • A new design method for the complex-valued multistate Hopfield associative memory

    Page(s): 891 - 899
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (1009 KB) |  | HTML iconHTML  

    A method to store each element of an integral memory set M ⊂ {1,2,...,K}n as a fixed point into a complex-valued multistate Hopfield network is introduced. The method employs a set of inequalities to render each memory pattern as a strict local minimum of a quadratic energy landscape. Based on the solution of this system, it gives a recurrent network of n multistate neurons with complex and symmetric synaptic weights, which operates on the finite state space {1,2,...,K}n to minimize this quadratic functional. Maximum number of integral vectors that can be embedded into the energy landscape of the network by this method is investigated by computer experiments. This paper also enlightens the performance of the proposed method in reconstructing noisy gray-scale images. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Combining support vector machine learning with the discrete cosine transform in image compression

    Page(s): 950 - 958
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (568 KB)  

    We present an algorithm for the application of support vector machine (SVM) learning to image compression. The algorithm combines SVMs with the discrete cosine transform (DCT). Unlike a classic radial basis function networks or multilayer perceptrons that require the topology of the network to be defined before training, an SVM selects the minimum number of training points, called support vectors, that ensure modeling of the data within the given level of accuracy (a.k.a. insensitivity zone ε). It is this property that is exploited as the basis for an image compression algorithm. Here, the SVMs learning algorithm performs the compression in a spectral domain of DCT coefficients, i.e., the SVM approximates the DCT coefficients. The parameters of the SVM are stored in order to recover the image. Results demonstrate that even though there is an extra lossy step compared with the baseline JPEG algorithm, the new algorithm dramatically increases compression for a given image quality; conversely it increases image quality for a given compression ratio. The approach presented can be readily applied for other modeling schemes that are in a form of a sum of weighted basis functions. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Monotonic convergence of fixed-point algorithms for ICA

    Page(s): 943 - 949
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (571 KB) |  | HTML iconHTML  

    We re-examine a fixed-point algorithm proposed by Hyvarinen for independent component analysis, wherein local convergence is proved subject to an ideal signal model using a square invertible mixing matrix. Here, we derive step-size bounds which ensure monotonic convergence to a local extremum for any initial condition. Our analysis does not assume an ideal signal model but appeals rather to properties of the contrast function itself, and so applies even with noisy data and/or more sources than sensors. The results help alleviate the guesswork that often surrounds step-size selection when the observed signal does not fit an idealized model. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • An improved algorithm for learning long-term dependency problems in adaptive processing of data structures

    Page(s): 781 - 793
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (1022 KB) |  | HTML iconHTML  

    Many researchers have explored the use of neural-network representations for the adaptive processing of data structures. One of the most popular learning formulations of data structure processing is backpropagation through structure (BPTS). The BPTS algorithm has been successful applied to a number of learning tasks that involve structural patterns such as logo and natural scene classification. The main limitations of the BPTS algorithm are attributed to slow convergence speed and the long-term dependency problem for the adaptive processing of data structures. In this paper, an improved algorithm is proposed to solve these problems. The idea of this algorithm is to optimize the free learning parameters of the neural network in the node representation by using least-squares-based optimization methods in a layer-by-layer fashion. Not only can fast convergence speed be achieved, but the long-term dependency problem can also be overcome since the vanishing of gradient information is avoided when our approach is applied to very deep tree structures. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Neural-network control of nonaffine nonlinear system with zero dynamics by state and output feedback

    Page(s): 900 - 918
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (1583 KB) |  | HTML iconHTML  

    This paper focuses on adaptive control of nonaffine nonlinear systems with zero dynamics using multilayer neural networks. Through neural network approximation, state feedback control is firstly investigated for nonaffine single-input-single-output (SISO) systems. By using a high gain observer to reconstruct the system states, an extension is made to output feedback neural-network control of nonaffine systems, whose states and time derivatives of the output are unavailable. It is shown that output tracking errors converge to adjustable neighborhoods of the origin for both state feedback and output feedback control. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • A new simple ∞OH neuron model as a biologically plausible principal component analyzer

    Page(s): 853 - 859
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (385 KB)  

    A new approach to unsupervised learning in a single-layer neural network is discussed. An algorithm for unsupervised learning based upon the Hebbian learning rule is presented. A simple neuron model is analyzed. A dynamic neural model, which contains both feed-forward and feedback connections between the input and the output, has been adopted. The, proposed learning algorithm could be more correctly named self-supervised rather than unsupervised. The solution proposed here is a modified Hebbian rule, in which the modification of the synaptic strength is proportional not to pre- and postsynaptic activity, but instead to the presynaptic and averaged value of postsynaptic activity. It is shown that the model neuron tends to extract the principal component from a stationary input vector sequence. Usually accepted additional decaying terms for the stabilization of the original Hebbian rule are avoided. Implementation of the basic Hebbian scheme would not lead to unrealistic growth of the synaptic strengths, thanks to the adopted network structure. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Helicopter trimming and tracking control using direct neural dynamic programming

    Page(s): 929 - 939
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (706 KB) |  | HTML iconHTML  

    This paper advances a neural-network-based approximate dynamic programming control mechanism that can be applied to complex control problems such as helicopter flight control design. Based on direct neural dynamic programming (DNDP), an approximate dynamic programming methodology, the control system is tailored to learn to maneuver a helicopter. The paper consists of a comprehensive treatise of this DNDP-based tracking control framework and extensive simulation studies for an Apache helicopter. A trim network is developed and seamlessly integrated into the neural dynamic programming (NDP) controller as part of a baseline structure for controlling complex nonlinear systems such as a helicopter. Design robustness is addressed by performing simulations under various disturbance conditions. All designs are tested using FLYRT, a sophisticated industrial scale nonlinear validated model of the Apache helicopter. This is probably the first time that an approximate dynamic programming methodology has been systematically applied to, and evaluated on, a complex, continuous state, multiple-input multiple-output nonlinear system with uncertainty. Though illustrated for helicopters, the DNDP control system framework should be applicable to general purpose tracking control. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Nonsymmetric PDF estimation by artificial neurons: application to statistical characterization of reinforced composites

    Page(s): 959 - 962
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (762 KB) |  | HTML iconHTML  

    We present a generalized adaptive activation function neuron structure which learns through an information-theoretic-based principle, which is able to estimate the probability density function of incoming input. It provides a low-order smooth robust estimate of the input signal probability density function. The presented method has been developed with reference to statistical characterization of polypropylene composites reinforced with vegetal fibers, that the proposed numerical experiments pertain to. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Performance analysis for a K-winners-take-all analog neural network: basic theory

    Page(s): 766 - 780
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (992 KB) |  | HTML iconHTML  

    In a previous work, the authors proposed an analog Hopfield-type neural network that identified the K largest components of a list of real numbers. In this work, we identify computable restrictions on the parameters, in order that the network can repeatedly process lists, one after the other, at a given rate. A complete mathematical analysis gives analytical bounds for the time required in terms of circuit parameters, the length of the lists, and the relative separation of list elements. This allows practical setting of circuit parameters for required clocking times. The emphasis is on high gain functioning of each neuron. Numerical investigations show the accuracy of the theoretical predictions, and study the influence of various parameters on performance. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Selective smoothing of the generative topographic mapping

    Page(s): 847 - 852
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (452 KB)  

    Generative topographic mapping is a nonlinear latent variable model introduced by Bishop et al. as a probabilistic reformulation of self-organizing maps. The complexity of this model is mostly determined by the number and form of basis functions generating the nonlinear mapping from latent space to data space, but it can be further controlled by adding a regularization term to increase the stiffness of the mapping and avoid data over-fitting. In this paper, we improve the map smoothing by introducing multiple regularization terms, one associated with each of the basis functions. A similar technique to that of automatic relevance determination, our selective map smoothing locally controls the stiffness of the mapping depending on length scales of the underlying manifold, while optimizing the effective number of active basis functions. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • A constructive algorithm for training cooperative neural network ensembles

    Page(s): 820 - 834
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (592 KB) |  | HTML iconHTML  

    Presents a constructive algorithm for training cooperative neural-network ensembles (CNNEs). CNNE combines ensemble architecture design with cooperative training for individual neural networks (NNs) in ensembles. Unlike most previous studies on training ensembles, CNNE puts emphasis on both accuracy and diversity among individual NNs in an ensemble. In order to maintain accuracy among individual NNs, the number of hidden nodes in individual NNs are also determined by a constructive approach. Incremental training based on negative correlation is used in CNNE to train individual NNs for different numbers of training epochs. The use of negative correlation learning and different training epochs for training individual NNs reflect CNNEs emphasis on diversity among individual NNs in an ensemble. CNNE has been tested extensively on a number of benchmark problems in machine learning and neural networks, including Australian credit card assessment, breast cancer, diabetes, glass, heart disease, letter recognition, soybean, and Mackey-Glass time series prediction problems. The experimental results show that CNNE can produce NN ensembles with good generalization ability. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • On the construction and training of reformulated radial basis function neural networks

    Page(s): 835 - 846
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (814 KB)  

    Presents a systematic approach for constructing reformulated radial basis function (RBF) neural networks, which was developed to facilitate their training by supervised learning algorithms based on gradient descent. This approach reduces the construction of radial basis function models to the selection of admissible generator functions. The selection of generator functions relies on the concept of the blind spot, which is introduced in the paper. The paper also introduces a new family of reformulated radial basis function neural networks, which are referred to as cosine radial basis functions. Cosine radial basis functions are constructed by linear generator functions of a special form and their use as similarity measures in radial basis function models is justified by their geometric interpretation. A set of experiments on a variety of datasets indicate that cosine radial basis functions outperform considerably conventional radial basis function neural networks with Gaussian radial basis functions. Cosine radial basis functions are also strong competitors to existing reformulated radial basis function models trained by gradient descent and feedforward neural networks with sigmoid hidden units. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Bias learning, knowledge sharing

    Page(s): 748 - 765
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (990 KB)  

    Biasing properly the hypothesis space of a learner has been shown to improve generalization performance. Methods for achieving this goal have been proposed, that range from designing and introducing a bias into a learner to automatically learning the bias. Multitask learning methods fall into the latter category. When several related tasks derived from the same domain are available, these methods use the domain-related knowledge coded in the training examples of all the tasks as a source of bias. We extend some of the ideas presented in this field and describe a new approach that identifies a family of hypotheses, represented by a manifold in hypothesis space, that embodies domain-related knowledge. This family is learned using training examples sampled from a group of related tasks. Learning models trained on these tasks are only allowed to select hypotheses that belong to the family. We show that the new approach encompasses a large variety of families which can be learned. A statistical analysis on a class of related tasks is performed that shows significantly improved performances when using this approach. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • A novel high-order associative memory system via discrete Taylor series

    Page(s): 734 - 747
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (2222 KB)  

    This paper proposes a novel high-order associative memory system (AMS) based on the discrete Taylor series (DTS). The mathematical foundation for the new AMS scheme is derived, three training algorithms are proposed, and the convergence of learning is proved. The DTS-AMS thus developed is capable of implementing error-free approximation to multivariable polynomial functions of arbitrary order. Compared with cerebellar model articulation controllers and radial basis function neural networks, it provides higher learning precision and less memory request. Furthermore, it offers less training computation and faster convergence rate than that attainable by multilayer perceptron. Numerical simulations show that the proposed DTS-AMS is effective in higher order function approximation and has potential in practical applications. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Kolmogorov's spline network

    Page(s): 725 - 733
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (677 KB) |  | HTML iconHTML  

    In this paper, an innovative neural-network architecture is proposed and elucidated. This architecture, based on the Kolmogorov's superposition theorem (1957) and called the Kolmogorov's spline network (KSN), utilizes more degrees of adaptation to data than currently used neural-network architectures (NNAs). By using cubic spline technique of approximation, both for activation and internal functions, more efficient approximation of multivariate functions can be achieved. The bound on approximation error and number of adjustable parameters, derived in this paper, favorably compares KSN with other one-hidden layer feedforward NNAs. The training of KSN, using the ensemble approach and the ensemble multinet, is described. A new explicit algorithm for constructing cubic splines is presented. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Independent component analysis of Gabor features for face recognition

    Page(s): 919 - 928
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (657 KB) |  | HTML iconHTML  

    We present an independent Gabor features (IGFs) method and its application to face recognition. The novelty of the IGF method comes from 1) the derivation of independent Gabor features in the feature extraction stage and 2) the development of an IGF features-based probabilistic reasoning model (PRM) classification method in the pattern recognition stage. In particular, the IGF method first derives a Gabor feature vector from a set of downsampled Gabor wavelet representations of face images, then reduces the dimensionality of the vector by means of principal component analysis, and finally defines the independent Gabor features based on the independent component analysis (ICA). The independence property of these Gabor features facilitates the application of the PRM method for classification. The rationale behind integrating the Gabor wavelets and the ICA is twofold. On the one hand, the Gabor transformed face images exhibit strong characteristics of spatial locality, scale, and orientation selectivity. These images can, thus, produce salient local features that are most suitable for face recognition. On the other hand, ICA would further reduce redundancy and represent independent features explicitly. These independent features are most useful for subsequent pattern discrimination and associative recall. Experiments on face recognition using the FacE REcognition Technology (FERET) and the ORL datasets, where the images vary in illumination, expression, pose, and scale, show the feasibility of the IGF method. In particular, the IGF method achieves 98.5% correct face recognition accuracy when using 180 features for the FERET dataset, and 100% accuracy for the ORL dataset using 88 features. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.

Aims & Scope

IEEE Transactions on Neural Networks is devoted to the science and technology of neural networks, which disclose significant technical knowledge, exploratory developments, and applications of neural networks from biology to software to hardware.

 

This Transactions ceased production in 2011. The current retitled publication is IEEE Transactions on Neural Networks and Learning Systems.

Full Aims & Scope