By Topic

Neural Networks, IEEE Transactions on

Issue 3 • Date May 2003

Filter Results

Displaying Results 1 - 23 of 23
  • Selecting a restoration technique to minimize OCR error

    Page(s): 478 - 490
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (986 KB) |  | HTML iconHTML  

    This paper introduces a learning problem related to the task of converting printed documents to ASCII text files. The goal of the learning procedure is to produce a function that maps documents to restoration techniques in such a way that on average the restored documents have minimum optical character recognition error. We derive a general form for the optimal function and use it to motivate the development of a nonparametric method based on nearest neighbors. We also develop a direct method of solution based on empirical error minimization for which we prove a finite sample bound on estimation error that is independent of distribution. We show that this empirical error minimization problem is an extension of the empirical optimization problem for traditional M-class classification with general loss function and prove computational hardness for this problem. We then derive a simple iterative algorithm called generalized multiclass ratchet (GMR) and prove that it produces an optimal function asymptotically (with probability 1). To obtain the GMR algorithm we introduce a new data map that extends Kesler's construction for the multiclass problem and then apply an algorithm called Ratchet to this mapped data, where Ratchet is a modification of the Pocket algorithm . Finally, we apply these methods to a collection of documents and report on the experimental results. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • A study of pattern recovery in recurrent correlation associative memories

    Page(s): 506 - 519
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (651 KB) |  | HTML iconHTML  

    In this paper, we analyze the recurrent correlation associative memory (RCAM) model of Chiueh and Goodman (1990, 1991). This is an associative memory in which stored binary memory patterns are recalled via an iterative update rule. The update of the individual pattern-bits is controlled by an excitation function, which takes as its argument the inner product between the stored memory patterns and the input patterns. Our contribution is to analyze the dynamics of pattern recall when the input patterns are corrupted by noise of a relatively unrestricted class. We show how to identify the excitation function which maximizes the separation (the Fisher discriminant) between the uncorrupted realization of the noisy input pattern and the remaining patterns residing in the memory. The excitation function which gives maximum separation is exponential when the input bit-errors follow a binomial distribution. We develop an expression for the expectation value of bit-error probability on the input pattern after one iteration. We show how to identify the excitation function which minimizes the bit-error probability. The relationship between the excitation functions which result from the two different approaches is examined for a binomial distribution of bit-errors. We develop a semiempirical approach to the modeling of the dynamics of the RCAM. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • An adaptive resource-allocating network for automated detection, segmentation, and classification of breast cancer nuclei topic area: image processing and recognition

    Page(s): 680 - 687
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (644 KB) |  | HTML iconHTML  

    This paper presents a unified image analysis approach for automated detection, segmentation, and classification of breast cancer nuclei using a neural network, which learns to cluster shapes and to classify nuclei. The proposed neural network is incrementally grown by creating a new cluster whenever a previously unseen shape is presented. Each hidden node represents a cluster used as a template to provide faster and more accurate nuclei detection and segmentation. Online learning gives the system improved performance with continued use. The effectiveness of the resulting system is demonstrated on a task of cytological image analysis, with classification of individual nuclei used to diagnose the sample. This demonstrates the potential effectiveness of such a system on diagnostic tasks that require the classification of individual cells. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Trajectory generation and modulation using dynamic neural networks

    Page(s): 520 - 533
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (699 KB) |  | HTML iconHTML  

    Generation of desired trajectory behavior using neural networks involves a particularly challenging spatio-temporal learning problem. This paper introduces a novel solution, i.e., designing a dynamic system whose terminal behavior emulates a prespecified spatio-temporal pattern independently of its initial conditions. The proposed solution uses a dynamic neural network (DNN), a hybrid architecture that employs a recurrent neural network (RNN) in cascade with a nonrecurrent neural network (NRNN). The RNN generates a simple limit cycle, which the NRNN reshapes into the desired trajectory. This architecture is simple to train. A systematic synthesis procedure based on the design of relay control systems is developed for configuring an RNN that can produce a limit cycle of elementary complexity. It is further shown that a cascade arrangement of this RNN and an appropriately trained NRNN can emulate any desired trajectory behavior irrespective of its complexity. An interesting solution to the trajectory modulation problem, i.e., online modulation of the generated trajectories using external inputs, is also presented. Results of several experiments are included to demonstrate the capabilities and performance of the DNN in handling trajectory generation and modulation problems. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • A robust approach to independent component analysis of signals with high-level noise measurements

    Page(s): 631 - 645
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (1364 KB) |  | HTML iconHTML  

    We propose a robust approach for independent component analysis (ICA) of signals where observations are contaminated with high-level additive noise and/or outliers. The source signals may contain mixtures of both sub-Gaussian and super-Gaussian components, and the number of sources is unknown. Our robust approach includes two procedures. In the first procedure, a robust prewhitening technique is used to reduce the power of additive noise, the dimensionality and the correlation among sources. A cross-validation technique is introduced to estimate the number of sources in this first procedure. In the second procedure, a nonlinear function is derived using the parameterized t-distribution density model. This nonlinear function is robust against the undue influence of outliers fundamentally. Moreover, the stability of the proposed algorithm and the robust property of misestimating the parameters (kurtosis) have been studied. By combining the t-distribution model with a family of light-tailed distributions (sub-Gaussian) model, we can separate the mixture of sub-Gaussian and super-Gaussian source components. Through the analysis of artificially synthesized data and real-world magnetoencephalographic (MEG) data, we illustrate the efficacy of this robust approach. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • An empirical investigation of bias and variance in time series forecasting: modeling considerations and error evaluation

    Page(s): 668 - 679
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (588 KB)  

    Bias and variance play an important role in understanding the fundamental issue of learning and generalization in neural network modeling. Several studies on bias and variance effects have been published in classification and regression related research of neural networks. However, little research has been done in this area for time-series modeling and forecasting. We consider modeling issues related to understanding error components given the common practices associated with neural-network time-series forecasting. We point out the key difference between classification and time-series problems in consideration of the bias-plus-variance decomposition. A Monte Carlo study on the role of bias and variance in neural networks time-series forecasting is conducted. We find that both input lag structure and hidden nodes are important in contributing to the overall forecasting performance. The results also suggest that overspecification of input nodes in neural network modeling does not impact the model bias, but has significant effect on the model variance. Methods such as neural ensembles that focus on reducing the model variance, therefore, can be valuable and effective in time-series forecasting modeling. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Neural associative memory storing gray-coded gray-scale images

    Page(s): 703 - 707
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (561 KB) |  | HTML iconHTML  

    We present a neural associative memory storing gray-scale images. The proposed approach is based on a suitable decomposition of the gray-scale image into gray-coded binary images, stored in brain-state-in-a-box-type binary neural networks. Both learning and recall can be implemented by parallel computation, with time saving. The learning algorithm, used to store the binary images, guarantees asymptotic stability of the stored patterns, low computational cost, and control of the weights precision. Some design examples and computer simulations are presented to show the effectiveness of the proposed method. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Algorithms for nonnegative independent component analysis

    Page(s): 534 - 543
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (1810 KB)  

    We consider the task of solving the independent component analysis (ICA) problem x=As given observations x, with a constraint of nonnegativity of the source random vector s. We refer to this as nonnegative independent component analysis and we consider methods for solving this task. For independent sources with nonzero probability density function (pdf) p(s) down to s=0 it is sufficient to find the orthonormal rotation y=Wz of prewhitened sources z=Vx, which minimizes the mean squared error of the reconstruction of z from the rectified version y+ of y. We suggest some algorithms which perform this, both based on a nonlinear principal component analysis (PCA) approach and on a geodesic search method driven by differential geometry considerations. We demonstrate the operation of these algorithms on an image separation problem, which shows in particular the fast convergence of the rotation and geodesic methods and apply the approach to a musical audio analysis task. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Automatic change detection of driving environments in a vision-based driver assistance system

    Page(s): 646 - 657
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (1004 KB) |  | HTML iconHTML  

    Detecting critical changes of environments while driving is an important task in driver assistance systems. In this paper, a computational model motivated by human cognitive processing and selective attention is proposed for this purpose. The computational model consists of three major components, referred to as the sensory, perceptual, and conceptual analyzers. The sensory analyzer extracts temporal and spatial information from video sequences. The extracted information serves as the input stimuli to a spatiotemporal attention (STA) neural network embedded in the perceptual analyzer. If consistent stimuli repeatedly innervate the neural network, a focus of attention will be established in the network. The attention pattern associated with the focus, together with the location and direction of motion of the pattern, form what we call a categorical feature. Based on this feature, the class of the attention pattern and, in turn, the change in driving environment corresponding to the class are determined using a configurable adaptive resonance theory (CART) neural network, which is placed in the conceptual analyzer. Various changes in driving environment, both in daytime and at night, have been tested. The experimental results demonstrated the feasibilities of both the proposed computational model and the change detection system. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Further results on adaptive control for a class of nonlinear systems using neural networks

    Page(s): 719 - 722
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (303 KB) |  | HTML iconHTML  

    Zhang et al. presented an excellent neural-network (NN) controller for a class of nonlinear control designs. The singularity issue is completely avoided. Based on a modified Lyapunov function, their lemma illustrates the existence of an ideal control which is important in establishing the NN approximator. In this paper, we provide a Lyapunov function to realize an alternative ideal control which is more direct and simpler. The major contributions of this paper are divided into two parts. First, it proposes a control scheme which results in a smaller dimensionality of NN than that of Zhang et al. In this way, the proposed NN controller is easier to implement and more reliable for practical purposes. Second, by removing certain restrictions from the design reported by Zhang et al., we further develop a new NN controller, which can be applied to a wider class of systems. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Clustering-based algorithms for single-hidden-layer sigmoid perceptron

    Page(s): 708 - 715
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (509 KB) |  | HTML iconHTML  

    Gradient-descent type supervised learning is the most commonly used algorithm for design of the standard sigmoid perceptron (SP). However, it is computationally expensive (slow) and has the local-minima problem. Moody and Darken (1989) proposed an input-clustering based hierarchical algorithm for fast learning in networks of locally tuned neurons in the context of radial basis function networks. We propose and analyze input clustering (IC) and input-output clustering (IOC)-based algorithms for fast learning in networks of globally tuned neurons in the context of the SP. It is shown that "localizing'' the input layer weights of the SP by the IC and the IOC minimizes an upper bound to the SP output error. The proposed algorithms could possibly be used also to initialize the SP weights for the conventional gradient-descent learning. Simulation results offer that the SPs designed by the IC and the IOC yield comparable performance in comparison with its radial basis function network counterparts. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Linear dependency between ε and the input noise in ε-support vector regression

    Page(s): 544 - 553
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (649 KB) |  | HTML iconHTML  

    In using the ε-support vector regression (ε-SVR) algorithm, one has to decide a suitable value for the insensitivity parameter ε. Smola et al. considered its "optimal" choice by studying the statistical efficiency in a location parameter estimation problem. While they successfully predicted a linear scaling between the optimal ε and the noise in the data, their theoretically optimal value does not have a close match with its experimentally observed counterpart in the case of Gaussian noise. In this paper, we attempt to better explain their experimental results by studying the regression problem itself. Our resultant predicted choice of ε is much closer to the experimentally observed optimal value, while again demonstrating a linear trend with the input noise. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Flexible neuro-fuzzy systems

    Page(s): 554 - 574
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (3570 KB) |  | HTML iconHTML  

    In this paper, we derive new neuro-fuzzy structures called flexible neuro-fuzzy inference systems or FLEXNFIS. Based on the input-output data, we learn not only the parameters of the membership functions but also the type of the systems (Mamdani or logical). Moreover, we introduce: 1) softness to fuzzy implication operators, to aggregation of rules and to connectives of antecedents; 2) certainty weights to aggregation of rules and to connectives of antecedents; and 3) parameterized families of T-norms and S-norms to fuzzy implication operators, to aggregation of rules and to connectives of antecedents. Our approach introduces more flexibility to the structure and design of neuro-fuzzy systems. Through computer simulations, we show that Mamdani-type systems are more suitable to approximation problems, whereas logical-type systems may be preferred for classification problems. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • On the number of multilinear partitions and the computing capacity of multiple-valued multiple-threshold perceptrons

    Page(s): 469 - 477
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (539 KB) |  | HTML iconHTML  

    We introduce the concept of multilinear partition of a point set V⊂Rn and the concept of multilinear separability of a function f:V→K={0,...,k-1}. Based on well-known relationships between linear partitions and minimal pairs, we derive formulae for the number of multilinear partitions of a point set in general position and of the set K2. The (n,k,s)-perceptrons partition the input space V into s+1 regions with s parallel hyperplanes. We obtain results on the capacity of a single (n,k,s)-perceptron, respectively, for V⊂Rn in general position and for V=K2. Finally, we describe a fast polynomial-time algorithm for counting the multilinear partitions of K2. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Pruning error minimization in least squares support vector machines

    Page(s): 696 - 702
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (340 KB) |  | HTML iconHTML  

    The support vector machine (SVM) is a method for classification and for function approximation. This method commonly makes use of an ε-insensitive cost function, meaning that errors smaller than ε remain unpunished. As an alternative, a least squares support vector machine (LSSVM) uses a quadratic cost function. When the LSSVM method is used for function approximation, a nonsparse solution is obtained. The sparseness is imposed by pruning, i.e., recursively solving the approximation problem and subsequently omitting data that has a small error in the previous pass. However, omitting data with a small approximation error in the previous pass does not reliably predict what the error will be after the sample has been omitted. In this paper, a procedure is introduced that selects from a data set the training sample that will introduce the smallest approximation error when it will be omitted. It is shown that this pruning scheme outperforms the standard one. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Global exponential stability of competitive neural networks with different time scales

    Page(s): 716 - 719
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (265 KB) |  | HTML iconHTML  

    The dynamics of cortical cognitive maps developed by self-organization must include the aspects of long and short-term memory. The behavior of such a neural network is characterized by an equation of neural activity as a fast phenomenon and an equation of synaptic modification as a slow part of the neural system. We present a new method of analyzing the dynamics of a biological relevant system with different time scales based on the theory of flow invariance. We are able to show the conditions under which the solutions of such a system are bounded being less restrictive than with the K-monotone theory, singular perturbation theory, or those based on supervised synaptic learning. We prove the existence and the uniqueness of the equilibrium. A strict Lyapunov function for the flow of a competitive neural system with different time scales is given and based on it we are able to prove the global exponential stability of the equilibrium point. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • COVNET: a cooperative coevolutionary model for evolving artificial neural networks

    Page(s): 575 - 596
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (783 KB) |  | HTML iconHTML  

    This paper presents COVNET, a new cooperative coevolutionary model for evolving artificial neural networks. This model is based on the idea of coevolving subnetworks that must cooperate to form a solution for a specific problem, instead of evolving complete networks. The combination of this subnetworks is part of a coevolutionary process. The best combinations of subnetworks must be evolved together with the coevolution of the subnetworks. Several subpopulations of subnetworks coevolve cooperatively and genetically isolated. The individual of every subpopulation are combined to form whole networks. This is a different approach from most current models of evolutionary neural networks which try to develop whole networks. COVNET places as few restrictions as possible over the network structure, allowing the model to reach a wide variety of architectures during the evolution and to be easily extensible to other kind of neural networks. The performance of the model in solving three real problems of classification is compared with a modular network, the adaptive mixture of experts and with the results presented in the bibliography. COVNET has shown better generalization and produced smaller networks than the adaptive mixture of experts and has also achieved results, at least, comparable with the results in the bibliography. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Two-stage clustering via neural networks

    Page(s): 606 - 615
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (796 KB) |  | HTML iconHTML  

    This paper presents a two-stage approach that is effective for performing fast clustering. First, a competitive neural network (CNN) that can harmonize mean squared error and information entropy criteria is employed to exploit the substructure in the input data by identifying the local density centers. A Gravitation neural network (GNN) then takes the locations of these centers as initial weight vectors and undergoes an unsupervised update process to group the centers into clusters. Each node (called gravi-node) in the GNN is associated with a finite attraction radius and would be attracted to a nearby centroid simultaneously during the update process, creating the Gravitation-like behavior without incurring complicated computations. This update process iterates until convergence and the converged centroid corresponds to a cluster. Compared to other clustering methods, the proposed clustering scheme is free of initialization problem and does not need to pre-specify the number of clusters. The two-stage approach is computationally efficient and has great flexibility in implementation. A fully parallel hardware implementation is very possible. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • A self-organizing map for adaptive processing of structured data

    Page(s): 491 - 505
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (1447 KB) |  | HTML iconHTML  

    Recent developments in the area of neural networks produced models capable of dealing with structured data. Here, we propose the first fully unsupervised model, namely an extension of traditional self-organizing maps (SOMs), for the processing of labeled directed acyclic graphs (DAGs). The extension is obtained by using the unfolding procedure adopted in recurrent and recursive neural networks, with the replicated neurons in the unfolded network comprising of a full SOM. This approach enables the discovery of similarities among objects including vectors consisting of numerical data. The capabilities of the model are analyzed in detail by utilizing a relatively large data set taken from an artificial benchmark problem involving visual patterns encoded as labeled DAGs. The experimental results demonstrate clearly that the proposed model is capable of exploiting both information conveyed in the labels attached to each node of the input DAGs and information encoded in the DAG topology. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • FPGA implementation of a pulse density neural network with learning ability using simultaneous perturbation

    Page(s): 688 - 695
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (2841 KB) |  | HTML iconHTML  

    Hardware realization is very important when considering wider applications of neural networks (NNs). In particular, hardware NNs with a learning ability are intriguing. In these networks, the learning scheme is of much interest, with the backpropagation method being widely used. A gradient type of learning rule is not easy to realize in an electronic system, since calculation of the gradients for all weights in the network is very difficult. More suitable is the simultaneous perturbation method, since the learning rule requires only forward operations of the network to modify weights unlike the backpropagation method. In addition, pulse density NN systems have some promising properties, as they are robust to noisy situations and can handle analog quantities based on the digital circuits. We describe a field-programmable gate array realization of a pulse density NN using the simultaneous perturbation method as the learning scheme. We confirm the viability of the design and the operation of the actual NN system through some examples. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Classification in a normalized feature space using support vector machines

    Page(s): 597 - 605
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (582 KB)  

    This paper discusses classification using support vector machines in a normalized feature space. We consider both normalization in input space and in feature space. Exploiting the fact that in this setting all points lie on the surface of a unit hypersphere we replace the optimal separating hyperplane by one that is symmetric in its angles, leading to an improved estimator. Evaluation of these considerations is done in numerical experiments on two real-world datasets. The stability to noise of this offset correction is subsequently investigated as well as its optimality. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • A dual neural network for redundancy resolution of kinematically redundant manipulators subject to joint limits and joint velocity limits

    Page(s): 658 - 667
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (561 KB) |  | HTML iconHTML  

    In this paper, a recurrent neural network called the dual neural network is proposed for online redundancy resolution of kinematically redundant manipulators. Physical constraints such as joint limits and joint velocity limits, together with the drift-free criterion as a secondary task, are incorporated into the problem formulation of redundancy resolution. Compared to other recurrent neural networks, the dual neural network is piecewise linear and has much simpler architecture with only one layer of neurons. The dual neural network is shown to be globally (exponentially) convergent to optimal solutions. The dual neural network is simulated to control the PA10 robot manipulator with effectiveness demonstrated. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • An efficient fully unsupervised video object segmentation scheme using an adaptive neural-network classifier architecture

    Page(s): 616 - 630
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (1189 KB) |  | HTML iconHTML  

    In this paper, an unsupervised video object (VO) segmentation and tracking algorithm is proposed based on an adaptable neural-network architecture. The proposed scheme comprises: 1) a VO tracking module and 2) an initial VO estimation module. Object tracking is handled as a classification problem and implemented through an adaptive network classifier, which provides better results compared to conventional motion-based tracking algorithms. Network adaptation is accomplished through an efficient and cost effective weight updating algorithm, providing a minimum degradation of the previous network knowledge and taking into account the current content conditions. A retraining set is constructed and used for this purpose based on initial VO estimation results. Two different scenarios are investigated. The first concerns extraction of human entities in video conferencing applications, while the second exploits depth information to identify generic VOs in stereoscopic video sequences. Human face/ body detection based on Gaussian distributions is accomplished in the first scenario, while segmentation fusion is obtained using color and depth information in the second scenario. A decision mechanism is also incorporated to detect time instances for weight updating. Experimental results and comparisons indicate the good performance of the proposed scheme even in sequences with complicated content (object bending, occlusion). View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.

Aims & Scope

IEEE Transactions on Neural Networks is devoted to the science and technology of neural networks, which disclose significant technical knowledge, exploratory developments, and applications of neural networks from biology to software to hardware.

 

This Transactions ceased production in 2011. The current retitled publication is IEEE Transactions on Neural Networks and Learning Systems.

Full Aims & Scope