By Topic

Information Theory, IEEE Transactions on

Issue 4 • Date July 1974

Filter Results

Displaying Results 1 - 22 of 22
  • Bibliography on estimation of misclassification

    Page(s): 472 - 479
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (1622 KB)  

    Articles, books, and technical reports on the theoretical and experimental estimation of probability of misclassification are listed for the case of correctly labeled or preclassified training data. By way of introduction, the problem of estimating the probability of misclassification is discussed in order to characterize the contributions of the literature. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Review of 'Markov Processes and Learning Models' (Norman, M. F.; 1972)

    Page(s): 562 - 563
    Save to Project icon | Request Permissions | PDF file iconPDF (454 KB)  
    Freely Available from IEEE
  • A lower bound on the probability of decoding error for the finite-state channel (Corresp.)

    Page(s): 549 - 551
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (440 KB)  

    A new lower bound on the probability of decoding error for transmission at high rates over a finite-state channel is obtained. It is a dual to the random coding bound of Yudkin and is a generalization of the Arimoto converse to the coding theorem for discrete memoryless channels. It also implies the strong converse to the coding theorem for indecomposable channels. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • A general time-discrete equivalent to a time-continuous Gaussian channel (Corresp.)

    Page(s): 544 - 549
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (912 KB)  

    Transmission of a time-discrete message over a time-continuous channel is considered. The channel is assumed to be stationary and Gaussian. The following results are shown. 1) An optimum receiver will always contain a matched matrix filter followed by a sampling unit. 2) Jointly optimized transmitting and receiving filters will always be strictly band limited to a set of Nyquist domains. It is shown that both these results are true for any kind of message and under any measure of performance. On the basis of these results a general time-discrete matrix channel equivalent to the original time-continuous scalar channel is derived. The significance of the results is that any optimization of transmitter and receiver is reduced from a time-continuous problem to a time-discrete one. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Digital transmission with coherent four-dimensional modulation

    Page(s): 497 - 502
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (808 KB)  

    This paper examines codes for four-dimensional (4-D) modulation and their performance for digital transmission. The signals are defined byMpoints inside a sphere in four-dimensional Euclidian space. Three classes of 4-D codes are presented, and an algorithm is given which yields good 4-D codes of any length. Bounds on symbol error probability are plotted versus symbol-energy-to-noise-density ratio. The performance is shown to exceed that of amplitude- and-phase modulation in two independent two-dimensional channels. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Further results on decoders for Q -ary output channels (Corresp.)

    Page(s): 552 - 554
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (544 KB)  

    Decoders for channels with an alphabetAand an output alphabetAprime supseteq Aare given. It is shown that, whenA = View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Generating cyclically permutable codes (Corresp.)

    Page(s): 554 - 555
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (336 KB)  

    Large sets of cyclically permutable (or cyclically inequivalent) codewords can be created by interleaving the cyclic shifts of several shorter words and selecting a cyclically inequivalent subset from the resulting set, This correspondence presents an efficient algorithm for generating the cyclically inequivalent subset directly. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Hypothesis testing and information theory

    Page(s): 405 - 417
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (1672 KB)  

    The testing of binary hypotheses is developed from an information-theoretic point of view, and the asymptotic performance of optimum hypothesis testers is developed in exact analogy to the asymptotic performance of optimum channel codes. The discrimination, introduced by Kullback, is developed in a role analogous to that of mutual information in channel coding theory. Based on the discrimination, an error-exponent functione(r)is defined. This function is found to describe the behavior of optimum hypothesis testers asymptotically with block length. Next, mutual information is introduced as a minimum of a set of discriminations. This approach has later coding significance. The channel reliability-rate functionE(R)is defined in terms of discrimination, and a number of its mathematical properties developed. Sphere-packing-like bounds are developed in a relatively straightforward and intuitive manner by relatinge(r)andE (R). This ties together the aforementioned developments and gives a lower bound in terms of a hypothesis testing model. The result is valid for discrete or continuous probability distributions. The discrimination function is also used to define a source code reliability-rate function. This function allows a simpler proof of the source coding theorem and also bounds the code performance as a function of block length, thereby providing the source coding analog ofE (R). View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • The effects of a visual fidelity criterion of the encoding of images

    Page(s): 525 - 536
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (2832 KB)  

    Shannon's rate-distortion function provides a potentially useful lower bound against which to compare the rate-versus-distortion performance of practical encoding-transmission systems. However, this bound is not applicable unless one can arrive at a numerically-valued measure of distortion which is in reasonable correspondence with the subjective evaluation of the observer or interpreter. We have attempted to investigate this choice of distortion measure for monochrome still images. This investigation has considered a class of distortion measures for which it is possible to simulate the optimum (in a rate-distortion sense) encoding. Such simulation was performed at a fixed rate for various measures in the class and the results compared subjectively by observers. For several choices of transmission rate and original images, one distortion measure was fairly consistently rated as yielding the most satisfactory appearing encoded images. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Source coding theorems without the ergodic assumption

    Page(s): 502 - 516
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (2312 KB)  

    Source coding theorems are proved for discrete-time stationary processes subject to a fidelity criterion. The alphabet of the process is assumed to be a separable metric space, but the process is not assumed to be ergodic. When the process is not ergodic, the minimum average distortion for a fixed-rate code is not given by the distortion-rate function of the source as usually defined. It is given instead by a weighted average of the distortion-rate functions of ergodic subsources comprising the ergodic decomposition of the source. Potential applications to universal source coding with a fidelity criterion are discussed. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • A construction method for path-invariant comma-free codes (Corresp.)

    Page(s): 555 - 559
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (792 KB)  

    Many classes of comma-free codes have been proposed. Among them, path-invariant comma-free codes have the advantage of relative simplicity of encoding and decoding. In this correspondence, we present a construction method for these codes which is a unified generalization of the known construction methods. By this method some new classes of path-invariant comma-free codes are obtained. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Finite-memory problems and algorithms

    Page(s): 440 - 455
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (2312 KB)  

    A formulation of finite-memory information processing problems is presented. The total state space of the system, including the "memory" of the source and processor, is assumed to be finite. A cost functional is specified over the trajectories of the system and a variational approach is used to minimize cost. There results a two-point boundary value problem and an associated improvement algorithm. Special attention is given to two types of cost functionals: finite-time problems, and time-average problems over an infinite time interval. Several examples are included. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Automatic selection of reference data for use in a nearest-neighbor method of pattern classification (Corresp.)

    Page(s): 541 - 543
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (400 KB)  

    Two methods for automatic selection of nearest-neighbor reference data have been compared in small-scale experimentation. One of the methods is original and appears to be more successful than Hart's well-known "condensed nearest-neighbor" method [1]. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • A modification of Huffman's impulse-equivalent pulse trains to increase signal energy utilization (Corresp.)

    Page(s): 559 - 561
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (376 KB)  

    It is shown by several examples that if D. A. Huffman's pulse train design is modified to allow one nonzero pair of sidelobes in addition to the unavoidable end sidelobes in the autocorrelation function, then a substantial increase in the central peak for the same maximum pulse energy can be achieved. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • On the \epsilon -entropy and the rate-distortion function of certain non-Gaussian processes

    Page(s): 517 - 524
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (1096 KB)  

    Letxi = {xi(t), 0 leq t leq T}be a process with covariance functionK(s,t)andE int_0^T xi^2(t) dt < infty. It is proved that for everyvarepsilon > 0thevarepsilon-entropyH_{varepsilon}(xi)satisfies begin{equation} H_{varepsilon}(xi_g) - mathcal{H}_{xi_g} (xi) leq H_{varepsilon}(xi) leq H_{varepsilon}(xi_g) end{equation} wherexi_gis a Gaussian process with the covarianeeK(s,t)andmathcal{H}_{xi_g}(xi)is the entropy of the measure induced byxi(in function space) with respect to that induced byxi_g. It is also shown that ifmathcal{H}_{xi_g}(xi) < inftythen, asvarepsilon rightarrow 0begin{equation} H_{varepsilon}(xi) = H_{varepsilon}(xi_g) - mathcal{H}_{xi_g}(xi) + o(1). end{equation} Furthermore, ff there exists a Gaussian processg = { g(t); 0 leq t leq T }such thatmathcal{H}_g(xi) < infty, then the ratio betweenH_{varepsilon}(xi)andH_{varepsilon}(g)goes to one asvarepsilongoes to zero. Similar results are given for the rate-distortion function, and some particular examples are worked out in detail. Some cases for whichmathcal_{xi_g}(xi) = inftyare discussed, and asymptotic bounds onH_{varepsilon}(xi), expressed in terms ofH_{varepsilon}(xi_g), are derived. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • An upper bound on the error probability in decision-feedback equalization

    Page(s): 490 - 497
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (856 KB)  

    An upper bound on the error probability of a decision-feedback equalizer which takes into account the effect of error propagation is derived. The bound, which assumes independent data symbols and noise samples, is readily evaluated numerically for arbitrary tap gains and is valid for multilevel and nonequally likely data. One specific result for equally likely binary symbols is that if the worst case intersymbol interference when the firstJfeedback taps are Set to zero is less than the original signal voltage, then the error probability is multiplied by at most a factor of2^Jrelative to the error probability in the absence of decision errors at highS/Nratios. Numerical results are given for the special case of exponentially decreasing tap gains. These results demonstrate that the decision-feedback equalizer has a lower error probability than the linear zero-forcing equalizer when there is both a highS/Nratio and a fast roll-off of the feedback tap gains. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Robust detection to stochastic signals (Corresp.)

    Page(s): 537 - 541
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (576 KB)  

    In this correspondence we evaluate both the large and small sample performance of a limiter-quadratic detector (LQD) for the detection of a Gaussian signal in non-Gaussian noise. The LQD is shown to be robust for small as well as large sample sizes. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Optimum transmitting filter in digital PAM systems with a Viterbi detector

    Page(s): 479 - 489
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (1560 KB)  

    Optimization of the transmitting filter in a PAM system using a Viterbi detector of constrained complexity is considered. The receiving filter is considered to be a whitened matched filter. A constraint on detector complexity is obtained by limiting the length of the system impulse response. The results are applied to a channel with coaxial cable characteristics. Comparison with other detectors shows that the Viterbi detector is preferable even when the length of the system impulse response is quite short. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • When should a learning machine ask for help?

    Page(s): 455 - 471
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (2880 KB)  

    We present a view of the learning phase of statistical pattern recognition as a problem in optimum mode switching for learning systems which can operate in the supervised and nonsupervised modes. We assume the standardJ-category statistical pattern recognition model, in which patterns are represented as points in Euclideann-space and the learning problem is to estimate the unknowns in the problem probability structure. More specifically, we assume each learning sample can be processed in either mode, but the machine incurs a cost for this processing--a larger cost for processing in the supervised mode than in the nonsupervised mode. The goal is to have the machine make the decision for each learning pattern concerning mode usage that results in minimum expected cost to learn the unknowns to a predetermined accuracy. We treat the parametric problem as a problem in stochastic control. Simple closed-form expressions partially describing system performance are derived for very general problem probability structures for the case of good learning, or, equivalently, large number of learning samples. Among the results obtained for identifiable probability structures for this case are i) expressions for purely supervised and purely nonsupervised learning costs; ii) a proof that supervised learning is always faster (though not necessarily less costly) than is nonsupervised learning; iii) an example showing that, depending on the relative costs of the two mode usages as well as on the problem probability structure, the learning cost of an optimum combined-mode learning system can be remarkably lower than that of a pure-mode learning system; iv) an argument to the effect that the a posteriori distribution of the unknown parameter vector is asymptotically Gaussian for a wide range of mode usage policies; v) a fairly simple functional equation that can be solved numerically for the optimum mode usage policy (for some probability structures the nature of the optimum mode usage policy can be inferred without resorting to computer calculation); vi) the conclusion that in general optimum mode usage involves mode switching, i.e., pure-mode learning is not optimum. For the most general discretized nonidentifiable prob- ability structure, we show that dual-mode learning may be significantly less costly than is purely supervised learning. This example also illustrates the effectiveness of making use of hard constraints, imposed by prior knowledge or experimentation, in reducing learning cost. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • The probability density function for the output of an analog cross-correlator with correlated bandpass inputs

    Page(s): 433 - 440
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (992 KB)  

    The probability density function (pdf) for the output of an analog cross-correlator with correlated bandpass inputs is derived. The pdf is derived by a "direct method" without resorting to the "characteristic function method," which usually requires contour integrations in a complex plane for inversion operations. The correlator consists of bandpass filters, a multiplier, and a zonal low-pass filter. We treat the general situation in which the two inputs are narrow-band signals of unequal power and of different phases. The bandpass input noises are assumed to be correlated and may have different powers. In the Appendix, another derivation for the pdf is given in the special case of equal power correlated noise. This derivation is based on the fact that the correlator output random variable is the difference of two independent noncentral chi-square variables of two degrees of freedom. We show that the two expressions for the pdf (one from the direct method and the other from the characteristic function method) are indeed equivalent. Finally, we discuss two major areas of application. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Derivation and evaluation of improved tracking filter for use in dense multitarget environments

    Page(s): 423 - 432
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (1584 KB)  

    When tracking targets in dense environments, sensor reports originating from sources other than the target being tracked (i.e., from clutter, thermal false alarms, other targets) are occasionally incorrectly used in track updating. As a result tracking performance degrades, and the error covariance matrix calculated on-line by the usual types of tracking filters becomes extremely unreliable for estimating actual accuracies. This paper makes three contributions in this area. First, a new tracking filter is developed that incorporates, in an a posteriori statistical fashion, all data available from sensor reports located in the vicinity of the track, and that provides both optimal performance and reliable estimates of this performance when operating in dense environments. The optimality of and the performance equations for this filter are verified by analytical and simulation results. Second, several computationally efficient classes of suboptimal tracking filters based on the optimal filter developed in this paper and on an optimal filter of another class that appeared previously in the literature are developed. Third, using an extensive Monte Carlo simulation, the various optimal and suboptimal filters as well as the Kalman filter are compared, with regard to the differences between the on-line calculated and experimental covariances of each filter, and with regard to relative accuracies, computational requirements, and numbers of divergences or lost tracks each produces. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • A likelihood ratio formula for two-dimensional random fields

    Page(s): 418 - 422
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (600 KB)  

    This paper is concerned with the detection of a random signal in white Gaussian noise when both the signal and the noise are two-dimensional random fields. The principal result is the derivation of a recursive formula for the likelihood ratio relating it to certain conditional moments of the signal. It is also shown that, except for some relatively uninteresting cases, a simple exponential formula for the likelihood ratio, such as one has in one dimension, is not possible. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.

Aims & Scope

IEEE Transactions on Information Theory publishes papers concerned with the transmission, processing, and utilization of information.

Full Aims & Scope

Meet Our Editors

Editor-in-Chief
Frank R. Kschischang

Department of Electrical and Computer Engineering