Recovery of sparse active inputs in general systems: A review

A staggering amount of attention was recently devoted to the study of compressed sensing and related areas using sparse priors in over parametrized linear models. The number of citations listed on the Rice University compressive sensing website within the last 5 years exceeds 600! The threshold phenomenon was empirically observed in early papers of Donoho et al : as the dimension of a random instance of a problem grows there is a sharp transition from successful recovery to failure as a function of the number of observations versus the dimension and sparsity of the unknown signal. We argue in this paper that there are strong parallels between this threshold behavior and earlier Shannon's justification for similar phenomenon in his notion of the capacity of the information transmission. Attempts of finding performance bounds for the compressive sensing with the standard information-theoretic tools were later made, e.g. by Wainwright et al. We outline sharp asymptotical bounds obtained in 1979 that were inspired by the Multi-Access Capacity Region theory and that can greatly enhance the current understanding of recovery of active inputs in sparse compressive sensing. These bounds were obtained for asymptotically optimal designs which include random designs and thus imply upper bounds for performance of recovery under arbitrary designs. These optimal designs are a natural choice in settings where the experiment can be predesigned, providing in addition a competitive performance of a simplified analysis method: separate testing of inputs (STI). STI originated in 1959 and can be further traced back to Fisher's idea of randomization in estimation. The STI capacity exceeds that of linear programming (LP) relaxation for randomized designs in a wide range of models and admits a straightforward generalization to nonparametric noisy models including `colored noise'. The latter observation inspires promising applications such as detecting the change-point in users profiles in a large computer network possibly caused by unauthorized intrusion into the system or monitoring either natural language corpora, or the phone call traffic in `sensitive' areas for their profiles matching those of special interest.