By Topic

On minimizing distortion and relative entropy

Sign In

Cookies must be enabled to login.After enabling cookies , please use refresh or reload or ctrl+f5 on the browser for the login options.

Formats Non-Member Member
$31 $13
Learn how you can qualify for the best price for this item!
Become an IEEE Member or Subscribe to
IEEE Xplore for exclusive pricing!
close button

puzzle piece

IEEE membership options for an individual and IEEE Xplore subscriptions for an organization offer the most affordable access to essential journal articles, conference papers, standards, eBooks, and eLearning courses.

Learn more about:

IEEE membership

IEEE Xplore subscriptions

2 Author(s)
Friedlander, M.P. ; Dept. of Comput. Sci., Univ. of British Columbia, Vancouver, BC, Canada ; Gupta, M.R.

A common approach for estimating a probability mass function w when given a prior q and moment constraints given by Aw≤b is to minimize the relative entropy between w and q subject to the set of linear constraints. In such cases, the solution w is known to have exponential form. We consider the case in which the linear constraints are noisy, uncertain, infeasible, or otherwise "soft." A solution can then be obtained by minimizing both the relative entropy and violation of the constraints Aw≤b. A penalty parameter σ weights the relative importance of these two objectives. We show that this penalty formulation also yields a solution w with exponential form. If the distortion is based on an ℓp norm, then the exponential form of w is shown to have exponential decay parameters that are bounded as a function of σ. We also state conditions under which the solution w to the penalty formulation will result in zero distortion, so that the moment constraints hold exactly. These properties are useful in choosing penalty parameters, evaluating the impact of chosen penalty parameters, and proving properties about methods that use such penalty formulations.

Published in:

Information Theory, IEEE Transactions on  (Volume:52 ,  Issue: 1 )