Scheduled System Maintenance:
Some services will be unavailable Sunday, March 29th through Monday, March 30th. We apologize for the inconvenience.
By Topic

Advanced Mean Field Methods:Theory and Practice

Cover Image Copyright Year: 2001
Author(s): Opper, M.; Saad, D.
Publisher: MIT Press
Content Type : Books & eBooks
Topics: Computing & Processing (Hardware/Software)
  • Print

Abstract

A major problem in modern probabilistic modeling is the huge computational complexity involved in typical calculations with multivariate probability distributions when the number of random variables is large. Because exact computations are infeasible in such cases and Monte Carlo sampling techniques may reach their limits, there is a need for methods that allow for efficient approximate computations. One of the simplest approximations is based on the mean field method, which has a long history in statistical physics. The method is widely used, particularly in the growing field of graphical models.Researchers from disciplines such as statistical physics, computer science, and mathematical statistics are studying ways to improve this and related methods and are exploring novel application areas. Leading approaches include the variational approach, which goes beyond factorizable distributions to achieve systematic improvements; the TAP (Thouless-Anderson-Palmer) approach, which incorporates correlations by including effective reaction terms in the mean field theory; and the more general methods of graphical models.Bringing together ideas and techniques from these diverse disciplines, this book covers the theoretical foundations of advanced mean field methods, explores the relation between the different approaches, examines the quality of the approximation obtained, and demonstrates their application to various areas of probabilistic modeling.

  •   Click to expandTable of Contents

    • Full text access may be available. Click article title to sign in or learn about subscription options.

      Front Matter

      Opper, M. ; Saad, D.
      Advanced Mean Field Methods:Theory and Practice

      Page(s): i - xiv
      Copyright Year: 2001

      MIT Press eBook Chapters

      This chapter contains sections titled: Half Title, Neural Information Processing Series, Title, Copyright, Contents, Series Foreword, Foreword, Contributors, Acknowledgments View full abstract»

    • Full text access may be available. Click article title to sign in or learn about subscription options.

      Introduction

      Opper, M. ; Saad, D.
      Advanced Mean Field Methods:Theory and Practice

      Page(s): 1 - 5
      Copyright Year: 2001

      MIT Press eBook Chapters

      A major problem in modern probabilistic modeling is the huge computational complexity involved in typical calculations with multivariate probability distributions when the number of random variables is large. Because exact computations are infeasible in such cases and Monte Carlo sampling techniques may reach their limits, there is a need for methods that allow for efficient approximate computations. One of the simplest approximations is based on the mean field method, which has a long history in statistical physics. The method is widely used, particularly in the growing field of graphical models.Researchers from disciplines such as statistical physics, computer science, and mathematical statistics are studying ways to improve this and related methods and are exploring novel application areas. Leading approaches include the variational approach, which goes beyond factorizable distributions to achieve systematic improvements; the TAP (Thouless-Anderson-Palmer) approach, which incorporates correlations by including effective reaction terms in the mean field theory; and the more general methods of graphical models.Bringing together ideas and techniques from these diverse disciplines, this book covers the theoretical foundations of advanced mean field methods, explores the relation between the different approaches, examines the quality of the approximation obtained, and demonstrates their application to various areas of probabilistic modeling. View full abstract»

    • Full text access may be available. Click article title to sign in or learn about subscription options.

      From Naive Mean Field Theory to the TAP Equations

      Opper, M. ; Saad, D.
      Advanced Mean Field Methods:Theory and Practice

      Page(s): 7 - 20
      Copyright Year: 2001

      MIT Press eBook Chapters

      This chapter contains sections titled: Introduction, The Variational Mean Field Method, The Linear Response Correction, The Field Theoretic Approach, When does MFT become exact?, TAP equations I: The cavity approach, TAP equations II: Plefka's Expansion, TAP equations III: Beyond the SK model, Outlook, References View full abstract»

    • Full text access may be available. Click article title to sign in or learn about subscription options.

      An Idiosyncratic Journey Beyond Mean Field Theory

      Opper, M. ; Saad, D.
      Advanced Mean Field Methods:Theory and Practice

      Page(s): 21 - 35
      Copyright Year: 2001

      MIT Press eBook Chapters

      This chapter contains sections titled: Introduction, Inference, Some Models from Statistical Physics, The Gibbs Free Energy, Mean Field Theory: The Variational Approach, Correcting Mean Field Theory, The Bethe Approximation, Belief Propagation, Kikuchi Approximations and Generalized Belief Propagation, Acknowledgments, References View full abstract»

    • Full text access may be available. Click article title to sign in or learn about subscription options.

      An Idiosyncratic Journey Beyond Mean Field Theory

      Opper, M. ; Saad, D.
      Advanced Mean Field Methods:Theory and Practice

      Page(s): 37 - 49
      Copyright Year: 2001

      MIT Press eBook Chapters

      This chapter contains sections titled: Introduction, Mean field theory, Boltzmann distributions, The general case, Sigmoid belief networks, Appendix A: The TAP equation for Boltzmann distributions and asymmetric networks, Acknowledgments, References View full abstract»

    • Full text access may be available. Click article title to sign in or learn about subscription options.

      The TAP Approach to Intensive and Extensive Connectivity Systems

      Opper, M. ; Saad, D.
      Advanced Mean Field Methods:Theory and Practice

      Page(s): 51 - 65
      Copyright Year: 2001

      MIT Press eBook Chapters

      This chapter contains sections titled: Introduction, The general framework, The TAP approach, Example - the Hopfield model, Summary, Acknowledgments, References View full abstract»

    • Full text access may be available. Click article title to sign in or learn about subscription options.

      TAP for Parity Check Error Correcting Codes

      Opper, M. ; Saad, D.
      Advanced Mean Field Methods:Theory and Practice

      Page(s): 67 - 83
      Copyright Year: 2001

      MIT Press eBook Chapters

      This chapter contains sections titled: Introduction, Noisy information transmission, Sparse parity-check codes, Decoding: Belief propagation, Decoding: the TAP approach, Experimental results, Summary, Acknowledgments, References View full abstract»

    • Full text access may be available. Click article title to sign in or learn about subscription options.

      Adaptive TAP Equations

      Opper, M. ; Saad, D.
      Advanced Mean Field Methods:Theory and Practice

      Page(s): 85 - 97
      Copyright Year: 2001

      MIT Press eBook Chapters

      This chapter contains sections titled: Introduction, Models with quadratic interactions, Deriving the TAP equations, Solving the TAP equations, Example I: The Hopfield model, Example II: Bayesian learning with a perceptron, Outlook, References View full abstract»

    • Full text access may be available. Click article title to sign in or learn about subscription options.

      Mean-field Theory of Learning: From Dynamics to Statics

      Opper, M. ; Saad, D.
      Advanced Mean Field Methods:Theory and Practice

      Page(s): 99 - 117
      Copyright Year: 2001

      MIT Press eBook Chapters

      This chapter contains sections titled: Introduction, Formulation, The Cavity Method, Simulation results, Steady-state behavior, The picture of rough energy landscapes, Conclusion, Acknowledgments, Appendix A: The Green's function, Appendix B: The fluctuation response relation, Appendix C: Macroscopic parameters in rough energy landscapes, References View full abstract»

    • Full text access may be available. Click article title to sign in or learn about subscription options.

      Saddle-point Methods for Intractable Graphical Models

      Opper, M. ; Saad, D.
      Advanced Mean Field Methods:Theory and Practice

      Page(s): 119 - 128
      Copyright Year: 2001

      MIT Press eBook Chapters

      This chapter contains sections titled: Introduction, Sigmoid belief networks, The saddle-point approximation, Numerical examples, Discussion and conclusions, References View full abstract»

    • Full text access may be available. Click article title to sign in or learn about subscription options.

      Tutorial on Variational Approximation Methods

      Opper, M. ; Saad, D.
      Advanced Mean Field Methods:Theory and Practice

      Page(s): 129 - 159
      Copyright Year: 2001

      MIT Press eBook Chapters

      This chapter contains sections titled: Introduction, Examples of variational methods, A brief introduction to graphical models, Variational mean field method, Structured variational approach, Local variational approach, Parameter estimation with variational methods, Variational Bayesian methods, Discussion, References View full abstract»

    • Full text access may be available. Click article title to sign in or learn about subscription options.

      Graphical Models and Variational Methods

      Opper, M. ; Saad, D.
      Advanced Mean Field Methods:Theory and Practice

      Page(s): 161 - 177
      Copyright Year: 2001

      MIT Press eBook Chapters

      This chapter contains sections titled: Introduction, Variational methods for maximum likelihood learning, Variational methods for Bayesian learning, Conjugate-Exponential Models, Belief Networks and Markov Networks, Examples, Sampling from Variational Approximations, Conclusion, Acknowledgments, References View full abstract»

    • Full text access may be available. Click article title to sign in or learn about subscription options.

      Some Examples of Recursive Variational Approximations for Bayesian Inference

      Opper, M. ; Saad, D.
      Advanced Mean Field Methods:Theory and Practice

      Page(s): 179 - 195
      Copyright Year: 2001

      MIT Press eBook Chapters

      This chapter contains sections titled: Approximate Bayesian inference using the variational approach, Some examples, Alternative approximations; recursive methods, Some results and discussion, Acknowledgment, References View full abstract»

    • Full text access may be available. Click article title to sign in or learn about subscription options.

      Tractable Approximate Belief Propagation

      Opper, M. ; Saad, D.
      Advanced Mean Field Methods:Theory and Practice

      Page(s): 197 - 211
      Copyright Year: 2001

      MIT Press eBook Chapters

      This chapter contains sections titled: Graphical Models, Undirected Belief Propagation, Directed Belief Propagation, Tractable Implementations of Directed Belief Propagation, Undirected vs Directed BP: An example application, Extensions, Discussion, Acknowledgments, References View full abstract»

    • Full text access may be available. Click article title to sign in or learn about subscription options.

      The Attenuated Max-Product Algorithm

      Opper, M. ; Saad, D.
      Advanced Mean Field Methods:Theory and Practice

      Page(s): 213 - 227
      Copyright Year: 2001

      MIT Press eBook Chapters

      This chapter contains sections titled: Introduction, Background, New results on computation trees, Max-product inference with Gaussian likelihoods, Skewed computation trees, The attenuated max-product algorithm, Conclusions, Acknowledgments, References View full abstract»

    • Full text access may be available. Click article title to sign in or learn about subscription options.

      Comparing the Mean Field Method and Belief Propagation for Approximate Inference in MRFs

      Opper, M. ; Saad, D.
      Advanced Mean Field Methods:Theory and Practice

      Page(s): 229 - 239
      Copyright Year: 2001

      MIT Press eBook Chapters

      This chapter contains sections titled: The setting: undirected graphs with pairwise potentuals, The algorithms, Simulation Results, Discussion, Acknowlegments, References View full abstract»

    • Full text access may be available. Click article title to sign in or learn about subscription options.

      Information Geometry of α-Projection in Mean Field Approximation

      Opper, M. ; Saad, D.
      Advanced Mean Field Methods:Theory and Practice

      Page(s): 241 - 257
      Copyright Year: 2001

      MIT Press eBook Chapters

      This chapter contains sections titled: Introduction, Geometry of Mean Field Approximation, Concepts from Information Geometry, Parametric model and Fisher information, Geometry of E, The α-projection and mean field approximation, α-trajectory, α-Hessian, Small w approximation of the α-trajectory, Conclusions, References View full abstract»

    • Full text access may be available. Click article title to sign in or learn about subscription options.

      Information Geometry of Mean Field Approximation

      Opper, M. ; Saad, D.
      Advanced Mean Field Methods:Theory and Practice

      Page(s): 259 - 273
      Copyright Year: 2001

      MIT Press eBook Chapters

      This chapter contains sections titled: Introduction, Problem spicifications, Boltzmann machines, Variational-Bayes framework, Mean-field approximation in EM algorithm, Discussion, Acknowledgments, References View full abstract»