Alternating Direction Method of Multipliers for Convex Optimization in Machine Learning - Interpretation and Implementation

The alternating direction method of multipliers (ADMM) is an important method to solve convex optimization problems. Due to the optimization tasks increased with the sort of machine learning applications, ADMM has gained much more attention recently. The principle of ADMM solves problems by breaking them into smaller pieces to specially limit the problem dimension. Each of the pieces are then easier to handle and speed up accordingly the total computational time to reach the optimum. With the speeding-up, it was widely adopted for optimization in a number of areas. In this paper, we start the explanation from the constrained convex optimization, and the relation between primal problem and dual problem. With the preliminary explanation, two optimization algorithms are introduced, including the dual ascent and the dual decomposition approaches. An introduction of augmented Lagrangian, the key to success ADMM, is also followed up ahead for elaboration. Finally, the main topic of ADMM is explained algorithmically based on the fundamentals, and an example code is outlined for implementation.