The Random Number Partitioning Problem: Overlap Gap Property and Algorithmic Barriers

We focus on the problem of algorithmically finding a near-optimal solution for the (random) number partitioning problem (NPP), a problem that is of great practical and theoretical significance. The NPP possesses a striking gap between the existential and best algorithmic guarantee: when its input has i.i.d. standard Gaussian entries, the optimal value of NPP is $\Theta \left( {\sqrt n {2^{ - n}}} \right)$ (w.h.p.); whereas the best polynomial-time algorithm achieves an exponentially worse value of only ${2^{ - \Theta \left( {{{\log }^2}n} \right)}}$ (w.h.p.). In this paper, we inquire into the origin of this gap by studying the landscape of the NPP through the lens of statistical physics and establish the presence of the Overlap Gap Property (OGP), a topological barrier for large classes of algorithms. We then leverage the OGP to establish that (a) sufficiently stable algorithms fail to find a near-optimal solution with value below $\left. {{2^{ - \omega (n\log - 1/5}}n} \right)$; and (b) a very natural Monte Carlo Markov Chain dynamics mixes slowly. A technical innovation of our paper is that we consider the overlap structure of m–tuples of near- optimal solutions where m itself grows in n. Our hardness result for stable algorithms is based on a Ramsey-theoretic argument from extremal combinatorics. To the best of our knowledge, this is the first usage of Ramsey Theory to show algorithmic hardness.