Coordinate Descent Algorithm for Nonlinear Matrix Decomposition with the ReLU Function

Nonlinear Matrix Decompositions (NMD) solve the following problem: Given a matrix X, find low-rank factors W and $H$ such that $X\approx f(WH)$, where $f$ is an element-wise nonlinear function. In this paper, we focus on the case when $f$ is the rectified linear unit (ReLU) activation, that is, when $f(\cdot)=\max(0,\ \cdot)$, which is referred to as ReLU-NMD. All state-of-the-art algorithms for ReLU-NMD have been designed to solve a reformulation of ReLU-Nmd. It turns out that this reformulation leads to a non-equivalent problem, and hence to suboptimal solutions. In this paper, we propose a coordinate-descent (CD) algorithm designed to solve ReLU-NMD directly. This allows us to compute more accurate solutions, with smaller error. This is illustrated on synthetic and real-world datasets.