A Lower Bound for Regret in Logistic Regression

We study logistic regression with binary features in which the number (or degree) of occurring features determines the label probability. This model fits one of social networks, where the number of friends determines the likelihood of outcomes instead of the identity of the friends, or more generally, a graph model, where the degree of a node can determine its structure. It includes the case in which weights can be viewed as i.i.d. (e.g., in Bayesian modeling). For such a model, we introduce the maximal minimax regret that we analyze using a unique combination of analytic combinatorics and information theory. More importantly, the resulting regret is a general lower bound for the pointwise regret of a general logistic regression over all algorithms (learning distributions). We show that the introduced worst case (maximum over feature sequences) maximal minimax regret grows asymptotically as $(d/2)\log(T/d)+(d/2)\log(\pi/2)+O(d/\sqrt{T})$ for dimensionality $d=o(\sqrt{T})$, which is a lower bound for a regret of a general logistic regression. We extend our results to loss functions other than logistic loss and non-binary labels.