Likelihood-Free Hypothesis Testing

Consider the problem of binary hypothesis testing. Given
Z
coming from either P
^⊗m
or Q
^⊗m
, to decide between the two with small probability of error it is sufficient, and in many cases necessary, to have
m
≍ 1/ε
²
, where ε measures the separation between P and Q in total variation (TV). Achieving this, however, requires complete knowledge of the distributions and can be done, for example, using the Neyman-Pearson test. In this paper we consider a variation of the problem which we call likelihood-free hypothesis testing, where access to P and Q is given through
n
i.i.d. observations from each. In the case when P and Q are assumed to belong to a non-parametric family, we demonstrate the existence of a fundamental trade-off between
n
and
m
given by
nm
≍
n
²
_GoF
(ε), where
n
_GoF
(ε) is the minimax sample complexity of testing between the hypotheses
H
₀
: P = Q vs
H
₁
: TV(P, Q) ≥ ε. We show this for three families of distributions, in addition to the family of all discrete distributions for which we obtain a more complicated trade-off exhibiting an additional phase-transition. Our results demonstrate the possibility of testing without fully estimating P and Q, provided
m
≫ 1/ε
²
.