A Time-Space Lower Bound for a Large Class of Learning Problems

We prove a general memory-samples lower bound that applies for a large class of learning problems and shows that for every problem in that class, any learning algorithm requires either a memory of quadratic size or an exponential number of samples. Our result is stated in terms of the norm of the matrix that corresponds to the learning problem. Let X, A be two finite sets. A matrix M : A × X → {-1, 1} corresponds to the following learning problem: An unknown element x ∈ X was chosen uniformly at random. A learner tries to learn x from a stream of samples, (a
₁
, b
₁
), (a
₂
, b
₂
) ..., where for every i, a
_i
∈ A is chosen uniformly at random and b
_i
= M(ai, x). Let σ
_max
be the largest singular value of M and note that always σ
_max
≤ |A|
^1/2
· |X|
^1/2
. We show that if σ
_max
≤ |A|
^1/2
· |X|
^1/2-ε
, then any learning algorithm for the corresponding learning problem requires either a memory of size at least Ω ((εn)
²
) or at least 2
^Ω(εn)
samples, where n = log
₂
|X|. As a special case, this gives a new proof for the memory-samples lower bound for parity learning [14].