Truncated Lottery Ticket for Deep Pruning

New deep learning models are massively over-parametrized, e.g., GPT-3 and Turing-NLG exceed 100Bs of parameters. Naturally, model reduction techniques such as pruning, quantization, and distillation have been proposed and deployed. In the pruning literature, the Lottery Ticket Hypothesis (LTH) is amongst the most cited. LTH provides a recipe for reducing the free parameters of a deep network by eliminating a large fraction of its edges by 1) remembering and reusing only the edges on the highest-traffic input-output paths and 2) resetting remaining parameters to their initial values after each round of elimination. Here, we extend this idea two steps further. We add a random buildup phase, starting from a small percentage of parameters to reach a predefined accuracy, often needing fewer than 50% of the parameters of a full network. Then we eliminate parameters using an LTH-like procedure combined with a binary search. We demonstrate the efficiency of the proposed technique using three well-known image data sets by sparsifying three standard deep nets. We exceed the elimination power of LTH (over 98% pruned) and effectively match the performance of full networks. We conclude that for both reducing computation and maintaining a sparse network for inference, the proposed scheme is effective and practical.