Continual Learning with Distributed Optimization: Does CoCoA Forget Under Task Repetition?

We focus on the continual learning problem where the tasks arrive sequentially and the aim is to perform well on the newly arrived task without performance degradation on the previously seen tasks. In contrast to the continual learning literature focusing on the centralized setting, we investigate the distributed estimation framework. We consider the well-established distributed learning algorithm CoCoA. Our results illustrate the importance of task repetition for continual learning with CoCoA. In particular, even when there is a shared solution for all tasks, task repetition may be necessary for satisfactory performance. We show how the dimensions of the offline and centralized problem affects the learning performance substantially; even though CoCoA processes only a subset of features and data samples at each iteration.