Communication-Learning Co- Design for Over-the-Air Federated Distillation

The rapid proliferation of artificial intelligence (AI) services gives rise to the development of federated learning (FL), enabling the cooperative learning among wireless devices (WDs) with only local model parameters communicated. Nevertheless, the current emergence of large AI models renders the existing FL approaches inefficient, due to the huge communication overhead. In this paper, we propose a novel over-the-air federated distillation (FD) framework by synergizing the strength of FL and knowledge distillation to avoid the heavy local model transmission. Instead of sharing model parameters, only WDs' model outputs, referred to as knowledge, are shared and aggregated over-the-air by exploiting the superposition property of the multiple-access channel. Accordingly, we study the communication-learning co-design in over-the-air FD, aiming to maximize the learning convergence rate while meeting the power constraints of the transceivers. The main challenge lies in the intractability of the learning performance analysis, as well as the non-convex nature and the optimization spanning the whole FD training period. To tackle this problem, we propose an efficient algorithm to jointly optimize the transmit power of the WDs, estimator for over-the-air aggregation, and receiver beamforming per training round. Numerical results demonstrate that the proposed over-the-air FD achieves significant communication overhead reduction, with only a slight compensation of testing accuracy compared to conventional FL benchmarks.