Model Sharing Mechanisms For Distributed Learning

In the domain of distributed learning, collaborative models are trained across diverse agents with varying data distributions or preferences. To minimize mean square error (MSE) in model training, an individual agent faces a bias-variance trade-off: joining a larger federation reduces model variance but increases model bias. When the system invites agents with both low and high data quality, this trade-off may lead to only low-quality data agents joining, while high-quality data agents reject the invitation. We propose two federation mechanisms to engage both groups and reduce the total MSE of the population. One mechanism charges low data quality agents and rewards high data quality agents. The other is non-monetary and optimally adjusts aggregation weights to protect high data quality agents. We also extend our mechanism to scenarios with incomplete information, where data quality is unknown. Our new Separate or Penalize (SOP) mechanism threatens to terminate the federation, successfully ensuring truthful reporting of data quality. Experiments demonstrate significant system improvements compared to non-intervention approaches.