Empirical analysis of collaborative filtering-based recommenders in temporally evolving systems

Recommender systems benefit people's daily lives at every moment. While considerable attentions have been drawn by performance in one-step recommendation and static user-item network, recommenders' performance on temporally evolving networks remains unclear. To address this issue, this paper firstly adopts a bipartite network to describe the online commercial system. We then propose a network evolution method to simulate the mutual feedback between recommender system and its users' decisions in the evolving network with time. To investigate the long-term performance of three state-of-the-art CF-based recommenders, i.e., the user-based collaborative filtering (UCF), item-based collaborative filtering (ICF) and latent factor-based model (LFM), this online network is evolving with time driven by each tested recommender. Besides using root mean squared error (RMSE) to evaluate prediction accuracy of recommender, we also calculate the intra-similarity and popularity to study the performance of recommendation, as well as Gini coefficient to evaluate the health of online network. Experiments on two real datasets, we find that during the temporal evolving process LFM's accuracy loss is less than that of UCF and ICF, besides LFM enjoys a high accuracy in one-step recommendation. Moreover, although LFM proves to be highly accurate and stable during the temporal evolving network, ICF shows a better performance than LFM in terms of recommendation diversity, and it simultaneously benefits the health of online system. Hence, these results provide insights for the design of a next generation of recommender systems, which would tradeoffs between short- and long-term performances.