Empirically Testing Deep and Shallow Ranking Models for Click-Through Rate (CTR) prediction

Recent studies have reported that deep learning models perform excellently for reranking the top recommendation items. However, we found that it is not easy to reproduce some of these results. In particular, we found that recommendations based on a simple neighbor-based model, on average, outperform the results generated by deep learning models based on two datasets from e-commerce websites (one open dataset and one private dataset from our collaborating partner). Moreover, we performed an error analysis to investigate when the deep learning models perform better than simple models and when they do not. Our analysis is especially useful for medium- and small-sized online retailers that may have a smaller training dataset.