Estimator performance for shallow and deep models

In recommendation system, users post-click conversion post great challenge for training a recommender because selection bias in data. Bias in recommendation system has become a severe issue for ranking performance. Currently, naïve, inverse propensity score and doubly robust estimator have been proposed to address the problem. However, the estimator have not been applied to state of the art deep learning models, which are more realistic in practical settings. In this study, to overcome those limitations, we explored the performance of those estimators on shallow models and deep models, and compared their results on same bunch of models. Extensive experiments shows that doubly robust estimator is more robust than inverse propensity score estimator in deep model.