Improving reliability of user preferences: Comparing instead of rating

More and more personalization systems are emerging to reduce the information overload of the Web. As a result, it has become vital to model users' preferences accurately. Today, users are often brought to express their preferences through ratings on a multi-point scale. However, several studies have highlighted problems with ratings, such as inconsistency through time. The preferences collected are thus not totally reliable and the quality of the models may be limited. To cope with this limitation, we propose a new preference modality whereby users compare items two-by-two (“I prefer x to y”). We conduct two user-studies with 100 users on the domain of movies. This preliminary work on comparisons shows promising results. First, we reveal that users are in favor of this new preference mechanism. Second, we demonstrate that preferences expressed through comparisons are almost 20% more stable over time than those conveyed through ratings, thus more reliable.