Finding Answer Passages with Rank Optimizing Decision Trees

The paper discusses the use of decision trees for probability-based ranking. Emphasis is placed on ranking problems in question answering, where the frequency of correct candidates is very low but a single correct answer at one of the top ranks is often sufficient. Since existing tree learners handle this task poorly, decision tree induction is reformulated in such a way that it directly optimizes a given measure of ranking quality (such as mean reciprocal rank or mean average precision). This change also makes it possible to incorporate a priori knowledge about the positive or negative effect of an attribute on ranking quality. Results are further improved by applying a stratified form of bagging. In a passage reranking task using factoid questions from the QA.CLEF evaluations, the new method outperforms existing tree induction techniques by a large margin.