The preferred alternative in a tradeoff study often depends on the technique used

When choosing amongst alternative designs, decision-makers would like to know that the choice of the preferred design is based on knowledge and experience, and not on some artifact of the decision process. But often, the apparent best alternative is just that, an artifact of the method that was chosen to do the tradeoff study. Many multicriterion decision-making techniques can be used for tradeoff studies: however, they often produce different recommendations. If the alternatives form a nonconvex set, then many techniques will have difficulty. If you got the "wrong answer", then you probably missed some requirements. Engineers should use formal, mathematical techniques to evaluate alternative system designs. Standards require it. Government organizations require it. Company policies require it. Common sense requires it. But when you do, be careful or mere artifacts will determine your recommendation. In our research we applied seven multicriterion decision-making techniques to some real-world data sets. We found that the preferred alternative depended on the technique that was used.