Recently, trust-based routing has received much attention as an effective way to improve security of wireless ad hoc networks (WANETs). Although various trust metrics have been designed and incorporated into the routing metrics, as far as we know, none of the existing works have used mathematical tools such as routing algebra to analyze the compatibility of trust related routing metrics and routing protocols in WANETs. In this paper, we first identify unique features of trust metrics compared with QoS-based routing metrics. Then, we provide a systematic analysis of the relationship between trust metrics and trust-based routing protocols by identifying the basic algebraic properties that a trust metric must have in order to work correctly and optimally with different generalized distance-vector or link-state routing protocols in WANETs. Moreover, we extend our framework to model the interactions between different trust-based routing protocols. Finally, our results are applied to check the compatibility of the trust metrics proposed in previous literature and the popular routing protocols used in WANETs.