Performance evaluation methods in ad hoc and wireless sensor networks: a literature study

Verification of theoretical analysis is a vital step in the development of an application or a protocol for wireless networks. Most proposals are evaluated through mathematical analysis followed by either simulation or experimental validation campaigns. In this article, we analyze a large set of statistics in articles published (674 papers in total) in the top representative ad hoc and wireless sensor networks related conferences (i.e., ACM/ IEEE IPSN, ACM MobiCom, ACM MobiHoc, and ACM SenSys) during the period 2008-2013. We focus on the evaluation methodologies provided by researchers. More specifically, our goal is to explore the role of simulators and testbeds in the theoretical analysis of a model throughout the protocol development procedure. We show that there is a tendency for more and more researchers to rely on custom or open testbeds in order to evaluate the performance of their proposals. Simulators indeed fail to reproduce actual environmental conditions of deployed systems. Experimentation with real hardware allows our research community to mind the gaps between simulation and real deployment. Still, as the experimental apFproach through custom testbeds results in a low reproducibility level (i.e., 16.5 percent), we investigate to what extent such performance evaluation methods will be able to bridge those gaps. We finally discuss experimental testbeds and their potential to replace simulators as the cornerstone of performance evaluation procedures.