A Comparative Analysis of Query Generation Methods for IoT Middleware Evaluation

The rapid expansion of Internet of Things (IoT) applications necessitates the demand for efficient IoT middleware platforms, especially Context Management Platforms (CMPs) for accessing real-time contextual information. Current IoT middleware evaluation methods lack accessibility, posing a significant challenge for standardized performance analysis of CMPs. This paper introduces a novel scenario-based context query generation approach for assessing the data retrieval performance of a CMP. The proposed approach enables scene creation using either IoT data or real-world images, employing both data-driven and image-driven scene generation methods. This paper presents the challenges, limitations, and applicability of each approach in scene generation through a qualitative evaluation. The scenes are transformed into scene graphs, enabling inference to identify situations. Both approaches generate queries reflecting dynamic real-world situations, allowing for random query loads to evaluate CMPs fairly. This paper validates the proposed approach using real-world IoT datasets and camera images with a focus on bicycle riding safety use cases. The evaluation provides insights into the contextual completeness of scene graph representations crucial for accurately capturing real-world situations. The proposed approach generates queries of varying complexity, with the only difference being the underlying mechanism of scenario generation, providing a streamlined and user-friendly method for evaluating the data retrieval performance of CMPs.