Quality of experience: the route to scalable performance evaluation of networked systems?

As computer simulation technologies enable larger, and more complex, networked systems to be modelled on high-end computing (HEC) resources within reasonable timeframes, the methodological challenges (e.g. of specifying suitable scenarios, efficiently exploring large-scale states spaces, and effectively summarizing performance) increase. In the context of communication networks, quality of experience (QoE) is a measure of performance that summarizes the end-user’s perspective of the service. Perhaps the most well known QoE measure is the Mean Opinion Score (MOS) for conversational quality in telephony service, which has been used extensively in the design and deployment of circuit-switched voice networks. In recent years, an analytical method (called the E-model) has been standardized by the ITU to enable voice quality to be predicted from various parameters describing the voice path across a network. This method generates an R-factor that has a one-to-one mapping to MOS and rates the quality of the voice connection on a scale of 0 to 100. The E-model is being used extensively in VoIP monitoring systems and tools, and there are similar efforts being made to develop equivalent models for combined audio-visual quality metrics. Thus, when studying the behaviour of converged IP networks, it is possible to report the performance of VoIP service as a single QoE value (the R factor), rather than using multiple measures of the underlying network impairments (i.e. packet loss, latency, jitter). This enables new ways of reporting, and visualizing, whole-of-network performance, and can change the focus of performance evaluation studies. Instead of a classic scenario comprising a tandem network with background cross traffic and a relatively small number of foreground streams for which multiple performance measures are gathered, it is possible to study, for example, an irregularly connected mesh network and measure the QoE across all source-destination paths. This talk describes how whole-of-network QoE measures have been developed that can offer new engineering insight in the study of wired and wireless network scenarios, with illustrations from simulation studies performed on national UK HEC resources.