A large-scale, long-term, user-centric evaluation of a commercial voice-over-IP application

To make cheap voice calls, Voice-over-IP (VoIP) services are often used as an alternative to the traditional telephone service providers. Also on the mobile platform, these VoIP services are becoming increasingly popular due to the increased capabilities and connectivity of mobile devices. Understanding the user's usage behavior and quality assessment of the VoIP service is crucial to optimize the Quality of Experience (QoE) and making the service to succeed. Whereas multimedia services are often evaluated in a controlled laboratorium setting, with a selected group of test subjects, and during a short evaluation period, this study analyzes the service usage and quality assessments in a real environment, with more than thousand users, and over a period of 120 days. The influence of various parameters (such as audio codec, handovers, platform, and manufacturer of the device) on the subjective quality of the service is validated by analyzing the quality assessments of users, provided after each voice call. The time of the day showed to have a significant influence on the number of calls, the duration, and the subjective quality assessment. Time-dependent patterns in the users' usage behavior are identified, thereby providing useful information to predict the system load. A regression analysis of the quality assessments over time shows that the perceived quality gradually decreases as users have utilized the service more, and get more familiar with it. In contrast, the mean duration of the calls increases as users get more familiar with it. This research is important in view of associating technical and contextual parameters with the QoE during service usage.