SPQER: Speech Quality Evaluation Using Word Recognition for VoIP Communication in Lossy and Mobile Networks

In this paper, we introduce SPQER (pronounced speaker), a novel approach to evaluate the quality of experience for real-time Voice over IP (VoIP) communication in mobile and lossy networks. Traditional speech quality metrics, e.g., Perceptual Evaluation of Speech Quality (PESQ) or the Hearing-Aid Speech Quality Index (HASQI), directly compare frequencies and amplitudes to calculate the received signal distortions. SPQER instead uses machine learning classification to evaluate the percentage of recognizable words in conjunction with a time-based decay function to penalize delay and cross-talking. So instead of evaluating noise, SPQER directly answers the question: What percentage of words is the recipient able to understand? We presented a sensitivity analysis, which is based on testbed experiments for different packet loss rates and simulated delays, to asses the impact of challenging link conditions. A final correlation analysis to a short user study shows that SPQER can better evaluate the amount of understandable words than PESQ and HASQI, while still giving a more precise indication about the voice quality than the Word Error Rate (WER) metric.