Unbiased Benchmarking in Mobile Networks: The Role of Sampling and Stratification

Cellular operators tightly monitor their networks to keep up with the market demand and frequently benchmark their performance against competitors. Typical benchmarking tests compare key performance indicators, quality of service, and quality of experience parameters on the city- and regional levels using user-collected crowdsourced data or drive test measurements. However, time-variant parameters and different user mobility patterns can bias the performance comparison. Designing a measurement sampling strategy that deals with such issues is critical for achieving a valid benchmark. Whether we would like to determine how many tiles of a map have to be measured in drive tests or how many samples we need from crowdsourced data to reach an estimate with the required accuracy, sampling theory can provide us with an answer. Since propagation conditions depend on user mobility and measurement environment, splitting the data set into groups or strata allows us to attain an unbiased estimate with fewer samples, thus allowing for a fair comparison to other mobile network operators with minimum effort measurements. In this work, we characterize the performance of different sampling methods on the simulated data set while investigating specific use cases to reveal scenarios where the stratification method pays off. We further analyze the sampling methods on two real-world crowdsourced data sets from a major Austrian operator. By stratifying the data into meaningful strata, we obtain the required number of areas and measurements in each stratum while remaining under the pre-set estimation error level. To our knowledge, this is the first study on sampling methodologies applied to real-world crowdsourced cellular measurements.