Network testing and benchmarking is an important way to improve operator performance and reputation.
Independent mobile network testing was developed during the 1990s to provide objective measurements of network quality. First used by operators in Europe and North America, it is now well established in emerging markets. Independent network testing has become an essential part of strategic reporting in addition to being a key element of customer experience and customer retention programs and marketing to attract new customers.
MNOs must ensure that they can deliver the levels service performance required by their licenses, regulators, shareholders and supervisory boards, in addition to being able to substantiate marketing claims made to existing and new customers.
There are a number of independent testing and benchmarking providers, including Ookla, Opensignal and Umlaut (formerly known as P3). Different methodologies are used by these providers; Ookla and Opensignal both exclusively use crowd sourcing, while Umlaut (P3) uses a mixture of drive tests, walk tests and crowd sourcing. These providers all offer high quality and reliable testing and benchmarking solutions for a range of connectivity and performance optimisation challenges. This article looks in more detail at the latest Umlaut score, released in November 2023.
The Umlaut score, which claims to be the de-facto global industry standard for MNOs, is a measure for management to compare their networks with others around the globe to enhance strategic decision making. Although the examples below relate to Western Europe, the framework used by Umlaut is based on a rule set applied globally, to provide a fair, transparent and independent evaluation of all networks. Umlaut conduct benchmarks in 70 countries with more than 180 mobile networks being evaluated with the Umlaut score.
The independent framework is the basis for measuring the actual customer experience as well as mobile network performance and capability, allowing technical analysis with a high level of detail. The Umlaut testing methodology and scoring models are regularly updated in response to technical developments.
The Umlaut score also takes into account growing customer expectations, expanding data volumes and rising data speeds by raising the requirements and thresholds of the tests each year. Despite this, most operators are able to improve their performance, demonstrating that Umlaut ’s challenging network tests contribute to the improvement of the networks.
The methodology of testing has evolved as networks themselves have changed, from 2G through to 5G. Testing has also responded to changing patterns of network usage, for example by analysing and scoring networks on their ability to handle data downloading via HTTPS, FTP and other protocols. Both data upload and download speeds are tested. Recently the Umlaut test has also included specific tests related to OTT traffic generated by, for example, YouTube and e-Gaming.
DACH 2023 Test Result Overview
The DACH (Germany /Austria /Switzerland) markets of Western Europe are recognised as pioneers in mobile network testing and tests have taken place in these countries on an annual basis for many years. The latest tests, performed in October 2023, cover a range of settings including drive and walk tests in all major cities, a large number of medium and smaller population centres, and in rural areas. Carrying out tests simultaneously in three separate countries means comparisons and benchmarks can be made cross border as well as within each country.
The tests look at both data and voice network performance in the different locations described, and also in specific settings including highway driving and rail travel. From this it is possible to identify the operators which have the best overall network performance in each country as well as in the individual settings. The ranking is a good advertisement for operators and provides regional consumers with useful advice while choosing a service provider.
In Germany the 2023 tests show that Deutsche Telecom won its 13th Championship in a row. Both Deutsche Telecom and Vodafone increased their overall scores significantly compared to the previous year, with the data category accounting for the largest gains. Third place operator Telefonica/O2 saw a more modest increase overall, but was able to improve its score significantly in Munich to become the leading operator in the city, overtaking country leader Deutsche Telecom and well ahead of Vodafone.
In Austria, Magenta extended its winning streak to six years in pole position. This is a remarkable achievement given that A1 and Hutchinson 3, Austria’s other two operators, have not been resting on their laurels; both have also seen improvements in their scores year-on-year in the 2023 tests.
In Switzerland all three operators Swisscom, Sunrise and Salt rank as outstanding networks. Swisscom and Sunrise are ranked in the top 5 operators worldwide, while Salt got its first ‘Outstanding’ result in 2023.
Apart from the reputational improvement to customers, these positive results show that operator efforts are paying off. In Deutsche Telecom’s annual report, the consecutive number 1 ranking of Deutsche Telecom (in Germany) and Magenta Telekom (in Austria) are referred on multiple occasions as one of the great achievements for the group, giving additional confidence to stakeholders.
Crowd Test
The separate crowd test crowdsources user experience data from large numbers of customers, meaning every user may be a part of the test, and delivering network performance feedback from many devices. This test is intended to supplement the measured values of the drive and walk tests, which are aimed at assessing peak performance.
Crowdsourced data is analysed to assess the quality of the everyday mobile phone performance delivered to large numbers of customers. In addition to voice, assessed categories include broadband coverage, active download and upload speeds, latency and stability. The results of crowdsourcing account for 25 per cent of the overall rating. Crowd tests show the network performance received by end users although the devices and tariffs used also have an impact on these results.
Network Reliability
The network reliability rating is not a separate category, but an additional view of the results of the other categories. For this purpose all measured values are divided into basic or everyday requirements, known as Qualifier KPIs, and values related to maximum performance, Differentiator KPIs.
The network reliability measurement only takes into account the Qualifier KPIs from the voice and data categories of the main tests as well as the basic results from the crowdsourcing. This makes it possible to investigate how well the networks fulfil the everyday requirements of end users.
Test Methodology
The DACH market testing program uses four test vehicles in each country, each equipped with nine smartphones. One Samsung Galaxy S23 per network operator is used to carry out the voice measurements, another S23 is used for the data tests and a third to establish the connections for the “Conversational app“ test case. For all measurements, the smartphones are set to “5G preferred” mode wherever this is supported by the network.
Network operators are given access to the testing framework conditions at an early stage, including the smartphones used, parameters taken into account and the testing schedule. Regular dialogue is intended to ensure firmware versions used on test smartphones optimally support technologies such as carrier aggregation and 5G variants such as DSS, NSA or SA.
Network Testing in Emerging Markets
More than 200 mobile networks in over 120 countries have now participated in mobile network testing programs similar to the DACH region one, including over 100 in emerging markets. Recently tested or benchmarked operators include Airtel, Jio, Globe, Movitel, MTN, Ooredoo, STC, Telkomsel, Tigo, TMCel, Vivo, Vodacom, Vodafone, Yettel and Zain in countries including Brazil, India, Indonesia, Kuwait, Mozambique, Nigeria, Qatar, Turkey, Saudi Arabia, Serbia, South Africa and Zambia.
Mobile network testing and benchmarking aims to fairly and transparently assess the global development of networks, to improve network quality and performance, and ultimately to deliver better service for customers, while improving operators’ reputations and increasing their business.
