In December 2020, Rohde & Schwarz did a benchmarking assessment of the cellular coverage in the Dallas/Fort Worth metro area with Signal Research Group. A year later, the tests were repeated to see if the wireless environment had changed in the interlude. Rohde & Schwarz held a webinar, called “Apples to Apples: 5G Network Benchmarking Part 2” to discuss the results of the follow-up measurements and analysis.
The 2022 setup was comparable to the previous assessment done in December 2020, covering nearly 2,500 miles of drive testing on the same area and routes in the Dallas/Fort Worth metro area. As a measurement device, technicians used the Samsung Galaxy S21 Plus smartphone and a network scanner to scan all available 5G and 4G frequencies. The operators compared in the test were AT&T, T-Mobile and Verizon. More than 1,000 voice calls and more than 14,000 data tests per network were conducted.
T-Mobile went from third place to first place with an overall score of 878.2, while AT&T dropped to second with 795.9. Verizon brought up the rear with 791.6. All operators lost points in voice, but AT&T maintained a slight lead.
“The 2022 benchmark assessment was marked by quite a bit of movement among the positionings of the operators,” said Nils Posegga, Solutions Manager, Rohde & Schwarz. “T-Mobile clearly is now number one due to massive improvements in data, including: downlink, uplink, and throughput, mainly. But also in the latencies, there is a lot of improvement.”
Verizon has seen improvement in data speeds, but its scores slightly decreased in the customer experience use cases, including browsing success rate and duration. It also lost ground in voice scores.
“One of the strengths of the applied methodology is the full comparability of the results,” Posegga said. “In this case, we are directly comparing the measurement results of two measurement rounds with almost one year apart.”
ETSI TR 103 559 Allows Robust Scoring of Network Performance
In its benchmark assessment, Rohde & Schwarz used the network performance score methodology according to ETSI Technical Report (TR) 103 559, which describes best practices for robust network quality of service benchmark testing and scoring for speech and multimedia transmission quality. It comprises a well-defined methodology, KPIs and guidance for scoring end-to-end networks. The weights and thresholds for the KPIs are defined and accepted in the market.
“The method we applied is fully transparent, and it’s free and publicly available,” Posegga said. “Rohde & Schwarz supported the TR 103 559 because we stand for open, transparent and harmonized methodologies that lead to comparable results. And the Rohde Schwarz implementation of the methodology is called the network performance score.”
The method used by Rohde & Schwarz delivers a true picture of customer perceived performance with more details than a simple scoring model based merely on maximum data rate. Key Performance Indicators (KPIs) are divided into service classes. For voice, the service class is telephony, and, for data, the service classes are divided into data transfer, video streaming, social media and web browsing. All service classes are weighted per population categories, typically highway and city. And then finally on the highest level, the different population categories are weighted into the overall network performance score.
“The aggregation will work all the way up from the technical details to a very easy, understandable single value that defines the quality of the radio network, but we don’t lose any details,” Posegga said. “In the other direction, we can drill down to the very detail and understand what is impacting the performance and what needs to be done to improve the network performance.” To cope with the higher 5G data speeds, Rohde & Schwarz has proposed the next generation of scoring methodology to allow for further differentiation.
An Applied Methodology for Testing Interactivity in 5G Networks
Even though mobile operators today are still putting the focus purely on downlink speed into activity, Rohde & Schwarz uses an applied methodology for testing interactivity in 5G networks. To get that measurement, UDP stream of packets is sent from a device to an active server instance that reflects the packets back to the client.
“The interactivity score combines latency, delay variation and packet loss into one single score,” Posegga said. “The interactivity score is specific per use case class with specific QoS requirements.”
The webinar drilled down into other network performance parameters, including call setup time, paging time, downlink performance, technology share and achieved throughputs, bandwidth share and achieved data throughputs, interactivity score results, and latency.
In summary, Posegga emphasized that the Network Performance Score following ETSI TR 103 559 provides a stable methodology for comparison over time. Interactivity tests are also a valuable addition to Network Performance Score and help the industry prepare for the real time and interactive demands in 5G networks.
“Rohde & Schwarz products and managed services can help you to identify and understand changes in network performance,” Posegga said. “Network scanners are always recommended as a valuable addition to every benchmarking activity. An apples-to-apples comparison is possible with the Network Performance Score and managed services from Rohde & Schwarz.”
To see the Rohde & Schwarz webinar, click here.
By J. Sharpe Smith Inside Towers Technology Editor
Reader Interactions