Voice Quality Testing for Wireless, VoIP and Analog Networks

Operations performed by POLQA

Temporal alignment

The basic concepts of the temporal alignment are:

• To split the signals into equidistant pairs of frames and to calculate a delay for each frame pair.
• Whenever possible, the matching counterparts of the degraded signal sections are searched for in the reference signal and not vice versa.
• Stepwise refinement of the delay per frame to avoid long search ranges (long search ranges require high computing power and are critical in combination with time scaled input signals).

Sample rate estimation

The sample rate ratio detection is required to compensate for perceptually irrelevant differences in the play-out speed of both, the reference and the degraded signal.

The detection of this effect as implemented in the ITU-T P.863 algorithm is based on the delay per frame vector and the detected active sections of the speech signals, as determined by the temporal alignment.

Resample

If the detected sample rate ratio is larger than 0.01, the signal with the highest sample rate will be down sampled and the entire processing starts from the beginning. This happens at most one time to avoid excessive looping in case of signals for which the sample rate ratio cannot be determined in a reliable manner.

Even if the sample rate determination cannot be made with perfect accuracy, e.g., in case of signals with additional variable delay, the detected sample rate ratio is still accurate enough to bring the signals back to the safe operating range of the temporal alignment.

Level alignment

The ITU-T P.863 algorithm is designed to take into account the impact of the play back level for the perceived quality prediction in super wideband mode; the playback level is calculated relative to a nominal level of –26 dBov, which represents 73 dB(A) SPL in dichotic presentation.

In narrowband operational mode the ITU-T P.863 algorithm is designed to determine the listening speech quality at a constant listening level of 79 dB(A) SPL.

Frequency response and time alignment

The ITU-T P.863 algorithm can operate in two modes, narrowband mode, and super wideband mode. In the narrowband mode, both the reference and degraded signals are pre-filtered with an IRS receive filter representing a listening situation in which subjects judge the quality of the speech signals over an IRS receive handset in monotic mode or over an IRS receive headset in monotic mode. In the super wideband mode, both the reference and degraded signals are not filtered, representing a listening situation in which subjects judge the quality of the speech signals over a diffuse field equalized headset in dichotic mode.

Results Provided by POLQA

Perceptual Results

MOS-LQO

The most eminent result of POLQA is the MOS-LQO. It directly expresses the voice quality on the MOS scale. It is important to understand and consider the two different operational modes supported by the ITU-T P.863 algorithm:

• super wideband mode for listening over super wideband headphones;
• narrow-band mode for listening over loosely coupled IRS type handsets.

In the super wideband mode the impact of play back level is modeled and the default calibration factor (C) of 2.8 has to be used in combination with the standard –26 dBov scaling for play back levels of 73 dB(A) SPL (dichotic). Play back levels down to 53 dB(A) SPL and up to 78 dB(A) SPL may be used and MOS-LQO scores should be reported in the format MOS-LQOsw (dB level). In narrowband mode only the play back level of 79dB(A) SPL (monotic) is supported. Narrowband mode MOS scores are referred to as MOS-LQO.

The maximum ITU-T P.863 MOS-LQO score is 4.5 in narrow-band while in super wideband mode this point lies at 4.75. Under some circumstances, when the reference signal contains noise or when the voice timbre is distorted, a transparent chain will not provide the maximum MOS score of 4.5 in narrowband mode or 4.75 in super wideband mode.

Below table compares PESQ and POLQA MOS scores:

G.107 R-Factor / Ie Value

POLQA also provides a mapping of the MOS-LQO score to the scale used by G.107 (e-model). The resulting parameter is equivalent to an Ie – Value. Many people also refer to it as an R-factor. The scale ranges from 0 (bad) up to 100 (best). All values below 60 indicate unacceptable quality.

Non-Perceptual Results

Attenuation

Especially all analog equipment modifies the level of the speech signal. A high attenuation generally leads to a worse perception of voice. In contrast to PESQ, POLQA does weight this as degradation of the signal. Knowing the value of the attenuation is also important for optimizing the overall system design. Attention should be paid to signals which show either a negative attenuation, or attenuations larger than approximately 10dB. In the first case, the signal was amplified instead of attenuated. This may eventually lead to level clipping during the transmission. In the second case, the quantization noise may become an important source of degradation, if low level analog signals are converted to the digital domain and are subsequently amplified in the digital domain. Depending on the test setup, both cases may be ok and intended, but this has to be decided on a case by case basis.

In order to calculate the attenuation, POLQA computes P.56 like active speech levels of the reference as well as the degraded signal in dB. The level of the degraded signal minus the level of the reference signal is then used as the attenuation.

Level and Background Noise Measurements

In transmission systems it is frequently important to know the exact levels of the signals. Especially for VoIP systems and voice activity detection (VAD) it becomes also important to know the signal level during the silent intervals as well as during active speech. It is important, that the received background noise does not exceed a certain limit. Levels can be measured in dB if you want to relate the level directly to a sound pressure or electrical level, or as loudness levels.

Signal to Noise Ratio (SNR)

POLQA calculates the SNR for the reference and the degraded signal independently. The noise as well as the signal level is calculated by the VAD which POLQA uses for the temporal alignment.

Active Speech Ratio (ASR)

ASR is calculated by POLQA based on the information calculated by the Voice Activity Detection (VAD) which is part of the temporal alignment. The ASR defines the ratio between active speech and the overall signal length.

POLQA support within Voice Quality Testing (VQT) and VQuad™

GL's enhanced VQuad™ and stand-alone Voice Quality Testing (VQT) software, both support the next-generation voice quality testing standard for fixed, mobile and IP-based networks using PESQ (ITU-T P.862), POLQA ver 2.4 and optional upgrade ver3 (ITU-T P.863). As part of the POLQA analysis, results include POLQA MOS, E-Model, Signal Level, Noise Level, and Jitter. Automated voice quality testing using POLQA can be performed through scripting in VQuad™.

The VQT software can work either independently, in conjunction with VQuad™ - Dual UTA HD, or in conjunction with VQuad™ Probe HD. Using GL's VQuad™ application along with the Dual UTA HD, or all-in-one VQuad™ Probe HD, voice, video, and data tests are supported for practically any end-point (customer experience) interfaces. The VQuad™ Probe HD (combines with VQuad™-Dual UTA HD - PC into one single box) provides multiple interfaces for connecting to any communication device on a network using an efficient, portable, single box solution.  The VQuad™ Probe HD can work with variety of networks, including Wireless (Bluetooth®, Wi-fi, 3G, 4G, LTE, PTT Mobile Radio), VoIP, Analog, and TDM, practically allowing any end-point interfaces to be tested for voice, data, and video quality. Various associated applications (Voice Quality, Video Quality, Data tests, Echo and Delay tests, Fax tests, VBA) works with VQuad™ Probe HD to provide "end-to-end assessment" with additional test and measuring capabilities. All the applications work in conjunction with the VQuad™ for automatically and remotely analysing the captured data and sending the test results to the central database. You can query results or control the tests remotely using web-based WebViewer™.

Third party Goldwave software (SA048) is required for playback, viewing, and analysis of audio from within VQT software.

The GL VQT software automatically analyzes the degraded voice files while sending the results to a central database. The results associated with the POLQA analysis include POLQA MOS, E-Model, Signal Level, Noise Level, and Jitter.

As an additional feature, POLQA is added directly to VQuad™ software with support for automated testing within the VQuad™ script. In this scenario, the degraded voice files remain at the VQuad™ node for analysis and display within the VQuad™ software.

For more details, please visit VQuad webpage.

VQT POLQA Measurements and Analysis

VQuad™ with POLQA Events Log

Automated VQT with POLQA through Scripts

Operations Performed by PESQ

The processing carried out by the PESQ algorithm includes the following stages.

Level Alignment

In order to compare the signals, the reference speech signal and the degraded signal should be at the same, constant power level. This is necessary because the reference signal does not have to be at a defined level and because the gain of the system under test is unknown before testing. PESQ assumes that the subjective listening level is a constant 79dB SPL at the ear reference point [ITU-T P.830, section 8.1.2]. A gain is applied to both the reference and degraded signals to bring them to this level.

Input Filtering

Analog connections often introduce some degree of filtering. For example, PESQ models the receive path of the telephone handset using an input filter. This takes account of the effect of the electrical and acoustic components of the handset. The filter used is similar to the standard "modified IRS receive characteristic" [ITU-T P.830]. It is generally accepted that this has less effect on quality than coding distortions do. PESQ compensates for any filtering that has taken place in the network.

Time Alignment

The system under test may include a delay, which may be variable. In order to compare the reference and degraded signals, they need to be lined up with each other. PESQ applies voice activity detection to the signals to identify those parts of the signal that are speech, ignoring noise. The PESQ time offset measurements do not take account of the delay of the test equipment generating or recording the signal. This means that a time offset reported by PESQ on a file collected will be dependent upon the way in which the test process is executed.

• First, PESQ aligns the overall speech signals (utterances). An utterance is a continuous speech burst identified by the voice activity detector that does not contain pauses longer than a pre-determined threshold (200ms). This process detects delay over major sections of the degraded signal compared to the reference signal.
• Second, PESQ aligns overlapping sections of the speech (frames). This process detects delay that is variable over the length of an utterance, as this can be significant in packet-based networks.
• The third stage does not occur immediately after the second stage, but is performed after the auditory transform has been calculated. The third stage realigns "bad intervals" (sections of the speech with very large disturbance), and improves the model's accuracy with a small number of files where delay changes are not correctly identified by the initial time alignment process.

Auditory Transform

In order to compare the reference and degraded signals, taking account of how a listener would have heard them, each is passed through an auditory transform that mimics certain key properties of human hearing. This gives a representation in time and frequency of the perceived loudness of the signal, known as the sensation surface.

Equalization

Part of the auditory transformation equalizes certain processes that have little subjective effect. First, the transfer function of the system is estimated, and is used to equalize the reference to the degraded in the auditory transform domain. This takes account of filtering in analogue components of the network such as telephone handsets. Second, the frame-by-frame amplitude gain of the system is estimated and used to equalize the auditory transform of degraded file to the reference. In both cases the equalization is partial - large amounts of filtering or gain variation are not cancelled, and therefore result in errors being measured.

Disturbance Processing

The difference between the sensation surfaces for the reference and degraded files is known as the error surface; this shows any audible differences introduced by the system under test. The error surface is analyzed by a process that takes account of the effect that small distortions in a signal are inaudible in the presence of loud signals (masking).

From the positive and negative errors, two disturbance parameters are calculated. They are calculated as non-linear averages over specific areas of the error surface. These disturbance parameters are:

• The absolute (symmetric) disturbance - a measure of absolute audible error
• The additive (asymmetric) disturbance - a measure of audible errors that are significantly louder than the reference

This analysis gives two error parameters that summarize the amount of each type of audible error. Finally, the error parameters are converted to a quality score, which is a linear combination of the average symmetric disturbance value and the average asymmetric disturbance value.

Results Provided by PESQ

PESQ (P.862)

PESQ returns a quality score, known as PESQ score, which conforms to ITU-T P.862. PESQ score lies on a scale from -0.5 to 4.5, though in most cases it is between 1 and 4.5. PESQ score correlates with subjective quality scores. However the PESQ score tends to be optimistic for poor quality speech and pessimistic for good quality speech. Alternative mappings for PESQ score have been developed which do exhibit a better correlation to subjective test scores. These are referred to as the PESQ-LQ and PESQ-LQO scores.

PESQ-LQ

PESQ-LQ scores are closer to the listening quality subjective opinion scale, which is standard in the industry and is defined in ITU-T P.861. Listening quality scores lie between 1 and 5. PESQ-LQ score lie between 1.0 and 4.5. This is because 4.5 is usually the maximum obtained in a subjective test.

Listening Quality Scale:

The score gives a measure of customers' perception of quality. The highest score, 4.5, means that no distortion is measured. As the amount of distortion increases the quality falls.

PESQ-LQO (P.862.1)

The aim of a separate recommendation ITU-T P.862.1 is to provide a single mapping from raw P.862 score to the Listening Quality Objective Mean Opinion Score (LQO-MOS). This latest ITU standard improves on the original PESQ (P.862) by correlating better to subjective test results.

Typical PESQ Score Comparisons

Based on simulations and real measurements, the table below represents the results of a number of typical networks and codecs with no errors or packet loss. In addition, it gives the scores that can be expected in some mobile network conditions where errors are significant.

For more details, please visit PESQ Measurement webpage

Command Line Interface (CLI) for remote access

VQuad™ provides full support for CLI on both Windows® and Linux environment. VQuad™ CLI also supports both IPv6 and IPv4.

The VQuad™ Command Line Interface (CLI) is designed to remotely access various application functionalities and thus controlling VQuad™ nodes located at various destinations.

The VQuad™ Remote Access (Client) allows VQuad™ operations to be remotely controlled by one (or several) VQuad™ clients over a LAN, WAN, or Internet.

All Call Control functions, and traffic simulation functions are supported. These includes - Load Call Control Script Configuration, Load VQuad™ Master Configuration, Start Stop Call Control Script, Place Call, Answer Call, Disconnect Call, Set and Get Log Files, and many other operations. It also supports traffic functions such as Send and Record Voice Files, Transmit and Detect Digits, Send and Detect Tones, Send and Receive Fax, OWD/RTD Correlation, and Stop Traffic.

VQuad™ CLI (Windows)

VQuad™ CLI (LINUX)

Windows® and Linux APIs for VQuad™ and VQT

GL has developed Windows® and Linux APIs for VQuad™ and VQT. Customers can now build custom, faster and cheaper applications using the features already available with GL applications. All important functionalities required to control VQuad™ and VQT remotely are available through APIs to ease the most complex application development activities.

The VQT API environment provides two-way traffic to the VQT server, ability to control the VQT server remotely and provides for instant feedback of VQT client actions.

The VQT functions that are generally available through APIs are listed below –

• Connect/disconnect the client applications with a VQT Server
• Automatically perform VQT with PESQ/POLQA algorithms
• Start/Stop VQT Auto-mode operations
• Capture individual VQT events to file in real-time
• Retrieve the VQT Statistics
• Load VQT auto measurement configuration files

The VQuad™ functions that are generally available through APIs are listed below –

• Connect/disconnect the client applications with a VQuad™ Server and GL Listener.
• Perform all general device configurations
• Send and receive files on the configured devices
• Load auto traffic configuration files and scripts
• Perform call control functions (place call, receive call, disconnect)
• Perform all traffic actions (send/detect tones/digits/files, send/record voice)
• Perform voice quality, delay, noise, and echo measurements
• Retrieve the all results and statistics
• Monitor status of all remote VQuad™ nodes