manual - streaming service test

WHEN QUALITY MATTERS

Diversity Streaming Service Test Manual

January 2011

SwissQual® License AG Allmendweg 8 CH-4528 Zuchwil Switzerland

t +41 32 686 65 65 f +41 32 686 65 66 e [email protected] www.swissqual.com

Part Number: 16-100-200635-2 Rev 1.2

SwissQual has made every effort to ensure that eventual instructions contained in the document are adequate and free of errors and omissions. SwissQual will, if necessary, explain issues which may not be covered by the documents. SwissQual’s liability for any errors in the documents is limited to the correction of errors and the aforementioned advisory services.

Copyright 2000 - 2011 SwissQual AG. All rights reserved.

No part of this publication may be copied, distributed, transmitted, transcribed, stored in a retrieval system, or translated into any human or computer language without the prior written permission of SwissQual AG.

Confidential materials.

All information in this document is regarded as commercial valuable, protected and privileged intellectual property, and is provided under the terms of existing Non-Disclosure Agreements or as commercial-in-confidence material.

When you refer to a SwissQual technology or product, you must acknowledge the respective text or logo trademark somewhere in your text.

SwissQual®, Seven.Five®, SQuad®, QualiPoc®, NetQual®, VQuad®, Diversity® as well as the following logos are registered trademarks of SwissQual AG.

Diversity Explorer™, Diversity Ranger™, Diversity Unattended™, NiNA+™, NiNA™, NQAgent™, NQComm™, NQDI™, NQTM™, NQView™, NQWeb™, QPControl™, QPView™, QualiPoc Freerider™, QualiPoc iQ™, QualiPoc Mobile™, QualiPoc Static™, QualiWatch-M™, QualiWatch-S™, SystemInspector™, TestManager™, VMon™, VQuad-HD™ are trademarks of SwissQual AG.

SwissQual acknowledges the following trademarks for company names and products:

Adobe®, Adobe Acrobat®, and Adobe Postscript® are trademarks of Adobe Systems Incorporated.

Apple is a trademark of Apple Computer, Inc.

DIMENSION®, LATITUDE®, and OPTIPLEX® are registered trademarks of Dell Inc.

ELEKTROBIT® is a registered trademark of Elektrobit Group Plc.

Google® is a registered trademark of Google Inc.

Intel®, Intel Itanium®, Intel Pentium®, and Intel Xeon™ are trademarks or registered trademarks of Intel Corporation.

INTERNET EXPLORER®, SMARTPHONE®, TABLET® are registered trademarks of Microsoft Corporation.

Java™ is a U.S. trademark of Sun Microsystems, Inc.

Linux® is a registered trademark of Linus Torvalds.

Microsoft®, Microsoft Windows®, Microsoft Windows NT®, and Windows Vista® are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries U.S.

NOKIA® is a registered trademark of Nokia Corporation.

Oracle® is a registered US trademark of Oracle Corporation, Redwood City, California.

SAMSUNG® is a registered trademark of Samsung Corporation.

SIERRA WIRELESS® is a registered trademark of Sierra Wireless, Inc.

TRIMBLE® is a registered trademark of Trimble Navigation Limited.

U-BLOX® is a registered trademark of u-blox Holding AG.

UNIX® is a registered trademark of The Open Group.


© 2000 - 2011 SwissQual AG

Contents | CONFIDENTIAL MATERIALS

ii

Contents 1 Introduction .......................................................................................................................................... 1

Video Streaming Service ....................................................................................................................... 1

2 Configuring a Video over IP Streaming Test .................................................................................... 3

Overview ................................................................................................................................................ 3 QuickTime and RealPlayer Settings ...................................................................................................... 3

Configuring QuickTime ..................................................................................................................... 4

Configuring RealPlayer .................................................................................................................... 6

Creating a Streaming Test ..................................................................................................................... 8

Configuring a Video No Reference Test .......................................................................................... 9 Configuring a Video Full Reference Test............................................................................................. 13

Configuration - Timings ....................................................................................................................... 17

3 Viewing Test Results in NQDI .......................................................................................................... 18

Stream and Session Results ............................................................................................................... 19

Video and Audio Content Results ........................................................................................................ 22

4 Video Full Reference Streams .......................................................................................................... 23

Available Video Full Reference Streams ............................................................................................. 23

Overview of VQuad Results ................................................................................................................ 24

QuickTime Player Results .............................................................................................................. 24

RealPlayer Results ......................................................................................................................... 27

5 Streaming From YouTube ................................................................................................................. 30

Source Signals ..................................................................................................................................... 30 YouTube Test Emulating PC ............................................................................................................... 30

YouTube Test Emulating a Mobile Phone ........................................................................................... 32

Uploading Video Material on YouTube ................................................................................................ 33

Referring to YouTube Video Clips in TestManager ....................................................................... 34

Figures Figure 1-1 Overview of Video Streaming test .................................................................................................... 1

Figure 2-1 Video Streaming - Layer model........................................................................................................ 3 Figure 2-2 QuickTime Audio settings ................................................................................................................ 4

Figure 2-3 QuickTime Streaming settings ......................................................................................................... 5

Figure 2-4 QuickTime Advanced settings .......................................................................................................... 5

Figure 2-5 RealPlayer Content / clip cache settings ......................................................................................... 6

Figure 2-6 RealPlayer Hardware settings / Video Card compatibility ............................................................... 6 Figure 2-7 RealPlayer Hardware settings / Sound Card compatibility .............................................................. 7

Figure 2-8 RealPlayer AutoUpdate setting ........................................................................................................ 7




iii

Figure 2-9 RealPlayer Message Center Preferences ........................................................................................ 8

Figure 2-10 Video Streaming – Job definition ................................................................................................... 8 Figure 2-11 Select "Video No Reference" Streaming test type ......................................................................... 9

Figure 2-12 Configuration of a Video No Reference test ................................................................................ 10

Figure 2-13 Overview of timings in a Video No Reference test....................................................................... 12

Figure 2-14 Image resolution options for a video streaming test .................................................................... 12

Figure 2-15 Key-word settings for a dynamic navigation ................................................................................ 13

Figure 2-16 Test selection of a Video Full Reference test .............................................................................. 14 Figure 2-17 Settings on the General tab of a Video Full Reference test ......................................................... 14

Figure 2-18 Schematic timing overview – Full Reference ............................................................................... 16

Figure 2-19 Schematic timing overview – Full Reference for high resolution / frame-rate video sequences . 16

Figure 2-20 Timings required for a Video Streaming test ................................................................................ 17

Figure 3-1 Importing a measurement file into NQDI ........................................................................................ 18 Figure 3-2 Data selection in NQDI ................................................................................................................... 19

Figure 3-3 Session, Service and Test Layer in NQDI ..................................................................................... 19

Figure 3-4 Session and Stream information in NQDI ...................................................................................... 20

Figure 3-5 Video Status trace providing video protocol information ................................................................ 20

Figure 3-6 Protocol information in NQDI ......................................................................................................... 21

Figure 3-7 Media information in NQDI ............................................................................................................. 21 Figure 3-8 Frame information in the media info tab-sheet .............................................................................. 22

Figure 3-9 Data analysis in NQDI .................................................................................................................... 22

Figure 5-1 Selection of Flash-Player in YouTube tests emulating PC ............................................................ 30

Figure 5-2 YouTube Full Reference tests emulating PC ................................................................................. 31

Figure 5-3 YouTube test emulating a mobile phone ....................................................................................... 32

Tables Table 2-1 Description of the settings on the General tab for a Video No Reference streaming test .............. 10 Table 2-2 Description of the settings on the General tab for a Video Full Reference streaming test ............. 14

Table 4-1 QuickTime player results from the “john” clip in QCIF resolution for different coding settings ....... 24

Table 4-2 QuickTime player results from the “laugh” clip in QCIF resolution for different coding settings ..... 25

Table 4-3 QuickTime player results from the “solothurn” clip in QCIF resolution for different coding settings 25

Table 4-4 QuickTime player results from the “john” clip in QVGA resolution for different coding settings ..... 26 Table 4-5 QuickTime player results from the “laugh” clip in QVGA resolution for different coding settings ... 26

Table 4-6 QuickTime player results from the “solothurn” clip in QVGA resolution for different coding settings ......................................................................................................................................................................... 27

Table 4-7 RealPlayer results from the “john” clip in QCIF resolution for different coding settings .................. 27

Table 4-8 RealPlayer results from the “laugh” clip in QCIF resolution for different coding settings................ 27

Table 4-9 RealPlayer results from the “solothurn” clip in QCIF resolution for different coding settings ......... 28 Table 4-10 RealPlayer results from the “john” clip in QVGA resolution for different coding settings .............. 28

Table 4-11 RealPlayer results from the “laugh” clip in QVGA resolution for different coding settings ............ 29




iv

Table 4-12 RealPlayer results from the “solothurn” clip in QVGA resolution for different coding settings ...... 29



Chapter 1 | Introduction CONFIDENTIAL MATERIALS

1

1 Introduction The Diversity Streaming Service Test Manual describes how the SwissQual Diversity software evaluates the quality of video streaming services as well as how to configure streaming service tests, that is, Diversity Streaming PC tests.

Note: As of Diversity 10.6, SwissQual renamed the former Job-Type ‘Streaming PC’ to ‘Video over IP’ that matches to the extended capabilities of this set of tests. Diversity 10.6 enables for first time the video streaming from YouTube using a FlashPlayer client. This emulates a PC user interface. Streaming from YouTube using a mobile phone ID allows the use of RealPlayer and QuickTime as in a phone as user interface.

Video Streaming Service Streaming refers to the ability of an application to play synchronized media streams, such as audio and video streams, in a continuous manner while the streams are being transmitted to the client over a data network. The software client plays the incoming multimedia stream in real time as the data is received.

Typical video streaming services include live and pre-recorded TV channels, informational content, such as traffic and weather, as well as surveillance and entertainment portals. Video streaming is already present in 2G and 2.5G networks and usually occurs over Packet Switched Services (PSS).

Common video streaming scenarios are on-demand and live information delivery applications. Examples of the first scenario are music and on-demand news applications, while live delivery of radio and television transmissions is an example of the second category.

In 3G systems, the 3G packet-switched streaming service (PSS) fills the gap between 3G MMS, for example, for download and conversation services.

Figure 1-1 Overview of Video Streaming test

Objective video streaming tests require a streaming server, a video client, and an objective algorithm for video quality assessment. SwissQual uses a Darwin Streaming Server to host the and stream the video clips for a test and the QuickTime and RealPlayer software on the client computer to play the streams. The client uses the RTSP or HTTP protocol to send a streaming connection request to the Darwin server. After the connection is established, the client starts the video capture and stores the output video sequence on the hard drive.



Chapter 1 | Introduction CONFIDENTIAL MATERIALS

2

An additional package enables the video streaming from YouTube either emulating a mobile phone or a PC as user interface.

In a full reference approach, the VQuad algorithm evaluates the difference between the output video sequence and the corresponding source video reference sequence on a psycho-visual scale to, predict the video quality. In a no reference approach, the VMon algorithm only evaluates the output video sequence.



Chapter 2 | Configuring a Video over IP Streaming Test CONFIDENTIAL MATERIALS

3

2 Configuring a Video over IP Streaming Test This chapter describes how to configure Video No Reference and Video Full Reference Streaming PC tests in Diversity TestManager. These tests are designed to evaluate a subscriber’s experience with a video streaming service.

Overview To set up and to maintain a video stream for a specific amount of time, for example, 2 minutes, a Streaming PC test requires a data connection. During the streaming process, the test performs a series of visual quality measurements and analyzes the transmission success of the incoming stream. These measurements provide a signal quality profile for the duration of the streaming activity.

The test uses a mobile phone as a dial-up device and a software video player, that is QuickTime or RealPlayer, on a computer to play back the video stream. This configuration enables full access to the entire IP layer as well as all of the stream and the media layer parameters.

The SwissQual VQuad algorithm objectively evaluates the quality of a full reference video capture whereas the VMon algorithm evaluates the quality of a no reference video capture. The tests capture the video from the display of a mobile phone.

Streaming PC streaming tests use the following layers to provide a set of corresponding results:

• Data Session layer: Contains the dial-up and the IP layers

• Service layer: Contains the media stream

• Test layer: Contains the content-based quality analysis

Figure 2-1 Video Streaming - Layer model

A Streaming PC test analyzes the data session for dial-up times, throughput on IP protocol information, and the success state of the dial up.

Note: You can also use a permanent IP connection during a test instead of a dial-up device.

The quality evaluation of the streaming service is based on connection and buffering times as well as the success state. A content analysis within the stream provides detailed information about the perceived quality and the main signal degradations.

Before you can create a Streaming PC test, you need to verify the QuickTime and RealPlayer settings.

QuickTime and RealPlayer Settings This section describes the QuickTime player and RealPlayer settings that Diversity requires to produce comparable measurement results.

Important: You need to reconfigure these settings whenever you update, repair, or reinstall QuickTime or RealPlayer. If you use Diversity in Optimizer mode, you also need to verify these settings before you start each measurement as some settings can change during a measurement.

Data Session DialUp

Streaming Test Connect

VQ VQ VQ VQ

Streaming Test

VQ VQ VQ VQ

Connect




4

Configuring QuickTime This section describes how to configure QuickTime for use with Diversity.

To configure QuickTime for Diversity 1. On the Start menu, click Control Panel, and then double-click QuickTime.

2. On the Audio tab, specify the following audio settings.

a In the Devices area, specify the following options and values:

• Select the Safe mode (waveOut only) option.

• In the Audio Output Device Latency (msec) box, enter 33.000.

b In the Sound Out area, specify the following values:

• In the Rate list, click 48 kHz.

• In the Size list, click 16 bit.

• In the Channels list, click Mono.

c In the Default Music Synthesizer list, click QuickTime Music Synthesizer.

Figure 2-2 QuickTime Audio settings

3. On the Streaming tab, click Automatic in the Streaming Speed list and clear the following check boxes:

• Override speed for downloads

• Enable Instant-On




5

Figure 2-3 QuickTime Streaming settings

4. On the Advanced tab, click Automatic in the Transport Setup list, and then specify the following options:

• Clear the RTSP Proxy Server check box.

• Drag the Download Cache slider to 0.

• In the Video area, click Safe mode (GDI only).

Note: DirectX strongly affects the appearance of the video image, which in turn has a negative impact on the measurement results.

• Clear the Enable encoding using legacy codecs and Install QuickTime icon in system tray check boxes.

Figure 2-4 QuickTime Advanced settings




6

Configuring RealPlayer This section describes how to configure RealPlayer for use with Diversity.

To configure RealPlayer for Diversity 1. On the Start menu, point to All Programs, point to Real, and then click RealPlayer SP.

Note: If a registration window opens, clear the check boxes, and then cancel the registration.

2. On the Tools menu, click Preferences.

3. Disable the temporary clip cache.

a In the Category tree, click Content.

b In the Cache settings area, clear the Enable caching check box.

Figure 2-5 RealPlayer Content / clip cache settings

4. Specify the video card compatibility.

a In the Category tree, click Hardware.

b In the Video Card Compatibility area, drag the slider to Most reliable, and then click OK.

Figure 2-6 RealPlayer Hardware settings / Video Card compatibility




7

5. Specify the playback sound quality.

a In the Category tree, click Hardware.

b In the Sound Card Compatibility area, click Settings.

c Clear the Disable Direct Sound check box.

d Drag the Playback Quality slider to Best quality, and then click OK.

Figure 2-7 RealPlayer Hardware settings / Sound Card compatibility

6. Disable the automatic update feature.

a In the Category tree, expand Automatic Services, and then click AutoUpdate.

b Clear the Automatically download and install important updates check box.

Figure 2-8 RealPlayer AutoUpdate setting

7. Disable the RealPlayer Message Center.




8

a In the Category tree, expand Automatic Services, and then click Message Center.

b In the Message Center window, clear the check boxes, and then click OK.

Figure 2-9 RealPlayer Message Center Preferences

Creating a Streaming Test A streaming service test requires either a permanent IP access or defined access points in the subscriber and provider configuration for dial-up connections.

To create a Video over IP test in Diversity TestManager 1. In TestManager, click Tasks on the Navigation panel.

2. Click Jobs and click Add job.

3. On the General tab, click Streaming PC in the Job type list.

4. In the Job name box, type a name for the streaming job, for example, "NoRef".

Figure 2-10 Video Streaming – Job definition

5. On the Video over IP tests tab and the Timings tab, specify the settings for the job.

Note: For more information, see the "Configuring a Video No Reference Test" and "Configuring a Video Full Reference Test" sections on pages 9 and 13 respectively.




9

You can use the settings on the Video over IP tests tab to define one of the following test types:

• Video No Reference: Video test for unknown content, such as public streaming servers

Important: This test does not evaluate the audio signal.

• Video Full Reference: Video test that requires a known reference video signal. You can only use this test to evaluate video signals from a streaming server that you control.

Important: In case you have enabled he YouTube test option you can select up to four further tests

• YouTube Full Reference (emulate PC): Video test that requires a known reference video signal pre-stored on YouTube. This test emulates a user at a PC approaching the YouTube site and watches the video using FlashPlayer.

• YouTube No Reference (emulate PC): Video test that streams a video signal from YouTube at a given URL. This test emulates a user at a PC approaching the YouTube site and watches the video using FlashPlayer.

• YouTube Full Reference (emulate Phone): Video test that requires a known reference video signal pre-stored on YouTube. This test emulates a user using a mobile phone approaching the YouTube site and identifies itself as a mobile phone. The stream will be enabled from a special YouTube site tailored for mobile phones and allowing RealPlayer or QuickTime as viewers.

• YouTube No Reference (emulate Phone): Video test that streams a video signal from YouTube at a given URL. This test emulates a user using a mobile phone approaching the YouTube site and identifies itself as a mobile phone. The stream will be enabled from a special YouTube site tailored for mobile phones and allowing RealPlayer or QuickTime as viewers.

Configuring a Video No Reference Test This section describes the settings that you can configure for a Video No Reference test on the Streaming PC tests tab for a job. The Video No Reference test uses the VMon algorithm to evaluate the quality of a video stream. However, since this test uses unknown content, VMon cannot evaluate the audio component of the stream.

Figure 2-11 Select "Video No Reference" Streaming test type

Important: Use the Video No Reference test type for unknown content, such as public streaming servers. Use the Video Full Reference test type when you evaluate a known reference video stream that originates on a server that you control.

Although Video No Reference tests can use video streams in VGA and SDTV resolution, the recording and evaluation of the video is limited by Diversity processing power restrictions. Diversity records and calculates session statistics, session-related KPIs as well as the IP throughput and Media Info of each measurement.

Note: VMon does not evaluate each frame in the spatial domain during image analysis and assumes that the skipped frames contain similar image artifacts as the evaluated frames.




10

General Tab

Figure 2-12 Configuration of a Video No Reference test

The following table describes the settings that you can configure on the General tab for a Video No Reference streaming test.

Table 2-1 Description of the settings on the General tab for a Video No Reference streaming test

Setting Description

Stream dur. Specifies the duration in seconds of the streaming test. If the test loses the stream during this time, the test status is set to DROPPED, otherwise the status is set to COMPLETED.

Capture time Specifies the length of the sample in seconds to capture for the quality analysis. For best results, enter a value in the range of 6 to 15 seconds.

The VMon algorithm uses the capture time to calculate a set of integrative quality values, for example, visual quality and blockiness, from the video capture sample. VMon seamlessly analyzes each portion of the complete capture. For an overview of the timing options for a video streaming test, see Figure 2-13 on page 12.

Connection TimeOut

Specifies the time after the test becomes aborted in case no connection to the video server can be established. The streaming session is set FAILED / Connection TimeOut in this case.

Stream Lost TimeOut

Specifies the time after the test becomes aborted in case no video packages are received anymore. The streaming session is set DROPPED / TimeOut in this case.

Player Specifies which video player to use for the test, that is, QuickTime or RealPlayer.

Important: Only QuickTime 7.1 and higher supports H.264 video streams.

URL Enter the URL for a network video stream.




11

Setting Description

Image Resolution

Select Auto if you want the video stream to play at the native resolution, which is defined in the session description of the stream. If the player cannot interpret this description, the video plays at a resolution of 176x144 pixels, that is, QCIF.

If you want to force the player to use a specific resolution, click a resolution in the list, for example, QCIF(176x144) or QVGA(320x240).

Figure 2-14 on page 12 for the available image resolution options.

Note: If the forced size does not match the native image resolution of the stream, the internal re-scaling functions of the player are applied to achieve the forced size, which can result in artifacts.

Stream This option is only for Video Full Reference tests.




12

Setting Description

Save A/V file Specifies the file saving options for a video stream during a measurement. Diversity saves the videos as *.AVI files.

The AVI files contain uncompressed video in RGB format at the original frame rate and resolution. Since the size of these files can be quite large, saving and copying the files can require a significant amount of CPU load and time.

Note: For best results in Diversity Q, click None.

Run Speech Evaluation

This option is only for Video Full Reference tests.

Video Recording Quality

The captured video becomes compresses in three quality levels. It saves storage capacity but may lead to compression artefacts by re-viewing the recording. The compression only affects the storage after evaluation. The evaluation is made on the un-compressed video.

Advanced Allows you to manually specify a dynamic link configuration by keywords to be spotted in websites and following the link behind.

Start VMon

Capture Time

Video evaluation

Report Vmon

Video evaluation

Report VMon

Start VMon

Video Stream

Figure 2-13 Overview of timings in a Video No Reference test

Figure 2-14 Image resolution options for a video streaming test




13

Advanced Tab In the Dynamic URL box, enter a series of keywords for dynamic link navigation. Add each new keyword to a new line.

Note: You can use an asterisk (*) as a wildcard to replace an undefined number of characters in a keyword.

The test searches the Web site that you specified in the URL box on the General tab, see Table 2-1 on page 10, for the first keyword in the list. If the keyword is found, the test opens page that contains the keyword and searches for the second keyword, and so on. The test only searches for the first occurrence of a keyword on a page.

Figure 2-15 Key-word settings for a dynamic navigation

Important: TestManager cannot retrieve a list of keywords from an HTTP page. Instead, you need to use external HTTP source interpreter to recognize the keywords, which you can then copy into the Dynamic URL box. Keywords are often hidden behind images.

Configuring a Video Full Reference Test This section describes the settings that you can configure for a Video Full Reference test on the Video over IP tests tab for a job. The Video Full Reference test uses the VQuad algorithm to evaluate the quality of a video stream. In addition SQuad-LQ can be used to evaluate the quality of the audio voice stream. The voice signal is evaluated in wideband mode, since video streaming is usually not restricted to telephony bandwidth.

Important: Video Full Reference clips in VGA and SD resolution are not part of the default Diversity installation as these clips require special environmental conditions. If you want to use these clips, you need to install the package, that is, sets 6 to 8, separately. For more information, contact SwissQual support.




14

Figure 2-16 Test selection of a Video Full Reference test

A Video Full Reference test requires a known reference video stream that is on a server, which you control. For a description on how to set up the server, see the Installation - RedHat Mediaserver.pdf document. For information on the available video streams, see the "Video Full Reference Streams" chapter on page 23.

Note: A blue sequence separates each content clip in the stream so that you can recognize the start and the end of a clip.

Figure 2-17 Settings on the General tab of a Video Full Reference test

Table 2-2 Description of the settings on the General tab for a Video Full Reference streaming test

Setting Description

Stream duration Specifies the duration in seconds for the streaming test. If the test loses the stream during this time, the test status is set to DROPPED, otherwise the status is set to COMPLETED.

Capture time Automatically displays the duration in seconds for the streaming sequence. The capture time is included in the video stream. The capture starts with a blue synchronization sequence and ends with the detection of the next blue marker. These sequences are between 05 and 1 seconds in length and use a sequence marker in the bottom lines to identify the next streaming sequence.

Connection timeout Specifies the time after the test becomes aborted in case no connection to the video server can be established. The streaming session is set FAILED / Connection TimeOut in this case.




15

Setting Description

Stream Lost timeout Specifies the time after the test becomes aborted in case no video packages are received anymore. The streaming session is set DROPPED / TimeOut in this case.

Player Specifies which video player to use for the test, that is, QuickTime or RealPlayer.

Important: Only QuickTime 7.1 and higher supports H.264 video streams.

URL Enter the URL that points to the server as well as the reference video sample that you want to stream.

Image resolution For a Video Full Reference test, the resolution of the captured video must be the same as the reference video stream, that is, select Auto. However, if the player cannot determine the resolution from the session description in the video stream, you need to force the player to use the correct resolution, that is, click the resolution in the list.

Important: If you do not select the same resolution as the reference video, the quality analysis of the measurement yields incorrect results.

Stream Select the reference video sample that you want to use for the test.

Save A/V file Specifies the file saving options for a video stream during a measurement. Diversity saves the videos as *.AVI files.

The AVI files contain uncompressed video in RGB format at the original frame rate and resolution. Since the size of these files can be quite large, saving and copying the files can require a significant amount of CPU load and time.

Note: For best results in Diversity Q, click None.

Run Speech Evaluation Evaluates the voice signal in a video stream with SQuad-LQ in wideband mode.

Video Recording Quality

The captured video becomes compresses in three quality levels. It saves storage capacity but may lead to compression artefacts by re-viewing the recording. The compression only affects the storage after evaluation. The evaluation is made on the un-compressed video.

Advanced Allows you to manually specify a dynamic link configuration by keywords to be spotted in websites and following the link behind.

The concept of the full reference video evaluation is slightly different from No Reference using VMon. Due to the comparison to the reference signal there is synchronization between the two signals required. For that reason the video sequence is interlaced with short blue sequences for synch-up. In addition, the evaluation of the video sequence requires a complete capturing and can’t be done in real-time as for VMon in the Video No Reference test.

Since, the full reference video evaluation is very resource consuming; the capturing of the following video stops so long the previous video is evaluated.




16

Start VQuad

Capture Time

Accumulating video frames

Report VQuad


Blue synch

Capture Time

Report VQuad

Fast evaluation (low resolution, low frame rate, high performance CPU) Figure 2-18 Schematic timing overview – Full Reference

The computation time depends on the CPU performance but also on image resolution and frame rate (the more frames and the bigger the frames are, the higher is the computation effort).

The computation can take between a few seconds until several 10s for QVGA or higher and frame rates > 20fps.

Capture Time


Blue synch

Report VQuad

Time consuming evaluation (high resolution, high frame rate, low performance CPU)

Accumulating video framesVQuad evaluation

Figure 2-19 Schematic timing overview – Full Reference for high resolution / frame-rate video sequences




17

Configuration - Timings

Figure 2-20 shows the required timings for the video streaming test. The influence of the individual timers can be seen in the graphics below the settings.

Figure 2-20 Timings required for a Video Streaming test



Chapter 3 | Viewing Test Results in NQDI CONFIDENTIAL MATERIALS

18

3 Viewing Test Results in NQDI This chapter describes how to view the results of a video streaming test in the NQDI post-processing software.

To import and view Streaming PC test data in NQDI 1. In NQDI, click the Data Management tab.

2. Click the Add files to the import list icon and browse to the folder that contains the measurement file.

3. Click Add and click Close.

4. On the toolbar, click the Start import process icon.

Figure 3-1 Importing a measurement file into NQDI

5. Select the data that you want to analyze.

a Click the Data Selection tab and click the Refresh icon on the Selection Criterion toolbar.

b In the Test List of Current Section area, double-click a test block to view the stream and session results.

Note: For more information, see the following section.




19

Figure 3-2 Data selection in NQDI

Stream and Session Results You can analyze each individual clip on the Analysis tab as well as view the Data Session, Service, and Test layers. The red elements in Figure 3-3 represent the clip, or rather, block of tests that you selected on the Data Selection tab, Stream-1 represents a full reference clip evaluation, and Stream-2 represents no reference clip evaluations.

Figure 3-3 Session, Service and Test Layer in NQDI

The figure also shows the related radio and transport layer information for a radio connection or, more specifically, HSDPA over UMTS 2100.

On the Session info tab of the Analysis tab, you can analyze the results of the streaming PC test session.




20

Figure 3-4 Session and Stream information in NQDI

The left panel of the Session info tab contains general information about the data session as well as the session status, that is, FAILED, DROPPED, or COMPLETED. The main panel of the tab contains the status information of the streaming service as well as other stream-related information such as defined player, URL, used protocol, image resolution, Duration, and time to first picture.

The Video Status Trace tab provides an overview of protocol-related negotiations and player information, that is, buffering status.

Figure 3-5 Video Status trace providing video protocol information

On the Protocol info tab of the Analysis tab, you can view the IP log and a graphical representation of the IP traffic.




21

Figure 3-6 Protocol information in NQDI

The Media info tab of the Analysis tab contains partially decoded MPEG4 part 2, H.263, and H.264 elementary streams.

Note: This media information, including the video codec in NQFI, is only decoded for RTSP streams.

Figure 3-7 Media information in NQDI




22

In Figure 3-7 and Figure 3-8, each bar of the top graph represents an individual frame, each colour represents a frame type, and the height of each bar represents the frame size in bytes. See an enlarged view in. The lower graph in Figure 3-7 is a copy of the IP-throughput graph and is included for direct comparison.

Figure 3-8 Frame information in the media info tab-sheet

Video and Audio Content Results You can view the video and audio measurement results of a clip on the Visual Quality and Listening Quality tabs in the Analysis window. For example, Figure 3-9 shows the VMon results for a clip.

Figure 3-9 Data analysis in NQDI

For a description and interpretation of the individual results, see the following documents:

• SQuad-LQ: Manual - SQuad Voice Test Result Description.pdf

• VMon and VQuad: Manual - VMon and VQuad Results Description.pdf

In a Video Full Reference test, the audio signal component of the AV signal is evaluated by SQuad-LQ with respect to the clean reference audio signal. In a Video No Reference test, the audio signal is not analysed regarding quality since the audio signal cannot be evaluated with respect to a pre-defined audio signal.



Chapter 4 | Video Full Reference Streams CONFIDENTIAL MATERIALS

23

4 Video Full Reference Streams This chapter describes the pre-encoded video streams that are available for Video Full Reference tests. The available streams consist of 3 different video sequences that have been encoded with different settings. Each stream is 5 minutes long and contains multiple repetitions of an 8 second long video clip, which are separated by one second blue sequences.

Available Video Full Reference Streams

Diversity has 8 sets of video streams for use with Video Full Reference tests. The audio tracks of the streams do match the content of the video component. Instead, each audio track is a highly qualitative wideband reference speech clip in American English that has been encoded with different settings.

In the following lists, each asterisk (*) is a placeholder for a short name that describes the content of the clip. Furthermore, the name “john” refers to a typical head-and-shoulder image, “Laugh” refers to a clip that has moderate movement, and “solothurn” refers to a clip that contains the most movement. The bit rates are approximate average content bit rates. However, the peak bit rates can be much higher. Due to the protocol overhead, the average bandwidth of the clips is higher than the sum of the codec bit rates. The size of the IP packets depends strongly on the bit rate and is usually in the range of 300 to 1400 bytes. The protocol overhead is around 40 bytes for each packet, which results in an overhead of 3 to 13 %.

• Set 1: 40kbps MPEG4/H.264 video, 8.333 / 12.5 fps, 8kbps AMR audio

o 5min_*_QCIF_8.333fps_40k-mpg4_8k-amr_08.3gp

o 5min_*_QCIF_12.5fps_40k-mpg4_8k-amr_08.3gp

o 5min_*_QCIF_12.5fps_40k-h264_8k-amr_08.3gp

• Set 2: 72kbps MPEG4/H.264 video, 12.5 fps, 24kbps AAC audio

o 5min_*_QCIF_12.5fps_72k-mpg4_24k-aac_08.3gp

o 5min_*_QCIF_12.5fps_72k-h264_24k-aac_08.3gp

• Set 3: 112kbps MPEG4/H.264 video, 12.5 fps, 24kbps AAC audio

o 5min_*_QCIF_12.5fps_112k-mpg4_24k-aac_08.3gp

o 5min_*_QCIF_12.5fps_112k-h264_24k-aac_08.3gp

o 5min_*_QVGA_12.5fps_112k-mpg4_24k-aac_08.3gp

o 5min_*_QVGA_12.5fps_112k-h264_24k-aac_08.3gp

• Set 4: 172kbps MPEG4/H.264 video, 25 fps, 32kbps AAC audio

o 5min_*_QCIF_25fps_172k-mpg4_32k-aac_08.3gp

o 5min_*_QCIF_25fps_172k-h264_32k-aac_08.3gp

o 5min_*_QVGA_25fps_172k-mpg4_32k-aac_08.3gp

o 5min_*_QVGA_25fps_172k-h264_32k-aac_08.3gp

• Set 5: 320kbps MPEG4/H.264 video, 25 fps, 48kbps AAC audio

o 5min_*_QVGA_25fps_320k-mpg4_48k-aac_08.3gp


• Set 6: 700kbps H.264 video, 25 fps, 64kbps AAC audio


o 5min_*_VGA_25fps_700k-h264_64k-aac_08.3gp

• Set 7: 1200kbps H.264 video, 25 fps, 128kbps AAC audio




24



• Set 8: 3100kbps/5500kbps H.264/MPEG2 video, 25 fps, 128kbps AAC audio


o 5min_*_SD_25fps_3100k-h264_128k-aac_08.3gp

o 5min_*_SD_25fps_5500k-h264_128k-aac_08.3gp

Overview of VQuad Results This section provides an overview of the QuickTime Player and RealPlayer Mean Opinion Scores (MOS) values and measurement results. The following types of artefacts result in a reduction of the VQuad and SQuad MOS values:

• Encoding artefacts in the video full reference stream on the server

• Transmission artefacts that the stream produces

The tables in this section present VQuad results for lossless transmission. These values represent the expectation that one has for the different individual results in perfect transmission conditions.

The tests were conducted on a Diversity system and involved streaming each video clip 10 to 20 times through a LAN connection. The tables also contain the 90th percentile values of the VQuad and SQuad results for each clip as well as the 10th percentile values of the individual degradations that were calculated by VQuad. The MOS values have an uncertainty of approximately +/- 0.05 MOS, while the other values have an uncertainty of approximately +/- 5 %.

Important: To obtain comparable results, you need to use the same QuickTime player and RealPlayer settings as described in "QuickTime and RealPlayer Settings" on page 3.

QuickTime Player Results

Video clip “john”, Resolution QCIF, Player QuickTime Table 4-1 QuickTime player results from the “john” clip in QCIF resolution for different coding settings

Encoding Settings VQuad MOS

Blockiness [%]

Blurring [%]

Jerkiness [%]

SQuad MOS

40kbps MPEG4 video, 8.333 fps, 8kbps AMR 2.12 81 25 9


40kbps H.264 video, 12.5 fps, 8kbps AMR 2.91 2 27 3

72kbps MPEG4 video, 12.5 fps, 24kbps AAC 2.98 62 17 3

72kbps H.264 video, 12.5 fps, 24kbps AAC 3.75 3 12 2



172kbps MPEG4 video, 25 fps, 32kbps AAC 3.85 34 11 0

172kbps H.264 video, 25 fps, 32kbps AAC 4.19 5 6 0




25

Video clip “laugh”, Resolution QCIF, Player QuickTime Table 4-2 QuickTime player results from the “laugh” clip in QCIF resolution for different coding settings


Blockiness [%]

Blurring [%]

Jerkiness [%]

SQuad MOS










Video clip “solothurn”, Resolution QCIF, Player QuickTime Table 4-3 QuickTime player results from the “solothurn” clip in QCIF resolution for different coding settings


Blockiness [%]

Blurring [%]

Jerkiness [%]

SQuad MOS













26

Video clip “john”, Resolution QVGA, Player QuickTime Table 4-4 QuickTime player results from the “john” clip in QVGA resolution for different coding settings


Blockiness [%]

Blurring [%]

Jerkiness [%]

SQuad MOS









Video clip “laugh”, Resolution QVGA, Player QuickTime Table 4-5 QuickTime player results from the “laugh” clip in QVGA resolution for different coding settings


Blockiness [%]

Blurring [%]

Jerkiness [%]

SQuad MOS












27

Video clip “solothurn”, Resolution QVGA, Player QuickTime Table 4-6 QuickTime player results from the “solothurn” clip in QVGA resolution for different coding settings


Blockiness [%]

Blurring [%]

Jerkiness [%]

SQuad MOS









RealPlayer Results

Video clip “john”, Resolution QCIF, Player RealPlayer Table 4-7 RealPlayer results from the “john” clip in QCIF resolution for different coding settings


Blockiness [%]

Blurring [%]

Jerkiness [%]

SQuad MOS






Video clip “laugh”, Resolution QCIF, Player RealPlayer Table 4-8 RealPlayer results from the “laugh” clip in QCIF resolution for different coding settings


Blockiness [%]

Blurring [%]

Jerkiness [%]

SQuad MOS





28


Blockiness [%]

Blurring [%]

Jerkiness [%]

SQuad MOS





Video clip “solothurn”, Resolution QCIF, Player RealPlayer Table 4-9 RealPlayer results from the “solothurn” clip in QCIF resolution for different coding settings


Blockiness [%]

Blurring [%]

Jerkiness [%]

SQuad MOS






Video clip “john”, Resolution QVGA, Player RealPlayer Table 4-10 RealPlayer results from the “john” clip in QVGA resolution for different coding settings


Blockiness [%]

Blurring [%]

Jerkiness [%]

SQuad MOS







29

Video clip “laugh”, Resolution QVGA, Player RealPlayer Table 4-11 RealPlayer results from the “laugh” clip in QVGA resolution for different coding settings


Blockiness [%]

Blurring [%]

Jerkiness [%]

SQuad MOS




Video clip “solothurn”, Resolution QVGA, Player RealPlayer Table 4-12 RealPlayer results from the “solothurn” clip in QVGA resolution for different coding settings


Blockiness [%]

Blurring [%]

Jerkiness [%]

SQuad MOS






Chapter 5 | Streaming From YouTube CONFIDENTIAL MATERIALS

30

5 Streaming From YouTube As of Diversity Release 10.6, a YouTube streaming option is also available. This option is an extension of the regular Diversity video streaming test; however, unlike the regular test, the video streams from the YouTube.com server and not from a video server that you control, for example, a Helix or Darwin server. The design and definition of a YouTube test is almost identical to the regular video streaming approach since the technical process of streaming, capturing and evaluation is essentially the same for both tests.

One difference between the tests is that the YouTube video streams are not on a single server or restricted to a single URL. YouTube provides uploaded video content in different resolutions and bitrates for different user clients and qualities. When you upload a video clip to the site, YouTube transcodes the clip automatically, an action that becomes part of the processing chain for SwissQual to measure. The dynamic nature of the device and user dependent quality of the uploaded clips results in Diversity YouTube tests that are close to the real user experience.

The YouTube test is an application of the SwissQual IP video streaming concept in that the KPIs and the results that VQuad and VMon produce are identical to the regular IP streaming tests.

Source Signals SwissQual provides a set of video clips on YouTube that have the resolutions and qualities of video content that people typically access through mobile phones. More specifically, the contents of the clips are the same as the clips that Diversity uses for the regular Video over IP tests on Darwin and Helix media servers, that is, ‘John’, ‘Laugh’ and ‘Solothurn’. These clips contain low motion head and shoulder, median motion, and high motion content.

Each of the contents is available with an aspect ratio of 4:3 in three of the natively supported YouTube resolutions, that is, QQVGA (160x120), QVGA (320x240), and ~HVGA (480x360). SwissQual provides high quality video clips without visible encoding distortions for upload to the YouTube Web site. YouTube performs the actual transcoding according to the available bandwidth.

YouTube Test Emulating PC

If YouTube recognizes that the user client is a PC, which is visible by the operating system information that is exchanged during the HTTP setup, YouTube redirects the request to the regular YouTube Web site. YouTube converts the video clip to a special Flash-based format and assigns the clip an HTTP URL.

Diversity YouTube tests emulate a PC to access this type of URL and video format. The video is displayed and captured by a completely controlled Flash Player implementation. As a result, you can only select the Flash Player for the ‘YouTube No Reference emulating PC’ and ‘YouTube Full Reference emulating PC’.

Figure 5-1 Selection of Flash-Player in YouTube tests emulating PC




31

YouTube No Reference: Uses the same settings as the regular No Reference video tests except for the URL that uses HTTP to link to the video stream on YouTube, that is, http://www.youtube.com/watch?v=0PaGUwhwK-0.

YouTube Full Reference: Uses the same settings as the regular Full Reference test except for the absence of a default stream selection. Unlike the regular test where the stream is stored on a media server that can be directly selected and assessed, YouTube processes and hosts the stream for the test.

Note: You need to enter the complete HTTP address in the URL box.

If you want the address to appear in the list of available URLs, you need to manually add the address to the SQTM reference file list. For more information, see the "Referring to YouTube Video Clips in " section on page 34.

Figure 5-2 YouTube Full Reference tests emulating PC

Important: Flash Player does not report the native image resolution of the clip, which can lead to incorrect results if the 'Auto' is the image size. In the event of a un-reported resolution, ‘Auto’ forces the player into QVGA resolution that might not be the correct resolution. As a result, you need to manually force the resolution whenever possible to ensure the correct native image resolution of the clip. For the Full-Reference test, this information should be available due to the uploaded clip. For the No Reference test, you need to determine this manually.

For demonstration purposes only, video examples are stored in a SwissQual test account on YouTube. These examples are stored in semi-private mode and cannot be found by searching YouTube.

http://www.youtube.com/watch?v=7Sibwe0HRYI // John QQVGA

http://www.youtube.com/watch?v=0PaGUwhwK-0 // John QVGA http://www.youtube.com/watch?v=kxqvGy0-d_c // John HVGA

http://www.youtube.com/watch?v=5TeNIDCBd-U // Laugh QQVGA

http://www.youtube.com/watch?v=V-A_UfaC2gg // Laugh QVGA http://www.youtube.com/watch?v=UmBNNZXUNqA // Laugh HVGA

http://www.youtube.com/watch?v=MPPuEZSW7oM // Solothurn QQVGA

http://www.youtube.com/watch?v=5HbMtM1CGVA // Solothurn QVGA

http://www.youtube.com/watch?v=teNsDytzN1M // Solothurn HVGA

SwissQual recommends that you create your own YouTube account for testing where you can upload the reference video clips from SwissQual. This approach lets you have complete control over the account, including the privacy settings and the number of parallel approaches to the links in the account.

Important: These links are only for YouTube tests that emulate PCs with a Flash Player and not for

http://www.youtube.com/watch?v=0PaGUwhwK-0

http://www.youtube.com/watch?v=7Sibwe0HRYI

http://www.youtube.com/watch?v=0PaGUwhwK-0

http://www.youtube.com/watch?v=kxqvGy0-d_c

http://www.youtube.com/watch?v=5TeNIDCBd-U

http://www.youtube.com/watch?v=V-A_UfaC2gg

http://www.youtube.com/watch?v=UmBNNZXUNqA

http://www.youtube.com/watch?v=MPPuEZSW7oM

http://www.youtube.com/watch?v=5HbMtM1CGVA

http://www.youtube.com/watch?v=teNsDytzN1M




32

YouTube tests that emulate mobile phones.

YouTube Test Emulating a Mobile Phone For a YouTube test that emulates a mobile phone user client by including the operating system information during the HTTP setup, the request is directed to a special YouTube site that is tailored to typical mobile phone display resolutions. You can view a similar site at: http://www.m.youtube.com.

The ‘YouTube Test emulating PC’ uses HTTP protocol and the Flash Player to access and play the Flash video clips on the YouTube site. The mobile phone emulation tests uses RTSP protocol and RealPlayer or QuickTime to access and play the 3GP files on the YouTube site.

Figure 5-3 shows an example of a YouTube No Reference test that emulates a mobile phone.

Figure 5-3 YouTube test emulating a mobile phone

The settings for this test are the same as for Diversity tests that emulate a PC, except for the video players and the URLs.

The following YouTube URLs are only for demonstration purposes of the mobile phone emulation mode. These video clip examples are saved in a SwissQual test account in semi-private mode and cannot be found by searching on the YouTube site. rtsp://v8.cache1.c.youtube.com/CjYLENy73wIaLQmCRQftwZso7RMYJCAkFEIJbXYtZ29vZ2xlSARSBXdhdGNoYJTs7Mr2jKfjTAw=/0/0/0/video.3gp // John QQVGA

rtsp://v5.cache2.c.youtube.com/CjYLENy73wIaLQntK3AIU4b20BMYJCAkFEIJbXYtZ29vZ2xlSARSBXdhdGNoYLnqoJDxy5vOTAw=/0/0/0/video.3gp // John QVGA

rtsp://v2.cache2.c.youtube.com/CjYLENy73wIaLQnld4EwII035RMYJCAkFEIJbXYtZ29vZ2xlSARSBXdhdGNoYLismYriqanjTAw=/0/0/0/video.3gp // Laugh QQVGA

rtsp://v2.cache6.c.youtube.com/CjYLENy73wIaLQkI2oL2UT_gVxMYJCAkFEIJbXYtZ29vZ2xlSARSBXdhdGNoYJu9-s23rJvOTAw=/0/0/0/video.3gp // Laugh QVGA

rtsp://v7.cache3.c.youtube.com/CjYLENy73wIaLQmD7paUEe7zMBMYJCAkFEIJbXYtZ29vZ2xlSARSBXdhdGNoYNqbhZ7nqKfjTAw=/0/0/0/video.3gp // Solothurn QQVGA

rtsp://v7.cache7.c.youtube.com/CjYLENy73wIaLQlQGULNtMx25BMYJCAkFEIJbXYtZ29vZ2xlSARSBXdhdGNoYPrukd3RgJvOTAw=/0/0/0/video.3gp // Solothurn QVGA

Note: The maximum supported resolution of the YouTube mobile site is QVGA. Do not use higher resolutions for the mobile phone emulation mode.

http://www.m.youtube.com/




33

Uploading Video Material on YouTube For best results, create your own YouTube account to act as a repository for the SwissQual video clips. You can also specify the privacy setting for the account.

For No Reference tests, you can upload video clips of your own choice. However, since YouTube does not accept uncompressed RAW AVI files; you need to encode the files beforehand. To avoid poor quality issues, encode the video with the H.264 codec at a high bitrate, that is, >1Mbps for QVGA. When you upload the video, YouTube trans-codes the clip into the target bitrates for the different quality ranges.

Full Reference tests require a special pre-processing of the video clips, such as a separation of chunks by blue sequences, as well as adding sequence marker patterns in the lowest lines of the image. Pre-processed reference clips that contain these sequences and lines are part of the SwissQual Diversity installation.

Important: You need to upload dedicated video clips to YouTube for each of the target formats, for example, QVGA. To avoid resizing by YouTube, you need to upload the clip in the target resolution. If YouTube does resize the video clip, the sequence marker patterns will become unreadable and the evaluation will fail.

Use the URL of the Flash formatted video on the YouTube Web site as the URL for the 'emulate PC' test. For videos in 3GP format, you need to access the file on YouTube with a mobile phone or a tool that mimics a mobile phone to retrieve the URL for the ‘emulate Phone’ tests. You can also go to http://www.m.youtube.com, locate the 3GP video clip, and then copy the HQ URL for the clip.

Note: QVGA is the maximum resolution that the YouTube mobile site supports. That is, do not use higher resolutions for the 'emulating phone' test scenario.

http://www.m.youtube.com/

34

Referring to YouTube Video Clips in TestManager The easiest way to refer to a YouTube link in TestManager is to copy and paste the complete URL of the uploaded video into the test description in TestManager, that is, the HTTP address for emulate PC tests or RTSP address for emulate Phone tests.

If you want to include the URL to the file selection list in TestManager, you need to manually add the URL of to the References.xml TestManager configuration file.

To add a YouTube URL to the file selection list in TestManager 1. Open the following file in a text editor:

C:\Program Files\SwissQual\Diversity\Specification\SwissQual\References\References.xml

2. Add the URL to the file.

a Search for the following passage:

<Video> <Sample>

b Add the following block of code to the YouTube PC emulation or YouTube Phone emulation video section of the file.



<Sample>

<DisplayName>YouTube PC emulation, john qqvga 25fps</DisplayName>

<FileName>http://www.youtube.com/watch?v=7Sibwe0HRYI</FileName>

<Duration>303088</Duration>

<Framerate>25</Framerate>

<BitRate>0</BitRate>

<BitRateClass>Low</BitRateClass>

<Format>QQVGA</Format>

<Quality>High</Quality>

<Complexity>Low</Complexity>

<ContainsAudio>true</ContainsAudio>

<VideoCodec>H264</VideoCodec>

<AudioCodec>AAC_128k</AudioCodec>

</Sample>

c Replace the DisplayName and FileName values in bold with the following relevant information:

• DisplayName: Name of the video stream to display in the TestManager selection list. Depending on the test type, the name must include the keywords "YouTube PC" or "YouTube Phone". The keyword check is not case sensitive.

• FileName: Enter the complete URL, including the protocol, of the video stream on the YouTube server. This URL is unique for each stream and YouTube account that you create.

Important: Keep a copy of the Reference.xml file as this file is overwritten when you update TestManager. After the update, you can replace the new file with your own.

manual - streaming service test

Documents

8kbps amr 40kbps

24kbps aac 112kbps

24kbps aac 72kbps

expand automatic services

public streaming servers

24kbps aac audio

64kbps aac 1200kbps

48kbps aac 320kbps