iptv qoe monitoring

www.dsp-ip.comFast Forward Your Development

IPTV QoS Monitoring


Overview

•QoS and QoE•Measurement and Monitoring Goals

▫Lab, pre-deployment, Monitoring, failure analysis

•Measurement Methods▫Objective, Subjective, payload/codec based

•Monitoring Levels▫Transport, Transaction, content

•Monitoring Domains ▫Head-end, network, Home


Challenges in providing QoE

Name Challenge Answer

network Overcome Packet loss, jitter, network congestion

Improved protocols

QoS mapping schemes

Bandwidth No bandwidth in the network and at the home

increased capacity inside the home and the network level

Monitoring Measure hundreds of video streams in Gb Eth & IP environment

Video over IP monitoring, new protocols & devices

CRM costs Lower MTBF, call center calls and “truck rolls”

Automatic maintenance, Automatic firmware update

Home network monitoring


User quality measurements•What does subscribers want?

▫Content Content quality Content pricing Content availability

▫Viewing experience Video Quality Channel Zapping delay

▫Integration of services Usability One bill New services

▫Customer services

Technical Factors


Voice/Video Quality?

We can look at quality in different ways:

1. Perceptual subjective Quality (MOS)2. Numerical Quality parameters (PSNR)3. How to connect (relation matrix

between 1 and 2)


MEASUREMENT & MONITORINGIn the Lab & In The FieldsFor Pre-Deployment/monitoring/Failure Analysis


Measurement & Monitoring phases

AnalysisProblem Solving

Tuning

PreDeployment

Testing

LabTesting

Design

24/7Monitoring

DeploymentDeploymentPhasePhase

Pre-Pre-DeploymentDeploymentPhasePhase


Measurement & Monitoring phases

•Design & Lab testing▫Simulation and Emulation of the network▫Lab and testing tools

•Pre Deployment Stage▫Work on actual network▫Load testing▫Lab, testing, diagnosing and monitoring tools

•Deployment (production) Phase▫Mostly monitoring (probes) equipment,

management systems, data filtering and diagnostics equipment


MEASUREMENT TAXONOMYSubjectiveObjectivePayload based, codec aware, codec anaware


Measurement methods review

•Subjective▫Accurate▫Expensive, not for monitoring

•Objective▫Repeatable▫For both testing and monitoring


Multimedia monitoring methods

ObjectiveSubjective

MOS(Voice)

BT500(Video)

Codec independentPacket based

MDI

MonitoringTesting

BroadcastWorld

HSI and Data World

NetworkMonitoring

Delay, JitterPacket loss

Codec awarePacket based

VQI

V-Factor

VQSTelchemy


Objective methods


Payload Based Methods


Full Reference: Video Quality AssessmentITU-T J.144 and ITU-R BT.1683 •Full-reference perceptual models•Digital TV•Rec. 601 image resolution (PAL/NTSC)•Bit rates: 768 kbps ~ 5 Mbps•Compression errors


Voice Quality Assessment – with/out reference

•ITU-T P.862 (Feb 2001) - Full Reference▫Full-reference perceptual model (PESQ)▫Signal-based measurement▫Narrow-band telephony and speech codecs▫P.862.1 provides output mapping for

prediction on MOS scale•ITU-T P.563 (May 2004)

▫No-reference perceptual model▫Signal-based measurement ▫Narrow-band telephony applications


Voice Quality Assessment

• ITU-T P.862.2 (Nov 2005):▫Extension of ITU-T P.862 ▫Wide-band telephony and speech codecs (5

~7Khz)• ITU-T P.VTQ (on-going):

▫Targeted at VoIP applications▫Minimum performance framework for no-

reference packet-based measurement▫Models analyze packet statistics; speech

payload is assumed▫Uses P.862 as a measurement reference


Codec Aware Methods

Codec awarePacket based

VQI

V-Factor

VQSTelchemy


Packet – Codec Aware•Monitoring technique•Codec dependent•Incorporates network parameters data

with codec behavior data•Scales- could monitor thousands of

channels•Examples:

▫ VQS (Telchemy)▫ VQI(Brix)▫ V-Factor (QoSMetrics)

0

5

10

15

20

25

30

35

0 5 10 15 20

Packet Loss (%)

PS

NR

(d

B)

Problem area

Robust codec

“Raw”codec

The need a codec aware metrics


Packet – Codec aware

0

20

40

60

80

100

0 10 20 30 40 50Time

Pack

et

Loss

/D

isca

rd R

ate

Packet loss/discard typicallyoccurs in high density periods

Average can bemisleading

1

2

3

4

5

0 5 10 15 20

Time

Mea

n O

pin

ion

Sco

re

Impact of Burst ofPacket Loss

Base quality level depends on frame rate, codec type, bit rate

Poor quality during burst of loss/discards

5-8 seconds 15-30

seconds

Subjective compensation for variance between human and testing equipment view of loss


Example V-Factor

•Based on MPQM (Moving Picture Quality Metrics) – high quality video measurement standard

•V = f(QER, PLR, R)▫QER – relative video codec quality▫PLR – Packet loss ratio (based on actual

packet loss, jitter data and jitter buffer model)

▫R – Image complexity factor (2-3)•Adopted by Spirnet


Packet – Codec Independent

•Monitoring only•Codec independent•Based on network parameters data only•Scales - could monitor thousands of

channels•Examples:

▫MDI IneoQuest standardized by IETF


MONITORING LEVELS


Measurement Levels

•Transport Level•Service (transaction) Level•Media Quality Level

Add Your Text

Add Your Text

Add Your Text

Add Your Text

Content Quality

Video Stream Quality

Transport Quality

Transaction Quality

Author

Add refrence to capex and opex expense


Transport Level

•Example: ▫Packet Loss▫Loss Patterns▫Jitter▫Delay

•Well understood•Defined by ITU and IETF


Transaction Level•Examples

▫Post-dial delay in PSTN/mobile networks▫Video start time

•Requires understanding in both network monitoring and signaling (IGMP, SIP) and in media coding (analysis of the media to discover dial tone or I frame)


Channel zapping delay

•Multicast saves bandwidth but creates signaling delays:▫Multicast Leave +▫Multicast Join + ▫First I Frame +▫Up to 2 seconds buffering time

Leave latency

Join latency

Signaling Latency

First I Frame

Media Latency

Total Channel zapping Latency

Buffering latency

First frame viewed


Microsoft improved channel zapping

•Unicast server is used to serve the stream till the second multicast stream has an “I Frame”▫Immediate I Frame▫Shorter buffering time

Leave latency

Join latency

Multicast stream

Total Channel zapping Latency

I Frame unicast

unicast stream

I Frame Multicast

Shorter buffering time

First frame viewed


VoD Signaling

Play to server

Total VoD Signaling Latency

Buffering latency

First frame viewed

•Unicast server is used to serve the stream till the second multicast stream has an “I Frame”▫Immediate I Frame▫Shorter buffering time


Content Level• Content quality is a payload based measurement.

▫ Requires decoding of the video stream▫ Understanding of the buffering and error concealment

algorithms of the decoder • CPU intensive – Does not scale• Accurate• Used mostly is Lab equipment and diagnostic

equipment• Examples:

▫ PSNR ▫ ITU-T J.144

• Usually requires the reference (original) stream• Tests:

▫ Source artifacts▫ Source quality


Standardization landscape

DSL

DSL Forum

TR-64, TR-69LAN and WAN monitoring standards

ITU

Study Group 12

Algorithms for end-to-end transmission performance

ITU

VQEG – Video Quality Expert Group

Video performance measurement based on Subjective tests Database

Used in Diagnostics / Lab

ATIS

IIF – IPTV Interoperability Forum

QoS Metrics Standardization

Used for Monitoring


Example: ATIS IIF Quality Metrics

VSTQ - Video Service Transmission Quality ▫ Transmission Quality - codec/ content independent▫ Based on the rate and distribution of effective packet loss and

discardVSPQ - Video Service Picture Quality

▫ Estimated viewing quality▫ Considers the impact of VSTQ, video codec type and rate,

resolutionVSAQ - Video Service Audio Quality

▫ audio listening quality▫ Considers the impact of VSTQ, audio codec type, sample rate, …..

VSMQ - Video Service Multimedia Quality ▫ overall user experience▫ Combined effect of VSPQ, VSAQ, audio-video synchronization..

VSCQ - Video Service Control (Plane) Quality ▫ Considers responsiveness and reliability of control plane (trick

play)


Monitoring levels• J.144 and PSNR examines the video content only (payload

measurements)• TR101290 examines only transport stream data and

coherence without examining the video content• V -Factor and VQS looks at packet loss, jitter and loss patterns

data and incorporate it with codec information and video header information

• MDI – Examines only packet loss and packet loss patterns without considering the codec or video information

TR101290MPEG2TS Headers

V-Factor, VQSMDI

J.144, PSNRVideo payload


MEASUREMENT DOMAINSHeadendNetworkVoDHome


DVB TR 101290

•DVB Transport stream monitoring standard•Alerts and KPI:

▫TS sync loss▫Sync byte error▫PAT error▫Continuity counter errors▫PMT error▫PID error▫Transport error

▫CRC error▫PCR error▫PCR

discontinuity▫PCR

accuracy error

▫PTS error


Network measurement points

• Head-end▫ VoD Servers▫ Encoders▫ TS muxers

• Network core▫ Routers

• Network edge▫ ATM switches▫ GBE Switches

• Access▫ DSLAM / BRAS

• CPE▫ STBs▫ Home network

Source: Brix-Video Quality Measurement Algorithms WP

• Network Probe interface GBE, OC3/12, STM1/4

• Raw Video input MPEG2 TS - DVB ASI,

DVBoIP Digital and analog video


Headend

•Video source quality•Video encoding quality•MPEG2 Transport stream parameters

▫TR100290 1st-3rd level priority alerts

•Inputs▫Raw analog video in a variety of forms

(SD/HD)▫DVB-ASI▫DVBoIP▫DVBoATM (AAL1, AAL5)


VoD

•Pre-deployment▫RTSP signaling latency (delay)▫Performance under stress (user load)▫Trick Mode (FF, REW)

Signaling delay Video quality Buffer state (overflow and underflow)

• Monitoring▫Loss of data (under stress)

•Interfaces:▫GBE▫10GBE


Home network monitoring challenges•Scalability – Millions of STBs

▫Cost – Monitoring hardware and/or software costs must not add to STB costs (multiply by number of STBs)

▫Data – Large amounts of data which needs RT processing, filtering, logging and analysis

•Unclear network structure•Non “standard” equipment•Installation, support and debugging

problems


Home network monitoring roots

•Home network monitoring is currently a minor feature in IPTV “Support Automation”

•Support Automation handles:▫Installation▫Preventive maintenance▫STB and CPE remote management (TR69 and

more)▫Service verification for field technician▫CRM support▫Interaction logging▫Home Network monitoring and repair


CPE network management•2Wire solution -Three data views

OSS/BSS Customer care Data Analysis

•Other companies in this field▫SupportSoft▫Enure ▫Motive


Low level network monitoring


Vendors specialization

Lab Testing Monitoring

Incumbent Telecom

Agilent

Ixia

Spirnet

Cable

Rhode & Schwartz

Telecom

Agilent

JDSU/ Acterna

Spirnet

Cable

Thomson

Snell&Wilcox

New entrants

Telecom Shenick

Telecom Telchemy

Bridge Technologies

iBrixQoSMetrix


DSP-IP Contact informationFor Course materials & lecture request contact: For Course materials & lecture request contact:

For projects development services contact: For projects development services contact:

Adi YakovAdi YakovTraining and Courses ManagerTraining and Courses [email protected]+972-9-8651933+972-9-8651933

www.dsp-ip.com Mail : Mail : [email protected] Phone: Phone: +972-9-8850956, +972-9-8850956, Fax : +972-50- 8962910Fax : +972-50- 8962910

Alona AshkenaziAlona AshkenaziTechnology Management Technology Management Services Services : : [email protected]+972-9-8850956+972-9-8850956

iptv qoe monitoring

Technology

measurement reference

codec packet loss

ip monitoring

monitoring goals lab

codec anawarefast

iptv qos monitoring

reference itut

qoe measurement