aims’99 workshop heidelberg, 11-12 may 1999 assessing audio visual quality p905 - aquavit...

AIMS’99 Workshop

Heidelberg, 11-12 May 1999

Assessing Audio Visual Quality

P905 - AQUAVIT

Assessment of Quality for audio-visual signals over Internet and UMTS

Geoff Morrison, BT Labs

AIMS’99 Workshop


Contents

• Why we are doing this

• What we want to achieve

• How we are going about it

• Who is doing it– BT, CSELT, Berkom

• When are we doing it– February 1999 to December 2000

• The Results

AIMS’99 Workshop


• Coding algorithms, such as MPEG-4 and H.263, can compress audio and video signals to bit rates<1 Mbit/s as available over mobile and IP networks.

• The impact of impairments, from compression and transmission, on users' opinion must be understood.

• Audiovisual communication must be evaluated under representative conditions and real tasks, such as conversational services and AV database retrieval.

• Network performance must be monitored to guarantee a minimum QoS. Need single-ended models to check audio and video transmission quality in real time.

Rationale

AIMS’99 Workshop


Is the resulting application usable for purpose?

How best to assign a given total bit rate between video and audio?

Compression Impairments• Digital TV in studios is 216 Mbit/s

• Lossless compression typically only halves the bit rate

• To reach very low rates requires lossy audio and video compression, introducing significant distortions

AIMS’99 Workshop


Transmission Impairments • AV bits <> data

• Audiovisual content has a “timeliness” aspect– Critical in conversational applications– Important in database retrieval applications

• Error detection and retransmission protocols not ideal for audiovisual content

– variable delay– variable throughput

• Compressed AV is very sensitive to transmission errors, packet loss

– more compression more sensitivity

AIMS’99 Workshop


Error Handling• Error resilience tools for encoder

– structure compressed data so that damage is localised– optimise encoder options for the network performance

• Forward Error Correction– improve apparent network performance seen by decoder

• Error concealment at decoder– reconstruct approximations for damaged parts of pictures

and sound

• Combinations of above

How well do these work in practice?

AIMS’99 Workshop


Project Objectives• Understand the potential of audio, video and AV

communication at bit rates < 1Mbit/s for UMTS and IP.

• Develop tools and techniques for simulation of complete audiovisual transmission systems.

• Develop quality evaluation methods for complete audiovisual transmission systems.

• Investigate the relationship between system parameters (network and terminal) and subjective quality.

• Provide guidelines for the system parameters to be used for new applications.

• Encourage the use of audiovisual communication on these dominant networks of the future

AIMS’99 Workshop


Work Items

• Define the tests to be performed

• Develop/implement a UMTS test bed

• Develop/implement an IP test-bed

• Conduct subjective tests

• Make objective measurements and relate them to subjective results

• Disseminate the findings

AIMS’99 Workshop


Definition of tests

• Literature survey for available test methods

• Select appropriate methods

AIMS’99 Workshop


UMTS Test Bed• Video codec (MPEG-4 and/or H.263)

• Audio codec (MPEG AAC, scalable and non-scalable)

• Error resilience tools (MPEG-4)

• MPEG-4 multiplex

• Transmission errors - simulated and perhaps actual

• Error recovery

• Demux

• Optimisation of parameters

• Tools to make objective measurements

• Production of material for subjective and objective tests

AIMS’99 Workshop


UMTS Test Bed

• Baseline non real-time test bed– MPEG-4 baseline video codec, AAC and G.723.1 audio, AL

layer and MPEG DMIF

• Enhanced non real-time test bed – Error resilience tools, layered audio

AIMS’99 Workshop


UMTS Test Bed

Server Application Audio decoder Video decoder

Network interface Network interface

Network

Server Client

AIMS’99 Workshop


IP Test Bed• Video codecs (H.263 and/or MPEG-4)

• Audio codecs (G.723.1, MPEG AAC scalable and n/s)

• Error resilience tools (ITU-T, MPEG, IETF)

• Packetisation

• Transmission errors - simulated, measured profiles and actual

• Error recovery

• Optimisation of parameters

• Tools to make objective data measurements

• Production of material for subjective and objective tests

AIMS’99 Workshop


IP Test Bed

• Baseline non real-time test bed– G.723.1, H.263, simplified RTP, simple network impairment

simulator

• Enhanced non real-time test bed– more codecs including MPEG, error resilience, complete

RTP, error concealment, enhanced network simulator

• Enhanced real-time test bed– real time image capture, encoding and decoding

AIMS’99 Workshop


IP Test Bed

Encoder Application

Audio Encoder

Server Application

Video Encoder

Network Interface

Network

Decoder Application

Audio Decoder Video Decoder

Network Interface

Encoder Decoder

AIMS’99 Workshop


Subjective tests

• “Conventional” subjective tests covering a suitably wide range of variables

– the codecs– bit rate, video frame rate, audio bandwidth– Error conditions (packet loss, delay variability, …)– Protocols (CBR, VBR, ABR, RSVP, ..)– Application (one-way, conversational)

• Users may accept lower quality in return for mobility. Testing will seek an estimation of the threshold of acceptability of audio and video quality in mobile applications

AIMS’99 Workshop


Objective measurements

• Objective assessment of audio quality– TOSQA, PAMS, ...

• Objective assessment of video quality– model of human vision– other algorithms to objectively estimate user’s opinion

• Compare subjective and objective assessments

• Investigate relationship between parameter values and subjective audio, video and AV quality assessments

AIMS’99 Workshop


Double Ended Model

• Measuring instrument needs both original input to encoder and delivered version at decoder output

SystemunderTest

Meter

AIMS’99 Workshop


Single Ended Model

• Measuring instrument needs only decoder output– more suitable for live performance monitoring

SystemunderTest

Meter

AIMS’99 Workshop


Project Resources

20 man-monthsProjectManagementDissemination ofResultsDefinition ofTestsUMTS Testbed

IP Testbed

Subjective Tests

Objective Tests

AIMS’99 Workshop


Results• Reports due in Q4 of 2000.

• The findings should be of benefit to at least 3 groups:– Network Providers. Results such as minimum desirable bit

rates and error characteristics should influence the planning and operation of IP and UMTS networks intended to carry audiovisual services.

– Application Providers. The results will provide reliable guidance about which applications and operating parameters will satisfactorily meet customers' expectations and those which will not.

– Researchers in the field of "human factors". It is expected that the project will advance the state of the art in perceptual modelling and in the understanding of cross-modal effects.

aims’99 workshop heidelberg, 11-12 may 1999 assessing audio visual quality p905 - aquavit...

Documents