the mpeg-7 standard - a brief tutorial - ali tabatabai sony us research laboratories february 27,...

The MPEG-7 Standard- A Brief Tutorial -

Ali Tabatabai

Sony US Research Laboratories

February 27, 2001

2

Outline

Objectives of the MPEG-7 Standard

Main Elements of MPEG-7

Scope of MPEG-7

MPEG-7 Application Areas

MPEG 7’s relation with other standards

3

Why do we need MPEG-7 ?

N e e d

• Content Management

• Fast & Accurate Access

• Personalized Content Production and Consumption

• Automation

+

Support for Advanced Query

•Visual•Audio•Sketch

4

MPEG: A Brief History (1)

MPEG: Moving Picture Experts Group

ISO / IEC/JTC1/SC29/WG11

A Working Group of ISO/IEC in charge of the Development of Standards for

Coded Representation of Digital Audio and Video

Established in 1988

5

MPEG: A Brief History (2)

MPEG-1: Interactive CD and MP3 11 / 1992

MPEG-2: DTV, STB, DVD 11 / 1994

MPEG-4: Web and Mobility ver1: 09 /1998

ver2: 11

/1999

MPEG-7: ??? 08 / 2001

MPEG-21: Multimedia Framework 11 / 2001

6

MPEG-7: What Is It ?

THE MPEG 7 THE MPEG 7 STANDARDSTANDARD THE MPEG 7 THE MPEG 7 STANDARDSTANDARD

Content Description of Various Audio Visual

Information

IS NOT a COMPRESSION Standardsimilar to MPEG-1/2/4 or their

Extension

IS NOT a STANDARD forFEATURE

EXTRACTION/MATCHING

Types of Audio Visual Information• Audio, Speech• Moving video, still pictures, graphics• Information on how objects are combined in scenes

Types of Audio Visual Information• Audio, Speech• Moving video, still pictures, graphics• Information on how objects are combined in scenes

MPEG-7: Application Areas Storage and retrieval of audiovisual databases (image, film,

radio archives) Broadcast media selection (radio, TV programs) Surveillance (traffic control, surface transportation, production

chains) E-commerce and Tele-shopping (searching for clothes /

patterns) Remote sensing (cartography, ecology, natural resources

management) Entertainment (searching for a game, for a karaoke) Cultural services (museums, art galleries) Journalism (searching for events, persons) Personalized news service on Internet (push media filtering) Intelligent multimedia presentations Educational applications Bio-medical applications

8

MPEG-7Description Scope for AV Content

Description Granularity Low-level

High-level

Form

Access

Classification

Link

Context

9

MPEG-7: Main Elements

Descriptors (D)

syntax and semantics of each feature representation

Description Schemes (DS)

structure and semantics of the relationships between

components

Description Definition Language (DDL)

creation of new DS’s

modification/extension of existing DS’s

10

MPEG-7: Major Functionalities

Systems (ISO / IEC 15938 -

1)

Description Definition Language (ISO / IEC 15938

- 2)

Visual (ISO / IEC 15938 -

3)

Audio (ISO / IEC 15938 -

4)

Multimedia Description Schemes (ISO / IEC 15938

- 5)

Reference Software (ISO / IEC 15938 -

6)

11

DS1

DS2 DS2

DS3 D1 D3 D2 Systems

0001100

<Object> <Label/> <Definition/>

. .

</Object>

DDL

Instantiation

MPEG-7: Main Elements (2)

12

MPEG-7: Systems

It defines tools to:

manage and protect intellectual property

synchronize between content and description

provide for efficient storage and transport

13

MPEG-7: DDL and its Components

Description Definition Language:

Creation of the Ds and DS’s: XML Schema & MPEG-7 Extensions

Instantiation of XML

XML Schema:

Data types

Simple and Complex types

Elements, attributes

Inheritance, Abstract types

MPEG-7 extensions:

Array and Matrix data type

14

MPEG-7: Audio

Sound Effects

Music Instrument Timbre

Spoken Content

Melody Contour

15

MPEG-7: Visual (1)

Color quantization, dominant, scalable, color-structure, layout,

GoF/GoP

Texture

Shape region-based, contour-based, 3D

Motion camera motion, motion trajectory, parametric motion,

motion activity

16

MPEG-7: Visual (2)

Localization spatio temporal

Others face recognition

17

MPEG-7: Basic Visual Structures

Grid Layout

2D-3D Multiple View

Time Series

Spatial 2D Coordinates

Temporal Interpolation

18

Low level Audio Visual descriptors

• Color • Camera motion• Motion activity• Mosaic

• Color • Motion trajectory• Parametric motion• Spatio-temporal

shape

• Color • Shape• Position• Texture

Video segments Still regions

Moving regions Audio segments

• Spoken content • Spectral

characterization• Music: timbre,

melody

19

MPEG-7: MMDS Basic Elements

Datatype &structures

Link & medialocalization Basic DSs

Basic elementsBasic elements

Schematools

Time, Duration, Medialocators Language

Annotation,Person, Place

Root, Top-level elements, Packages

20

MPEG 7: Content Management and Description

Content descriptionContent description

Content managementContent management

Creation &production

Media ContentUsage

Conceptualaspects

Structuralaspects

Title, Creator, Creation location & date, Purpose,

Classification, Genre, Review, Parental guidance,

etc. (Author generated)

Format, Coding, Instances, Identification, Transcoding

Hint, etc.(Several instances)

Rights holder, Access rights, Usage Record, Financial aspects,

etc. (Evolution)

Datatype &structures

Link & medialocalization Basic DSs

Basic elementsBasic elements

Schematools

Viewpoint of the structure: Segments• Spatial / temporal structure• Audio, video low-level Ds

• Elementary semantic information.

Viewpoint of conceptual notions• Events, objects, abstract concepts, and

their relation

21

Foreground

Example of Segment trees

SR1: Creation, Usage meta

information Media description Textual annotation Color histogram, Texture

SR2: Shape Color Histogram Textual annotation

SR6: Color Histogram Textual annotation

SR5: Shape Textual annotation

SR4: Shape Color Histogram Textual annotation

SR3: Shape Color Histogram Textual annotation

Background

22

Segment Tree

Shot1 Shot2 Shot3

Segment 1

Sub-segment 1

Sub-segment 2

Sub-segment 3

Sub-segment 4

segment 2

Segment 3

Segment 4

Segment 5

Segment 6

Segment 7

Semantic DS (Events)

• Introduction

• Summary

• Program logo

• Studio

• Overview

• News Presenter

• News Items

• International

• Clinton Case

• Pope in Cuba

• National

• Twins

• Sports

• Closing

TimeAxis

23

MPEG 7: Navigation and Access

Navigation &Navigation &AccessAccess

Summary

Variation

Content descriptionContent description

Content managementContent management

Creation &production

Media ContentUsage

Conceptualaspects

Structuralaspects

Datatype &structures

Link & medialocalization Basic DSs

Basic elementsBasic elements

Schematools

Efficient support of: discovery, browsing, navigation, visualization

Substitution of the original contentAdaptation to terminal, network, or

user preferences

24

MPEG 7: Hierarchical summary

A-VData

HierarchicalSummary

HighlightLevel HighlightLevel

HighlightSegment

HighlightSegment

HighlightSegment

HighlightSegment

HighlightSegment

HighlightSegment

HighlightSegment

25

MPEG 7: Sequential summary

SequentialSummary

A-VData

TextProperty

TextProperty

FrameProperty

FrameProperty

FrameProperty

SoundProperty

SoundProperty

SoundProperty

26

MPEG 7: Variation

VIDEO IMAGE TEXT AUDIO

Modality

Fidelity

Source

Variation

A

IH

GFE

DCB

Universal Multimedia Access Adapt delivery to network and terminal characteristics (QoS)

27

MPEG-7: Content Organization

Navigation &Navigation &AccessAccess

Summary

Variation

AnalyticModel

Content organizationContent organization

Content descriptionContent description

Content managementContent management

Creation &production

Media ContentUsage

Conceptualaspects

Structuralaspects

Description and organization of collection of documents

Collection &Classification

Datatype &structures

Link & medialocalization Basic DSs

Basic elementsBasic elements

Schematools

28

MPEG-7: Collection

29

MPEG 7: User Interaction

Navigation &Navigation &AccessAccess

Summary

Variation

AnalyticModel

Content organizationContent organization

Content descriptionContent description

Content managementContent management

Creation &production

Media ContentUsage

Conceptualaspects

Structuralaspects

Collection &Classification

UserUserInteractionInteraction

User preferences

Datatype &structures

Link & medialocalization Basic DSs

Basic elementsBasic elements

Schematools

User identification and preferences:Filtering, search and browsing

30

MPEG-7 Its Relation with other standards

AHG on “Metadata harmonization”:

SMPTE: Metadata dictionary, KLV encoding

Dublin Core Metadata Initiative

European Broadcast Union

AHG on TV AnyTime Application

Large number of Liaisons:

SMPTE

Dublin Core

W3C (XML Schema)

etc.

Competition:• Individual work• Definition scope and r

MPEG-7: TimeLine - The Work Plan

Call for proposalsCall for proposals

20012000199919981996

Working draftWorking draft Committee draftCommittee draft

Draft internationalstandard

Draft internationalstandard

Final committeedraft

Final committeedraft

InternationalstandardInternationalstandard

Divergence Convergence

32

Conclusions on AV Content Description and MPEG-7

MPEG-7:

AV content description for interoperable

application

Description Definition Language:

XML Schema (flexibility) + Binary version

(efficiency)

Description Schemes:

Library of description tools

Covers a wide range of generic needs