media extensions

Media extensionsMarcin Stankiewicz and Matthew HowardDevelopment - Multimedia Platform TeamRichard FricksWindows Scenario Adoption TeamSession 4-104

Engaging with rich media—whether watching a movie, video chatting, or playing music—is one of the most prevalent and enjoyable things we do on our PCs today.

Drag picture to placeholder or click icon to add

Architecture overviewMedia Foundation pipelineMedia extensionsVisual Studio 2012Media source demoStreaming media to JavaScript

Agenda

Media Application Architecture

Media Foundation

Media enginePlayback and preview

Windows Store app

Capture engineCapture

Sharing enginePlayTo

Transcode

Windows Runtime (WinRT)

C#MediaElement

JavaScript – HTML5Video tag, audio tag

Media namespace• Capture• Transcode• PlayTo• PlaybackManager• MediaExtensionMana

Media source

Video decoder

Video effect

Video encoder Video sink

Audio decoder

Audio effect

Audio encoder Audio sink

Media Foundation pipeline

Media app architecture

Media enginePlayback and preview

Capture engineCapture

Sharing enginePlay To

Transcode

Media Foundation

Media source

Decoder Effect SinkEncoder

Media app architecture

Media source

Video decoder

Video effect

Audio decoder

Audio effect

Media Foundation pipelineMedia source

Decoder Effect SinkEncoder

http://msdn.microsoft.com/en-us/library/windows/apps/hh452756.aspx

• Can be either native C++ DLL or a native C++/CX DLL• The DLL must ship in your app package• You can call WinRT APIs but you can use ONLY those desktop APIs that are

marked for Windows Store apps:

Registration processMedia source

Video decoder

Video effect

Audio decoder

Audio effect

Windows.Media.MediaExtensionManager

.RegisterAudioDecoder()

.RegisterVideoDecoder()

.RegisterAudioEncoder()

.RegisterVideoEncoder()

.RegisterSchemeHandler()

.RegisterByteStreamHandler()?

Media source

Video decoder

Video effect

Audio decoder

Audio effect

Effects are registered using the class your code uses to process the media:MediaElement AddAudioEffect() VideoTag msInsertAudioEffect()

AddVideoEffect() msInsertVideoEffect()

MediaTranscoder AddAudioEffect() AudioTag msInsertAudioEffect()

AddVideoEffect()

MediaCapture AddEffectAsync()

Registration process: Effects

Media source

Video decoder

Video effect

Audio decoder

Audio effect

Sinks are supported only by MediaCapture:MediaCapture StartRecordToCustomSinkAsync()

StartPreviewToCustomSinkAsync()

Registration process: Sinks

Media source

Video decoder

Video effect

Audio decoder

Audio effect

Windows.Media.MediaExtensionManager

.RegisterByteStreamHandler()?

.RegisterByteStreamHandler()

Registration Process: Media source

ByteStreamHandler• New container format (WebM, MKV)

SchemeHandler• Generating your own streams• Media stored in a location not

accessible to inbox ByteStream• Custom streaming protocol

Which to use?

IMFByteStreamHandlerInterface

Method DescriptionBeginCreateObject Begins an asynchronous request to create a media source from

a byte stream.CancelObjectCreation Cancels the current request to create a media source.

EndCreateObjectCompletes an asynchronous request to create an object from a URL. Completes an asynchronous request to create a media source.

GetMaxNumberOfBytesRequiredForResolution

Retrieves the maximum number of bytes needed to create the media source or determine that the byte stream handler cannot parse this stream.

HRESULT BeginCreateObject( [in] IMFByteStream *pByteStream,[in] LPCWSTR pwszURL,[in] DWORD dwFlags,[in] IPropertyStore *pProps,[out] IUnknown **ppIUnknownCancelCookie,[in] IMFAsyncCallback *pCallback,[in] IUnknown *punkState );

MediaElement.Source = new Uri(“video.mv4");

HRESULT EndCreateObject( [in] IMFAsyncResult *pResult,[out] MF_OBJECT_TYPE *pObjectType,[out] IUnknown **ppObject );

typedef enum MF_OBJECT_TYPE { MF_OBJECT_MEDIASOURCE,MF_OBJECT_BYTESTREAM,MF_OBJECT_INVALID } MF_OBJECT_TYPE;

SchemeHandlers and Byte-StreamHandlers

Byte-StreamHandler

Media source

Video decoder

Video effect

Audio decoder

Audio effect

IMFSchemeHandlerInterface

Method DescriptionBeginCreateObject Begins an asynchronous request to create an object from a

URL.CancelObjectCreation Cancels the current request to create an object from a URL.

EndCreateObject Completes an asynchronous request to create an object from a URL.

HRESULT BeginCreateObject( [in] LPCWSTR pwszURL,[in] DWORD dwFlags,[in] IPropertyStore *pProps,[out] IUnknown

**ppIUnknownCancelCookie,[in] IMFAsyncCallback *pCallback,[in] IUnknown *punkState );

MediaElement.Source = new Uri("myscheme://square");

HRESULT EndCreateObject( [in] IMFAsyncResult *pResult,[out] MF_OBJECT_TYPE *pObjectType,[out] IUnknown **ppObject );

typedef enum MF_OBJECT_TYPE { MF_OBJECT_MEDIASOURCE,MF_OBJECT_BYTESTREAM,MF_OBJECT_INVALID } MF_OBJECT_TYPE;

SchemeHandlers and Byte-StreamHandlers

Byte-StreamHandler

Media source

Video decoder

Video effect

Audio decoder

Audio effect

SchemeHandlers and Byte-StreamHandlersSchemeHandler

Byte-StreamHandler

Media source

Video decoder

Video effect

Audio decoder

Audio effect

ByteStream

What is a ByteStream?

Represents a byte stream from some data source:• Local file• Network file• Some other source

This interface supports typical stream operations such as reading, writing, and seeking.

IMFByteStream Interface

Method GetCurrentPositionBeginRead GetLengthBeginWrite IsEndOfStreamClose ReadEndRead SeekEndWrite SetCurrentPositionFlush SetLengthGetCapabilities Write

Byte-StreamHandler

Media source

Video decoder

Video effect

Audio decoder

Audio effect

ByteStream

Byte-StreamHandler

Media source

Video decoder

Video effect

Audio decoder

Audio effect

ByteStream

SourceResolver

Media source

Video decoder

Video effect

Audio decoder

Audio effect

SchemeHandler

Byte-StreamHandlerByteStream

SourceResolver

Media source

Video decoder

Video effect

Audio decoder

Audio effect

SchemeHandler

SourceResolver

Media source

Video decoder

Video effect

Audio decoder

Audio effect

SchemeHandler

SourceResolver

Marcin Stankiewicz Development - Multimedia Platform Team

Demo: Building a media extension• How to build an extension using C++/CX• The proper use of WRL to implement a

SchemeHandler• Instantiating a media source from a SchemeHandler • Using the PropertySet class to implement two-way

communication between the app and the extension• PropertySet change notification to support dynamic

updates• Direct3D rendering

Recap of Demo• How to build an extension using C++/CX• The proper use of WRL to implement a

SchemeHandler• Instantiating a media source from a

SchemeHandler • Using the PropertySet class to implement two-

way communication between the app and the extension

• PropertySet change notification to support dynamic updates

• Direct3D rendering

Media source

Video decoder

Video effect

Audio decoder

Audio effect

SchemeHandler

SourceResolver

Matthew HowardDevelopment - Multimedia Platform Team

Demo: Streaming media to an app• How to implement a sink media extension for

audio• Capturing uncompressed audio• Processing audio buffers in a JavaScript app• Writing audio to a wave file• Asynchronous processing techniques

Recap of Demo• How to implement a sink media extension for

audio• Capturing uncompressed audio• Processing audio buffers in a JavaScript app• Writing audio to a wave file• Asynchronous processing techniques

Need more information?

Come to our question-and-answer session!3-106 Chalk Talk: Media appsThursday 4:15pm, B33 Hood

Media-related sessions3-117 10/31/2012 17:15300 - Advanced B33 Hood Key technologies for building advanced media apps3-122 11/1/2012 14:30 300 - Advanced B33 Hood Media app case studies3-106 11/1/2012 16:15 300 - Advanced B33 Hood Chalk Talk: Media apps

Resources

Windows 8 media overviewhttp://blogs.msdn.com/b/b8/archive/2012/06/08/building-a-rich-and-extensible-media-platform.aspx

Media extensions SDK samplehttp://code.msdn.microsoft.com/windowsapps/Media-extensions-sample-7b466096

Windows 8 multimedia desktop APIs available to Windows Store appshttp://msdn.microsoft.com/en-us/library/windows/apps/hh452756.aspx

SchemeHandlers and ByteStreamHandlershttp://msdn.microsoft.com/en-us/library/windows/desktop/aa371872(v=vs.85).aspx

Writing a Custom Media Sourcehttp://msdn.microsoft.com/en-us/library/windows/desktop/ms700134(v=vs.85).aspx

• Develop: http://msdn.microsoft.com/en-US/windows/apps/br229512

• Design: http://design.windows.com/

• Samples: http://code.msdn.microsoft.com/windowsapps/Windows-8-Modern-Style-App-Samples

• Videos: http://channel9.msdn.com/Windows

Resources

Please submit session evals by using the Build Windows 8 appor at http://aka.ms/BuildSessions

© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

media extensions

Documents

ngi summer project streaming media extensions to open mash...

extensions of mendelian genetics extensions of mendelian...

preview of “fanfare cover” - mit media...

title: different types of file extensions, for example:...

extensions substation extensions for high-voltage ... ·...

sundancetv & storycentrallabs digital extensions story lab @...

esy 2064430 1670983466 - eyelash extensions › media ›...

mcluhan marshall understanding media the extensions of man

fme extensions - magento...fme extensions media gallery &...

understanding media: the extensions of man

amazing ppc tactics ppc tactics 062811.pdf · types of ad...

unesco, -...

hair extensions - before & after -...

top 10 google chrome extensions for your social media...

cilk plus: multicore extensions for c and...

multi-media extensions in super-pipelined micro ... ·...

extensions & frameworks geospatial extensions for esri

media playerremove mediaplayersvideos browser extensions...

all concepts are kan extensions: kan extensions as the ......

windows protocols errata...release: june 30, 2015...