Camera, Media and Audio in Windows Phone 8.1

Click to place personal photo

Click to place personal photo

Click to place personal photo

2

In this module…

Opening, editing and saving photosLoading or taking photo with the File Open Picker, saving with the File Save PickerEditing with the Nokia Imaging SDK

Capturing and Screen RecordingBasic capturing of photos and videosAdvanced capturing with the new VariablePhotoSequenceCaptureScreen recording with the new ScreenCapture API

Video EditingUsing the new MediaComposition API

Background AudioUsing the new MediaPlayer API

Photo Editing

4

Getting a photo

Where’s the CameraCaptureUI?Windows CameraCaptureUI is not available on PhoneWindows Phone 8 PhotoChooserTask and CameraCaptureTask still work for Silverlight 8.1, but not for Windows Phone XAML-based apps

FileOpenPicker is the new PhotoChooserTaskPicking single or multiple media files, not only photosTaking photo or video with Windows Phone’s built-in camera UICovers most scenarios of the CameraCaptureUI as wellAvailable for Silverlight 8.1 and Windows Phone XAML

For details, see: Building Apps for Windows Phone 8.1 Jump Start: (13) Sharing

5

Getting or storing media files

Full media library I/O access!Read and write access of photos, videos, etc. with UI via FileOpenPicker and FileSavePicker

Read and write access of photos, videos, etc. without UI via KnownFolders.VideosLibrary, KnownFolders.PicturesLibrary, KnownFolders… Including transparent access to data stored on SD card

Capability declaration needed in Package.appxmanifest: <Capability Name="videosLibrary" />...

Available for Silverlight 8.1 and WinRT apps

6

Editing photos

Nokia Imaging SDKEasy to use imaging SDK from Nokia for free http://bit.ly/ImageSdk

Lots of features and samples• 50+ filters and effects• Crop, resize, rotate• Very fast and efficient • Real-time manipulation• Available for Silverlight 8.1, WinRT Phone and Windows apps

WriteableBitmapExOpen source library with helpful extension methods http://bit.ly/wbmpex Available for Silverlight 8.1, WinRT Phone, Windows apps, …

Editing a photo with the Nokia Imaging SDK// Create NOK Imaging SDK effects pipeline and run itvar imageStream = new BitmapImageSource(image.AsBitmap());using (var effect = new FilterEffect(imageStream)){     // Define the filters list var filter = new AntiqueFilter();    effect.Filters = new[] { filter };

    // Render the filtered image to a WriteableBitmap.    var renderer = new WriteableBitmapRenderer(effect, editedBitmap);    editedBitmap = await renderer.RenderAsync();    editedBitmap.Invalidate();

    // Show image in Xaml Image control    Image.Source = editedBitmap;}

8

Demo

PhotoEditing

Capturing and Screen Recording

10

Basic Capturing

Windows.Media.CaptureWindows Phone 8.0 provided Windows.Phone.Media.Capture with PhotoCaptureDevice and AudioVideoCaptureDevice for photo, audio and video recording with real-time preview

Windows Phone 8.1 MediaCapture from Windows.Media.Capture API replaces the Windows.Phone.Media.Capture APISupports low lag photos (burst mode)

Mostly common API for Windows and Phone apps (WinRT and Silverlight 8.1) with a couple of phone-only additions

Rene SchulteFace Lens

11

Basic Capturing

Showing the camera stream in your Xaml UI• Using the CaptureElement in Windows XAML-based apps• Using the VideoBrush in Silverlight 8.1 apps• MediaCapturePreviewSink for camera preview stream from MediaCapture API• using Microsoft.Devices

for new VideoBrush.SetSource extension to connect MediaCapturePreviewSink with VideoBrush

Real-time parameter controlAudio/VideoDeviceController to change parameters like focus, zoom, exposure, RegionOfInterest. …

Required capabilities <Capability Name=“webcam" /><Capability Name=“microphone" />

12

Implementing a Capturing Solution

Initializing MediaCapture

// Create MediaCapture and initmediaCapture = new MediaCapture();await mediaCapture.InitializeAsync();

// Assign to Xaml CaptureElement.Source and start preview

PreviewElement.Source = mediaCapture;await mediaCapture.StartPreviewAsync();

Windows and Windows Phone XAML Windows Phone Silverlight 8.1// Create MediaCapture and initmediaCapture = new MediaCapture();await mediaCapture.InitializeAsync();

// Interop between new MediaCapture API and SL 8.1 to show// the preview in SL XAML as Rectangle.Fillvar previewSink = new MediaCapturePreviewSink();var videoBrush = new VideoBrush();videoBrush.SetSource(previewSink);PreviewElement.Fill = videoBrush; 

// Find the supported preview sizevar vdc = mediaCapture.VideoDeviceController;var availableMediaStreamProperties = vdc.GetAvailableMediaStreamProperties(MediaStreamType.VideoPreview);// More LINQ selection happens here in the demo to find closest formatvar previewFormat = availableMediaStreamProperties.FirstOrDefault();

// Start Preview streamawait  vdc.SetMediaStreamPropertiesAsync(MediaStreamType.VideoPreview, previewFormat);await mediaCapture.StartPreviewToCustomSinkAsync( new MediaEncodingProfile { Video = previewFormat }, previewSink);

Capturing a photo

public async Task CapturePhoto(){    // Create photo encoding properties as JPEG and set the size that should be used for capturing    var imgEncodingProperties = ImageEncodingProperties.CreateJpeg();    imgEncodingProperties.Width = 640;    imgEncodingProperties.Height = 480;

    // Create new unique file in the pictures library and capture photo into it    var photoStorageFile = await KnownFolders.PicturesLibrary.CreateFileAsync("photo.jpg",  CreationCollisionOption.GenerateUniqueName);    await mediaCapture.CapturePhotoToStorageFileAsync(imgEncodingProperties, photoStorageFile);}

Supporting hardware camera buttons

HardwareButtons.CameraHalfPressed += HardwareButtonsOnCameraHalfPressed;HardwareButtons.CameraPressed += HardwareButtonsOnCameraPressed;

private async void HardwareButtonsOnCameraHalfPressed(object sender, CameraEventArgs cameraEventArgs){    await mediaCapture.VideoDeviceController.FocusControl.FocusAsync();}

private async void HardwareButtonsOnCameraPressed(object sender, CameraEventArgs cameraEventArgs){    await CapturePhoto();}

Recording and playing a videopublic async Task StartVideoRecording(){    // Create video encoding profile as MP4     var videoEncodingProperties = MediaEncodingProfile.CreateMp4(VideoEncodingQuality.Vga);                // Create new unique file in the videos library and record video!     var videoStorageFile = await KnownFolders.VideosLibrary.CreateFileAsync("video.mp4",  CreationCollisionOption.GenerateUniqueName);    await mediaCapture.StartRecordToStorageFileAsync(videoEncodingProperties, videoStorageFile);}

public async Task StopVideoRecording(){    await mediaCapture.StopRecordAsync();

// Start playback in MediaElement var videoFileStream = await videoFile.OpenReadAsync(); PlaybackElement.SetSource(videoFileStream, videoFile.ContentType);}

Changing camera parameters

var zoomControl = mediaCapture.VideoDeviceController.Zoom;if (zoomControl != null && zoomControl.Capabilities.Supported){    SliderZoom.IsEnabled = true;    SliderZoom.Maximum = zoomControl.Capabilities.Max;    SliderZoom.Minimum = zoomControl.Capabilities.Min;    SliderZoom.StepFrequency = zoomControl.Capabilities.Step;    SliderZoom currentValue;    if (zoomControl.TryGetValue(out currentValue))    {        SliderZoom.Value = currentValue;    }    SliderZoom.ValueChanged += (s, e) => zoomControl.TrySetValue(SliderZoom.Value);}else{    SliderZoom.IsEnabled = false;}

18

Demo

BasicCapturing

19

Advanced Capturing

Phone is the better point ‘n’ shootWindows.Media.Capture.Core ships some phone-only APIs Available for Silverlight 8.1 and WinRT-based phone apps

VariablePhotoSequenceCaptureBurst shot with variable parameters for each photo Enables HDR and more cool scenarios without tripod

Just add the desired parameters for each photo via the VariablePhotoSequenceController.DesiredFrameControllers thenvpc = await PrepareVariablePhotoSequenceCaptureAsync()

and finally start sequence capturingvpc.StartAsync()

25

Screen Capturing

Another gemWindows.Media.Capture also provides a new phone-exclusive API to record the current screen

ScreenCaptureJust route the current screen as input for the MediaCaptureAutomatically asks for user approval upon first usage Personal information like notifications or DRM content is left outSuper easy to use

Useful for bug reporting Or recording progress in gamesOr Replay functionalityOr …

Initializing and running ScreenCapture

// Wire up current screen as input for the MediaCapturevar screenCapture = ScreenCapture.GetForCurrentView();mediaCapture = new MediaCapture();await mediaCapture.InitializeAsync(new MediaCaptureInitializationSettings{    VideoSource = screenCapture.VideoSource,    AudioSource = screenCapture.AudioSource,});

// Create video encoding profile as MP4 var videoEncodingProperties = MediaEncodingProfile.CreateMp4(VideoEncodingQuality.Vga);

// Create new unique file in the videos library and record video! var videoStorageFile = await KnownFolders.VideosLibrary.CreateFileAsync("screenrecording.mp4",  CreationCollisionOption.GenerateUniqueName);await mediaCapture.StartRecordToStorageFileAsync(videoEncodingProperties, videoStorageFile);

27

Demo

ScreenRecording

Video Editing

29

Video Editing

Windows.Media.EditingAnother new phone-only API for easy video editingEnables simple scenarios like video trimming or even complex timeline editing applicationsAvailable for Silverlight 8.1 and WinRT-based phone apps

MediaCompositionCombination of source audio and video MediaClips MediaClip can be trimmed by begin, end and volume changedAlso supports Media Foundation Transforms effectsAdditional background music can be addedPlayback of the composition result via MediaElementRendering of the composed video result to a StorageFileBuilt-In persistence support (load and save MediaComposition)

30

Implementing a simple video editor

Initializing MediaComposition and adding ClipsmediaComposition = new MediaComposition();foreach (StorageFile videoFile in videoFiles){    // Create Clip and add to composition    var clip = await MediaClip.CreateFromFileAsync(videoFile);    mediaComposition.Clips.Add(clip);

    // Create thumbnail to show as placeholder in an UI ListView    var thumbnail = await videoFile.GetThumbnailAsync(ThumbnailMode.VideosView);    var image = new BitmapImage();    image.SetSource(thumbnail);

    // Add to a viewmodel used as ItemsSource for a ListView of clips    videoClips.Add(new VideoClip(clip, image, videoFile.Name));}

Trimming and adding background music

// Trimming. Skip 1 second from the beginning of the clip and 2 from the endvar clipVm = videoClips.SelectedClip;clipVm.Clip.TrimTimeFromStart = TimeSpan.FromSeconds(1);clipVm.Clip.TrimTimeFromEnd = clipVm.Clip.OriginalDuration.Subtract(TimeSpan.FromSeconds(2));

// Add MP3 as background which was deployed together with the AppX packagevar file = await Package.Current.InstalledLocation.GetFileAsync("mymusic.mp3");var audio = await BackgroundAudioTrack.CreateFromFileAsync(file);mediaComposition.BackgroundAudioTracks.Add(audio);

Playback and rendering to a file

// Play the composed result including all clips and the audio track in a Xaml MediaElementvar w = (int) MediaElement.ActualWidth;var h = (int) MediaElement.ActualHeight;MediaElement.SetMediaStreamSource(mediaComposition.GeneratePreviewMediaStreamSource(w, h)));

// Create output filevar vidLib = KnownFolders.VideosLibrary;var resultFile = await vidLib.CreateFileAsync("myComposition.mp4",  CreationCollisionOption.ReplaceExisting);// Encode new composition as MP4 to the filevar mediaEncodingProfile = MediaEncodingProfile.CreateMp4(VideoEncodingQuality.Vga);await mediaComposition.RenderToFileAsync(resultFile, MediaTrimmingPreference.Fast, mediaEncodingProfile);

34

Demo

VideoEditing

Background Audio

36

Background Audio Playback

Windows.Media.BackgroundPlaybackNew MediaPlayer class replaces the BackgroundAudioPlayerOnly available for WinRT-based phone apps, not for Silverlight 8.1

MediaPlayerUses two-process model with foreground UI and background playbackContinues playback via background process when app navigates awayCan also be used in foreground for playback even without MediaElement

Different than Windows 8.1 model:MediaElement.AudioCategory="BackgroundCapableMedia"

37

Background Audio ProcessDefined in a WinRT componentEntry point declared in AppxManifest

Custom BackgroundAudioTask class implementsIBackgroundTask and its Run method

Access to global instance through BackgroundMediaPlayer.Current

Inter-process communication via BackgroundMediaPlayer .SendMessageToBackground(…) .SendMessageToForeground(…)

38

Universal Volume Control

SystemMediaTransportControlsThe SystemMediaTransportControls (SMTC) API provides access to the device’s Universal Volume Control (UVC)

SMTC allows to define which buttons are enabled in the UVCSMTC DisplayUpdater provides a way to set the UVC text like title, etc.

ButtonPressed event of the SMTC needs to be synced to the MediaPlayer and vice versaEvent is triggered in background process as well

39

Implementing Background Audio

MediaPlayer Play, Pause and Stop

// Attach event handler to background player to update UIBackgroundMediaPlayer.Current.CurrentStateChanged += MediaPlayerStateChanged;

// Uri could also be ms-appx:/// for package-local tracksBackgroundMediaPlayer.Current.SetUriSource(new Uri("http://foo.bar/my.mp3")); // Starts play since MediaPlayer.AutoPlay=true by default

// Or trigger manually play when AutoPlay=falseBackgroundMediaPlayer.Current.Play();

// Pause.BackgroundMediaPlayer.Current.Pause();

// Stop. There's no MediaPlayer.Stop() methodBackgroundMediaPlayer.Current.Pause();BackgroundMediaPlayer.Current.Position = TimeSpan.FromSeconds(0);

Updating UI in MediaPlayer.CurrentStateChangedprivate async void MediaPlayerStateChanged(MediaPlayer sender, object args){    // This event is called from a background thread, so dispatch to UI     await Dispatcher.RunAsync(CoreDispatcherPriority.Normal, () =>    {        switch (BackgroundMediaPlayer.Current.CurrentState)        {            case MediaPlayerState.Playing:                AppBarBtnPause.IsEnabled = true;                // Pass params on to background process, so it can update the UVC text there                BackgroundMediaPlayer.SendMessageToBackground(new ValueSet                { {"Title", "My Super MP3"}, {"Artist", "Foo Bar"} });                break;            case MediaPlayerState.Paused:                AppBarBtnPause.IsEnabled = false;                break;        }    });}

BackgroundAudioTask wiring up SMTC and UVCpublic void Run(IBackgroundTaskInstance taskInstance){    // Initialize SMTC object to talk with UVC    // This is intended to run after app is paused, // therefore all the logic must be written to run in background process    systemmediatransportcontrol = SystemMediaTransportControls.GetForCurrentView();    systemmediatransportcontrol.ButtonPressed += SystemControlsButtonPressed;    systemmediatransportcontrol.IsEnabled = true;    systemmediatransportcontrol.IsPauseEnabled = true;    systemmediatransportcontrol.IsPlayEnabled = true;

    // Add handlers to update SMTC when MediaPlayer is used in foreground    BackgroundMediaPlayer.Current.CurrentStateChanged += BackgroundMediaPlayerCurrentStateChanged;    BackgroundMediaPlayer.MessageReceivedFromForeground += BackgroundMediaPlayerOnMessageReceived;}

BackgroundAudioTask updating UVC text

private void BackgroundMediaPlayerOnMessageReceived(object sender, MediaPlayerDataReceivedEventArgs e){    // Update the UVC text    systemmediatransportcontrol.DisplayUpdater.Type = MediaPlaybackType.Music;    systemmediatransportcontrol.DisplayUpdater.MusicProperties.Title = e.Data["Title"].ToString();    systemmediatransportcontrol.DisplayUpdater.MusicProperties.Artist = e.Data["Artist"].ToString();    systemmediatransportcontrol.DisplayUpdater.Update();}

44

Demo

BackgroundAudio

45

Summary

46

Adding it all up – Media I/O

Windows Phone 8 Windows Phone 8.1Silverlight

Windows Phone 8.1WinRT

Windows 8.1

Getting photos PhotoChooserTaskXNA MediaLibrary

PhotoChooserTaskXNA MediaLibraryFileOpenPickerKnownFolders

FileOpenPickerKnownFolders

CameraCaptureUI

FileOpenPickerKnownFolders

Storing photos XNA MediaLibrary XNA MediaLibraryFileSavePickerKnownFolders

FileSavePickerKnownFolders

FileSavePickerKnownFolders

Getting videos NoFileOpenPickerKnownFolders

FileOpenPickerKnownFolders

CameraCaptureUIFileOpenPickerKnownFolders

Storing videos No FileSavePickerKnownFolders

FileSavePickerKnownFolders

FileSavePickerKnownFolders

47

Adding it all up – Capturing

Windows Phone 8 Windows Phone 8.1Silverlight

Windows Phone 8.1WinRT

Windows 8.1

Windows.Phone.Media.Capture

Yes No No No

Windows.Media.Capture

No Yes Yes Yes

VariablePhoto-SequenceCapture

No Yes Yes No

ScreenCapture No Yes Yes No

48

Adding it all up – Media Editing

Windows Phone 8 Windows Phone 8.1Silverlight

Windows Phone 8.1WinRT

Windows 8.1

Windows.Media.Editing

No Yes Yes No

Windows.Media.Transcoding

No Yes Yes Yes

49

Adding it all up – Background Audio

Windows Phone 8 Windows Phone 8.1Silverlight

Windows Phone 8.1WinRT

Windows 8.1

Windows.Media.BackgroundPlayback

No No Yes No

Microsoft.Phone.BackgroundAudio

Yes No No No

MediaElement AudioCategory="BackgroundCapableMedia"

No No No Yes

©2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, Office, Azure, System Center, Dynamics and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.