http://bobmccune.com
Composing & Editing Media with
AV Foundation
Bob McCuneAbout...
‣MN Developer and Instructor‣Owner of TapHarmonic, LLC.‣Founded Minnesota CocoaHeads in 2008
What will I learn?Agenda
‣ AV Foundation Overview‣ Media Playback‣ Media Editing‣ Composing Media‣ Mixing Audio‣ Building Video Transitions‣ Layering Visual Content
OverviewAV Foundation
‣ Apple’s advanced Objective-C framework for working with timed-media‣ High performance, asynchronous processing‣ Hardware accelerated handling of AV media
‣ Available in its current form since iOS 4‣ Signi!cant additions and enhancements iOS 5 and 6‣ Default media framework on Mac OS X since 10.7 Lion
‣ Apple’s focus for media apps on both iOS and Mac
iOS Media OptionsWhere does it !t?
MediaPlayer
UIKit
Simple
CoreAudio
CoreMedia
CoreVideo
CoreAnimation
Complex
AVFoundation
Awesome
AV Essentials
Understanding AssetsMedia Assets
‣ AVAsset is an abstract representation of media resource modeling the static aspects of the media.‣ Abstracts away the type and location
‣ AVAssetTrack models the static aspects of the individual media streams within an asset‣ Tracks are of a uniform type (video, audio, etc.)
AVAssetTrack (Video)
AVAssetTrack (Audio)
What can I do with an asset?Using AVAssets
‣ Inspect‣ Generate Images‣ Transcode and Export‣ Playback
Media Playback
Playback ControllerAVPlayer
‣ AVPlayer is a controller for managing playback‣ play‣ pause‣ rate‣ seekToTime:
‣ Use KVO to observe playback readiness and state‣ status
‣ Timed Observations‣ addPeriodicTimeObserverForInterval:queue:usingBlock‣ addBoundaryTimeObserverForInterval:queue:usingBlock
Static
Static vs Dynamic ModelsPlaying Media
‣ AV Foundation distinguishes between static and dynamic aspects of media
D ynamic
AVPlayerItemAVPlayerItemTrack
AVPlayerItemTrackAVPlayerItemTrack
AVAssetAVAsset
AVAssetAVAssetTrack
Core Media EssentialsUnderstanding Time
CMTime‣ Rational number representing time‣ 64-bit integer time value (numerator)‣ 32-bit integer time scale (denominator)
CMTime fiveSeconds = CMTimeMake(5, 1);CMTime oneSample = CMTimeMake(1, 44100);CMTime zeroTime = kCMTimeZero;
‣ Large number of utility functions in Core Media:‣ CMTimeAdd,CMTimeSubtract,CMTimeCompare, etc.
Core Media EssentialsUnderstanding Time
CMTimeRange‣ Core Media struct containing start time and duration
‣ Like CMTime, there are many Core Media functions:‣ CMTimeRangeEqual, CMTimeRangeContainsTime, CMTimeRangeGetEnd, CMTIMERANGE_ISVALID, etc.
CMTimeRange zeroRange = kCMTimeRangeZero;CMTimeRange assetRange = CMTimeRangeMake(kCMTimeZero, asset.duration);
AVPlayerLayerVideo Playback
AVPlayerAVPlayerItem
AVPlayerItemTrackAVPlayerItemTrackAVPlayerItemTrack
AVAsset
AVAssetAVAssetAVAssetTrack
AVPlayerLayerVideo Playback
AVPlayerAVPlayerItem
AVPlayerItemTrackAVPlayerItemTrackAVPlayerItemTrack
AVPlayerLayer
Demo
Composing Media
AVCompositionComposing Assets
‣Concrete extension of AVAsset‣Composes asset segments on a timeline
Tracks and SegmentsComposing Assets
AVComposition
AVMutableComposition *composition = [AVMutableComposition composition];
AVComposition
Tracks and SegmentsComposing Assets
CMPersistentTrackID trackID = kCMPersistentTrackID_Invalid;
AVMutableCompositionTrack *videoTrack = [composition addMutableTrackWithMediaType:AVMediaTypeVideo preferredTrackID:trackID];
AVMutableCompositionTrack *audioTrack = [composition addMutableTrackWithMediaType:AVMediaTypeAudio preferredTrackID:trackID];
AVCompositionTrack (Video)
AVCompositionTrack (Audio)
AVCompositionTrack (Video)
AVCompositionTrack (Audio)
AVComposition
Tracks and SegmentsComposing Assets
AVCompositionTrack (Video)
AVCompositionTrack (Audio)
AVCompositionTrackSegment
Seconds 10-30 of “redpanda.m4v”
AVCompositionTrackSegment
Seconds 20-60 of “waves.m4v”
AVCompositionTrackSegment
Seconds 0-60 of “soundtrack.mp3”
AVAssetTrack *srcVideoTrack1 = // source video track 1[videoTrack insertTimeRange:timeRange ofTrack:srcVideoTrack1 atTime:startTime error:&error];
AVAssetTrack *srcVideoTrack2 = // source video track 2[videoTrack insertTimeRange:timeRange ofTrack:srcVideoTrack2 atTime:startTime error:&error];
AVAssetTrack *srcAudioTrack = // source audio track[audioTrack insertTimeRange:timeRange ofTrack:srcAudioTrack atTime:startTime error:&error];
Demo
Mixing Audio
AVAudioMixAudio Mixing
‣ Composition tracks play at their natural volume‣ AVAudioMix applies track-level volume adjustments
‣ Composed of AVAudioMixInputParameters‣ Parameters control individual track volume over time
CMTime CMTimeRange
CMTime fadeTime = CMTimeMake(5, 1);CMTime fadeInStartTime = kCMTimeZero;CMTime fadeOutStartTime = CMTimeSubtract(asset.duration, fadeTime);
CMTimeRange fadeInRange = CMTimeRangeMake(fadeInStartTime, fadeTime);CMTimeRange fadeOutRange = CMTimeRangeMake(fadeOutStartTime, fadeTime);
AVMutableAudioMixInputParameters *parameters = [AVMutableAudioMixInputParameters audioMixInputParameters];parameters.trackID = assetTrack.trackID;[parameters setVolumeRampFromStartVolume:0.0 toEndVolume:1.0 timeRange:fadeInRange];[parameters setVolumeRampFromStartVolume:1.0 toEndVolume:0.0 timeRange:fadeOutRange];
AVMutableAudioMix *audioMix = [AVMutableAudioMix audioMix];audioMix.inputParameters = @[parameters];playerItem.audioMix = audioMix;
ExampleAVAudioMix
Demo
Video Transitions
AVVideoCompositionVideo Transitions
AVVideoCompositionDefines how two or more video tracks are composited together
Configured through collection of composition instructions describing compositing behavior
AVVideoCompositionInstructionAVVideoCompositionInstructionAVVideoCompositionInstruction
AVVideoCompositionInstructionVideo Transitions
AVVideoComposition
AVVideoCompositionInstructionAVVideoCompositionInstructionAVVideoCompositionInstructionDefines the time range of compositing behavior
Composed of layer instructions describing compositing behavior
AVAssetAVAssetAVVideoCompositionLayerInstruction
AVVideoCompositionLayerInstructionVideo Transitions
AVVideoComposition
AVAssetAVAssetAVVideoCompositionLayerInstruction
AVVideoCompositionInstructionAVVideoCompositionInstructionAVVideoCompositionInstruction
Defines the transform and opacity ramps of input layers
Transform and opacity changes modified over given time range
Conceptual StepsBuilding Transitions
Stagger LayoutBuilding Transitions 1
A
B
De!ne Overlapping RegionsBuilding Transitions 2
A
B
De!ne Time RangesBuilding Transitions
A
B
Transition
Passthrough
Transition
Passthrough Passthrough
* Time ranges must not have gaps or overlap* Total duration must not be shorter than composition
3
Con!gure InstructionsBuilding Transitions 4
// Build transition instructionsAVMutableVideoCompositionInstruction *transitionInstruction = ...;transitionInstruction.timeRange = transitionTimeRange;
AVMutableVideoCompositionLayerInstruction *fromLayerInstruction = ...;AVMutableVideoCompositionLayerInstruction *toLayerInstruction = ...;
// Cross Disolve[fromLayerInstruction setOpacityRampFromStartOpacity:1.0 toEndOpacity:0.0 timeRange:transitionTimeRange];
NSArray *instructions = @[fromLayerInstruction, toLayerInstruction];transitionInstruction.layerInstructions = instructions;[instructions addObject:transitionInstruction];
Set sizes and applyBuilding Transitions 5
AVMutableVideoComposition *videoComposition = [AVMutableVideoComposition videoComposition];
// Set instructions on AVVideoComposition instancevideoComposition.instructions = instructions;videoComposition.frameDuration = FRAME_RATE;videoComposition.renderSize = RENDER_SIZE;
AVPlayerItem *playerItem = [AVPlayerItem playerItemWithAsset:[composition copy]];playerItem.videoComposition = videoComposition;
#define FRAME_RATE CMTimeMake(1, 30)#define RENDER_SIZE CGSizeMake(1280, 720)
videoCompositionWithPropertiesOfAsset:
New in iOS 6AVVideoComposition
• Calculates all required passthrough and transition time ranges
• Builds appropriate composition and layer instructions for time ranges
• Sets the appropriate render size
• Sets the appropriate frame rate
Automagic Setup:
Demo
Layering Content
Core AnimationLayering Content
Core Animation a natural choice‣ High performance, inherently time-based‣ CALayer subclasses used for all video rendering
CALayer: used to layer images and text
CAAnimation: used to animate layered content
CABasicAnimation
CAKeyframeAnimation
AVSynchronizedLayerAnimation Timing
‣ Core Animation operates on host time‣ Starts at boot, marches towards in!nity
‣ Timeline animations need to use movie time‣ Starts at kCMTimeZero and runs to duration‣ Can be started, stopped, rewound, etc.
‣ Use AVSynchronizedLayer to use movie time‣ Confers player item timing on to its sublayer tree
AVSynchronizedLayer
CATextLayer CABasicAnimation
AVPlayerItem
Timeline vs Realtime AnimationsCore Animation
‣ Exactly the same, but different:‣ Animations with zero beginTime won’t be seen‣ Set beginTime = AVCoreAnimationBeginTimeZero
‣ Animations removed by default‣ Set removedOnCompletion = NO‣ Unable to use CAAnimationGroup?
Natural Choice, Awkward ImplementationCore Animation
‣ Different conceptual models for timeline editing‣ CMTime and CMTimeRange for asset items‣ Seconds and milliseconds for layers‣ Build abstraction to help bridge the gap
‣ Usage differs in playback and export scenarios‣ AVSynchronizedLayer for playback‣ Attach to player’s view hierarchy/layer tree
‣ AVVideoCompositionCoreAnimationTool for export
Demo
AV Foundation Rocks!Summary
‣ Powerful tools for audio and video playback‣ AVPlayer, AVPlayerItem, AVPlayerLayer
‣ Powerful tools for composing/editing media:‣ AVComposition, AVAudioMix, AVVideoComposition, AVSynchronizedLayer
‣ Powerful utility classes:‣ AVAssetImageGenerator‣ AVExportSession
‣ Steep learning curve, but worth the investment!
ResourcesPresentation Materialshttp://www.speakerdeck.com/bobmccune/http://www.slideshare.net/bobmccune/https://github.com/tapharmonic/AVFoundationEditor
Learning AV Foundationhttp://www.speakerdeck.com/bobmccune/https://github.com/tapharmonic/AVFoundationDemos
WWDC 2011: Exploring AV Foundationhttps://developer.apple.com/videos/wwdc/2011/?id=405
WWDC 2011: Working with Media in AV Foundationhttps://developer.apple.com/videos/wwdc/2011/?id=415
BobMcCune.com @bobmccune