brad ford, purveyor of deep thoughts - devstreaming-cdn.apple · pdf fileredistribution or...
TRANSCRIPT
![Page 1: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/1.jpg)
#WWDC17
© 2017 Apple Inc. All rights reserved. Redistribution or public display not permitted without written permission from Apple.
Brad Ford, Purveyor of Deep Thoughts
•Capturing Depth in iPhone Photography • Session 507
Media
![Page 2: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/2.jpg)
![Page 3: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/3.jpg)
![Page 4: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/4.jpg)
•Depth and disparity on iPhone 7 Plus •Streaming depth data from the camera •Capturing photos with depth data •Dual photo capture
![Page 5: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/5.jpg)
•Depth and disparity on iPhone 7 Plus •Streaming depth data from the camera •Capturing photos with depth data •Dual photo capture
![Page 6: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/6.jpg)
•Depth and disparity on iPhone 7 Plus •Streaming depth data from the camera •Capturing photos with depth data •Dual photo capture
![Page 7: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/7.jpg)
•Depth and disparity on iPhone 7 Plus •Streaming depth data from the camera •Capturing photos with depth data •Dual photo capture
![Page 8: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/8.jpg)
•Depth and disparity on iPhone 7 Plus •Streaming depth data from the camera •Capturing photos with depth data •Dual photo capture
![Page 9: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/9.jpg)
![Page 10: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/10.jpg)
Telephoto lensWide-angle lens
![Page 11: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/11.jpg)
Dual Camera Zoom
Switches between wide and tele automatically
Matches exposure, focus, and frame rate
Compensates for parallax shift to smooth the transition
![Page 12: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/12.jpg)
Portrait Mode
![Page 13: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/13.jpg)
![Page 14: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/14.jpg)
![Page 15: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/15.jpg)
![Page 16: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/16.jpg)
![Page 17: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/17.jpg)
![Page 18: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/18.jpg)
![Page 19: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/19.jpg)
![Page 20: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/20.jpg)
![Page 21: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/21.jpg)
![Page 22: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/22.jpg)
![Page 23: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/23.jpg)
•Learning•Deep
![Page 24: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/24.jpg)
Depth Map
Image Plane (Sensor)
Observed Objects
Pinhole Camera
![Page 25: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/25.jpg)
Depth Map
Image Plane (Sensor)
Observed Objects
Pinhole Camera
![Page 26: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/26.jpg)
Depth Map
Image Plane (Sensor)
Pinhole Camera
![Page 27: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/27.jpg)
Depth Map
Image Plane (Sensor)
Focal Point
Pinhole Camera
![Page 28: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/28.jpg)
Depth Map
Image Plane (Sensor)
Focal Length
Focal Point
Pinhole Camera
![Page 29: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/29.jpg)
Depth Map
Time of Flight Camera
![Page 30: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/30.jpg)
Depth Map
Time of Flight Camera
![Page 31: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/31.jpg)
•Disparity
![Page 32: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/32.jpg)
![Page 33: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/33.jpg)
![Page 34: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/34.jpg)
Disparity Map
Camera 1
Camera 2
![Page 35: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/35.jpg)
Disparity Map
Focal Length
Camera 1
Camera 2
![Page 36: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/36.jpg)
Disparity Map
Focal Length
Optical Center
Optical Center
Camera 1
Camera 2
![Page 37: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/37.jpg)
Disparity Map
Focal Length
Optical Center
Optical Center
Camera 1
Camera 2
Baseline
![Page 38: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/38.jpg)
Disparity Map
Focal Length
Optical Center
Optical Center
Camera 1
Camera 2Observed Point
Baseline
![Page 39: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/39.jpg)
Disparity Map
Focal Length
Optical Center
Optical Center
Camera 1
Camera 2Observed Point
Baseline
“Z” (Depth)
![Page 40: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/40.jpg)
Disparity Map
Focal Length
Optical Center
Optical Center
Camera 1
Camera 2Observed Point
Baseline
“Z” (Depth)
![Page 41: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/41.jpg)
Disparity Map
Focal Length
Optical Center
Optical Center
Camera 1
Camera 2Observed Point
Baseline
“Z” (Depth)
![Page 42: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/42.jpg)
Disparity Map
Focal Length
Optical Center
Optical Center
Camera 1
Camera 2Observed Point
Baseline
“Z” (Depth)
![Page 43: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/43.jpg)
Disparity Map
Focal Length
Optical Center
Optical Center
Camera 1
Camera 2Observed Point
Baseline
“Z” (Depth)
![Page 44: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/44.jpg)
Disparity Map
Camera 1 Camera 2
![Page 45: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/45.jpg)
Disparity Map
Camera 1 Camera 2
Disparity
![Page 46: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/46.jpg)
Removing Despair from Disparity
Observed Point
Camera 1
Camera 2
![Page 47: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/47.jpg)
Removing Despair from Disparity
Observed Point
Camera 1
Camera 2
![Page 48: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/48.jpg)
Removing Despair from Disparity
Observed Point
Camera 1
Camera 2
![Page 49: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/49.jpg)
Removing Despair from Disparity
Z (meters)
Baseline (meters)
![Page 50: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/50.jpg)
Removing Despair from Disparity
focal length (pixels)
disparity (pixels)
Z (meters)
Baseline (meters)
![Page 51: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/51.jpg)
Removing Despair from Disparity
focal length (pixels)
disparity (pixels)
Z (meters)
Baseline (meters)
![Page 52: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/52.jpg)
___
Removing Despair from Disparity
focal length (pixels)
disparity (pixels)
Z (meters)
Baseline (meters)
____bz
dfl=
![Page 53: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/53.jpg)
___
Removing Despair from Disparity
focal length (pixels)
disparity (pixels)
Z (meters)
Baseline (meters)
____bz
dfl=
![Page 54: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/54.jpg)
___
Removing Despair from Disparity
focal length (pixels)
disparity (pixels)
Z (meters)
Baseline (meters)
____bz
dfl=
![Page 55: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/55.jpg)
___
Removing Despair from Disparity
____bz
dfl=b b
focal length (pixels)
disparity (pixels)
Z (meters)
Baseline (meters)
![Page 56: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/56.jpg)
___
Removing Despair from Disparity
____bz
dfl=b b
focal length (pixels)
disparity (pixels)
Z (meters)
Baseline (meters)
![Page 57: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/57.jpg)
___
Removing Despair from Disparity
focal length (pixels)
disparity (pixels)
Z (meters)
Baseline (meters)
____1z
dfl= b
![Page 58: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/58.jpg)
___
Normalized Disparity
focal length (pixels)
disparity (pixels)
Z (meters)
Baseline (meters)
____1z
dfl= b
![Page 59: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/59.jpg)
Disparity vs. Depth
iPhone 7 Plus Dual Camera system is disparity based
![Page 60: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/60.jpg)
Disparity vs. Depth
iPhone 7 Plus Dual Camera system is disparity based
Disparity is a proxy for depth
![Page 61: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/61.jpg)
Disparity vs. Depth
iPhone 7 Plus Dual Camera system is disparity based
Disparity is a proxy for depth
Normalized disparity is the inverse of depth
![Page 62: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/62.jpg)
A Deep Thought
![Page 63: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/63.jpg)
![Page 64: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/64.jpg)
![Page 65: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/65.jpg)
New Term: Depth Data
![Page 66: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/66.jpg)
Introducing AVDepthData
The canonical representation of depth on iOS, macOS, and tvOS
Class in the AVFoundation framework
Represents depth or disparity maps
Converts between depth/disparity formats
NEW
![Page 67: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/67.jpg)
Introducing AVDepthData
public var kCVPixelFormatType_DisparityFloat16: OSType { get } /* 'hdis' */
public var kCVPixelFormatType_DisparityFloat32: OSType { get } /* 'hdis' */
public var kCVPixelFormatType_DepthFloat16: OSType { get } /* 'hdep' */
public var kCVPixelFormatType_DepthFloat32: OSType { get } /* 'fdep' */
![Page 68: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/68.jpg)
Introducing AVDepthData
public var kCVPixelFormatType_DisparityFloat16: OSType { get } /* 'hdis' */
public var kCVPixelFormatType_DisparityFloat32: OSType { get } /* 'hdis' */
public var kCVPixelFormatType_DepthFloat16: OSType { get } /* 'hdep' */
public var kCVPixelFormatType_DepthFloat32: OSType { get } /* 'fdep' */
![Page 69: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/69.jpg)
Introducing AVDepthData
public var kCVPixelFormatType_DisparityFloat16: OSType { get } /* 'hdis' */
public var kCVPixelFormatType_DisparityFloat32: OSType { get } /* 'hdis' */
public var kCVPixelFormatType_DepthFloat16: OSType { get } /* 'hdep' */
public var kCVPixelFormatType_DepthFloat32: OSType { get } /* 'fdep' */
![Page 70: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/70.jpg)
Introducing AVDepthData
@available(iOS 11.0, *)open class AVDepthData: NSObject {
open var depthDataType: OSType { get }
open var depthDataMap: CVPixelBuffer { get }
open var isDepthDataFiltered: Bool { get }
open var depthDataAccuracy: AVDepthDataAccuracy { get }
}
![Page 71: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/71.jpg)
Introducing AVDepthData
@available(iOS 11.0, *)open class AVDepthData: NSObject {
open var depthDataType: OSType { get }
open var depthDataMap: CVPixelBuffer { get }
open var isDepthDataFiltered: Bool { get }
open var depthDataAccuracy: AVDepthDataAccuracy { get }
}
![Page 72: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/72.jpg)
Introducing AVDepthData
@available(iOS 11.0, *)open class AVDepthData: NSObject {
open var depthDataType: OSType { get }
open var depthDataMap: CVPixelBuffer { get }
open var isDepthDataFiltered: Bool { get }
open var depthDataAccuracy: AVDepthDataAccuracy { get }
}
![Page 73: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/73.jpg)
Introducing AVDepthData
@available(iOS 11.0, *)open class AVDepthData: NSObject {
open var depthDataType: OSType { get }
open var depthDataMap: CVPixelBuffer { get }
open var isDepthDataFiltered: Bool { get }
open var depthDataAccuracy: AVDepthDataAccuracy { get }
}
![Page 74: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/74.jpg)
Holes
Camera 1
Camera 2
![Page 75: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/75.jpg)
Holes
Camera 1
Camera 2Observed Point
![Page 76: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/76.jpg)
Holes
Camera 1
Camera 2Observed Point
![Page 77: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/77.jpg)
Holes
Camera 1
Camera 2Observed Point
![Page 78: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/78.jpg)
Holes
Camera 1
Camera 2Observed Point
??
??
![Page 79: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/79.jpg)
Hole-Reporting in AVDepthData
NaN = Invalid depthDataMap value
open var isDepthDataFiltered: Bool { get }
![Page 80: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/80.jpg)
Calibration Errors
Camera 1 Camera 2
![Page 81: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/81.jpg)
Calibration Errors
Camera 1 Camera 2
![Page 82: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/82.jpg)
Calibration Errors
Camera 1 Camera 2Baseline
![Page 83: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/83.jpg)
Calibration Errors
Camera 1 Camera 2
Baseline
![Page 84: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/84.jpg)
Calibration Errors
Camera 1 Camera 2
Baseline
![Page 85: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/85.jpg)
Calibration Errors
Camera 1 Camera 2
Baseline
![Page 86: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/86.jpg)
Calibration Errors
Optical Image Stabilization
Gravity
Focus Coil
![Page 87: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/87.jpg)
Depth Data Accuracy
extension AVDepthData { public enum Accuracy: Int { case relative case absolute }}
![Page 88: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/88.jpg)
^d = d + e
Relative disparityAbsolute disparity
Fixed error
![Page 89: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/89.jpg)
^d - d = (d + e) - (d + e)
^1 2 1 2
![Page 90: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/90.jpg)
+ e - e^d - d = d - d
^1 2 1 2
^d - d = (d + e) - (d + e)
^1 2 1 2
![Page 91: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/91.jpg)
+ e - e^d - d = d - d
^1 2 1 2
![Page 92: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/92.jpg)
+ e - e^d - d = d - d
^1 2 1 2
![Page 93: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/93.jpg)
^d - d = d - d
^1 2 1 2
![Page 94: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/94.jpg)
^d - d = d - d
^1 2 1 2
![Page 95: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/95.jpg)
•Streaming Depth Data
![Page 96: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/96.jpg)
•Demo •AVCamPhotoFilter
![Page 97: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/97.jpg)
Introducing AVCaptureDepthDataOutput
AVCaptureSession
![Page 98: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/98.jpg)
Introducing AVCaptureDepthDataOutput
AVCaptureDeviceInput
AVCaptureDevice (DualCamera)
AVCaptureSession
![Page 99: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/99.jpg)
Introducing AVCaptureDepthDataOutput
AVCaptureDeviceInput
AVCaptureDevice (DualCamera)
AVCaptureSession
AVCaptureDepthDataOutput
![Page 100: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/100.jpg)
Introducing AVCaptureDepthDataOutput
AVCaptureDeviceInput
AVCaptureDevice (DualCamera)
AVCaptureSession
AVCaptureDepthDataOutput
![Page 101: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/101.jpg)
Introducing AVCaptureDepthDataOutput
AVCaptureDeviceInput
AVCaptureDevice (DualCamera)
AVCaptureSession
AVCaptureDepthDataOutput
AVDepthDataAVDepthDataAVDepthDataAVDepthData
![Page 102: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/102.jpg)
AVCaptureDepthDataOutput
Only supported on Dual Camera
NEW
![Page 103: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/103.jpg)
AVCaptureDepthDataOutput
Only supported on Dual Camera
Dual Camera auto-zooms to 2x (zoom is disabled)
NEW
![Page 104: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/104.jpg)
AVCaptureDepthDataOutput
Only supported on Dual Camera
Dual Camera auto-zooms to 2x (zoom is disabled)
Dual Camera video formats have a supportedDepthDataFormats property
NEW
![Page 105: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/105.jpg)
AVCaptureDepthDataOutput
Only supported on Dual Camera
Dual Camera auto-zooms to 2x (zoom is disabled)
Dual Camera video formats have a supportedDepthDataFormats property
AVCaptureDevice has new activeDepthDataFormat property
NEW
![Page 106: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/106.jpg)
Supported Depth Resolutions for Streaming
Preset or Active Format AVCaptureVideoDataOutput AVCaptureDepthDataOutput
.photo 1440x1080 @ 30320x240 @ 24 fps 160x120 @ 24 fps
1280x720@3-30 FPS 1280x720 @ 30320x180 @ 24 fps 160x90 @ 24 fps
.vga640x480 640x480 @ 30320x240 @ 24 fps 160x120 @ 24 fps
![Page 107: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/107.jpg)
Supported Depth Resolutions for Streaming
Preset or Active Format AVCaptureVideoDataOutput AVCaptureDepthDataOutput
.photo 1440x1080 @ 30320x240 @ 24 fps 160x120 @ 24 fps
1280x720@3-30 FPS 1280x720 @ 30320x180 @ 24 fps 160x90 @ 24 fps
.vga640x480 640x480 @ 30320x240 @ 24 fps 160x120 @ 24 fps
![Page 108: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/108.jpg)
Supported Depth Resolutions for Streaming
Preset or Active Format AVCaptureVideoDataOutput AVCaptureDepthDataOutput
.photo 1440x1080 @ 30320x240 @ 24 fps 160x120 @ 24 fps
1280x720@3-30 FPS 1280x720 @ 30320x180 @ 24 fps 160x90 @ 24 fps
.vga640x480 640x480 @ 30320x240 @ 24 fps 160x120 @ 24 fps
![Page 109: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/109.jpg)
Supported Depth Resolutions for Streaming
Preset or Active Format AVCaptureVideoDataOutput AVCaptureDepthDataOutput
.photo 1440x1080 @ 30320x240 @ 24 fps 160x120 @ 24 fps
1280x720@3-30 FPS 1280x720 @ 30320x180 @ 24 fps 160x90 @ 24 fps
.vga640x480 640x480 @ 30320x240 @ 24 fps 160x120 @ 24 fps
![Page 110: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/110.jpg)
Depth Frame Rate Examples
VideoDataOutput DepthDataOutput
1440x1080 @ 24 320x240 @ 24 fps
1440x1080 @ 30 320x240 @ 15 fps
![Page 111: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/111.jpg)
Depth Frame Rate Examples
VideoDataOutput DepthDataOutput
1440x1080 @ 24 320x240 @ 24 fps
1440x1080 @ 30 320x240 @ 15 fps
![Page 112: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/112.jpg)
Depth Frame Rate Examples
VideoDataOutput DepthDataOutput
1440x1080 @ 24 320x240 @ 24 fps
1440x1080 @ 30 320x240 @ 15 fps
![Page 113: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/113.jpg)
Depth Filtering with AVCaptureDepthDataOutput
open var isFilteringEnabled: Bool
![Page 114: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/114.jpg)
Unsynchronized Data Output
AVCaptureDeviceInput
AVCaptureDevice (Microphone)
AVCaptureSession
AVCaptureDeviceInput
AVCaptureDevice (DualCamera)
![Page 115: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/115.jpg)
Unsynchronized Data Output
AVCaptureVideoDataOutput
sbufsbufsbufvideo sbufs
AVCaptureDeviceInput
AVCaptureDevice (Microphone)
AVCaptureSession
AVCaptureDeviceInput
AVCaptureDevice (DualCamera)
![Page 116: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/116.jpg)
Unsynchronized Data Output
AVCaptureVideoDataOutput
sbufsbufsbufvideo sbufs
sbufsbufsbufaudio sbufs
AVCaptureAudioDataOutput
AVCaptureDeviceInput
AVCaptureDevice (Microphone)
AVCaptureSession
AVCaptureDeviceInput
AVCaptureDevice (DualCamera)
![Page 117: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/117.jpg)
Unsynchronized Data Output
AVCaptureVideoDataOutput
sbufsbufsbufvideo sbufs
sbufsbufsbufaudio sbufs
AVCaptureAudioDataOutput
AVCaptureDeviceInput
AVCaptureDevice (Microphone)
sbufsbufsbuffaces or codes
AVCaptureMetadataOutput
AVCaptureSession
AVCaptureDeviceInput
AVCaptureDevice (DualCamera)
![Page 118: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/118.jpg)
Unsynchronized Data Output
AVCaptureVideoDataOutput
sbufsbufsbufvideo sbufs
sbufsbufsbufaudio sbufs
AVCaptureAudioDataOutput
AVCaptureDeviceInput
AVCaptureDevice (Microphone)
sbufsbufsbufdepth data
AVCaptureDepthDataOutput
sbufsbufsbuffaces or codes
AVCaptureMetadataOutput
AVCaptureSession
AVCaptureDeviceInput
AVCaptureDevice (DualCamera)
![Page 119: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/119.jpg)
Synchronized Data Output
AVCaptureVideoDataOutput AVCaptureAudioDataOutput
AVCaptureDeviceInput
AVCaptureDevice (Microphone)
AVCaptureDepthDataOutput
AVCaptureMetadataOutput
AVCaptureSession
AVCaptureDeviceInput
AVCaptureDevice (DualCamera)
![Page 120: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/120.jpg)
AVCaptureDataOutputSynchronizer
Synchronized Data Output
AVCaptureVideoDataOutput AVCaptureAudioDataOutput
AVCaptureDeviceInput
AVCaptureDevice (Microphone)
AVCaptureDepthDataOutput
AVCaptureMetadataOutput
AVCaptureSession
AVCaptureDeviceInput
AVCaptureDevice (DualCamera)
![Page 121: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/121.jpg)
AVCaptureSynchronizedDataCollectionAVCaptureSynchronizedDataCollectionAVCaptureSynchronizedDataCollectionAVCaptureSynchronizedDataCollection
AVCaptureDataOutputSynchronizer
Synchronized Data Output
AVCaptureVideoDataOutput AVCaptureAudioDataOutput
AVCaptureDeviceInput
AVCaptureDevice (Microphone)
AVCaptureDepthDataOutput
AVCaptureMetadataOutput
AVCaptureSession
AVCaptureDeviceInput
AVCaptureDevice (DualCamera)
![Page 122: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/122.jpg)
func dataOutputSynchronizer(_ synchronizer: AVCaptureDataOutputSynchronizer, didOutput synchronizedDataCollection: AVCaptureSynchronizedDataCollection) {
// Iterate through an AVCaptureSynchronizedDataCollection using fast enumeration
for syncedData in synchronizedDataCollection { if let syncedDepthData = syncedData as? AVCaptureSynchronizedDepthData { // ... } } }
![Page 123: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/123.jpg)
func dataOutputSynchronizer(_ synchronizer: AVCaptureDataOutputSynchronizer, didOutput synchronizedDataCollection: AVCaptureSynchronizedDataCollection) {
// Iterate through an AVCaptureSynchronizedDataCollection using fast enumeration
for syncedData in synchronizedDataCollection { if let syncedDepthData = syncedData as? AVCaptureSynchronizedDepthData { // ... } } }
![Page 124: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/124.jpg)
func dataOutputSynchronizer(_ synchronizer: AVCaptureDataOutputSynchronizer, didOutput synchronizedDataCollection: AVCaptureSynchronizedDataCollection) {
// Iterate through an AVCaptureSynchronizedDataCollection using fast enumeration
for syncedData in synchronizedDataCollection { if let syncedDepthData = syncedData as? AVCaptureSynchronizedDepthData { // ... } } }
![Page 125: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/125.jpg)
func dataOutputSynchronizer(_ synchronizer: AVCaptureDataOutputSynchronizer, didOutput synchronizedDataCollection: AVCaptureSynchronizedDataCollection) {
// Use dictionary-esque subscripting to find a particular data
if let synDepth = synchronizedDataCollection[self.ddo] as? AVCaptureSynchronizedDepthData { // ... } }
![Page 126: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/126.jpg)
func dataOutputSynchronizer(_ synchronizer: AVCaptureDataOutputSynchronizer, didOutput synchronizedDataCollection: AVCaptureSynchronizedDataCollection) {
// Use dictionary-esque subscripting to find a particular data
if let synDepth = synchronizedDataCollection[self.ddo] as? AVCaptureSynchronizedDepthData { // ... } }
![Page 127: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/127.jpg)
•AVCamPhotoFilter •Sample code
![Page 128: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/128.jpg)
Streaming Camera Intrinsics
Focal Length
Optical Center
NEW
![Page 129: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/129.jpg)
Streaming Camera Intrinsics
Focal Length
Optical Center
open class AVCaptureConnection : NSObject {
open var isCameraIntrinsicMatrixDeliveryEnabled: Bool
}
NEW
![Page 130: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/130.jpg)
Focal Length
Optical Center
Streaming Camera Intrinsics
K =
f s x
0 f y
0 0 1
x 0
0y
![Page 131: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/131.jpg)
Focal Length
Optical Center
Streaming Camera Intrinsics
K =
f s x
0 f y
0 0 1
x 0
0y
![Page 132: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/132.jpg)
Focal Length
Optical Center
Streaming Camera Intrinsics
K =
f s x
0 f y
0 0 1
x 0
0y
![Page 133: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/133.jpg)
kCMSampleBufferAttachmentKey_CameraIntrinsicMatrix
Focal Length
Optical Center
Streaming Camera Intrinsics
K =
f s x
0 f y
0 0 1
x 0
0y
![Page 134: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/134.jpg)
•Capturing Photos with Depth
![Page 135: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/135.jpg)
•AVCam •Wiggle Me
![Page 136: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/136.jpg)
Photos with Depth
• Flash • Auto still image stabilization • Auto exposure brackets • Live photos
![Page 137: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/137.jpg)
Capturing Photos with Depth
AVCaptureDeviceInput
AVCaptureDevice (Camera)
AVCaptureSession
AVCapturePhotoOutput
![Page 138: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/138.jpg)
Capturing Photos with Depth
AVCaptureDeviceInput
AVCaptureDevice (Camera)
AVCaptureSession
AVCapturePhotoOutput
![Page 139: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/139.jpg)
Capturing Photos with Depth
AVCaptureDeviceInput
AVCaptureDevice (Camera)
AVCaptureSession
AVCapturePhotoOutput
![Page 140: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/140.jpg)
Capturing Photos with Depth
AVCaptureDeviceInput
AVCaptureDevice (Camera)
AVCaptureSession
AVCapturePhotoOutput
![Page 141: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/141.jpg)
Capturing Photos with Depth
AVCaptureDeviceInput
AVCaptureDevice (Camera)
AVCaptureSession
AVCapturePhotoOutput
![Page 142: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/142.jpg)
func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto photo: AVCapturePhoto, error: Error?)
![Page 143: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/143.jpg)
Requesting Depth with Photos
![Page 144: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/144.jpg)
Requesting Depth with Photos
Before starting the capture session, opt in!photoOutput.isDepthDataDeliveryEnabled = true
![Page 145: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/145.jpg)
Requesting Depth with Photos
Before starting the capture session, opt in!photoOutput.isDepthDataDeliveryEnabled = true
Request depth on a per-photo basisphotoSettings.isDepthDataDeliveryEnabled = true
![Page 146: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/146.jpg)
Requesting Depth with Photos
Before starting the capture session, opt in!photoOutput.isDepthDataDeliveryEnabled = true
Request depth on a per-photo basisphotoSettings.isDepthDataDeliveryEnabled = true
Work with the resulting AVDepthData in your AVCapturePhotoopen class AVCapturePhoto: NSObject { open var depthData: AVDepthData? { get } }
![Page 147: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/147.jpg)
High Res Photo Depth Maps
Preset or Active Format Streaming Video Resolution
High Res Photo Resolution
Streaming Depth Resolution
Photo Depth Map Resolution
.photo 1440x1080 4032x3024 320x240160x120 768x576
1280x720@3-30 FPS 1280x720 4096x2304320x180 160x90
768x432
.vga640x480 640x480 4032x3024320x240 160x120
768x576
![Page 148: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/148.jpg)
High Res Photo Depth Maps
Preset or Active Format Streaming Video Resolution
High Res Photo Resolution
Streaming Depth Resolution
Photo Depth Map Resolution
.photo 1440x1080 4032x3024 320x240160x120 768x576
1280x720@3-30 FPS 1280x720 4096x2304320x180 160x90
768x432
.vga640x480 640x480 4032x3024320x240 160x120
768x576
![Page 149: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/149.jpg)
High Res Photo Depth Maps
Preset or Active Format Streaming Video Resolution
High Res Photo Resolution
Streaming Depth Resolution
Photo Depth Map Resolution
.photo 1440x1080 4032x3024 320x240160x120 768x576
1280x720@3-30 FPS 1280x720 4096x2304320x180 160x90
768x432
.vga640x480 640x480 4032x3024320x240 160x120
768x576
![Page 150: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/150.jpg)
Rectilinear vs. Lens Distorted Images
![Page 151: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/151.jpg)
Rectilinear vs. Lens Distorted Images
![Page 152: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/152.jpg)
Rectilinear vs. Lens Distorted Images
![Page 153: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/153.jpg)
Depth Map Distortions
Tele
Wide
![Page 154: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/154.jpg)
Depth Map Distortions
Tele
Wide
![Page 155: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/155.jpg)
Depth Map Distortions
Tele
Wide
![Page 156: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/156.jpg)
Depth Map Distortions
Rectilinear Disparity Map
![Page 157: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/157.jpg)
Depth Map Distortions
Rectilinear Disparity Map
Tele Warped Disparity Map
![Page 158: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/158.jpg)
Depth Map Distortions
Depth data maps work well for applying effects to the images
Don't work well for 3D-scene reconstruction
Maps and images can be made rectilinear
![Page 159: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/159.jpg)
Depth in Image Files
HEIF HEVC (HEIC) • Map is an auxiliary image • Encoded as monochrome HEVC • Metadata is XMP in the auxiliary image
![Page 160: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/160.jpg)
Depth in Image Files
JPEG • Map is 8-bit lossy JPEG (filtered) or 16-bit
lossless (unfiltered) MPO • Metadata is XMP in the second image
![Page 161: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/161.jpg)
•Dual Photo Capture
![Page 162: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/162.jpg)
Dual Photo Capture
AVCapturePhotoOutput
12 MP Wide 12 MP Tele
NEW
![Page 163: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/163.jpg)
Requesting Dual Photo Delivery
![Page 164: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/164.jpg)
Requesting Dual Photo Delivery
Before starting the capture session, opt in!photoOutput.isDualCameraDualPhotoDeliveryEnabled = true
![Page 165: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/165.jpg)
Requesting Dual Photo Delivery
Before starting the capture session, opt in!photoOutput.isDualCameraDualPhotoDeliveryEnabled = true
Request Dual Photo capture on a per-request basisphotoSettings.isDualCameraDualPhotoDeliveryEnabled = true
![Page 166: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/166.jpg)
Requesting Dual Photo Delivery
Before starting the capture session, opt in!photoOutput.isDualCameraDualPhotoDeliveryEnabled = true
Request Dual Photo capture on a per-request basisphotoSettings.isDualCameraDualPhotoDeliveryEnabled = true
Number of AVCapturePhoto callbacks doubleslet sourceDeviceType = photo.sourceDeviceType
![Page 167: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/167.jpg)
Dual Photo Capture
Supports • Flash captures • Auto still image stabilization • Auto exposure brackets • Depth delivery
![Page 168: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/168.jpg)
Dual Photo Capture + Zoom
Area outside of the preview field of view is blackened
Clean aperture rectangle defines the area with valid pixel data
12 MP Wide 12 MP Tele
![Page 169: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/169.jpg)
Dual Photo Capture
Dual Photos can be delivered with camera calibration data
Make your own depth maps!
Augment reality!
![Page 170: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/170.jpg)
Introducing AVCameraCalibrationData
Property of AVDepthData
NEW
![Page 171: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/171.jpg)
Introducing AVCameraCalibrationData
Property of AVDepthData
Property of AVCapturePhoto (if you opt in):photoSettings.isCameraCalibrationDeliveryEnabled = true
NEW
![Page 172: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/172.jpg)
AVCameraCalibrationData
open var intrinsicMatrix: matrix_float3x3 { get } Focal Length
Optical CenterK =
f s x
0 f y
0 0 1
x 0
0y
![Page 173: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/173.jpg)
AVCameraCalibrationData
open var intrinsicMatrix: matrix_float3x3 { get } Focal Length
Optical CenterK =
f s x
0 f y
0 0 1
x 0
0y
![Page 174: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/174.jpg)
AVCameraCalibrationData
open var intrinsicMatrix: matrix_float3x3 { get } Focal Length
Optical CenterK =
f s x
0 f y
0 0 1
x 0
0y
![Page 175: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/175.jpg)
AVCameraCalibrationData
open var intrinsicMatrix: matrix_float3x3 { get }
open var intrinsicMatrixReferenceDimensions: CGSize { get }
Focal Length
Optical CenterK =
f s x
0 f y
0 0 1
x 0
0y
![Page 176: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/176.jpg)
open var extrinsicMatrix: matrix_float4x3 { get }
AVCameraCalibrationData
![Page 177: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/177.jpg)
open var extrinsicMatrix: matrix_float4x3 { get }
AVCameraCalibrationData
[R | t] =
r r r1,1 1,2 1,3
r r r
r r r
2,1 2,2 2,3
3,1 3,2 3,3
t1
t2
t3
![Page 178: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/178.jpg)
open var extrinsicMatrix: matrix_float4x3 { get }
AVCameraCalibrationData
[R | t] =
r r r1,1 1,2 1,3
r r r
r r r
2,1 2,2 2,3
3,1 3,2 3,3
t1
t2
t3
Rotation
![Page 179: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/179.jpg)
open var extrinsicMatrix: matrix_float4x3 { get }
AVCameraCalibrationData
[R | t] =
r r r1,1 1,2 1,3
r r r
r r r
2,1 2,2 2,3
3,1 3,2 3,3
t1
t2
t3
Translation
![Page 180: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/180.jpg)
open var extrinsicMatrix: matrix_float4x3 { get }
AVCameraCalibrationData
[R | t] =
r r r1,1 1,2 1,3
r r r
r r r
2,1 2,2 2,3
3,1 3,2 3,3
t1
t2
t3
Tele camera is the world origin
![Page 181: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/181.jpg)
AVCameraCalibrationData
![Page 182: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/182.jpg)
AVCameraCalibrationData
open var lensDistortionCenter: CGPoint { get }
![Page 183: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/183.jpg)
AVCameraCalibrationData
open var lensDistortionCenter: CGPoint { get }
open var lensDistortionLookupTable: Data { get }
![Page 184: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/184.jpg)
AVCameraCalibrationData
open var lensDistortionCenter: CGPoint { get }
open var lensDistortionLookupTable: Data { get }
![Page 185: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/185.jpg)
AVCameraCalibrationData
open var inverseLensDistortionLookupTable: Data { get }
![Page 186: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/186.jpg)
AVCameraCalibrationData
open var inverseLensDistortionLookupTable: Data { get }
![Page 187: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/187.jpg)
•Straighten Up
![Page 188: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/188.jpg)
Summary
iPhone 7 Plus Dual Camera is a disparity system
![Page 189: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/189.jpg)
Summary
iPhone 7 Plus Dual Camera is a disparity system
The canonical representation for depth is AVDepthData
![Page 190: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/190.jpg)
Summary
iPhone 7 Plus Dual Camera is a disparity system
The canonical representation for depth is AVDepthData
Intrinsic, extrinsic, and lens distortion info are stored in AVCameraCalibrationData
![Page 191: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/191.jpg)
Summary
iPhone 7 Plus Dual Camera is a disparity system
The canonical representation for depth is AVDepthData
Intrinsic, extrinsic, and lens distortion info are stored in AVCameraCalibrationData
Filtered/unfiltered depth is streamed using AVCaptureDepthDataOutput
![Page 192: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/192.jpg)
Summary
iPhone 7 Plus Dual Camera is a disparity system
The canonical representation for depth is AVDepthData
Intrinsic, extrinsic, and lens distortion info are stored in AVCameraCalibrationData
Filtered/unfiltered depth is streamed using AVCaptureDepthDataOutput
Depth is captured in photos using AVCapturePhotoOutput
![Page 193: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/193.jpg)
Summary
iPhone 7 Plus Dual Camera is a disparity system
The canonical representation for depth is AVDepthData
Intrinsic, extrinsic, and lens distortion info are stored in AVCameraCalibrationData
Filtered/unfiltered depth is streamed using AVCaptureDepthDataOutput
Depth is captured in photos using AVCapturePhotoOutput
Dual Camera Dual Photo Delivery produces wide and tele plus calibration
![Page 194: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/194.jpg)
AVCam
AVCamPhotoFilter
Wiggle Me
Sample Code
![Page 195: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/195.jpg)
More Informationhttps://developer.apple.com/wwdc17/507
![Page 196: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/196.jpg)
Related Sessions
Photography Get Together Technology Lab J Wednesday 6:30PM
Image Editing with Depth Hall 2 Thursday 11:00AM
Working with HEIF and HEVC Hall 2 Friday 11:00AM
![Page 197: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/197.jpg)
Labs
Photo Extensions & PhotoKit Lab Technology Lab A Thur 9:00AM–12:00PM
AVFoundation Lab Technology Lab F Thur 12:00PM–3:00PM
Photos Editing & Core Image Lab Technology Lab F Thur 3:10PM–6:00PM
Photos Depth & Capture Lab Technology Lab A Thur 3:10PM–6:00PM
HEIF and HEVC Lab Technology Lab F Fri 12:00PM–1:50PM
Vision Lab Technology Lab A Fri 1:50PM–4:00PM
Photos Depth & Capture Lab Technology Lab F Fri 1:50PM–4:00PM
![Page 198: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/198.jpg)
Labs
Photo Extensions & PhotoKit Lab Technology Lab A Thur 9:00AM–12:00PM
AVFoundation Lab Technology Lab F Thur 12:00PM–3:00PM
Photos Editing & Core Image Lab Technology Lab F Thur 3:10PM–6:00PM
Photos Depth & Capture Lab Technology Lab A Thur 3:10PM–6:00PM
HEIF and HEVC Lab Technology Lab F Fri 12:00PM–1:50PM
Vision Lab Technology Lab A Fri 1:50PM–4:00PM
Photos Depth & Capture Lab Technology Lab F Fri 1:50PM–4:00PM
![Page 199: Brad Ford, Purveyor of Deep Thoughts - devstreaming-cdn.apple · PDF fileRedistribution or public display not permitted without written permission from Apple. Brad Ford, Purveyor of](https://reader035.vdocuments.mx/reader035/viewer/2022081417/5a786f277f8b9aa3688eb588/html5/thumbnails/199.jpg)