web standards for ar workshop at ismar13
TRANSCRIPT
https://buildAR.com - image credit
Web Standards for AR
An intro to the latest developmentson the Augmented Web Platform
http://www.flickr.com/photos/37380823@N03/3442372197/sizes/o/in/photostream/
Who's making these outlandish claims?
Rob Manson @nambor
CEO of MOB-labs the creators of buildAR.com
Chair of the W3C's Augmented Web Community Group
Invited Expert with the ISO, W3C & the Khronos Group
Author of Getting started with WebRTC - (Packt Publishing)
https://buildAR.com
What is the Augmented Web?
https://buildAR.com
It's what comes after HTML5
https://buildAR.com
https://buildAR.com - image credit
http://www.flickr.com/photos/hbart/3417599994/sizes/o/in/photostream/
Key Concept:
1992-2000WWW: Page based websiteskey innovation: HREF
2000-2013Web 2.0: AJAX driven social serviceskey innovation: XHR
2013-...AWE: Sensor driven real-time web of thingskey innovation: ArrayBuffer
https://buildAR.com
https://buildAR.com - image credit
http://www.flickr.com/photos/congvo/301678287/sizes/o/in/photostream/
https://buildAR.comKey Concept:
https://buildAR.comJSARToolkit demo
https://buildAR.comFAST algorithm feature detection demo
Test your own device at: theAWEsomeWEB.com
https://buildAR.comhttp://www.khronos.org/streaminput/
https://buildAR.com - image credit
http://www.flickr.com/photos/eltman/353241574/sizes/o/in/photolist-xdspY-pTuti-5z6PrP-5WtYra-7TraYS-bPXMfB-5hVt66-frMcHT-8sTZzS-7Bjs8T-6MKBmQ-8PU77j-33fJed-Ht9xU-99aKN-cUrcA-6yEAWp-7Ezwo8-aet2qK-4d5TfF-rqwWj-9rRjqk-5Wyg3b-5WtYV4-wwJ1Z-6EdA7y-5f79J2-6G8fja-3yZu6S-4Y6cn2-rjKsj-dwh5ee-4D1juK-7uidYY-nkcwB-4VG9yz-3jUx3T-cu4KkC-cu4PmE-7498qy-6wrgXr-7Rubrv-348E4E-5yBq3E-8e96oG-bWZwfy-6QSudj-6QNEER-6QSug5-6QNEDp-6QNEFX/
First lets look at the broadStream processing pipeline concept
https://buildAR.com
Key Concept:
https://buildAR.com
So what the hell is aStream processing pipeline?
https://buildAR.comThe vision we proposed in 2010 is now here!
ARStandards Workshop in Seoul 2010 Rob Manson
https://buildAR.comStream processing pipelines
1. Get Stream
2. Connect to a Stream pipeline
A way to connect a Stream to an ArrayBuffer
3. Get ArrayBuffer
5. Populate a scene Array with Array Buffer Views
6. Move through Views to process data
7. Output events and metadata
8. Update UI and/or send requests
Now lets look at the more specificMediaStream processing pipeline
https://buildAR.com
https://buildAR.comMediaStream processing pipelines
1. Get MediaStreamMediaStream from getUserMedia or RTCPeerConnection
2. Connect to a MediaStream pipeline Canvas/Video, Image Capture, Recording, ScriptNodePorcessor or Video/Shader pipelines
3. Get ArrayBuffernew ArrayBuffer()
5. Populate a scene Array with Array Buffer Viewsvar scene = [new Uint8ClampedArray(buffer), ...]
6. Move through Views to process datafor each (var view in scene) { process(view)... }
7. Output events and metadatareturn results
8. Update UI and/or send requestsWave hands inspriationally here!
https://buildAR.comThis is where the APIs are still evolving
1. Get MediaStreamMediaStream from getUserMedia or RTCPeerConnection
2. Connect to a MediaStream pipeline Canvas/Video, Image Capture, Recording, ScriptNodePorcessor or Video/Shader pipelines
3. Get ArrayBuffernew ArrayBuffer()
6. Move through Views to process datafor each (var view in scene) { process(view)... }
7. Output events and metadatareturn results
8. Update UI and/or send requestsWave hands inspriationally here!
5. Populate a scene Array with Array Buffer Viewsvar scene = [new Uint8ClampedArray(buffer), ...]
https://buildAR.comThis is where the Augmented Web comes to life
1. Get MediaStreamMediaStream from getUserMedia or RTCPeerConnection
2. Connect to a MediaStream pipeline Canvas/Video, Image Capture, Recording, ScriptNodePorcessor or Video/Shader pipelines
3. Get ArrayBuffernew ArrayBuffer()
6. Move through Views to process datafor each (var view in scene) { process(view)... }
7. Output events and metadatareturn results
8. Update UI and/or send requestsWave hands inspriationally here!
5. Populate a scene Array with Array Buffer Viewsvar scene = [new Uint8ClampedArray(buffer), ...]
https://buildAR.comThis is where the fun is!
1. Get MediaStreamMediaStream from getUserMedia or RTCPeerConnection
2. Connect to a MediaStream pipeline Canvas/Video, Image Capture, Recording, ScriptNodePorcessor or Video/Shader pipelines
3. Get ArrayBuffernew ArrayBuffer()
6. Move through Views to process datafor each (var view in scene) { process(view)... }
7. Output events and metadatareturn results
8. Update UI and/or send requestsWave hands inspriationally here!
5. Populate a scene Array with Array Buffer Viewsvar scene = [new Uint8ClampedArray(buffer), ...]
webcamtoy.com
https://buildAR.com
https://buildAR.com - image credit
http://www.flickr.com/photos/xerones/101094011/sizes/o/in/photostream/
Key Concept:
#$!%@#%@#$@$@!!!!!
https://buildAR.com
https://buildAR.comBrowser's have always had a binary barrier
The Video element
stream ->
https://buildAR.comNow we've broken through the binary barrier
The Video/Canvas pipeline
stream -> -> -> image data -> array buffer -> process -> output
https://buildAR.comNow we've broken through the binary barrier
The Video/Canvas pipeline
stream -> -> -> image data -> array buffer -> process -> output
The Mediastream Image Capture API pipeline
stream -> track -> image capture -> image data -> array buffer -> process -> output
https://buildAR.comNow we've broken through the binary barrier
The Video/Canvas pipeline
stream -> -> -> image data -> array buffer -> process -> output
The Mediastream Image Capture API pipeline
stream -> track -> image capture -> image data -> array buffer -> process -> output
The MediaStream Recording API pipeline
stream -> recorder -> blob -> file reader -> array buffer -> process -> output
https://buildAR.comNow we've broken through the binary barrier
The Video/Canvas pipeline
stream -> -> -> image data -> array buffer -> process -> output
The Mediastream Image Capture API pipeline
stream -> track -> image capture -> image data -> array buffer -> process -> output
The MediaStream Recording API pipeline
stream -> recorder -> blob -> file reader -> array buffer -> process -> output
Plus other non-MediaStream pipelines:
????? -> Web Sockets/XHR/File/JS -> array buffer -> process -> output
Now lets take a closer look at the
Video/Canvas MediaStream pipeline
https://buildAR.com
https://buildAR.comThe Video/Canvas MediaStream pipeline
Access the camera (or some other stream) sourcea. getUserMedia()
Setup element in the DOMa. declaratively then via getElementById or similarb. createElement(video) (no need to appendChild())
Pipe camera stream into a. video.src = stream
Setup element in the DOMa. declaratively then via getElementById or similarb. createElement(canvas) then appendChild()
Get 2D drawing contexta. canvas.getContext('2d');
Draw frame onto a. canvas.drawImage(video, top, left, width, height);
Get RGBA Uint8ClampedArray of the pixelsa. context.getImageData(top, left, width, height).data.buffer;
Burn CPU (not GPU) cyclesa. fore each(var obj in view) { }NOTE: Integrate other streams & sensor data here
Render resultsa. using HTML/JS/CSSb. using another and drawImage()c. using WebGLd. a combination of all
1. Get MediaStreamMediaStream from getUserMedia or RTCPeerConnection
2. Connect to a MediaStream pipeline Canvas/Video, Image Capture or Recording pipelines
3. Get ArrayBuffernew ArrayBuffer()
5. Populate a scene with Array Buffer Viewsvar scene = [new Uint8ClampedArray(buffer), ...]
6. Move through Views to process datafor each (var view in scene) { process(view)... }
7. Output events and metadatareturn results
8. Update UI and/or send requestsWave hands inspriationally here!
Follow along on github: github.com/buildar/getting_started_with_webrtc
https://buildAR.comThe Video/Canvas MediaStream pipeline
Follow along on github: github.com/buildar/getting_started_with_webrtc
Access the camera (or some other stream) sourcea. getUserMedia()
Setup element in the DOMa. declaratively then via getElementById or similarb. createElement(video) (no need to appendChild())
Pipe camera stream into a. video.src = stream
Setup element in the DOMa. declaratively then via getElementById or similarb. createElement(canvas) then appendChild()
Get 2D drawing contexta. canvas.getContext('2d');
Draw frame onto a. canvas.drawImage(video, top, left, width, height);
Get RGBA Uint8ClampedArray of the pixelsa. context.getImageData(top, left, width, height).data.buffer;
Burn CPU (not GPU) cyclesa. fore each(var obj in view) { }NOTE: Integrate other streams & sensor data here
Render resultsa. using HTML/JS/CSSb. using another and drawImage()c. using WebGLd. a combination of all
setInterval()RequestAnimationFrame()setTimeout()
Now lets compare that to theWebGLSL based Video/Shader pipeline
https://buildAR.com
https://buildAR.comThe Video/Shader MediaStream pipeline
Access the camera (or some other stream) sourcea. getUserMedia()
Setup element in the DOMa. declaratively then via getElementById or similarb. createElement(video) (no need to appendChild())
Pipe camera stream into a. video.src = stream
Compile shaders from text in elements in the DOMa. declaratively then via getElementById or similarb. createElement(canvas) then appendChild()
Get 3D drawing contexta. canvas.getContext('webgl|experimental-webgl');
Load frame into fragment shadera. gl.texImage2D(..., video)
Burn GPU cyclesa. for each(var obj in view) { }NOTE: Integrate other streams & sensor data here
Render resultsa. using HTML/JS/CSSb. using another and drawImage()c. using WebGLd. a combination of all
setInterval()RequestAnimationFrame()setTimeout()
https://buildAR.com - image credit
http://www.calwatchdog.com/wp-content/uploads/2011/04/JohnCleeseSillyWalk1.jpg
Key Concept:
ArrayBuffers and minimising copiesare the key to an efficient walk
https://buildAR.com
https://buildAR.comViews are moveable windowsinto the raw data streams
1. Get streamMediaStream from getUserMedia or RTCPeerConnection
2. Connect to a MediaStream pipeline Canvas/Video, Image Capture or Recording pipelines
3. Get ArrayBuffernew ArrayBuffer()
6. Move through Views to process datafor each (var view in scene) { process(view)... }
7. Output events and metadatareturn results
8. Update UI and/or send requestsWave hands inspriationally here!
5. Populate a scene Array with Array Buffer Viewsvar scene = [new Uint8ClampedArray(buffer), ...]
https://buildAR.com - image credit
http://www.flickr.com/photos/ekcragg/5649296082/sizes/o/in/photostream/
Key Summary:
Where is the Augmented Webup to right now?
https://buildAR.com
https://buildAR.comMobile AR vs. the Augmented Web
Mobile AR at end of 2009
Augmented Web at end of 2013
https://buildAR.comMobile AR vs. the Augmented Web
Apple had not yet opened up camera access on iOS through objective-c.See the Letter from Ori
Mobile AR at end of 2009
Augmented Web at end of 2013
https://buildAR.comMobile AR vs. the Augmented Web
Apple had not yet opened up camera access on iOS through objective-c.See the Letter from Ori
Apple has not yet opened up camera access by adopting getUserMedia()/WebRTC.See 1.0 spec within ~5mths
Mobile AR at end of 2009
Augmented Web at end of 2013
https://buildAR.comMobile AR vs. the Augmented Web
Apple had not yet opened up camera access on iOS through objective-c.See the Letter from Ori
QR codes, Geo AR and Fiducial Marker tracking were commonly available but Natural Feature Tracking was still evolving.
Apple has not yet opened up camera access by adopting getUserMedia()/WebRTC.See 1.0 spec within ~5mths
Mobile AR at end of 2009
Augmented Web at end of 2013
https://buildAR.comMobile AR vs. the Augmented Web
Apple had not yet opened up camera access on iOS through objective-c.See the Letter from Ori
QR codes, Geo AR and Fiducial Marker tracking were commonly available but Natural Feature Tracking was still evolving.
Apple has not yet opened up camera access by adopting getUserMedia()/WebRTC.See 1.0 spec within ~5mths
QR codes, Geo AR and Fiducial Marker tracking are now available and Natural Feature Tracking is quickly evolving in our R&D.
Mobile AR at end of 2009
Augmented Web at end of 2013
https://buildAR.comMobile AR vs. the Augmented Web
Apple had not yet opened up camera access on iOS through objective-c.See the Letter from Ori
QR codes, Geo AR and Fiducial Marker tracking were commonly available but Natural Feature Tracking was still evolving.
There was limited if any support for rich 3D on all the mainstream AR browsers.
Apple has not yet opened up camera access by adopting getUserMedia()/WebRTC.See 1.0 spec within ~5mths
QR codes, Geo AR and Fiducial Marker tracking are now available and Natural Feature Tracking is quickly evolving in our R&D.
Mobile AR at end of 2009
Augmented Web at end of 2013
https://buildAR.comMobile AR vs. the Augmented Web
Apple had not yet opened up camera access on iOS through objective-c.See the Letter from Ori
QR codes, Geo AR and Fiducial Marker tracking were commonly available but Natural Feature Tracking was still evolving.
There was limited if any support for rich 3D on all the mainstream AR browsers.
Apple has not yet opened up camera access by adopting getUserMedia()/WebRTC.See 1.0 spec within ~5mths
QR codes, Geo AR and Fiducial Marker tracking are now available and Natural Feature Tracking is quickly evolving in our R&D.
Rich 3D is supported in Chrome and Firefox on Android. Safari supports WebGL, and iOS devices support OpenGL, but iSafari does not have WebGL enabled
Mobile AR at end of 2009
Augmented Web at end of 2013
https://buildAR.comMobile AR vs. the Augmented Web
Apple had not yet opened up camera access on iOS through objective-c.See the Letter from Ori
QR codes, Geo AR and Fiducial Marker tracking were commonly available but Natural Feature Tracking was still evolving.
There was limited if any support for rich 3D on all the mainstream AR browsers.
Apple has not yet opened up camera access by adopting getUserMedia()/WebRTC.See 1.0 spec within ~5mths
QR codes, Geo AR and Fiducial Marker tracking are now available and Natural Feature Tracking is quickly evolving in our R&D.
Rich 3D is supported in Chrome and Firefox on Android. Safari supports WebGL, and iOS devices support OpenGL, but iSafari does not have WebGL enabled (except for advertisers!!!?).
Mobile AR at end of 2009
Augmented Web at end of 2013
https://buildAR.comWhat's in the near future?
Integrating WebRTC and Visual Search
Using WebGL/GLSL to utilise GPU parallelism
Khronos Group's OpenVX
Khronos Group's Camera Working Group
Lots more demos to share! 8)
https://buildAR.comWhat's in the near future?
Integrating WebRTC and Visual Search
Using WebGL/GLSL to utilise GPU parallelism
Khronos Group's OpenVX
Khronos Group's Camera Working Group
Lots more demos to share! 8)
https://buildAR.comWhat's in the near future?
Integrating WebRTC and Visual Search
Using WebGL/GLSL to utilise GPU parallelism
Khronos Group's OpenVX
Khronos Group's Camera Working Group
Lots more demos to share! 8)
https://buildAR.comWhat's in the near future?
Integrating WebRTC and Visual Search
Using WebGL/GLSL to utilise GPU parallelism
Khronos Group's OpenVX
Khronos Group's Camera Working Group
Lots more demos to share! 8)
https://buildAR.comJoin the W3C'sAugmented Web Community Grouphttp://w3.org/community/ar
https://buildAR.com - image credit
Web Standards for AR
An intro to the latest developmentson the Augmented Web Platform
http://www.flickr.com/photos/37380823@N03/3442372197/sizes/o/in/photostream/