jooink - devfest piemonte 2013
DESCRIPTION
WebRTC, and most of HTML5 tecnologies as File API, WebGL, WebWorkers, ...,answer to a common problem, i.e. to make 'in browser' apps able to do operations impossible before, as access to the video and audio stream by a javascript api.TRANSCRIPT
![Page 1: JooinK - DevFest Piemonte 2013](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554b9cf2b4c905b8618b469f/html5/thumbnails/1.jpg)
WebRTC... GWT & Browser Performance
Alberto Mancini, Francesca TosiJooinK.com
![Page 2: JooinK - DevFest Piemonte 2013](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554b9cf2b4c905b8618b469f/html5/thumbnails/2.jpg)
WebRTCPlug-in free realtime communication …
WebRTC is a free, open project that enables web browsers with Real-Time Communications (RTC) capabilities via simple Javascript APIs.
source: webrtc.org
![Page 3: JooinK - DevFest Piemonte 2013](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554b9cf2b4c905b8618b469f/html5/thumbnails/3.jpg)
WebRTC
Just another JS API
or
WebRTC is a new front in the long war for an open and unencumbered webBrendan Eich (Mozilla CTO and inventor of JavaScript)
?
![Page 4: JooinK - DevFest Piemonte 2013](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554b9cf2b4c905b8618b469f/html5/thumbnails/4.jpg)
Browser PiercingOnce upon a time ...
LDP: Firewall Piercing mini-HOWTOFrançois-René Rideauv0.97, 24 November 2001
![Page 5: JooinK - DevFest Piemonte 2013](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554b9cf2b4c905b8618b469f/html5/thumbnails/5.jpg)
Browser Piercing
a Javascript Application cannot
➔ access the filesystem and then File API
➔ open full-duplex socket connection and then WebSockets
![Page 6: JooinK - DevFest Piemonte 2013](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554b9cf2b4c905b8618b469f/html5/thumbnails/6.jpg)
Browser Piercing
a Javascript Application cannot
➔ use graphics accelerator and then WebGL
➔ span multiple threads and then WebWorkers
![Page 7: JooinK - DevFest Piemonte 2013](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554b9cf2b4c905b8618b469f/html5/thumbnails/7.jpg)
Browser Piercing
a Javascript Application cannot
➔ acquire audio and video and then WebRTC
➔ communicate P2P and then WebRTC
![Page 8: JooinK - DevFest Piemonte 2013](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554b9cf2b4c905b8618b469f/html5/thumbnails/8.jpg)
WebRTC across platforms● Chrome (mobile too)● Firefox (mobile too)● Opera● Native Java and Obj-C
![Page 9: JooinK - DevFest Piemonte 2013](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554b9cf2b4c905b8618b469f/html5/thumbnails/9.jpg)
WebRTC JS-APIAcquiring video and audio
navigator.getUserMedia(constraints,
successCallback,
errorCallback);
![Page 10: JooinK - DevFest Piemonte 2013](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554b9cf2b4c905b8618b469f/html5/thumbnails/10.jpg)
WebRTC JS-APIAcquiring video and audio
navigator.getUserMedia(constraints, successCallback,
errorCallback);
![Page 11: JooinK - DevFest Piemonte 2013](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554b9cf2b4c905b8618b469f/html5/thumbnails/11.jpg)
WebRTC JS-APIConstraints
● Controls the contents of the MediaStream● Media type, resolution, frame rate
var constraints = {video: true};
video: { mandatory: { ... }, optional [{... }]}
![Page 12: JooinK - DevFest Piemonte 2013](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554b9cf2b4c905b8618b469f/html5/thumbnails/12.jpg)
WebRTC JS-APIConstraints
● Controls the contents of the MediaStream● Media type, resolution, frame rate
var constraints = {video: true};
video: { mandatory: { chromeMediaSource: ‘screen’ }, optional [{... }]}
![Page 13: JooinK - DevFest Piemonte 2013](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554b9cf2b4c905b8618b469f/html5/thumbnails/13.jpg)
WebRTC JS-APIAcquiring video and audio
navigator.getUserMedia(constraints,
successCallback,
errorCallback);
![Page 14: JooinK - DevFest Piemonte 2013](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554b9cf2b4c905b8618b469f/html5/thumbnails/14.jpg)
WebRTC JS-APIerrorCallback
function errorCallback(error) { console.log("error: ", error);}
;-)
![Page 15: JooinK - DevFest Piemonte 2013](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554b9cf2b4c905b8618b469f/html5/thumbnails/15.jpg)
WebRTC JS-APIAcquiring video and audio
navigator.getUserMedia(constraints,
successCallback,errorCallback);
![Page 16: JooinK - DevFest Piemonte 2013](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554b9cf2b4c905b8618b469f/html5/thumbnails/16.jpg)
WebRTC JS-APIsuccessCallback
function successCallback(stream) { ...}
stream: MediaStream; a flux of audio- or video-related data.
![Page 17: JooinK - DevFest Piemonte 2013](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554b9cf2b4c905b8618b469f/html5/thumbnails/17.jpg)
WebRTC JS-APIsuccessCallback
function successCallback(stream) { var video = ... video.src =
window.URL.createObjectURL(stream);
}
(W3C File API)
![Page 18: JooinK - DevFest Piemonte 2013](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554b9cf2b4c905b8618b469f/html5/thumbnails/18.jpg)
WebRTC JS-APIfull sample
var constraints = {video: true};
function successCallback(stream) { var video = document.querySelector("video"); video.src = window.URL.createObjectURL(stream);}
function errorCallback(error) { console.log("navigator.getUserMedia error: ", error);}
navigator.getUserMedia(constraints, successCallback, errorCallback);
![Page 19: JooinK - DevFest Piemonte 2013](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554b9cf2b4c905b8618b469f/html5/thumbnails/19.jpg)
WebRTC JS-APIDo not forget
● to play the video
● to polyfill getUserMedia (moz*,webkit*,ms*)
![Page 20: JooinK - DevFest Piemonte 2013](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554b9cf2b4c905b8618b469f/html5/thumbnails/20.jpg)
WebRTC JS-APIRTCPeerConnection
→getUserMedia
+RTCPeerConnection
←
![Page 21: JooinK - DevFest Piemonte 2013](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554b9cf2b4c905b8618b469f/html5/thumbnails/21.jpg)
WebRTC JS-APIRTCDataChannel
Bidirectional communication of arbitrary data
● low latency● API similar to WebSockets
![Page 22: JooinK - DevFest Piemonte 2013](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554b9cf2b4c905b8618b469f/html5/thumbnails/22.jpg)
RTCPeerConnection samplepc = new RTCPeerConnection(null);pc.onaddstream = gotRemoteStream;pc.addStream(localStream);pc.createOffer(gotOffer);
function gotOffer(desc) { pc.setLocalDescription(desc); sendOffer(desc);}
function gotAnswer(desc) { pc.setRemoteDescription(desc);}
function gotRemoteStream(e) { attachMediaStream(remoteVideo, e.stream);}
![Page 23: JooinK - DevFest Piemonte 2013](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554b9cf2b4c905b8618b469f/html5/thumbnails/23.jpg)
RTCPeerConnection
![Page 24: JooinK - DevFest Piemonte 2013](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554b9cf2b4c905b8618b469f/html5/thumbnails/24.jpg)
RTCPeerConnection
.
![Page 25: JooinK - DevFest Piemonte 2013](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554b9cf2b4c905b8618b469f/html5/thumbnails/25.jpg)
RTCPeerConnection
.
![Page 26: JooinK - DevFest Piemonte 2013](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554b9cf2b4c905b8618b469f/html5/thumbnails/26.jpg)
STUN
.
![Page 27: JooinK - DevFest Piemonte 2013](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554b9cf2b4c905b8618b469f/html5/thumbnails/27.jpg)
TURN
.
![Page 28: JooinK - DevFest Piemonte 2013](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554b9cf2b4c905b8618b469f/html5/thumbnails/28.jpg)
Yes BUTVideoChat already exists and we are too late to build skype without skype ...
● Video chat:○ SimpleWebRTC○ easyRTC○ webRTC.io
● Peer-to-peer data:○ PeerJS○ Sharefest
![Page 29: JooinK - DevFest Piemonte 2013](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554b9cf2b4c905b8618b469f/html5/thumbnails/29.jpg)
Yes BUT
● getUserMedia: gives us real time stream from the cam
● we can draw the <video/> into a <canvas/>
● from the canvas we can extract pixel data getImageData()
and then elaborate.
![Page 30: JooinK - DevFest Piemonte 2013](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554b9cf2b4c905b8618b469f/html5/thumbnails/30.jpg)
Computing (GWT 4 us)goals- Almost native performance- Manage complex algorithms and applications
![Page 31: JooinK - DevFest Piemonte 2013](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554b9cf2b4c905b8618b469f/html5/thumbnails/31.jpg)
Computing (GWT 4 us)goals- Almost native performance- Manage complex algorithms and applications
our approach- javascript as a target language (GWT) - hand written optimized pieces of code
![Page 32: JooinK - DevFest Piemonte 2013](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554b9cf2b4c905b8618b469f/html5/thumbnails/32.jpg)
JooinK’s WebRTC sample
Sample
http://picshare.jooink.com
Demo …
![Page 33: JooinK - DevFest Piemonte 2013](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554b9cf2b4c905b8618b469f/html5/thumbnails/33.jpg)
PicShareTecnologies
● WebGL for 3D graphics● WebRTC for MediaSteams● ImgUR for pictures store● AppEngine as a glue
![Page 34: JooinK - DevFest Piemonte 2013](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554b9cf2b4c905b8618b469f/html5/thumbnails/34.jpg)
FOSS
gwt-nyartoolkit.googlecode.com
gwt-webgl.googlecode.com
The strategy: compile a java-lib with GWT into javascript
![Page 35: JooinK - DevFest Piemonte 2013](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554b9cf2b4c905b8618b469f/html5/thumbnails/35.jpg)
What’s nextChallanges
● real-time on mobile apps● real world recognition: computer vision
(BoofCV)
![Page 36: JooinK - DevFest Piemonte 2013](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554b9cf2b4c905b8618b469f/html5/thumbnails/36.jpg)
What’s nextOur approach
● use TypedArrays everywhere ● offload computation to the graphic
accelerator through WebGL● help the JIT compiler/optimizer by asm.js
![Page 38: JooinK - DevFest Piemonte 2013](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554b9cf2b4c905b8618b469f/html5/thumbnails/38.jpg)
Markers