Boundary Front end tech talk: how it works

Download Boundary Front end tech talk: how it works

Post on 05-Dec-2014




0 download

Embed Size (px)




<ul><li> 1. Tech Talk - FrontendStephen Boak ( Mahoney ( King ( </li> <li> 2. When we say real-time we mean it: JSON beingpushed to the browser once a second andvisualizations built to constantly move and adaptto streaming data </li> <li> 3. Boundary | Architecture </li> <li> 4. Meters ! Lightweight and highly scalable, these sit on the cloud as well as private data centers across virtual or physical servers. </li> <li> 5. Intercept Meter data via TransportCollectors Layer Security (TLS) Authentication. </li> <li> 6. Collect all of the data at high resolution to gain rich insight into complexData Store environments &amp; problems before they can impact critical business services. </li> <li> 7. Real-time continuous data streaming atStreaming high resolution, low latency intervals. </li> <li> 8. Stream data to your dashboard with sub-second latency,Streaming UI providing you with intuitive, powerful dashboard. </li> <li> 9. How We Get Data CometD server (streaker)! Pulls in data from multiple backend services and streams it out to clients! Can also aggregate and lter data on demand </li> <li> 10. Subscriptions 1 2 3 4 5 Make a Get back a Subscribe to Receive Receivesubscription request subscription ID new query state-dump add/remove (query and lters) (Query ID) (unique channel) (inserts &amp; schema) messages !! ! ! ! ! </li> <li> 11. Data Structure Schema and key (which elds are concatenated for each record) State dump with schema for requested time window (up to 100kb state dumps) Save bandwidth with N-tuples indexed by the schema Adds/removes update the state with keys (subset of schema) </li> <li> 12. JSONExample port:protocol subscription State Object{ { channel:""/query/ac3941b8924a2f73/custom9 "1331234598000:80:6":"{ query92f536bbc93f159418b98f1a973b5dc4e78379 "epochMillis":=1331234598000, filter979564d6c94c9394f479abae906ff95a3411e9 "portProtocol":="80:6", aggregation9b5cc18d49098d949c29a08497037b08a30e6" "ingressPackets":"100, data:"{ "ingressOctets":"8463, insert:"[[1331234598000,"80:6,"100,"8463,"93," "egressPackets":"93, 9672],"[1331234925000,"8080:6,"242,"39657,"243," "egressOctets":"9672 40865],"], }, keys:"[epochMillis,"portProtocol], "1331234925000:8080:6":"{ schema:"[epochMillis,"portProtocol,"ingressPackets," "epochMillis":=1331234925000, ingressOctets,"egressPackets,"egressOctets], "portProtocol":="8080:6", timestamp:"1331235119001 "ingressPackets":"242, } "ingressOctets":"39657,} "egressPackets":"243, "egressOctets":"40865 } } </li> <li> 13. DataSource.js Negotiates between multiple data sources and subscribers on a page Subscriptions One data source for N subscribers Data Sources Abstracts CometD subscription process for JS development Subscribers Smart enough to resubscribe and noties subscribers (so that state dumps arent doubled up) </li> <li> 14. bndry.datasource repo//"Receives"object"with"properties"for"state,"inserts"and"removes."function"updateData(data)"{"" console.log(data.added);"}"//"Create"a"handle"to"a"new"data"source"var"volumeDataSource"="bndry.dataSource(volume_1s);"//"Add"a"subscriber"and"start"getting"updates."volumeDataSource.addSubscriber(updateData); </li> <li> 15. Demo Time </li> <li> 16. Data and Subcription Problems Monolithic, Multi-Purpose Queries Large State Dumps cause CometD timeout (5-10MBs per second) No Resubscription Aggregated all data on front-end </li> <li> 17. Data and Subcription Solutions Stratied, Filterable Queries and server-side aggregation Top-N Limitations (constrained output) Resubscription &amp; ltering Web workers havent helped much Time smoothing (1-second ticks) </li> <li> 18. VisualizationDOM manipulation is expensive is tweening SVG Canvas </li> <li> 19. Whats Next WebSockets Historical Data and long term data storage HTML5 local storage (store data) Machine-learning for real-time network visualization </li> <li> 20. Thank You </li> </ul>