Download - Boundary Front end tech talk: how it works
Tech Talk - FrontendStephen Boak ([email protected])Mark Mahoney ([email protected])Matt King ([email protected])
When we say real-time we mean it: JSON being pushed to the browser once a second and visualizations built to constantly move and adapt to streaming data
Boundary | Architecture
Meters Lightweight and highly scalable, these sit on the cloud as well as private data centers across virtual or physical servers.
!
Collectors Intercept Meter data via Transport Layer Security (TLS) Authentication.
Data Store Collect all of the data at high resolution to gain rich insight into complex environments & problems before they can impact critical business services.
Streaming Real-time continuous data streaming at high resolution, low latency intervals.
Streaming UI Stream data to your dashboard with sub-second latency,providing you with intuitive, powerful dashboard.
How We Get Data
• CometD server (streaker)!
• Pulls in data from multiple backend services and streams it out to clients!
• Can also aggregate and filter data on demand
Subscriptions
54321
Receiveadd/remove
messages
Receivestate-dump
(inserts & schema)
Subscribe tonew query
(unique channel)
Get back a subscription ID
(Query ID)
Make asubscription request
(query and filters)
!
!
!
! !!
Data Structure
• Schema and key(which fields are concatenated for each record)
• State dump with schema for requested time window(up to 100kb state dumps)
• Save bandwidth with N-tuples indexed by the schema
• Adds/removes update the state with keys (subset of schema)
JSON
{channel:""/query/ac3941b8924a2f73/custom9query92f536bbc93f159418b98f1a973b5dc4e78379filter979564d6c94c9394f479abae906ff95a3411e9aggregation9b5cc18d49098d949c29a08497037b08a30e6"data:"{insert:"[[1331234598000,"80:6,"100,"8463,"93,"9672],"[1331234925000,"8080:6,"242,"39657,"243,"40865],"…],keys:"[epochMillis,"portProtocol],schema:"[epochMillis,"portProtocol,"ingressPackets,"ingressOctets,"egressPackets,"egressOctets],timestamp:"1331235119001
}}
{"1331234598000:80:6":"{"epochMillis":=1331234598000,"portProtocol":="80:6","ingressPackets":"100,"ingressOctets":"8463,"egressPackets":"93,"egressOctets":"9672
},"1331234925000:8080:6":"{"epochMillis":=1331234925000,"portProtocol":="8080:6","ingressPackets":"242,"ingressOctets":"39657,"egressPackets":"243,"egressOctets":"40865
}…
}
Example port:protocol subscription State Object
DataSource.js
• Negotiates between multiple data sources and subscribers on a page
• One data source for N subscribers
• Abstracts CometD subscription process for JS development
• Smart enough to resubscribe and notifies subscribers(so that state dumps aren’t doubled up)
DataSources
Subscribers
Subscriptions
bndry.datasource repo
//"Receives"object"with"properties"for"state,"inserts"and"removes."function"updateData(data)"{"" console.log(data.added);"}"
//"Create"a"handle"to"a"new"data"source"var"volumeDataSource"="bndry.dataSource('volume_1s');"
//"Add"a"subscriber"and"start"getting"updates."volumeDataSource.addSubscriber(updateData);
Demo Time
Data and Subcription Problems
• Monolithic, Multi-Purpose Queries• Large State Dumps cause CometD timeout
(5-10MBs per second)• No Resubscription• Aggregated all data on front-end
Data and Subcription Solutions
• Stratified, Filterable Queries and server-side aggregation• Top-N Limitations
(constrained output)• Resubscription & filtering• Web workers haven’t helped much• Time smoothing (1-second ticks)
Visualization
DOM manipulation is expensive ...so is tweening
CanvasSVG
What’s Next
• WebSockets
• Historical Data and long term data storage
• HTML5 local storage (store data)
• Machine-learning for real-time network visualization
Thank You
