transport for london - using data to keep london moving
TRANSCRIPT
Transport for LondonUsing data to keep London moving
Sriskandarajah SuhothayanAssociate Director / ArchitectWSO2
OVERVIEW
• Based on the presentation presented at– Strata Data Conference, London, 2017.
• Co-presented with :
– Roland MajorEnterprise ArchitectFormerly Transport of London
2
• Introduction of Transport for London• Surface Intelligent Transport System• Conceptual Architecture
• Introduction to WSO2 technologies• Data in Motion - Hack Week
• ‘Live Journey Planner’ prototype• How It’s Done?
• Data Driven Operational Applications• Game Changing Visualizations• Learning Outcomes• What's next in WSO2 Analytics
AGENDA
• 30 million journeys daily• In addition to all road and rail
transport, they look after rivers, assisted travel, taxi and private hireregulation
• They also do much more fromeducation to running the world’slargest out of home advertising estateof its type
• They are 150 years old and a chock fullof heritage and design assets
BREADTH OF TRANSPORT FOR LONDON
SponsorshipOwn the improvement strategy for the Transport for London Road NetworkEngage with a wide range of customers and external stakeholdersDeliver user benefits that are clearly defined and measured
Outcome DeliveryHighways design and engineeringTraffic modelling capabilityIntelligent control of traffic signalsMonitor, analyse and optimise road network performance
Operations24/7 Operation to keep London movingReal-time incident management through LSTCCAssess and coordinate works and schemes to minimise disruption on TfL’s roads
ROAD SPACE MANAGEMENT
SITS will
• provide the capability to unlock significant additional effective capacity onthe road network for the future
• enable and support delivery of a multi-modal approach to transportmanagement by using and allocating existing and new capacity
• enable and support delivery of a Balanced Scorecard approach to transportmanagement by using and allocating existing and new capacity based onlocal modal demands
SURFACE INTELLIGENT TRANSPORT SYSTEM
Key CharacteristicsBringing information and insightcloser to decisions
Locale, map based UITimely, event driven not batchTrusted, consistent and accurate
Public cloud hostedReusable commodity platformsOpen StandardsMiddleware Platform - WSO2
CONCEPTUALARCHITECTURE
Integration
Data Hub
Collaboration and Innovation
Analytics and Visualisation
Data DrivenOperationalApplications
Secure Enclave
• Enabler for Digital Transformation
• 100% Open Source Middleware Platform
• Offices in : Mountain View, New York, London, Sao Paolo, Colombo
• 350+ Customers
• 450 People, 300 Engineers
WSO2
OPEN TECHNOLOGY FOR AGILE DIGITAL BUSINESS
100% open source
September 26-29, 2016Objective : Managing the Capacity of London’s TransportNetwork
• Maximizing capacity on the public transportnetwork
• Maximizing capacity on the roads network• Improving air quality
Datasets• TfL APIs (Realtime and Historical)• SCOOT sensor reading (Realtime)• Passenger flow (Historical)• Air quality (from KCL) (Historical)
Solution• ‘Live Journey Planner’ Prototype
DATA IN MOTION - HACK WEEK
TfL has about 14,000 sensors measuring roads approaching junctionsThis data is currently used by the Real Time control to manage optimizationRequirements to process
780 Million events per dayLatency to data center circa 1 SecondAt data capturing resolution of 250ms scans
TRAFFIC CONTROL SENSORS
JunctionSCOOT Sensor
CHANCE OF GET A SEAT IN THE TRAIN?
Summarized Oyster Card Data
REALTIME TRAFFIC
High Traffic
Low Traffic
Collect SCOOT DataLearning traffic patterns using R
Building Random Forest Classification (it’s 88% accurate for this usecase) !
Exported the model as PMMLUse the model to predict traffic in realtime with WSO2 DAS
LOOKING INTO THE FUTURE
INCIDENT !
Via Local and Waze
WALKING AND CYCLING
Via Local and Waze
ARCHITECTURE
• Realtime data processing pipeline
HOW IT’S DONE ?
Detect Headway and Vehicle Length
Traffic and Flow
Calculation
Integrating Historic
Summarization
Predicting Traffic
Potential Incident Analysis
On raw SCOOT data stream
Sample stream: 1111100000111111100000
define stream ScootStream (scootId string, time long, reading int, seqId long);
from every e1=ScootStream[reading==1], e2=ScootStream[reading==0]+, e3=ScootStream[reading==1]+, e4=ScootStream[reading==0]
select e3[0].seqId - e2[0].seqId as headway, e4.seqId - e3[0].seqId as vehicleLength, ...
insert into DetectorStream;
DETECT HEADWAY AND VEHICLE LENGTH
Detect Headway and Vehicle Length
Traffic and Flow Calculation
Integrating Historic Summarization Predicting Traffic Potential Incident
Analysis
Pattern Matching
On all SCOOT Detectors in London region.
from DetectorStream[str:split(scootId, "-", 0) == ‘London’]#window.time(’1 min’)select count(*)/60 as flow,
avg(vehicleLength)/60 as traffic, 1/avg(headway) as density, ...
group by scootIdinsert into TrafficStream;
The results are mapped to links and presented via APIs for visual representation
TRAFFIC AND FLOW CALCULATION
Detect Headway and Vehicle Length
Traffic and Flow Calculation
Integrating Historic Summarization Predicting Traffic Potential Incident
Analysis
Filtering
Sliding Time Window
Group By Aggregations
Historic data analysis with Apache SparkJoining With Summarized Data@from(table=‘rdbms’, url=‘...’, ...)define table TrafficSummery (scootId string, week int, day string, traffic long);
from TrafficStream as ts join TrafficSummery as tton ts.week == tt.week and ts.day == tt.day
select ts.traffic as currentTraffic, tt.traffic as usualTraffic, ...insert into SummeryTrafficStream;
INTEGRATING WITH HISTORIC SUMMARIZATION
Detect Headway and Vehicle Length
Traffic and Flow Calculation
Integrating Historic Summarization Predicting Traffic Potential Incident
Analysis
Join
Predicting traffic in next 15 minutes
from SummeryTrafficStream#pmml:predict(’wso2das-3.1.0/marbel_model.pmml')
select * insert into PredictedTrafficStream;
PREDICTIONS
Detect Headway and Vehicle Length
Traffic and Flow Calculation
Integrating Historic Summarization Predicting Traffic Potential Incident
Analysis
Predict Function
Increasing trend in traffic hikes
from PredictedTrafficStreamselect currentTraffic - historicTraffic as currentHike,
predictedTraffic – currentTraffic as predictedHike, ...having currentTrafficHike > 0 and predictedTrafficHike > 0insert into TrafficHikeStream;
from every e1=TrafficHikeStream -> e2=TrafficHikeStream[ (e2.currentHick – e1.currentHike)*100.0 / e1.currentHike > 20 and
(e2.predictedHike – e1.predictedHike)*100.0 / e1.predictedHike > 20]with in 15 min insert into PotentialIncidentStream;
POTENTIAL INCIDENT ANALYSIS
Detect Headway and Vehicle Length
Traffic and Flow Calculation
Integrating Historic Summarization Predicting Traffic Potential Incident
Analysis
Pattern Matching
London Works 2• Central Register – a pan London system enabling visibility and management of works and
related activities in London• Traffic Management Act Notifications (TMAN) - A dedicated interface between London
boroughs and TfL enabling the balanced delivery of major schemes and works on the TLRNand SRN
• Forward Planning Tool - An advance planning tool that allows promoters to provide earlyvisibility of road and street works
Data Driven Operational Applications
GAME CHANGING VISUALIZATIONS
Situational Awareness
Situational Awareness
• Realtime analytics on data provides edge• Keep focus on usability• Use the right tool for the right task• Skills are the biggest hurdle• Bringing information sets together encourages new thinking• Using Agile approaches has transformed outcomes• Removing system fragmentation has a big impact on organization• Flattening delivery structures & small staged initiatives• Platform approach• Early indications of making data easily shared and integrated is improving
decisions
LEARNING POINTS
Available by end ofyear 2017
34
WHATS NEXT IN WSO2 ANALYTICS
• Support for Machine Learning model execution• Online machine learning, PMML, and Java models
• Support to consume and publish data to and from various transports andin multiple data formats• HTTP, Kafka, Thrift, TCP, JMS, MQTT, RabbitMQ, Email• XML, JSON, Text, CSV, Binary
• Support for connecting to various data stores• RDBMs, MongoDB, Cassandra, HBase, Solr
• Incremental long running data aggregation• Without a need of spark cluster
35
SUPPORTED FUNCTIONALITY
• Rich developer tool• With auto completion, simulation and debugging support
• System monitoring support• Business user friendly configurations• Realtime dashboard
• With gadget generation support
36
TOOLING SUPPORT
37
DEVELOPER STUDIO
38
BUSINESS RULES
39
DASHBOARD
• Scalable deployment with Apache Kafka• Exactly once data processing• Zero data loss
• Minimum high available deployment with 2 nodes• Multi data center support• Zero downtime
40
SCALABILITY AND HIGH AVAILABILITY
• Strata Talk on “Transport for London: Using data to keep London moving” https://conferences.oreilly.com/strata/strata-eu-2017/public/schedule/detail/57582
• WSO2 Analytics https://wso2.com/analytics• WSO2 Data Analytics Server https://wso2.com/analytics/features• WSO2 Stream Processor (Will be released on end of 2017)
https://github.com/wso2/product-sp/releases• WSO2 Siddhi https://wso2.github.io/siddhi/• WSO2 Support https://wso2.com/support/
41
USEFUL LINKS
THANK YOU
wso2.com