heiko koziolek, decrc/i1 ladenburg, germany, 2014-11-13 6
TRANSCRIPT
© ABB Group December 3, 2014 | Slide 1
6 Years of Performance Modelingat ABB Corporate Research
Heiko Koziolek, DECRC/I1 Ladenburg, Germany, 2014-11-13
2008: Performance Modeling with PalladioOverview
1) Measure
System instrumentation
& performance tests
using load drivers
(custom tooling)
2) Model
Component-based
model with annotated
flow charts entered into
Eclipse-based tooling
3) Predict
Running
analytic solvers /
simulators, varying
model parameters to
test different situations
© ABB Group December 3, 2014 | Slide 3
Model derivation (manually)
Prediction (automatically)
2010: Q-ImPrESSModels & Tools
© ABB Group December 3, 2014 | Slide 5
SoMoX / Sissy for Reverse Engineering Component Models from C++
Windows Performance Monitor for Performance Measurement
Self-implemented C#-Client as load driver
Q-ImPreSS Workbench for Modelling (meta model similar to Palladio)
LQN solver / Palladio SimuCom for Performance Prediction
PerOpteryx for Design Space Exploration
Q-ImPress WorkbenchSoMoX
2010: Q-ImPrESSResults
© ABB Group December 3, 2014 | Slide 6
Koziolek, Schlich et al.
An industrial case study on
quality impact prediction for
evolving service-oriented
software.
In Proc. ICSE 2011 SEIP,
pp. 776-785. ACM, May 2011.
2010: Q-ImPrESSLessons Learned
Successes
Large performance model build with Q-ImPreSS tooling
Models validated through measurements (<30% error)
First experiments with design space exploration
Challenges
Not enough inputs on new ABB system available,
had to fallback to model older version
predictions for older system are not really actionable
as the older version will not be changed
Modeling tools disconnected from the tools currently
used during development (e.g., Enterprise Architect)
creating models with the tools from scratch required
high effort
Static code analysis challenged by Microsoft C++ code© ABB Group December 3, 2014 | Slide 7
2012: Performance Modeling for ABB RoboticsModels & Tools
Dynatrace for distributed performance profiling
Neoload as load driver
Palladio Workbench for modelling ‚
(all manual no static code analysis)
LQN/SimuCom for performance prediction
PerOpteryx for design space exploration© ABB Group December 3, 2014 | Slide 9
Palladio Workbench
LQN Solver
2012: Performance Modeling for ABB RoboticsResults
© ABB Group December 3, 2014 | Slide 10
Thijmen de Gooijer, Anton Jansen, Heiko Koziolek, and Anne Koziolek. An industrial case study of performance and cost
design space exploration. In Proc. 3rd Int. Conf. on Performance Engineering (ICPE'12), pp 205-216. ACM, April 2012.
2012: Performance Modeling for ABB RoboticsLessons Learned
Successes
Due to performance fixes based on the measurements
the performance could be improved by 50%
Roadmap for extending the system cost-effectively was devised
based on the models
Large-scale industrial case study on design space exploration of
a distributed, component-based system
ABB Robotics integrated Dynatrace into their development
environment
Challenges
Information extraction for the models took a long time, lots of
calibration needed, several assumptions required for models
Expensive measurement & testing tools (>30K€)
© ABB Group December 3, 2014 | Slide 11
2014: Automation CloudModels & Tools
Amazon Web Services / Own Cloud Server as test
environment (up to 36 AWS m1.large instances)
KairosDB, OpenTSDB, Databus as time-series databases
Apache Cassandra and Hbase as distributed DBMS
Netflix Priam / Apache Whirr for quick deployment
Visual Studio Ultimate Web Load Test as load driver
[No modeling, only benchmarks!]© ABB Group December 3, 2014 | Slide 13
2014: Automation CloudResults
© ABB Group December 3, 2014 | Slide 14
Limited overload WITH
AWS Autoscaling
Linear scalability
for KairosDB
Avg. Roundtrip Time: 193ms
for 15 customers
ABB Phasor Measurement Unit
used in Power Grids
ABB Smart Meter
Thomas Goldschmidt, Anton Jansen, Heiko Koziolek, Jens Doppelhamer, and Hongyu Pei-Breivold. Scalability
and Robustness of Time-Series Databases for Cloud-Native Monitoring of Industrial Processes. In Proceedings
7th IEEE Int. Conf. on Cloud Computing (IEEE CLOUD 2014) Industry Track. IEEE, July 2014.
2014: Automation CloudLessons Learned
Successes
Showed technical feasibility for several scenarios from
industrial automation in a cloud computing environment
Created benchmarks for time-series databases based
on realistic workloads from ABB products
Created elasticity metrics and benchmark
Challenges
Better testing needed to improve robustness
Component-based model did not well fit with databases
/ cloud platform (e.g., auto-scaling?)
Limited insights expected from modeling due to focus
on initial technical feasibility
© ABB Group December 3, 2014 | Slide 15
2015 Outlook: Collaboration with Uni WürzburgAutomatic Construction of Architectural Perf. Models
© ABB Group December 3, 2014 | Slide 16
Kieker + C# adapter / JNBridge for distributed profiling
LibReDe for resource demand estimation
.NET Bookstore / Pet Shop (C#) for testing, later ABB system
Palladio / Descartes / PerOpteryx for modeling / prediction
6 Years of Performance Modeling at ABBConclusions
Performance modeling has matured over the last 6 years
But: to get wider adoption
lower costs and higher benefits
are required.
© ABB Group December 3, 2014 | Slide 17
6 Years of Performance Modeling at ABBFuture Work
Future work for lower costs
Better integration between measurement and modeling tools
Faster modeling via more convenient software tools
Faster modeling via reusable model libraries
Future work for higher benefits
More performance questions to be answered
Decision support and incorporation of heuristics
Better integration into existing development processes & tools
© ABB Group December 3, 2014 | Slide 18