why we model: using mbd effectively in critical...
Post on 06-Feb-2018
223 Views
Preview:
TRANSCRIPT
Why We Model: Using MBD Effectively in Critical DomainsMike WhalenProgram Director, UMSECUniversity of Minnesota
5/19/2013 Why We Model - Mike Whalen 1
AcknowledgementsAcknowledgements
� Rockwell Collins (Darren Cofer, Andrew Gacek,Steven Miller, Lucas Wagner)
� UPenn: (Insup Lee, Oleg Sokolsky)� UMN (Mats P. E. Heimdahl)� NASA Langley (Ricky Butler)� Lockheed Martin (Walter Storm, Greg Tallant,
Peter Stanfill)
5/19/2013 Why We Model - Mike Whalen 2
Note: all incorrect or controversial opinions are mine only J
Outline of PresentationOutline of Presentation
Introduction
Why use Model-Based Development?RequirementsDesignImplementation: Code GenerationVerification and Validation
Pitfalls
5/19/2013 Why We Model - Mike Whalen 3
How we Develop SoftwareHow we Develop Software
4
Concept Formation
Requirements Specification
Design
Implementation
Integration
System
Unit Test
Integration Test
System Test
Analysis
Test
Object Code
What is ModelWhat is Model--Based Development?Based Development?
Why We Model - Mike Whalen 5
SpecificationModel
Visualization PrototypingTestingAnalysis
Properties
Test oracle
Code
Code/test generation
ModelModel--Based Development ToolsBased Development Tools
� Esterel Studio and SCADE Studio from EsterelTechnologies
� Rhapsody from I-Logix� Simulink and Stateflow
from Mathworks Inc.� Rose Real-Time from
Rational� I will focus on Statecharts
and Dataflow notations.
5/19/2013 Why We Model - Mike Whalen 6
SystemSpecification/Model
How we How we WillWill Develop Develop SoftwareSoftware((iin theory)n theory)
5/19/2013 Why We Model - Mike Whalen 7
Concept Formation
Requirements
Implementation
Integration
PropertiesAnalysis
Integration Test
System Test
Specification Test
ModelModel--Based Development ExamplesBased Development ExamplesCompany Product Tools Specified & Autocoded Benefits Claimed Airbus A340 SCADE
With Code Generator
· 70% Fly-by-wire Controls · 70% Automatic Flight Controls · 50% Display Computer · 40% Warning & Maint Computer
· 20X Reduction in Errors · Reduced Time to Market
Eurocopter EC-155/135 Autopilot
SCADE With Code Generator
· 90 % of Autopilot
· 50% Reduction in Cycle Time
GE & Lockheed Martin
FADEDC Engine Controls
ADI Beacon · Not Stated
· Reduction in Errors · 50% Reduction in Cycle Time · Decreased Cost
Schneider Electric
Nuclear Power Plant Safety Control
SCADE With Code Generator
· 200,000 SLOC Auto Generated from 1,200 Design Views
· 8X Reduction in Errors while Complexity Increased 4x
US Spaceware
DCX Rocket MATRIXx · Not Stated
· 50-75% Reduction in Cost · Reduced Schedule & Risk
PSA Electrical Management System
SCADE With Code Generator
· 50% SLOC Auto Generated · 60% Reduction in Cycle Time · 5X Reduction in Errors
CSEE Transport
Subway Signaling System
SCADE With Code Generator
· 80,000 C SLOC Auto Generated · Improved Productivity from 20 to 300 SLOC/day
Honeywell Commercial Aviation Systems
Primus Epic Flight Control System
MATLAB Simulink
· 60% Automatic Flight Controls · 5X Increase in Productivity · No Coding Errors · Received FAA Certification
8Slide courtesy of Steve Miller in “Proving the Shalls” © 2006 Rockwell Collins, Inc. All rights reserved.
Does ModelDoes Model--Based Development Based Development ScaleScale??
Systems Developed Using MBD
� Flight Control
� Auto Pilot
� Fight Warning
� Cockpit Display
� Fuel Management
� Landing Gear
� Braking
� Steering
� Anti-Icing
� Electrical Load Management
9
Airbus A380Length 239 ft 6 in
Wingspan 261 ft 10 in
Maximum Takeoff Weight 1,235,000 lbs
Passengers Up to 840
Range 9,383 miles
Slide courtesy of Steve Miller in “Proving the Shalls” © 2006 Rockwell Collins, Inc. All rights reserved.
…But it is not all roses…But it is not all roses� Many MBD projects fail to meet their original
goals of cost, productivity◦ These tend not to get as much publicity!
� Clear eyed understanding of why you model and what you expect is necessary
5/19/2013 Why We Model - Mike Whalen 10
A Personal AnecdoteA Personal Anecdote� Part of two large projects using Model-Based
Development◦ Same company, similar quality developers◦ One great success� Significant cost reductions� Improvement in quality� Excellent customer satisfaction
◦ One great failure� Large cost overruns� Models considered less
useful than code� Group abandoned MBD
5/19/2013 Why We Model - Mike Whalen 11
Outline of PresentationOutline of Presentation
Introduction
Why use Model-Based Development?RequirementsDesignImplementation: Code GenerationVerification and Validation
Pitfalls
5/19/2013 Why We Model - Mike Whalen 12
What are your models What are your models for?for?� Possible to use MBD for many different purposes:� Requirements � Design� Simulation � Visualization� Testing◦ Test Generation◦ Test Oracle
� Formal Verification� Code Generation◦ Complete implementation◦ Code skeleton
� Prototyping� Communication with Customer
5/19/2013 Why We Model - Mike Whalen 13
You must understand, up front, what you expect to do with models in order to successfully adopt MBD.
Major opportunity for improvement in V&V
MBD Models as RequirementsMBD Models as Requirements
� Are MBD models requirements?
� Notations in this talk are executable; good at describing how system works
5/19/2013 Why We Model - Mike Whalen 14
� Lots of design detail� Difficult to see “full system” behavior.� Straightforward to generate code
5/19/2013 Why We Model - Mike Whalen 15
Outline of PresentationOutline of Presentation
Introduction
Why use Model-Based Development?RequirementsDesignImplementation: Code GenerationVerification and Validation
Pitfalls
5/19/2013 Why We Model - Mike Whalen 16
The Most Important Issue The Most Important Issue for for Successful Adoption of MBDSuccessful Adoption of MBD
� Block diagrams are very natural for control problems� Statecharts are very natural for description of system
modes & mode transitions� Both block diagrams and statecharts are very unnatural
for representing complex data structures� Neither notation naturally supports iteration or
recursion◦ It can be “faked”, but not well
5/19/2013 Why We Model - Mike Whalen 17
Do the Domain-Specific Notations provide a natural representation for your problem?
Just…No…Just…No…
5/19/2013 Why We Model - Mike Whalen 18
Stateflow model of Tetris game (included in the Stateflow Demo models from the Mathworks!).
Diagram is essentially a control-flow graph of a program that implements tetris.
*Much* harder to read and modify than an equivalent program.
Model © The Mathworks, 2007
ToolsTools MatterMatter� Often notations are much more cumbersome
to use than text◦ No diff / merge capabilities◦ Adding information requires many clicks
� Expressible != Easy� Anecdote: Simulink vs. SCADE at Rockwell
Collins in 2006◦ SCADE had formal pedigree, strong analysis � But tools kept crashing on our Windows boxes
◦ Simulink had better tools and better salespeople
5/19/2013 Why We Model - Mike Whalen 19
Outline of PresentationOutline of Presentation
Introduction
Why use Model-Based Development?RequirementsDesignImplementation: Code GenerationVerification and Validation
Pitfalls
5/19/2013 Why We Model - Mike Whalen 20
Analysis PyramidAnalysis Pyramid
5/19/2013 Why We Model - Mike Whalen 21
Optimistic Inaccuracy
Pessimistic Inaccuracy Simplified
Properties
Perfect VerificationExhaustive Testing
(Infinite Effort)
Typical TestingPrecise Analysis
of Simple Syntactic Properties
Simplistic Program Analysis
Pyramid Adopted from Dr. Michal Young
Theorem Proving
Model Checking Temporal PropertiesOf Finite systems.
Data Flow Analysis
MCDC Testing
What We NeedWhat We Need
5/19/2013 Why We Model - Mike Whalen 22
Optimistic Inaccuracy
Pessimistic Inaccuracy Simplified
Properties
Perfect VerificationExhaustive Testing
(Infinite Effort)
Typical TestingPrecise Analysis
of Simple Syntactic Properties
Simplistic Program Analysis
Theorem Proving
Model Checking Temporal PropertiesOf Finite Systems
Data Flow Analysis
Access to ManyTools and
Techniques
MCDC Testing
MBD Is a MBD Is a V&VV&V--Enabling Enabling TechnologyTechnology
� Strong simulation and analysis capabilities built into most tools ◦ Demo: Stateflow Elevator � (Help: Stateflow/Demos/Large-Scale Modeling/Modeling an Elevator
System)
� Even stronger simulation capabilities in external tools◦ Demo: Reactis step simulation with Microwave
� Allows straightforward “Build a little, test a little” philosophy◦ Consistent with incremental development philosophy
5/19/2013 Why We Model - Mike Whalen 23
ModelModel--Driven Test Generation (v1)Driven Test Generation (v1)
5/19/2013 Why We Model - Mike Whalen 24
Source Code
Generated Tests
while(a<0) {a=a-1;b=b*a;
}printf(“%d”,
b);
Test Case Generator
Compiler
Coverage Metric
Object Code
?Possible to generate test suites that satisfy very
rigorous structural coverage metrics
MBD Model
Model results must match source code for
tests to pass
ModelModel--Driven Test Generation (v2)Driven Test Generation (v2)
5/19/2013 Why We Model - Mike Whalen 25
MBD Model
Generated Tests
Test Case Generator
Code Generator+ Compiler
Coverage Metric
Object Code
?Model should match source code exactly
ModelModel--Driven Test Generation (v2)Driven Test Generation (v2)
5/19/2013 Why We Model - Mike Whalen 26
MBD Model
Generated Tests
Test Case Generator
Code Generator+ Compiler
Coverage Metric
Object Code
Model should match source code exactly
Oracle
Where does Oracle come from?
What is a good oracle?
Use Requirements as OracleUse Requirements as Oracle
5/19/2013 Why We Model - Mike Whalen 27Slide courtesy of Steve Miller in “Proving the Shalls” © 2006 Rockwell Collins, Inc. All rights reserved.
Static Analysis and Model CheckingStatic Analysis and Model Checking
5/19/2013 Why We Model - Mike Whalen 28
Analysis Tool
Oracle
PropertyTrue
Property False: Test Case
MBD Model
FCS 5000 Flight Control Mode LogicFCS 5000 Flight Control Mode Logic
5/19/2013 Why We Model - Mike Whalen 29
6.8 x 1021 Reachable States
Mode Controller B
Mode Controller A
Counterexample Found inLess than Two Minutes
Found 27 Errors
Example RequirementMode A1 => Mode B1
Modeled in SimulinkTranslated to NuSMV
Slide © Rockwell Collins, 2008
ADGS 2100 Adaptive Display and ADGS 2100 Adaptive Display and Guidance System Guidance System
5/19/2013 Why We Model - Mike Whalen 30
Example Requirement:Drive the Maximum Number of Display Units
Given the Available Graphics Processors
Counterexample Found in 5 Seconds
Checked 573 Properties -Found and Corrected 98 Errors
in Early Design Models
Modeled in SimulinkTranslated to NuSMV
4,295 Subsystems16,117 Simulink Blocks
Over 1037 Reachable States
Slide © Rockwell Collins, 2008
CerTACerTA FCS Phase IFCS Phase I� Sponsored by AFRL◦ Wright Patterson VA
Directorate
� Compare FM & Testing◦ Testing team & FM team
� Lockheed Martin UAV ◦ Adaptive Flight Control
System
◦ Redundancy Management Logic
◦ Modeled in Simulink
◦ Translated to NuSMVmodel checker
5/19/2013 Why We Model - Mike Whalen 31
Testing
Model-Checking 1240%
060%
ErrorsFound
Effort(% total)
Phase I Results
4input_sel
3totalizer_cnt
2persistence_cnt
1failure_report
pc
trigger
input_a
input_b
input_c
DST_index
input_sel
triplex_input_selector
input_a
input_b
input_c
trip_level
persist_lim
MS
failreport
pc
tc
triplex_input_monitor
trip_leveltrip_level1
persist_limpersistence limit
[DSTi]
[C]
[B]
[status_c]
[status_b]
[status_a]
[A]
[trigger]
[DSTi][MS]
[MS]
[DSTi][A]
[prev_sel]
[prev_sel]
[DSTi]
[trigger]
[trigger]
[status_c]
[status_b]
[status_a]
[A]
[A]
IndexVector
[C]
[B]
[C]
[B]
[C]
[B]
failure_report
dst_index
Failure_Processing
mon_failure_report
status_a
status_b
status_c
prev_sel
input_a
input_b
input_c
failure_report
Failure_Isolation
Extract Bits[0 3]
Extract Bits
DOCText
double
DST
Data StoreRead
8dst_index
7status_c
6status_b
5status_a
4input_c
3input_b
2input_a
1sync
persist_lim
totalizer_cnt<tc>
trip_level
persistence_cnt<pc>
sync<>
failreport
Slide © Rockwell Collins, 2008
MBD Formal Analysis EffortsMBD Formal Analysis Efforts
5/19/2013 Why We Model - Mike Whalen 32
Outline of PresentationOutline of Presentation
Introduction
Why use Model-Based Development?RequirementsDesignImplementation: Code GenerationVerification and Validation
Pitfalls
5/19/2013 Why We Model - Mike Whalen 33
Problem Problem 1:1:Using Models Where They Don’t FitUsing Models Where They Don’t Fit
If MBD notation doesn’t provide a better representation of your
problem than code, you’re wasting your time.
5/19/2013 Why We Model - Mike Whalen 34
5/19/2013 Why We Model - Mike Whalen 35
MBD notations can be awful programming languages…
Model © The Mathworks, 2007
RemediesRemedies� Perform honest assessment of where MBD
notations can be used◦ They do not do everything◦ Recursive data structures are especially difficult
to model.◦ Use models where they are a good
representation.� Create a partitioning strategy between
models and code for applications that contain both complex mode logic and complex data.
5/19/2013 Why We Model - Mike Whalen 36
Problem Problem 22Believing Testing Can be EliminatedBelieving Testing Can be Eliminated
Testing will always be a crucial (and costly) component
5/19/2013 Why We Model - Mike Whalen 37
SystemSpecification/Model
Testing Does not go AwayTesting Does not go Away
5/19/2013 Why We Model - Mike Whalen 38
Concept Formation
Requirements
Implementation
Integration
Properties
Extensive Testing (MC/DC)
SystemSpecification/Model
It Simply MovesIt Simply Moves
5/19/2013 Why We Model - Mike Whalen 39
Concept Formation
Requirements
Implementation
Integration
Properties
Extensive Testing (MC/DC)
SystemSpecification/Model
Do it the Right WayDo it the Right Way
5/19/2013 Why We Model - Mike Whalen 40
Concept Formation
Requirements
Implementation
Integration
Properties
Analysis
Integration Test
System Test
Specification Test
Unit Test
Problem Problem 33Believing the Model is EverythingBelieving the Model is Everything
The model is never enough
5/19/2013 Why We Model - Mike Whalen 41
Modeling is so much fun
Properties
Specification/Model
Modeling FrenzyModeling Frenzy
5/19/2013 Why We Model - Mike Whalen 42
Concept Formation
Requirements
Implementation
IntegrationHow do we know the model is “right”?
SystemTest
RemediesRemedies� Recognize the Role of Software
Requirements◦ The model is not everything
� Development Methods for Model-Based Development Badly Needed◦ Model-Based Software Development Process
� Develop Tools and Techniques for Model, Properties, and Requirements Management
� Develop Inspection Checklists and Style Guidelines for Models
5/19/2013 Why We Model - Mike Whalen 43
Problem Problem 44Trusting VerificationTrusting Verification
To really mess things up,you need formal verification
5/19/2013 Why We Model - Mike Whalen 44
Property or Model: Who is Right?Property or Model: Who is Right?
5/19/2013 Why We Model - Mike Whalen 45
AG(Onside_FD_On -> Mode_Annunciations_On)
The Mode Annunciations shall be turned onwhen the Flight Director is turned on
AG( (Is_This_Side_Active & Onside_FD_On) -> Mode_Annunciations_On)
If this side is active, the Mode Annunciations shall be turned on when the Flight Director is turned on
If this side is active and the Mode Annunciations are off, the Mode Annunciations shall be turned on when the Flight Director is turned onAG( ! Mode_Annunciations_On ->
AX ((Is_This_Side_Active & Onside_FD_On) -> Mode_Annunciations_On)))
RemediesRemedies� Develop techniques to determine adequacy of model and
property set◦ How do we know they are any “good”
� Techniques for management of invariants◦ How do we validate the assumptions we make
� Methodology and guidance badly needed ◦ Tools with training wheels◦ “Verification for Dummies”
All we need is one high-profile verified systemto fail spectacularly to set us back
a decade or more
5/19/2013 Why We Model - Mike Whalen 46
ConclusionsConclusions� MBD can significantly improve developer productivity, cost,
schedule, and quality� …or it can make your life miserable� The important thing is to know why you’re doing it!◦ Know the limitations of what can be modeled using the DSNs
◦ Know which capabilities you hope to use� Design and quality of models depends on this
� V & V receives the largest benefit of the MBD approach◦ Mature tools for test-case generation
◦ Starting to see model checking built into commercial tools: SCADE Verifier, Simulink Design Verifier
� There are many other things to discuss! Versioning, diff, semantics, tool costs, training, structuring, vendor “lock in”
5/19/2013 Why We Model - Mike Whalen 47
Questions?Questions?
5/19/2013 Why We Model - Mike Whalen 48
ReferencesReferences
5/19/2013 Why We Model - Mike Whalen 49
M. Whalen, D. Greve, L. Wagner, S. Miller, Model Checking Information Flow. In Design and Verification of Microprocessor Systems for High-Assurance Applications. D. Hardin, Ed. Springer, 2010.
M. Whalen, P. Godefroid, L. Mariani, A. Polini, N. Tillman, and W. Visser. FITE: Future Integrated Testing Environment. Workshop on the Future of Software Engineering Research 2010 (FoSER), Santa Fe, New Mexico, November 7-8, 2010.
S. Miller, M. Whalen, and D. Cofer. Software Model Checking Takes Off. Communications of the ACM, Volume 53, No 2, February 2010.
D. Hardin, T. D. Hiratzka, D. R. Johnson, L. Wagner, and M. Whalen. Development of Security Software: A High-Assurance Methodology. Proceedings of the 11th International Conference of Formal Engineering Methods (ICFEM 2009), Rio de Janeiro, Brazil, December, 2009.
M. Whalen, D. Cofer, S. Miller, B. Krogh, and W. Storm. Integration of Formal Analysis into a Model-Based Software Development Process. 12th International Workshop on Industrial Critical Systems (FMICS 2007), Berlin, Germany, July, 2007.
S. Miller, A. Tribble, M. Whalen, and M.P.E. Heimdahl. Proving the Shalls: Early Validation of Requirements through Formal Methods, Journal of Software Tools for Technology Transfer. Volume 8 Issue 4, August 2006.
M.P.E. Heimdahl, Y. Choi, and M. Whalen. Deviation Analysis: A New Use for Model Checking, Automated Software Engineering, Volume 12, Number 3, July, 2005.
M. Whalen, B. Fischer, and J. Schumann. Certifying Synthesized Code. Proceedings of Formal Methods Europe 2002, Copenhagen, Denmark, July 2002
M. Whalen, B. Fischer, and J. Schumann. AutoBayes/CC – Combining Program Synthesis with Automatic Code Certification. Proceedings of Conference on Automated Deduction 18, Copenhagen, Denmark, July 2002.
Medical Cyber-Physical SystemsResearch directions:• Medical device interoperability• High-confidence development
– Model-driven design– V&V, regulatory approval
Participants• University of Pennsylvania• U. Penn Hospital System• University of Minnesota• CIMIT/MGH
Improving patient treatment by coordinated systems of medical devices• Smart alarms and decision support• Physiological closed-loop control
Supported by NSF CNS-1035715http://rtg.cis.upenn.edu/MDCPS/
Networked Blood Glucose Control System
Safety-critical, closed-loop MCPSResearch issues:• Identifying new risks and
hazards• Mitigation strategies• Validation• Control designPursue model-driven approach
Smart alarm systems
Model driven development and assurance casesCoordination framework for medical devices• Build high-confidence middleware
– Rely on formal methods and static analysis• Design a language for executable
clinical scenarios– Specify information flows– Identify timing constraints– Ensure non-interference
• Reduction of irrelevant alarms for CABG patients– Based on aggregation of
multiple vital signs and fuzzy logic
• On-going research:– Prediction of vasospasm
in neuro-ICU patients
High-assurance development:• Modeling, code synthesis• Model-level verification,
code-level validationAssurance case constructionreflects development processstructureApplied to pacemaker, PCA pump
50Why We Model - Mike Whalen5/19/2013
top related