SSRR 2019 November 19, 2019 1
Digital Engineering Metrics
Sponsor: OUSD(R&E)By
Tom McDermott, Nicole Hutchison, Mark Blackburn, Annie Yu, Neal Chen (Stevens)Eileen Van Aken, Alejandro Saledo, Kaitlin Henderson (Va Tech)
11th Annual SERC Sponsor Research ReviewNovember 19, 2019
FHI 360 CONFERENCE CENTER1825 Connecticut Avenue NW, 8th Floor
Washington, DC 20009
www.sercuarc.org
SSRR 2019 November 19, 2019 2
• 2018: SERC Project RT-182 conceptually modeled the 5 goals of the DoD DE Strategy to identify necessary acquisition enterprise changes
• 2019: Addressing multiple OUSD/RE research priorities: ―DE Metrics (WRT-1001), determine critical ROI measures and improved SE
value indicators―Model Curation
(WRT-1009), curationpractices, enablers,and technicalinnovation opportunities
―DE Workforce(WRT-1006), DEcompetency modelfor DAU
―DE Policy – buildinga model using DocGen
Presented at inaugural DEIXWG meeting, INCOSE IS2018
Change the nature of AoAsChange how
we curate data and models
Change what Systems
Engineers do
Change Reqs
process
Enterprise Modeling of the DoD Digital Information Exchange Process
SSRR 2019 November 19, 2019 3
WRT-1001 Digital Engineering Metrics Project Guiding Questions
• If you had a “Program Office Guide to Successful DE Transition” what would that look like?―Extend previous SERC work on DE enterprise transformation to the
program office level.• How can the value and effectiveness of DE be described and measured?―Determine appropriate metrics for evaluating the benefits of DE
transformation.• Are there game-changing methods and/or technologies that would
make a difference?―Analyze the DE Innovation System (methods, processes, and tools) to
identify gaps and challenges and potential paths for innovation.• Can we describe an organizational performance model for DE
transformation?―Generalize the data and results.
SSRR 2019 November 19, 2019 4
WRT-1001 DE Metrics Project Activities
Horizon Scanning
•Digital Business Trends
•DevOps trends•Technologies &
Metrics
Innovation System Analysis
•Problem Space definition
•Innovation Areas•Innovation
Enablers/Barriers
RT-182 Results
•Initial enterprise transition model
•Acquisition Outcomes Focus
•Candidate Metrics
Literature Review
•Published DE/MBSE value –> benefit –> gain
•Candidate Metrics
DE/MBSE Maturity Survey
•Assessment of relative maturity -> value -> benefits
•Candidate Metrics
DoD Workshops
•DoD Program Office Focus
•Evaluation of Metrics
Enterprise Performance
Analysis
Program Success
Guidance
SSRR 2019 November 19, 2019 5
DoD Digital Engineering Strategy
STRATEGIC MANAGEMENT SYSTEMDigital Engineering (DE) Vision: Modernize how the Department designs, develops, delivers, operates, and sustains systems. DE Mission: Securely and safely connect people, processes, data, and capabilities across an end-to-end digital enterprise.
WAY
S /I
NIT
IATI
VES
END
S END
SW
AYS / INITIATIVES
MEA
NS M
EANS
An enduring, authoritative
source of truth is used over the
lifecycle
Plan and develop the authoritative source
of truth
Govern the authoritative source
of truth
Use technological innovation to
improve engineering practices
Infuse technological innovations to enable the end-to-end digital
enterprise
Advance human-machine interactions
Make use of data to improve awareness,
insights, and decision making
Models are used to inform enterprise
and program decision making
Formalize the planning for models to support engineering activities and decision making across the lifecycle
Use the authoritative source of truth across
the lifecycle
Use models to support engineering activities and decision making across the lifecycle
Improve the digital engineering
knowledge base
Transform culture and workforce
engineering across the lifecycle
Build and prepare the workforce
Lead and support digital engineering
transformation efforts
Process, Methods, Tools, Technology, People
Infrastructure and environments support improved
communication and collaboration
Develop, mature, and use digital
engineering IT infrastructures
Secure IT infrastructure and protect intellectual
property
Develop, mature, and use digital
engineering methodologies
Formally develop, integrate,
and curate models
SSRR 2019 November 19, 2019 6
Focus: Enterprise Transformation Measures
• Gartner: “Select just 5 to 9 metrics to track, report and act on. The value of a metric lies in its ability to influence business decision making.”
• The best metrics:―Have a defined and defensible causal relationship to a business outcome―Work as a leading, not lagging, indicator―Address a specific defined audience – in a way they can understand―Drive action when they change from green to yellow to red
• There are no universal metrics – must be enterprise specific• Good Digital Transformation metrics have some traits―Measure people adoption, and enterprise process adoption―Analyze breadth of usability, and issues with usability―Measure productivity indicators―Generate new value to the enterprise (revenue, operational efficiency, etc.)
https://www.gartner.com/smarterwithgartner/how-to-measure-digital-transformation-progress/
SSRR 2019 November 19, 2019 7
Agile/DevOps and Digital Business
Agile/DevOps
Digital Business
Horizon Scanning
Traction:• Adoption/Scale• Engagement• Active Usage• Productivity
• Talent & Skills
Finance:• Change in Business or
Operating Model• New Value
• New revenues
Time to Market:• Frequency• Lead time
• Deployment time• Customer tickets
• Volume
Predictability:• Escapes
• Failed Deployments• Error rates
• Costs
User Experience:• Features
• System usage• Limitations
• Change detection• Availability
• Recovery time• Security
SSRR 2019 November 19, 2019 8
DE Success Measures from RT-182
Quality:• Defects/Design Escapes• AoA coverage• Design space explored• SE rigor• CM
Knowledge Transfer:• Data/Model reuse• Link to Mission Eng• Depth of review• Expanded Visualization• Innovation
An enduring, authoritative
source of truth is used over the
lifecycle
Use technological innovation to
improve engineering
practices
Models are used to inform enterprise
and program decision making
Transform culture and workforce
engineering across the lifecycle
Infrastructure and environments support improved
communication and collaboration
Adoption:• Pace of adoption• Infrastructure investment• Enterprise process & tool
integration• Tool/model interoperability• Role/Skill transition
Velocity/Agility:• Data/model reuse• Decision times• Cycle time/Agility• Data search time• Standards
User Experience:• Collaboration• Automation• Interoperability
RT-182 Results
SSRR 2019 November 19, 2019 9
• Reduce defects & design escapes• More robust, SoS-based AoAs• Increase design space exploration• Improved SE rigor & maturity• SE has more time for quality control• Comprehensive, effective CM• Increase speed of finding & using data• Improve agility• More rapid program decision cycles• Data & model reuse• Incorporation of standards • Improved collaboration across
disciplines & locations• Improved reasoning/inference – leads
to better automation• Improve modularity and interoperability
• Manage Complexity – knowledge sharing & transfer
• Repeatable link to mission level simulation, capabilities
• Expanded visualizations for pattern analysis & decision making
• Amount of system reviewed in SETR process
• Enable innovation• Pace of DE/MBSE adoption• Infrastructure investment• Enterprise process & tool integration• Tool/model interoperability successes• Replace tech writers with modelers
Outcome driven metrics from RT-217RT-182 Results
SSRR 2019 November 19, 2019 10
Initial Literature Review Results
• Searched 20 journals and conference proceedings for any paper that mentions Model-Based Systems Engineering. Identified papers that mention a benefit of MBSE and what the source of that benefit was: measured gains, observed gains, perceived gains (no source for benefit), reference ―Total Papers that mention MBSE: 852oPapers that mention benefits: 361―Measured gains: 3―Observed gains: 27―Perceived gains: 236―Reference: 114―Misc.: 2
*Kaitlin Henderson (VT) PhD studies
Literature Review
SSRR 2019 November 19, 2019 11
MBSE Benchmarking Survey
Objective • Assess value and effectiveness of MBSE adoption for improving business outcomes (gov’t, industry) – benefits vs. traditional methods. Develop a profile of MBSE use and meeting expectations across the life cycle.
• Where are we as organizations, and as an industry? Building models, or using models? Applying what we learn.
• Enable adopters to conduct a qualitative or quantitative assessment of their progress against MBSE best practices and guidance on developing an improvement roadmap
Method • Conduct an industry survey of MBSE capability. Align with INCOSE draft DE Capabilities Definition matrix.
• Characterizing MBSE practices, capability, value, benefits. • Probe alignment and integration with other adopter initiatives (e.g., PLM, DevOps,
cross-discipline)• Collect and share best practices and assets on MBSE benefits/value from community
Organizational Involvement
• Participation call through industry associations: INCOSE (lead), NDIA, …• Government sponsorship and support: DoD (OUSD R&E), FFRDCs (SERC)• Survey administration by DoD SERC (Stevens Institute) - “honest broker” to protect
proprietary data.
Schedule • Survey: define (Sep-Oct); instrument (Oct-Nov); distribute (Dec-Jan); analyze (Feb-Mar)
Core Team • INCOSE: Garry Roedler; Troy Peterson• NDIA: M&S Committee (Chris Schreiber); SE Division (Joe Elm, Geoff Draper; Garry
Roedler)• SERC: Tom McDermott, Nicole Hutchinson
DE/MBSE Maturity Survey
SSRR 2019 November 19, 2019 12
MBSE Survey OverviewTopics Summary of Survey Questions7. Model Sharing and Reuse
19. Teams establish, share, reuse org model libraries20. Org interface around models for stakeholder use21. Shared models used to consistently manage
programs across lifecycle22. Q: org implementation for data/model discovery,
reuse?
8. Modeling Environments
23. Modeling environment security24. Modeling environment protects IP25. Cross-discipline processes for tools, data
interoperability26. Q: value from collaborating on models across
disciplines
9. Organizational Implementation
27. Q: most challenging org obstacles for MBSE?28. Q: Best organizational enablers for MBSE?29. Q: Biggest changes our org needs forMBSE?
10. Workforce 30. Organization defined critical roles to support MBSE31. Q: Top MBSE roles in your organization?32. Org staffing adequate to fill MBSE-related roles?
11. MBSE Skills 33. Defined critical skills for MBSE34. Q: The most critical skills for MBSE?
12. Demographics
Organizational size, domain, MBSE experience
Topics Summary of Survey Questions1. MBSE Usage
1. MBSE strategy documented at enterprise level2. MBSE processes & tools integrated, inform enterprise
staff3. Q: Primary value of cross-functional MBSE integration?
2. Model Manage-ment
4. Taxonomy for modeling across organization5. Well-defined processes/tools for model management.6. Standard org guidance for model management/tools7. Q: Business value from consistent model management?
3. Technical Manage-ment
8. Modeling basis for enterprise org processes9. MBSE process support for technical reviews10. Q: Value of MBSE (or digital engrg) in technical
reviews?
4. Metrics 11. Modeling provides measurable improvement across projects
12. Consistent metrics across programs/enterprise?13. Q: Most useful metrics?
5. Model Quality
14. Defined processes/tools for V&V of models15. Defined processes/tools for data/model quality
assurance
6. Data Manage-ment
16. Org approach for data interface between tools17. Data managed independent of tools for portability18. Q: Data management roles/processes?
Survey content is derived from the draft INCOSE Digital Engineering Capabilities Definition
DE/MBSE Maturity Survey
SSRR 2019 November 19, 2019 13
Draft INCOSE DE Capabilities Definition Matrix
• Basis from The Aerospace Corporation MBSE Community Roadmap and the NASA MSFC MBSE Maturity Matrix
• Developed through a series of workshops with INCOSE and NDIA to form a proposed comprehensive Model-Based Enterprise Capability Matrix
• Will complete and release as an INCOSE DE Capabilities Document, Jan 2020Model-Based
Capability Stages Stage 0 Stage 1 Stage 2 Stage 3 Stage 4
Tools & IT Infrastructure
CollaborationE-mail, telecom.
System Model File Exchange.
Various organizations working on different parts of model. Full model integrated by a single organizations.
Partial On-line, real-time collaboration amongst distributed teams
On-line, real-time collaboration amongst distributed teams
Disparate Database/Tool interoperability None
Tool-to-Tool, ad hoc interoperability
Partial Federated Database Management System (FDBMS)
Main tools interoperable. Supporting tools interact through file transfer.
Fully Federated w/ standard "plug-and-play" interfaces. Data is interchanged among tools
Inter-Database/Tool Data Item Associations
Databases/tools are independent
Inter-Database/Tool Data Item associations defined
Inter-Database/Tool Data Item associations defined, captured, managed
Inter-Database/Tool Data Item associations among all data items defined, captured, managed, and traceable
Inter-Database/Tool Data Item associations among all data items defined, captured, managed, and traceable where changes in one data source alerts owners of other data sources of intended updates
User IF, Viewpoint/Views N/A Doc Gen UI draws from Model app
UI draws from multiple models/DBs
UI supports Interrogation; multiple configs
DE/MBSE Maturity Survey
SSRR 2019 November 19, 2019 14
• DE incorporated into organization policy & work instructions
• Appropriate tools, environments, methods, resources available
• Organizational lexicon & taxonomies integrated into data repositories
• Model management activities in place• DE basis incorporated into SE process
descriptions for each phase• Model CM applied• Digital Threads and Digital Twin artifact
baselines maintained in process• Requirements traceable across programs at
the enterprise level• Model development practices in place• DE basis incorporated into SETR criteria,
processes, and artifacts• Have a quality and improvement program
that incorporates modeling• Have a DE-based SE metrics program
including model metrics• Model reuse standards, processes, and
activities
• Model development standards, processes, and activities
• Model assurance standards, processes, and activities
• Fully federated data & IT infrastructure• Data interchange across tools• Data change notification and traceability• Data and model libraries established and
shared• Data interrogation standards, processes, and
activities• Simulation standards, processes, and activities• Data & information supports discoverable
knowledge (data-driven decision processes)• Enterprise planning & decisions from digital
artifacts (threads and twins)• Tool research & improvement forums in place• Model exchange standards, processes, and
activities with acquirers and subcontractors• Training programs• Model development standards• Roles, competencies, and skills in place
Organizational Maturity CapabilitiesDE/MBSE Maturity Survey
SSRR 2019 November 19, 2019 15
• Requirements – growth, volatility, discovery
• System Definition Change Backlog –rate, resolution time, closure rate
• Interface Trends - discovery• Requirements Validation - completed• Requirements Verification - completed• Work Product Approval - in-work,
rework• Review Action Item Closure - burndown• Risk Exposure – number, burndown• Risk Treatment - #actions• Technology Maturity – TRLs and change• Technical Measurement - TPMs• SE Staffing & Skills – plan/actual
• Process Compliance - discrepancies• Facility & Equipment Availability• Defects/Errors – discovery, closure,
phase containment• System Affordability – cost & confidence• Architecture – base measures, maturity
by review cycle• Schedule & Cost Pressure – budget,
schedule, risks
* Effected by DE
WRT 1008 Model Curation –INCOSE SE Leading Indicators Guide
DE/MBSE Maturity Survey
SSRR 2019 November 19, 2019 16
Which of These Digital Innovations willTransform the Engineering Disciplines?
• 5G mobility – enhanced bandwidth and connectivity mobile services• Collaborative telepresence – Highly realistic, haptics enabled video conferences• AI and ML – Artificial Intelligence and Machine Learning• Immersive Realities – Human, Augmented, and/or Virtual Reality technology integration• Blockchain – Blockchain derived technologies to manage workflows• Cloud Evolution – Evolving cloud computing architectures• NL/Chatbots/social robots – true human realistic natural language interfaces • IoT – Internet of Things sensors and architectures• Low-code SW – Domain specific design languages/visual composition design methods• DevSecOps – Secure Continuous development and deployment environments• Quantum computing – Evolving non-binary computing architectures• Advanced Manufacturing – Rapid programming/realization of hardware design• DNA-based Data Storage – High speed, ultra-high capacity storage devices• Digital Identities – Computer (not human) determines identity verification
* SERC Project WRT-1001 presented to the Digital Engineering Working Group 08/2019
Horizon Scanning
SSRR 2019 November 19, 2019 17
Competing Views of Innovation System Evolution
The wave model (Dahmann, etal.)
Multilevel View of technology transitions (Geels)
Innovation System Analysis
SSRR 2019 November 19, 2019 18
GPI Cards
GPI Application GeneratorStart with a pressing challenge, then generate ideas
How might we improve tool and model interoperability?
challenge:
Innovation System Analysis
SSRR 2019 November 19, 2019 19
Summary and Completion
Horizon Scanning
•Digital Business Trends
•DevOps trends•Technologies &
Metrics
Innovation System Analysis
•Problem Space definition
•Innovation Areas•Innovation
Enablers/Barriers
RT-182 Results
•Initial enterprise transition model
•Acquisition Outcomes Focus
•Candidate Metrics
Literature Review
•Published DE/MBSE value –> benefit –> gain
•Candidate Metrics
DE/MBSE Maturity Survey
•Assessment of relative maturity -> value -> benefits
•Candidate Metrics
DoD Workshops
•DoD Program Office Focus
•Evaluation of Metrics
Enterprise Performance
Analysis
Program Success
Guidance
SSRR 2019 November 19, 2019 20
Questions?
Thank you!