1 model testing © 2015 carnegie mellon university model testing: testing executable requirements,...

1Model Testing © 2015 Carnegie Mellon University

Model Testing:Testing Executable Requirements, Architecture, and Design Models

Software Engineering InstituteCarnegie Mellon UniversityPittsburgh, PA 15213

Donald G. FiresmithPrincipal Engineer Engineer

2Model Testing

Copyright 2015 Carnegie Mellon University

This material is based upon work funded and supported by the Department of Defense under Contract No. FA8721-05-C-0003 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development center.Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the United States Department of Defense.

NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON AN “AS-IS” BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT.

This material has been approved for public release and unlimited distribution except as restricted below.

This material may be reproduced in its entirety, without modification, and freely distributed in written or electronic form without requesting formal permission. Permission is required for any other use. Requests for permission should be directed to the Software Engineering Institute at permission@sei.cmu.edu.

DM-0002518

3Model Testing

Verification and Validation Methods

Traditionally, V&V ≈ Analysis + Demonstration + “Inspection” + Test.There are actually more methods for verification and validation.Testing is similar to other dynamic V&V methods.

Verification and Validation (V&V) MethodsQuality Control (QC)

Test Evaluation OtherIn Scope

T&E

Analysis

StaticAnalysis

DynamicAnalysis

Demonstration

Desk Checking

Inspection ReviewWalk-

Through

Peer Review

Formal Review

“Inspection” Certification Reuse Warantee

Legend

Static Dynamic

4Model Testing

Model Testing

Model testing tests executable requirements, architecture, and/or design models the same way traditional testing tests software (and systems).A requirements, architecture, or design model of a system is executable if:•It describes the system in sufficient detail (e.g., components,

functions, relations, processes, and status changes) that it can be executed as a simulation of that system.

•A means exists (e.g., modeling tool, simulation engine, execution engine, or human brain) to run the model.

Model testing is NOT the same thing as Model-Based Testing (MBT)!•Model testing uses testing to uncover defects in the models.•MBT uses models to automate testing to uncover defects in the software / systems.

5Model Testing

Model-Based Shift Left Testing

System RequirementsModel Tests

Architectural Model Tests

Component Design

Model Tests

Unit Design Model Tests

User Requirements Model Tests

Production System

Subsystem Integration

Tests

Component Integration

Test

Unit Tests

System Tests

System Requirements

Models

Architectural Models

Component DesignModels

Unit Design Models

User Requirements

Models

Executable Work Products

Tests

Legend

↑ Validation↓ Verification

xMBSLx

User Needs Model Tests

User Needs Models

Operational System

Operational Tests

Units

Components

Subsystems

Integrated System

xMBSLx

xMBSLx

xMBSLx

Model-Based Shift Left (MBSL)x

MBSL

Acceptance Tests

6Model Testing

Shift Left Testing

Major trend in software testing.

Move testing earlier in development cycle (left on timeline)

Four types of shift left testing:

•Traditional Shift Left Testing

Move emphasis from system-level GUI testing via record/playback to API, component integration, and unit testing

• Incremental Shift Left Testing

Few deliveries/releases with testing of multiple large increments

•Agile/DevOps Shift Left Testing

Many deliveries/releases with testing of multiple small increments

•Model-Based Shift Left Testing ←Test executable requirements, architecture, and/or design models(move testing left to models)

7Model Testing

Model Testing

Model Testing

Requirements ModelTesting

Design Model Testing

Architecture Model Testing

Model Testing via Execution

Engine

Testing via Symbolic Execution

Model Testing via Simulation

Engine

Model Testing with Executionby Computer

Model Testing with Execution

by Human

Test Model Testing

8Model Testing

Model TestingClient Actors

(roles + systems)real or simulated

Actor 1

Actor 2

Test Inputs

Server Actors(roles + systems)real or simulated

Software Under Test

(SUT)Test Outputs

Test InputsTest Outputs

Test Outputs

Test Input(Exception)

StubTest Outputs

Actor 3

Client Actors(roles + systems)

simulated

Actor 1

Actor 2

Test Inputs

Server Actors(roles + systems)

simulated

Model Under Test

(MUT)Test Outputs

Test InputsTest Outputs

Test Outputs

Test Input(Exception)

StubTest Outputs

Actor 3

Software Testing

Model Testing

9Model Testing

Model Testing

verify the creates

the

is input for the

1

Executable Requirements, Architecture, or Design Model

Under Test (MUT)

Testeror

Test Case Generator Tool

2

Test Cases

3

Simulation Engine, Execution Engine, or

Testerexecutes the

executes the

4

10Model Testing

Testable Models – Requirements Models

Example executable models include:•ConOps storyboards (simulations)

•Use case path sequence diagrams

•Decision trees, finites state machines, and Petri nets

•Executable requirements languages

•Requirements prototypes

11Model Testing

Testable Models – Architecture/Design Models

Examples executable models include:•Architecture Analysis and Design Language (AADL) models

•Unified Modeling Language (UML) models and System Modeling Language (SysML) models:

–Communication diagrams, interaction diagrams, sequence diagrams, state machine diagrams, activity diagrams, and timing diagrams,

–Object Constraint Language (OCL) or Action Semantic Language (ASL)

•Unified Profile for DODAF and MODAF (UPDM 2.0)

•Hardware Description Languages

•Architecture prototypes

•Design models for automatic code generation