iv&v facility assertion–based requirements validation patrick theeke (pmp, csqe) john ryan...
DESCRIPTION
IV&V Facility 1/18/ Assertion–Based Requirements Validation Agenda Briefly describe Model-Based Requirements Validation Define the process for Assertion-Based Requirements Validation Present experiences of pilot project Present observations Discuss additional details via panelTRANSCRIPT
IV&V FacilityAssertion–BasedRequirements Validation
Patrick Theeke (PMP, CSQE)John [email protected]
September 8, 2008
IV&V Facility
05/03/23 2
Assertion–Based Requirements ValidationObjectives• Describe the process of using executable
models for requirements validation • Discuss experiences and characterize
outcomes from a pilot project employing the technique
• Identify lessons learned to date and possible next steps
IV&V Facility
05/03/23 3
Assertion–Based Requirements ValidationAgenda• Briefly describe Model-Based Requirements
Validation• Define the process for Assertion-Based
Requirements Validation• Present experiences of pilot project• Present observations• Discuss additional details via panel
IV&V Facility
05/03/23 4
Requirements ValidationGoals of Requirements Validation• Assure that the documented requirements fully specify the capabilities and
characteristics necessary for the system being defined to meet its goals• In particular, NASA IVV is interested in critical capabilities• Requirements defects include
– Missing – Superfluous– Ambiguous– Incomplete– Inconsistent– Unverifiable– Incorrect
IV&V Facility
05/03/23 5
Requirements ValidationValidation Approach• Build a standard of reference for comparison
against the documented requirements of the system of interest
• This standard is the System Reference Model (SRM), which represents IVV’s understanding of the system of interest
Verifiable RequirementsRequirements
Software RequirementsSRM Behaviors
SRM Correct?Requirements Complete?Valid?
Validated Requirements
SRM Correct?Requirements in Scope?Valid?
Correct, Complete,
Consistent, Unambiguous, and
Verifiable Requirements
IV&V Facility
05/03/23 6
Requirements ValidationSRM components• Behaviors
– Use cases and UML diagrams (e.g. activity diagrams)
• Structure– Placeholder for named physical and logical components of the system– Identify component roles and events important to understanding the
system
• Independent Modeling Assertions (IMA)– Formalized IMA extracted from the other SRM components– Collection of unit tests to validate the IMAs
• System Scenarios– Scenarios that describe various sequences of events
• Nominal sequences (what the system is supposed to do)• Unwanted behaviors (what the system is not supposed to do)• Detection and response to adverse conditions (how the systems responds to
adverse conditions)
IV&V Facility
05/03/23 7
Assertion–Based Requirements Validation Pilot Assertion Statechart Definition
• Each Statechart Assertion is a formal specification of a single requirement.
• A statechart assertion is fundamentally a monitoring device that observes system behavior and determines whether that behavior is valid
• Observed behavior is valid when it matches the behavior specification coded into the assertion, and invalid when it violates the specification
• An assertion is run against observable behavior, typically supplied by some executable artifact running under a test scenario
IV&V Facility
05/03/23 8
Requirements ValidationCorrelation-Based Requirements Validation• Capture independent understanding of system
behavior or characteristic in a SRM• Correlate project requirements with SRM behaviors• Analyze results to identify
– Ambiguous, incorrect, incomplete, inconsistent, or non-verifiable requirements that correspond to SRM behaviors
– Requirements without matching SRM behavior– SRM behavior without matching requirements
SRM
Correlationswith SRM
Project Requirements
Analyze Results
Findings
IV&V Facility
05/03/23 9
Assertion–Based Requirements ValidationMotivation• Formalization of natural language requirements reveal errors in
the specification• Leveraging formal assertions for requirements validation
increases the utility of formalized requirements– Formalization of requirements is a necessary step in leveraging the
SRM for verification
• Utilizing assertions more closely mimics the highly parallel threads of execution of real time, reactive systems, including timing and synchronization, allowing more in-depth examination of complex behavioral interactions
• Discovering requirements and representing them as assertions is more readily done early in the modeling process while the SRM is under development – so we might as well use them!
IV&V Facility
05/03/23 10
Assertion–Based Requirements ValidationDependencies• Assertion Based Validation Depends on:
– SRM was built to the appropriate level of depth and details
– The requirements were traced to the SRM– The set of system scenarios provided an appropriate
coverage of the system behaviors– The naming convention for the assertion events
match the events of the formalized unit tests, system scenarios, and assertion state charts
IV&V Facility
05/03/23 11
Assertion–Based Requirements Validation ProcessAssertion-Based Requirements Validation• Formally capture developer’s documented
requirements as Project Based Assertions (PBA)• Exercise PBAs and IMAs using validated System
Scenarios• Analyze results
SRM
Correlationswith SRM
Project Requirements
Analyze Results
FindingsDevelop & Validate PBAs
Execute Scenarios in Context of PBAs
Develop & Validate IMAs
Develop System Scenarios
IV&V Facility
05/03/23 12
Assertion–Based Requirements Validation Process Develop & Validate Project–Based Assertions • Translate the documented requirements from the development
project into a formal representation that is conducive to comparison with the SRM
• Requires detailed understanding of the requirements • Exposes errors in the requirements specifications
SRM
Correlationswith SRM
Project Requirements
Analyze Results Findings
Develop & Validate PBAs
Execute Scenarios in Context of PBAs
Identify Relevant Documented requirements
Interpret Requirements
Formalize Requirements as PBAs
Validate PBAs
IV&V Facility
05/03/23 13
Assertion–Based Requirements Validation Process Execute scenarios in Context of PBAs• We are interested in how the PBAs react to the
system scenarios– Which system scenarios ended as expected? – Which system scenarios ended differently than expected?– Which PBAs were not exercised?
SRM
Correlationswith SRM
Project Requirements
Analyze Results Findings
Develop & Validate PBAs
Execute Scenarios in Context of PBAs
Include PBAs in executable environment
Execute system scenarios
Collect results for further analysis
IV&V Facility
05/03/23 14
Assertion–Based Requirements Validation Process Analyze Results• Examine each assertion failure for:
– Errors in constructing the SRM (system scenarios, IMAs, etc.)– Errors in constructing the PBA(s)– Errors in a requirement
• Fix the errors in the SRM• Rerun test until no SRM or PBA construction errors are
discovered. Therefore all unexpected results indicate findings
SRM
Correlationswith SRM
Project Requirements
Analyze Results
FindingsDevelop & Validate PBAs
Execute Scenarios in Context of PBAs
Ensure system scenario set is correct and complete
Ensure PBA set is complete
Ensure that PBA is in scope
Ensure PBA is correct
IV&V Facility
05/03/23 15
Assertion–Based Requirements Validation PilotDevelop and Validate IMAs• Examine behavior in UML activity diagrams
– Begin with preconditions, triggers, constraints– Assert important sequences that must occur in order (under all
conditions or adverse conditions)– Constraints on looping (will continue to try until …)– Important executions if a critical behavior is unavailable– Any possible/impossible executions that will occur during off-
nominal events– Look at the whole picture and observe modeling issues as well
SRM
Correlationswith SRM
Project Requirements
Analyze Results Findings
Develop & Validate PBAs
Execute Scenarios in Context of PBAs
Develop System Scenarios
Develop & Validate IMAs
Identify Candidate IMAs
Analyze Candidate IMAs
Represent Candidate IMAs in Natural Language
Formalize IMAs as assertion state charts
Validate IMAs
IV&V Facility
05/03/23 16
Assertion–Based Requirements Validation Pilot Identify candidate IMAs
-Actively Damp Nutation[After burn maneuver, need to stabilize damping before selecting Idle Mode to downlink data]
Select Damping Mode Determine Attitude Configure Damping Filter Estimate Nutation Angle IF Angle > target: Then compute damping torque prior to controlling fired thruster burn IF Angle < target: continue process
Estimate target AngleCheck if < targetTrack time passing
Once stablized for minimal time, GNC subsytem can Select Idle Mode
IV&V Facility
05/03/23 17
Assertion–Based Requirements Validation PilotAnalyze candidate IMAs• Analysis
No te: The nutation angle must be less than or equal to target nutation angle for the minimum amount of stable time to maintain an acceptable and stable nutation angle. [P <= R and T1 >= T2 (Tmin)]
Q1. The nutation angle must (/can?) be less than or equal to target nutation angle for the minimum amount of stable time to select Idle Mode.
Domain Answer – Actually both Ground and Fault Protection Mode can select Idle Mode at any time even if the above is not true ([P <= R and T 1 >= T2 (Tmin)]).
Assumption: If there was a satellite timeout occurring, and if the fault protection mode has not been entered (due to # attempts or timeout), then the above is true.
Q1b. The Reference Orientation (GN&C sub system) must maintain a nutat ion angle that is less than or equal to the target nutation angle for a minimal amount of stable time to select Idle Mode. (See Figure 16 and 20)
Q2. What occurs if the goal is met (nutation angle stabilized and maintainable) and something causes the angle to exceed the nutation angle again?
Domain Answer – The issue is monitored and the active damp nutation behavior is called again.
IV&V Facility
05/03/23 18
Assertion–Based Requirements Validation PilotAnalyze candidate IMAs• Analyze each candidate assertion and create good and bad
scenarios to represent the requirements• Use the diagrams in the SRM to derive scenarios or unit tests that
will cause the requirement to succeed and fail• Use the scenarios to challenge the assertion against
– redundancy of events– improper sequencing– Unknowns– misuse of system boundaries– Dependencies– and constraints (i.e time)
• Determine whether or not the sequencing, repetition, looping, constraints, and concurrence always (“must”) occur this way or conditionally throughout the entire SRM
IV&V Facility
05/03/23 19
Assertion–Based Requirements Validation PilotAnalyze candidate IMAs
Good and Bad Scenarios for Q1B: Ex
ceed
Tar
get
Nuta
tion
Angl
e
Reac
h Ta
rget
Nut
atio
nAn
gle
Tim
e pa
ssed
= 2
Tim
e pa
ssed
= 3
Sele
ct Id
le M
ode
Exce
ed T
arge
t Nu
tatio
nAn
gle
Reac
h Ta
rget
Nut
atio
nAn
gle
Tim
e pa
ssed
= 4
Sele
ct Id
le M
ode
Good Scenario Bad Scenario
Timer Tmin is the minimal amount of stable time required to reach a mainta inable angle state and allow ability to select Idle Mode. Tmin = 5.
Exce
ed T
arge
t Nu
tatio
nAn
gle
Reac
h Ta
rget
Nut
atio
nAn
gle
Tim
e pa
ssed
= 4
The Reference Orientation must maintain a nutation angle that is less than or equal to the target nutation angle for a minimal amount of stable time to select Idle Mode.
IV&V Facility
05/03/23 20
Assertion–Based Requirements Validation PilotRepresent the candidate IMAs as NLR • Natural Language Requirement for Q1b
– The Reference Orientation (GN&C sub system) shall achieve a stable nutation angle that is less than or equal to an acceptable target nutation angle for a minimal amount of time prior to selecting Idle Mode.
IV&V Facility
05/03/23 21
Assertion–Based Requirements Validation Pilot Formalize each Natural Language Requirement
IV&V Facility
05/03/23 22
Assertion–Based Requirements Validation Pilot Validate each formalized IMA• Validate the IMA is unambiguous, correct, and consistent
with the SRM behavior and scenarios.– Validate each model-based formalized requirement has been modeled
correctly to represent the intended requirement
• Create formalized unit tests for the scenarios
• Validate the IMA against the formalized unit tests
public void testDampNutation_IMA001_sc001(){// this is testing a good scenario for RO selecting Idle ModeprintState(IMA001);IMA001.reachTargetNutationAngle();printState(IMA001);IMA001.incrSimTime(1);printState(IMA001);IMA001.exceedTargetNutationAngle();printState(IMA001);IMA001.reachTargetNutationAngle();printState(IMA001);IMA001.incrSimTime(5);printState(IMA001);IMA001.incrSimTime(1);printState(IMA001);IMA001.ROselectIdleMode();printState(IMA001);assertTrue(IMA001.isSuccess());System.out.println("Completed testDampNutation_IMA001_sc001");System.out.println();System.out.println("-------------------------");System.out.println();}
public void testDampNutation_IMA001_sc002(){// this is testing a b scenario for RO selecting Idle ModeprintState(IMA001);IMA001.reachTargetNutationAngle();printState(IMA001);IMA001.incrSimTime(1);printState(IMA001);IMA001.exceedTargetNutationAngle();printState(IMA001);IMA001.reachTargetNutationAngle();printState(IMA001);IMA001.incrSimTime(4);printState(IMA001);IMA001.ROselectIdleMode();printState(IMA001);assertFalse(IMA001.isSuccess());System.out.println("CompletedtestDampNutation_IMA001_sc002");}
IV&V Facility
05/03/23 23
Assertion–Based Requirements Validation PilotDevelop & Validate Project–Based Assertions • Translate the documented requirements from the development
project into a formal representation that is conducive to comparison with the SRM
• Requires detailed understanding of the requirements • Exposes errors in the requirements specifications
SRM
Correlationswith SRM
Project Requirements
Analyze Results Findings
Develop & Validate PBAs
Execute Scenarios in Context of PBAs
Develop & Validate IMAs
Develop System Scenarios
Identify Relevant Documented requirements
Interpret Requirements
Formalize Requirements as PBAs
Validate PBAs
IV&V Facility
05/03/23 24
Assertion–Based Requirements Validation PilotDevelop & Validate Project–Based Assertions
IV&V Facility
05/03/23 25
public void testTorqueVector_IMA001(){printState(pba001);pba001.beginJOIPhase();printState(pba001);pba001.SRUUpdate();printState(pba001);pba001.incrSimTime(4);pba001.beginFireThrusters();printState(pba001);pba001.endFireThrusters();printState(pba001);
assertTrue(pba001.isSuccess());System.out.println("Completed testTorqueVector_pba001");System.out.println();System.out.println("-------------------------");System.out.println();
}
Assertion–Based Requirements Validation PilotValidate each formalized PBAValidate the PBA is unambiguous, correct, and consistent.
Validate each project-based formalized requirement has been modeled correctly to represent the intended requirement
Create formalized unit tests for the scenarios
IV&V Facility
05/03/23 26
Assertion–Based Requirements Validation PilotAdd system scenarios to SRM• Derive system scenarios
– Nominal, expected behavioral sequences– Behavioral sequences that should not be permitted– Detection and responses to adverse conditions
SRM
Correlationswith SRM
Project Requirements
Analyze Results Findings
Develop & Validate PBAs
Execute Scenarios in Context of PBAs
Develop & Validate IMAs
Develop System Scenarios
Identify events
Define system scenarios
Implement system scenarios
Validate system scenarios
IV&V Facility
05/03/23 27
Assertion–Based Requirements Validation PilotIdentify Events• Identify events in the activity diagrams
– Activities model the control of execution of one or more actions. Activities do not model sequences of event occurrences.
– Activities do not often model timing constraints– Action names and control flow guards are defined in
natural language and thus prone to ambiguity
IV&V Facility
05/03/23 28
Assertion–Based Requirements Validation PilotDefine System Scenarios• “Good” and “Bad” Scenarios
– Used the semantics of UML sequence diagrams to define a scenario, by modeling sequence of event occurrences (i.e. action execution start/finish events)
– Used guard conditions in activity diagrams that should not hold true in the context of the goal as means of creating “bad scenarios”
IV&V Facility
05/03/23 29
Assertion–Based Requirements Validation PilotImplement System Scenarios• Scenarios are implemented in J-Unit and
reference events in assertions
IV&V Facility
05/03/23 30
Assertion–Based Requirements Validation PilotValidate System Scenarios• Use the IMAs to ensure that the scenarios
return the expected results– Q1 scenarios complete without errors, as expected– Q2 scenarios do not cause any assertions to “pop”– Q3 scenarios result in detection of an adverse
condition and respond in a correct way, as expected
• Scenarios represent a test oracle
IV&V Facility
05/03/23 31
Assertion–Based Requirements Validation Pilot Execute scenarios in Context of PBAs• Interested in how the PBAs react to the system
scenarios– Which system scenarios ended as expected? – Which system scenarios ended differently than expected?– Which PBAs were not exercised?
SRM
Correlationswith SRM
Project Requirements
Analyze Results Findings
Develop & Validate PBAs
Execute Scenarios in Context of PBAs
Develop & Validate IMAs
Develop System Scenarios
Include PBAs in executable environment
Execute system scenarios
Collect results for further analysis
IV&V Facility
05/03/23 32
Assertion–Based Requirements Validation Process Analyze Results• Examine each assertion failure for:
– Errors in constructing the SRM (system scenarios, IMAs, etc.)– Errors in constructing the PBA(s)– Errors in a requirement
• Fix the errors in the SRM• Rerun test until no SRM or PBA construction errors are
discovered. Therefore all unexpected results indicate findings
SRM
Correlationswith SRM
Project Requirements
AnalyzeResults Findings
Develop & Validate PBAs
Execute Scenarios in Context of PBAs
Develop & Validate IMAs
Develop System Scenarios
Ensure system scenario set is correct and complete
Ensure PBA set is complete
Ensure that PBA is in scope
Ensure PBA is correct
IV&V Facility
05/03/23 33
Assertion–Based Requirements Validation PilotObservations• Developing IMAs helps identify defects and
deficiencies in the SRM. Doing this early in the SRM development process yields early benefits
• The process of examining the documented requirements helps to identify omissions in the correlation process
• The depth of understanding required to create IMAs and PBAs results in more rapid understanding of the system of interest
• Assertions best represent requirements of specific behaviors
• When creating PBAs for high level requirements such as “The system shall have the capability to do X”, these behavioral details need to be supplied from domain knowledge and analysis
IV&V Facility
05/03/23 34
Assertion–Based Requirements Validation PilotObservations• It was easier than expected to:
– Discover early questions and concerns that could lead to good assertions
– Develop a high level understanding of what the system is supposed to do
– Create higher level system scenarios that describe what the system is supposed to do
– Develop Formal Assertion State Charts from Natural Language– Observe the lack of model depth, details– Discover possible missing behavioral IMAs– Write unit tests for IMAs and PBAs (although easier for IMAs if the
SRM provides more information)– Write formalized Junit Tests– Test assertions against the System Scenarios
IV&V Facility
05/03/23 35
Assertion–Based Requirements Validation PilotObservations• It was harder than expected to:
– Vet good/worthwhile NL (informal assertions) from the list of answered questions
– Create System Scenarios with the lack of model depth and details– Create System Scenarios that describe what the system is not supposed
to do, and what the system is supposed to do under adverse conditions– Correlate system scenario events, IMA events, and PBA events without a
data dictionary or complete Domain Model– Discover worthwhile PBAs from documented requirements that define and
monitor a single event, and/or provide limited details and constraints– Discover matching IMAs and PBAs when there are limited requirement
details– Discover matching IMAs and PBAs when the model is not developed to
needed level of coverage/detail– Validate Requirements when the model is not developed to the needed
level of coverage/detail– Set up the test rig for the first time
IV&V Facility
05/03/23 36
Assertion–Based Requirements ValidationRecommendations• Develop IMAs early in the SRM development process• Consider developing IMAs as part of the process for
reviewing activity diagrams• Continue refining techniques for using assertion-
based methods for requirements validation
IV&V Facility
05/03/23 37
Assertion–Based Requirements ValidationSummary• Model–Based Requirements Validation is part of the SRM
Validation process– The purposes for using assertions include:
• Have ability to test conflicts and validate complex requirements• Provides an advantage of using formal requirements over natural language• Sets groundwork for automated testing
– Use assertions to validate requirements that correspond to SRM behaviors and IMAs as unambiguous, correct, complete, consistent, and verifiable.
– Use assertions to ensure that the right behaviors have been defined and the behaviors are of high quality. The right behaviors are those that adequately describe:
• What the system is supposed to do• What the system in not supposed to do• What the system is supposed to do under adverse conditions
Assertion–based validation leverages our independent Assertion–based validation leverages our independent modeling to find issues earlier in the life–cyclemodeling to find issues earlier in the life–cycle
IV&V Facility
05/03/23 38
Assertion–Based Requirements ValidationPanel Discussion• Michael Facemire• John Ryan• Bill Stanton• Patrick Theeke
IV&V Facility
05/03/23 39
References• D. Drusinsky, “From AD to Assertions”; informal presentation,
2008. • D. Drusinsky, J. B. Michael, and M. Shing, “A Framework for
Computer-aided Validation”; Naval Postgraduate School, NPS-CS-07-008, 08/2007.
• K. Hurley, and D. Kranz, “Executable Reference Model”; informal presentation, 2008.
• Juno IVV Team, “Juno Level-4 Flight Software Requirements Validation Report”; NASA IV&V, 2008.
• D. Kranz, and J. Ryan, “Juno and Assertion Based Validation”; NASA IV&V, white paper, 2008.
• NASA IV&V Facility, “System Level Procedure 09-1”; 2007.• S. Raque, “SRM Example Overview”; informal presentation ,
2008.• P. Theeke, “A process architecture for reference model based
validation”; NASA IV&V, white paper, 2007.• P. Theeke, “System Reference Models and Executable
Assertions for Requirements Validation”; NASA IV&V, white paper, 2007.