m&s–based system development and testing in a net-centric environment

51
M&S–Based System Development and Testing In a Net-Centric Environment Bernard P. Zeigler, Ph.D., Arizona Center for Integrative Modeling and Simulation and Joint Interoperability Test Command Fort Huachuca, AZ 85613-7051[email protected]

Upload: taji

Post on 07-Jan-2016

20 views

Category:

Documents


1 download

DESCRIPTION

M&S–Based System Development and Testing In a Net-Centric Environment. Bernard P. Zeigler, Ph.D., Arizona Center for Integrative Modeling and Simulation and Joint Interoperability Test Command Fort Huachuca, AZ 85613-7051 [email protected]. Motivation from a Testing Perspective. - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: M&S–Based System Development and   Testing In a Net-Centric Environment

M&S–Based System Development and

Testing In a Net-Centric EnvironmentBernard P. Zeigler, Ph.D.,

Arizona Center for Integrative Modeling and Simulation and

Joint Interoperability Test CommandFort Huachuca, AZ [email protected]

Page 2: M&S–Based System Development and   Testing In a Net-Centric Environment

Motivation from a Testing Perspective

• Traditional interoperability concepts and test practices are facing severe challenges from several sources:

– a) complexity within each new system, as well as composition into families of

systems and systems of systems, – b) extensive use of modeling and simulation in simulation-based acquisition, and– c) the move to service oriented architecture (SOA) to support composable

services over the Global Information Grid (GIG).

• Need a methodology and process to integrate development and testing in a net-centric environment

– combines with and goes beyond current software development paradigms – rests upon and exploits dynamic systems theory, a modeling and simulation

(M&S) framework, and model-continuity development concepts

• Relationship to other software development processes will be open for discussion at the end

Page 3: M&S–Based System Development and   Testing In a Net-Centric Environment

Part 1 Modeling and Simulation Background

Briefly review the M&S framework and theory

• provide background for integrated system development and testing process

• overview the Discrete Event Systems Specification (DEVS) formalism which provides the computational support for the M&S framework and theory

• DEVS can serve as the basic medium for formalization of standards, analysis of the resulting models, automation of the test cases, and execution of the test drivers

Page 4: M&S–Based System Development and   Testing In a Net-Centric Environment

Part 2. M&S-Based Development and Testing: DEVS Methodology

• Illustrate M&S-Based testing methodology with an application drawn from a current effort to provide automated test generation for an important tactical command and control standard, MIL-STD 6016c

• Goal: develop a “formalized” version of the MIL-STD 6016C rule sets with the objective– to complete an unambiguous description of the specification,– enable more automated test development

• Result: Automated Test Case Generator (ATC-Gen) is a methodology and tool-set based on the DEVS formalism.

• Success: ATC-Gen will be one of the primary test vehicles for the first assessment of an integrated air picture system based on MIL-STD 6016c to occur this fall

Page 5: M&S–Based System Development and   Testing In a Net-Centric Environment

Framework for M&S

Systems theory-based framework

• provides an ontology for M&S that recognizes the essential dynamic character of simulation models

• distinguishes the elements in the M&S enterprise (such as models, simulators, and experimental frames) and the relationships (such as validity and correctness) that connect such elements in meaningful ways related to the objectives of simulation exercises,

• provides a rigorous mathematical theory that supports manipulations of the elements in their real-world incarnations in order to achieve the desired relationships

Page 6: M&S–Based System Development and   Testing In a Net-Centric Environment

M&S Entities and Relations

Real WorldReal World SimulatorSimulatorSimulatorSimulator

modelingrelation

simulationrelation

Each entity is represented as a dynamic system

Each relation is represented by a homomorphism or other equivalence

Data: Input/output relation pairs

structure for generating behaviorclaimed to represent real world

Device forexecuting model

Model

Page 7: M&S–Based System Development and   Testing In a Net-Centric Environment

M&S Framework: Continuous case

Real WorldReal World SimulatorSimulatorSimulatorSimulator

modelingrelation

simulationrelation

Model

d q(t) / dt = x(t)

Numerical Integration:•Accuracy•Error effects

Validity:•Accuracy of -retro-diction-prediction

Page 8: M&S–Based System Development and   Testing In a Net-Centric Environment

M&S Framework: Discrete Event case

Real WorldReal World SimulatorSimulatorSimulatorSimulator

simulationrelation

Model

x0 x1X

t0 t1 t2

first second third passiveFirstDuration

secondDuration

thirdDuration

active

0

modelingrelation

Concurrent Processing•Correctness• Randomness

Validity:•Accuracy of -retro-diction-prediction

sMake a transition

s’

Time advance

output

Page 9: M&S–Based System Development and   Testing In a Net-Centric Environment

DEVS within the Framework for M&S

• DEVS (Discrete Event Systems Specification) formalism

– is a constructive framework for M&S– based on mathematical systems theory

• DEVS enables applications resting on rigorous theory

– modeling discrete/continuous heterogeneously composed networked systems – universality of representation

– hierarchical, modular model construction –closure under coupling

– verifiably correct parallel and distributed simulation of ultra-large scale networks

– model-continuity from simulation to execution – based on abstract simulator

– automated test generation for net-centric services – based on behavior representation from systems theory

Page 10: M&S–Based System Development and   Testing In a Net-Centric Environment

DEVS Hierarchical Modular Composition – Key to Constructing/Orchestrating Complex Systems

Atomic: lowest level model, contains structural dynamics -- model level modularity

Atomic

Atomic Atomic

Atomic

+ coupling

Atomic

Atomic

Atomic

Coupled: composed of one or more atomic and/or coupled models hierarchical

construction

Page 11: M&S–Based System Development and   Testing In a Net-Centric Environment

Atomic Models

OrdinaryDifferentialEquations

Pulse BasedModels

(varGen, Sum)

Quantum Based Models

(DEVS Integrator,instantaneous

Functions

Coupled Models

Phase BasedModels

Cellular Automata

1,2 Dim Cell Space

PartialDifferentialEquations

Self Organized Criticality

Models

Processing/Queuing/

Coordinating

DigraphModels

NetworksCollaborations Physical

Space

1 Dim State Space

2 Dim State Space

Types of Models and their DEVS Representations

can becomponents in a coupled model

MultiAgent

Systems

Discrete Time/

Automata

Page 12: M&S–Based System Development and   Testing In a Net-Centric Environment

DEVS Theory: Nutaro's optimistic simulation algorithm and its correctness proof

• Nutaro developed a time warp variant that correctly simulates all discrete event models

– Previous variants of the time warp algorithm have been based on the logical process world view.

– These time warp derivatives have been widely used for simulating discrete event models on parallel computers,

– But, they can not correctly simulate DEVS coupled models with zero-time interactions

– employs a 2-D distributed clock

• A correctness proof for the algorithm was constructed– The correctness proof demonstrates that the parallel algorithm simulates exactly

the same system as the canonical DEVS abstract simulator. – Consequently, parallel and sequential simulation of the same DEVS model will

always produce the same results.

• This is a strong proof of correctness that no logical-processor-based proof has been able to rival – depends strongly on model/simulator separation

Page 13: M&S–Based System Development and   Testing In a Net-Centric Environment

DEVS-Based Model Continuity

•Model continuity is retained between models of different design stages

•reduces the occurrence of design discrepancies along the development process

•increases the confidence that the final system realizes the specification

•Allows component models of a distributed real-time system to be tested incrementally then deployed to a distributed environment for execution

•Based on replacing DEVS simulator with DEVS Real Time executor

RobotModel

RobotModel

Virtual Environment

Virtual Sensor

VirtualActuator

RobotModel

RealRobot

Mixed Environment

Virtual Sensor

VirtualActuator

Virtual Sensor

HIL Actuator

RealRobot

RealRobot

Real Environment

Real Sensor

RealActuator

DEVS Distributed Simulator

DEVS Distributed RT Executor

Page 14: M&S–Based System Development and   Testing In a Net-Centric Environment

Model Continuity Mixed Virtual/Real Environment – Any number of virtual

robots can interact with a few real robots

Page 15: M&S–Based System Development and   Testing In a Net-Centric Environment

Summary

DEVS is an overarching approach

– supports all levels of system interoperability

– using a Systems Theoretic worldview

– component-based characterization of dynamical systems in discrete and continuous forms

– methods for modular, hierarchical model composition targeting reusability and composability

Page 16: M&S–Based System Development and   Testing In a Net-Centric Environment

Part 2. M&S-Based Development and Testing: DEVS Methodology

• DoD acquisition policy requires testing throughout the systems development process to ensure interoperability and conformance to standards.

• The Joint Interoperability Test Command (JITC) is striving to improve traditional standards conformance and interoperability testing methodologies to

– keep pace with the behavioral complexity of new systems that are increasingly reliant on advanced information technology

– DoD mandated use of M&S throughout the development life cycle, requires testing methods that are themselves M&S-based

• In 2003, the JITC started development of a “formalized” version of the MIL-STD 6016C rule sets with the objective

– to complete an unambiguous description of the specification, and – enable more automated test development.

• illustrate the methodology with an application drawn from a current effort to provide automated test generation for an important tactical command and control standard, MIL-STD 6016c

– Automated Test Case Generator (ATCGen), which is a methodology and tool-set based on the DEVS formalism.

– ATC-Gen will be one of the primary test vehicles for the first assessment of an integrated air picture system based on MIL-STD 6016c to occur this fall.

• working towards a proposal for how the DEVS-based approach enables– 1) simulation-based model continuity in developing and testing specifications that stem from

the DoD Architectural Framework (DoDAF), and– 2) a standard and an infrastructure for developing and testing web-based services for

DoD’s emerging Net-Centric Enterprise Services on the Global Information Grid.

Page 17: M&S–Based System Development and   Testing In a Net-Centric Environment

Outline

• JITC responsibility for Link-16 and successors

• Link-16: Challenges to Implementation and Testing

• M&S–based Approach to Automated Test Case Generation

• Application to Link-16 in the IABM SIAP Context

Page 18: M&S–Based System Development and   Testing In a Net-Centric Environment

JITC is the Responsible Test Organization for TDL Standards

• JITC is responsible for ensuring systems that implement Tactical Data Link (TDL) (Link 11/11B/16 and Variable Message Format (VMF)) are interoperable and in compliance with the applicable joint standards.

• This is accomplished by conducting the following types of tests: – Joint / NATO /Combined Interoperability – Performance Assessment in Operational Environments – Standards Validation – Standards Conformance

• JITC employs a variety of tools that provide its analysts the ability to evaluate TDL system performance in both the lab and live environments.

source: http://jitc.fhu.disa.mil

Page 19: M&S–Based System Development and   Testing In a Net-Centric Environment

Link-16: Challenges to Implementation and Testing

Joint Single Link Implementation Requirements Specification JSLIRS is an evolving standard (MIL-STD-6016c) for tactical data link

information exchange and networked command/control of radar systems

• Presents significant challenges to automated conformance testing:– The specification document states requirements in natural language– open to ambiguous interpretations– The document is voluminous – many interdependent chapters and appendixes– labor intensive and prone to error– potentially incomplete and inconsistent.

• Problem: how to ensure that a certification test procedure – is traceable back to specification– completely covers the requirements– can be consistently replicated across the numerous contexts– military service, inter-national, and commercial companies

Page 20: M&S–Based System Development and   Testing In a Net-Centric Environment

SIAP/IABM —Successor to Link-16

• SIAP (Single Integrated Air Picture) Goal: Improve the adequacy and fidelity of information to form a shared understanding of tactical situation

• Integrated Architecture Behavior Model (IABM) requires that all sensors utilize a standard reference frame for conveying information about the location of targets.

• Developed by the Joint SIAP System Engineering Organization (JSSEO), Arlington, Va., a sub-office of the Assistant Secretary of the Army for Acquisition, Logistics and Technology.

• Development costs $160 million over five years; the individual services will spend an estimated $600 million

• First major test – Config05 – next week -- ATC-Gen will be the basis for testing link-16 and extended IABM requirements

http://www.navyleague.org/sea_power/mar_04_18.php

Page 21: M&S–Based System Development and   Testing In a Net-Centric Environment

Automated Test Case Generator (ATC-Gen)

• IABM is an extension of Link-16 developed in HLA environment and requires HLA simulation-based testing

• JITC has taken the initiative to integrate modeling and simulation into the automation of the testing process

• Funded the development of Automated Test Case Generator (ATC-Gen) led by ACIMS using DEVS (Discrete Event System Specification) technology.

• In R&D of two years, proved the feasibility and the general direction

• The requirements have evolved to a practical implementation level, with help from conventional testing personnel.

Page 22: M&S–Based System Development and   Testing In a Net-Centric Environment

Discrete Event Nature of Link-16 Specification

Constraints(Exception)Rules

Stop

Modify C2Record for TN

1 2 3

RuleProcessing

Stop, Do Nothing,Alerts, Or jump to other

Transaction

TrackDisplay

Operatordecisions

Validity checking

TransmitMsg

Other ConsequentProcessing

Jumps (stimuli) to other

Transactions of specification

Transaction Level - example P.1.2 = Drop Track Transmit

Preparation Processing

Timeouts

PeriodicMsg

Input to

systemDEVS

Output from

system

t1

t2 t

3t4

Page 23: M&S–Based System Development and   Testing In a Net-Centric Environment

Automated Test Case Generation: Goals and Approach

Objective:Automate Testing

Capture Specification as DEVS

Analyze DEVSI/O Behavior

Synthesize Test Models

Test Driver executesmodels to induce testable behavior in SUT

Interactwith SUT over Middleware

Goals:

• Increase the productivity and effectiveness of the conformance testing• Apply DEVS systems theory, modeling and simulation concepts, and current software technology to (semi-)automate portions of conformance testing

Formalized approach for converting standards documents into test models to run directly against a system, automating the process to the extent possible

Network

DEVS Simulator

Test Driver

HLA

SystemUnder

Test (SUT)

HLA

Page 24: M&S–Based System Development and   Testing In a Net-Centric Environment

Benefits of Formalization and Automation

• Provides traceability to original specification

• Reduces ambiguity from textual specification

• Facilitates integrating Modeling & Simulation into the testing process

• Enables testing of complex:– Standards– Systems– Functions– Families of systems

Page 25: M&S–Based System Development and   Testing In a Net-Centric Environment

ATC Gen Overall Process

• MIL-STD-6016C is a description of a DEVS that mandates the outcome of tests and sequences of tests

• ATC Gen analysts convert if-then rules from the MIL-STD into XML, defining state variables and vectors

• XML if-then rules are used to generate test sequences

• Test models are derived from test sequences

• Test Driver runs test models against SUT in distributed simulation

Page 26: M&S–Based System Development and   Testing In a Net-Centric Environment

XMLRule Set

Rule capture

6016C

XMLRules

XMLRule Set

Analyze Rules Define State Variables

Reference Model

Compile executableReference model

Rule Compiler Executable

paths

Test case models

Simulations of test cases

Verification of Test Sequences

Test sequences

ExecutableTests

Test StimulatorReference modelLogging

Test Execution

Test Driver

Test ProceduresTest Results

Rule capture andDependencyAnalysis

Rule Formalization

Test Generation

ATC-Gen Process Lines

Analyst:Interpret the text to extract state variables and rules, where rules are written in the form:

If P is true nowthen do action A

laterunless Q occurs in the interim

The Dependency Analyzer (DA):determines the relationship between rules by identifying shared state variablesGenerates test sequences and test modelsTest Engineer:Develop sets of test sequencesConvert test sequences to executable simulation modelsConvert test sequences to executable simulation models

Test Driver:Interacts with and connects to SUT via HLA or Simple J interfacesPerforms conformance testing of SUTs

Page 27: M&S–Based System Development and   Testing In a Net-Centric Environment

MIL-STD-6016C Micro-Structure

Stimuli ConstraintsProcessing

MessagePreparation

Message Transmission

Message Reception

InitiatorInitiator ResponderResponderRoles

AppendixStructure

MessagePhase

Page 28: M&S–Based System Development and   Testing In a Net-Centric Environment

Interpretation and Translation

Appendix E.1.1

Identify RolePreparation of message set as

Initiator Role

Identify Type of Stimulus Operator action

Translation to XML – Variable Standards

Role E1.Init=True

Type of Stimulus VOI

Page 29: M&S–Based System Development and   Testing In a Net-Centric Environment

Interpretation and Translation

Constraints in Appendix E.1.1.2.1

If Reference TN equals 0, 77, 176, 177, 777 or 77777

ThenHost System:

– Performs no further processing– Alerts operator (CAT 4)

Translation to XML – Variables Standards

ConditionE1.Init == True and

VOI.TNref == 0 or 63 or 126 or 127 or 4095 or 118783

Action E1.Init = False

Output CAT 4 Alert

Page 30: M&S–Based System Development and   Testing In a Net-Centric Environment

Detailed XML Example: Traceability, Quality Assurance

4.11.13.12 Execution of the Correlation. The following rules apply to the disposition of the Dropped TN and the retention of data from the Dropped TN upon origination or receipt of a J7.0 Track Management message, ACT = 0 (Drop Track Report), for the Dropped TN. The correlation shall be deemed to have failed if no J7.0 Track Management message, ACT = 0 (Drop Track Report), is received for the dropped TN after a period of 60 seconds from the transmission of the correlation request and all associated processing for the correlation shall be cancelled.

a. Disposition of the Dropped Track Number: (2) If own unit has R2 for the Dropped TN, a J7.0 Track Management message, ACT = 0 (Drop Track Report), shall

be transmitted for the Dropped TN. If the Dropped TN is reported by another IU after transmission of the J7.0 Track Management message, own unit shall retain the dropped TN as a remote track and shall not reattempt to correlate the Retained TN and the Dropped TN for a period of 60 seconds after transmission of the J7.0 Track Management message.

• XML Translation:<rule trans="4.11.13" stimulus="4.11.13.12" reference="4.11.13.12.a.2" ruleName="R2 Unit transmits J7.0">

<condition txt="Check for R2 own unit" expression="AutoCor==True and (CRair.TNcor.CORtest==3 and J32.TNref.CORtest==3) and CRair.R2held==1 AND J72.MsgTx==True">

</condition><action txt="Prepare J7.0 Drop Air Track message" expression="J70.TNsrc=TNown; J70.TNref=TNdrop;

J70.INDex=0; J70.INDcu=0; J70.ACTVair=0; J70.SZ=0; J70.PLAT=0; J70.ASchg=0; J70.ACTtrk=0; J70.ENV=0; MsgTx(J70)">

</action><output txt="Transmit J7.0" outType="Message" outVal="J70"></output>

</rule><QA>

<revisit name="DHF" date="10/16/04" status="Open">need to add timer for a period of 60 seconds in which correlation is not reattempted</revisit>

</QA>

Page 31: M&S–Based System Development and   Testing In a Net-Centric Environment

Quality Assurance (QA)

• Provides a history of rule changes• Identifies possible Change Requests and

other problematic portions of the standard• QA tags:

– Info: Explanation / reasoning for rule interpretation– Revisit: Rule needs further analysis; set to Open

or Closed– Resolution: Solution to question / issue

Page 32: M&S–Based System Development and   Testing In a Net-Centric Environment

Appendix N Functional Area, e.g. Track Management

Function P.N e.g. C2 Drop Track

Transaction P.1.N C2 Drop Track Transmit

Step 1 – Stimuli, Preparation, Constraints

Step 2 – Processing, Operator Decisions

Step 3 – Update Database, Output

AtomicLevelUnit

• Transactions are atomic units

• Functions are collections of Transactions

• Appendices group related Functions

• Java processing restructures rules into a Transaction Module Hierarchy

MIL-STD-6016C Macro-Structure

Page 33: M&S–Based System Development and   Testing In a Net-Centric Environment

• Organized according to MIL-STD-6016C macro-structure hierarchy

• RuleCombo folders store aggregations/abstractions of lower level rules

• Value added:– Provides a basis for further analysis– Removes ambiguity – Annotates problems areas, improving the

ability to find and fix issues– Provides organization for indexing states,

rules, and variables– Supports test generation and executable rule

construction

XML Repository – GIG/SOA Asset

Page 34: M&S–Based System Development and   Testing In a Net-Centric Environment

Dependency Analysis:Visualization of Rule Dependencies

Clicking will reveal rule text

Dependencies revealed by hovering

Page 35: M&S–Based System Development and   Testing In a Net-Centric Environment

DependencyBetween Modules

Desired SequenceC.1 Receive PPLID.1 Receive TrackU.1 Correlate

Test Sequence Generation Transaction Level Paths

C.1ModuleN

active

= infinity

D.1ModuleN

active

= infinity

U.1ModuleN

passive[out1]

= infinity

Potential Test

Sequence

Page 36: M&S–Based System Development and   Testing In a Net-Centric Environment

Determining initial state and message field values required to drive SUT through sequence

Analyst:• Determine the data

needed to execute a test sequence

• Set state variables and field values accordingly

Page 37: M&S–Based System Development and   Testing In a Net-Centric Environment

Test Simulatlon Architecture

J-SeriesMessage

Generator

J-SeriesMessageAcceptor

Test Model

System Under Test

Produces Stimulus

Checks for Proper Response

Page 38: M&S–Based System Development and   Testing In a Net-Centric Environment

holdSend(Jx1,data1,t1) holdSend (Jx2,data2,t2)

holdSend (Jx3,data3,t3) waitReceive(Jx4,data4)

receiveAndProcess(Jx1,data1) receiveAndProcess(Jx2,data2)

receiveAndProcess(Jx3,data3) transmit(Jx4,data4)

Jx1,data1Jx2,data2Jx3,data3

Jx4,data4

t1 t2 t3 t4time

Test ModelTest Model

SUT

Test Model Construction:

Translate test sequences from the DA to derive a composed model for the Test Driver

Page 39: M&S–Based System Development and   Testing In a Net-Centric Environment

Test Driver Composition and Deployment

Middleware

Test ModelTest Model

Jx1,data1Jx2,data2Jx3,data3

Jx4,data4 Jx1,data1Jx2,data2Jx3,data3

Jx4,data4 Jx1,data1Jx2,data2Jx3,data3

Jx4,data4

Sequence 1

Sequence 2

Sequence 3

SUT

Page 40: M&S–Based System Development and   Testing In a Net-Centric Environment

IABM/Link 16 Correlation Testing

• ATC Gen automated correlation/decorrelation tool provides in-depth compliance testing

• Mathematical algorithm of the proximity of the tracks

• Logic for prohibitions and constraints

• Test for post-correlation (dropped and retained tracks)

• Discriminates between types of failures encountered during correlation testing

Page 41: M&S–Based System Development and   Testing In a Net-Centric Environment

ATC Gen Summary: What Can (not) be Automated?

Original Standard(MIL-STD-6016C)

Repository (XML)

XML Translation

Natural Language

XML (Computer Language)

XML Validation &Verification (DTD)

Validated XML

Dependency Web(Java GUI)

DependencyAnalyzer

Test Sequence Step& Test ModelGeneration

XML

Class Files

XML

Test ModelsTest Driver

(Event Coordinator)

DEVS Model

Manually Derived Paths

Simple JInterface

HLAInterface

DEVS Messages

SUT

XML Capture of Standard

Dependency Analysis & Test Generation

Process Legend:

Manual

Semi-Automated

Automated

Executable TestSequences

C Files

Java Files

Simple J

DEVS Messages

HLA Updates& Interactions

Testing of Systems

Page 42: M&S–Based System Development and   Testing In a Net-Centric Environment

Discussion

Can the DEVS-based M&S approach enable

• simulation-based model continuity in developing and testing specifications that stem from the DoD Architectural Framework (DoDAF), and

• a standard and an infrastructure for developing and testing web-based services for DoD’s emerging Net-Centric Enterprise Services on the Global Information Grid.

Page 43: M&S–Based System Development and   Testing In a Net-Centric Environment

Systems Engineering Perspectives for M&S Applications (Robert Marcus)

DoD System Life-Cycle

ConceptRefinement

TechnologyDevelopment

SystemDevelopment & Demonstration Production &

DeploymentOperations &

Support

Concept Analysis Design Develop Test Deploy Maintain

Proofof

ConceptSystem

SimulationSystems

EmulationModelBased

LVCSimulation

DemosLVC

TrainingDiagnosticEmulation

ConceptsTesting

PrototypeTesting

SystemValidation

DesignTesting

SystemVerification

UseabilityTesting

DiagnosticTesting

IndustrySystemLife-Cycle

Page 44: M&S–Based System Development and   Testing In a Net-Centric Environment

DODAF

Operational View

Systems View

Technical View

Packaging

Messaging

Communication

Service D

iscovery

Service D

escription

Service C

omposition

Service T

estingLevel

Name What we specify at this level

4 Coupled Systems

System built up by several component systems which are coupled together

3 I/O System System with state and state transitions to generate the behavior

2 I/O Function

Collection of input/output pairs constituting the allowed behavior partitioned according to the initial state the system is in when the input is applied

1 I/O Behavior

Collection of input/output pairs constituting the allowed behavior of the system from an external Black Box view

0 I/O Frame Input and output variables and ports together with allowed values

Simulator

Model

Experimental Frame

Hierarchy of System Specifications

ArchitectureFrameworkStandards

DevelopmentProcessStandards Service

OrientedArchitectureStandards

MiddlewareStandardsM&S

Standards

ConceptRefineme

nt

Technology

Development

SystemDevelopme

nt & Demonstrat

ion

Production &

Deployment

Operations &

Support

Concept

Analysis

Design

Develop Test Depl

oyMaintain

Proofof

Concept

System

Simulation

Systems

Emulation

ModelBase

d

LVCSimulationDemo

s

LVCTraini

ng

DiagnosticEmulation

ConceptsTesting

PrototypeTesting

SystemValidation

DesignTesting

SystemVerification

UseabilityTesting

DiagnosticTesting

IndustrySystemLife-Cycle

Interlocking Standards – How Do These Play Together?

Page 45: M&S–Based System Development and   Testing In a Net-Centric Environment

Models and Methodologies –How Do These Play Together?

IDEFIntegrated Definition

for Function Modeling

PetriNets

UMLUnified Modeling

Language

MDAModel Driven Architecture

MDDModel Driven Development

IDEFØ is a method designed to model the decisions, actions, and activities of an organization or system. IDEFØ was derived from a well-established graphical language, the Structured Analysis and Design Technique (SADT). The United States Air Force commissioned the developers of SADT to develop a function modeling method for analyzing and communicating the functional perspective of a system. Effective IDEFØ models help to organize the analysis of a system and to promote good communication between the analyst and the customer. IDEFØ is useful in establishing the scope of an analysis, especially for a functional analysis. As a communication tool, IDEFØ enhances domain expert involvement and consensus decision-making through simplified graphical devices. As an analysis tool, IDEFØ assists the modeler in identifying what functions are performed, what is needed to perform those functions, what the current system does right, and what the current system does wrong. Thus, IDEFØ models are often created as one of the first tasks of a system development effort.

This is a 4GL notation tailored to OO software development. It is primarily a graphical notation. The standard is managed by the OMG.integrates the concepts of Booch, OMT and OOSE by fusing them into a single, common and widely usable modelling language. UML aims to be a standard modelling language which can model concurrent and distributed system

"model-driven" development methods, based on greater use of automation compared to traditional methods, have already demonstrated their potential for radical improvements in the quality of software

The MDA defines an approach whereby you can separate the system functionality specification from its implementation on any specific technology platform. That way you can have an architecture that will be language, vendor and middleware neutral.

Petri nets were invented by Carl Adam Petri to model concurrent systems and the network protocols used with these systems.

Page 46: M&S–Based System Development and   Testing In a Net-Centric Environment

Semi-automated test suite design based on Experimental Frames

BehaviorRequirements at one or morelevels of SystemSpecification

Reference Master Model

Formalization by mapping into DEVS representationsCorresponding levels ofSystem Specification

Real-timeimplement-ation and execution

Simulation based testing in net-centric environment

Ideal Bifurcated Development Process

Page 47: M&S–Based System Development and   Testing In a Net-Centric Environment

AV-1,2OV 1,5,6,2

Bifurcated Development Process in the DODAF/GIG Context

OV Activities classified into Activity Components

OV-3,8,9DEVS

Formalization

Activity characterizationand System scope refinement

Integrated Functionalities transported to System viewsusing:1. Model Continuity2. Web services3. COTS components

OV-7,4

Semi-automated test suite design based on NLspecified vignette scenarios

Simulation based testing using DEVSTest Federations

OV-8 information mapped to WSDL and vice versa. Any Web Service can become a component Activity of OV-8 and be constituted as a ‘functionality’ in DoDAF

WEB SERVICES

Model Design &Repository

Page 48: M&S–Based System Development and   Testing In a Net-Centric Environment

Transfering DEVS-based Testing Methodology to SOA

DEVS Simulator

DEVS Model

Packaging:FOM

Messaging:Interactions,Updates

Communication: RTI

HLA

Service Discovery: UDDI

DEVS Simulator

DEVS Model

Sevice Description: WSDL

Packaging:XML

Messaging:SOAP

Communication: HTTP

SOA

Page 49: M&S–Based System Development and   Testing In a Net-Centric Environment

Refining OV-6a (IDEF) to OV-8 (DEVS)

LOGIC CODE IN OV-6a (Rules of Engagement/Activation of further Activities done with boolean decision making)

IF Significant Movement of target Then Monitor Target/Target Status

Project Target Movement Target Vector = . . . .. ?

Else Monitor for Movement

•Inherent problems with IDEF process (&,||,!),•Timing, a critical factor in real-time decision making• unaddressed by CPNs, IDEF processes,

Automated Code generation of DEVS model thru OV-8

IDEF

DEVS

Page 50: M&S–Based System Development and   Testing In a Net-Centric Environment

DEVS-based Web-Services Testing

GES DEVS Test Federation

LiveTest

Player

ServiceUnderTest

DEVS Simulator

Node

SOAP-XML

DEVS Test

Player

Page 51: M&S–Based System Development and   Testing In a Net-Centric Environment

Bernard P. Zeigler ACIMS

www.acims.arizona.edu

Contact: