1

Software Reliability Software Reliability Analysis ToolsAnalysis Tools

Joel Henry, Ph.D.Joel Henry, Ph.D.University of MontanaUniversity of Montana

2

OutlineOutline

BackgroundBackground MATLAB Automated Testing ToolMATLAB Automated Testing Tool Graphical Input Specification ToolGraphical Input Specification Tool Real-Time Analysis Testing ToolReal-Time Analysis Testing Tool StatusStatus ConclusionConclusion

3

System DevelopmentSystem DevelopmentExample: Wind Tunnel SoftwareExample: Wind Tunnel Software

Controls devices that control:Controls devices that control: Wind generatorWind generator Model supportModel support Tunnel atmosphereTunnel atmosphere

Emphasizes reliability and safetyEmphasizes reliability and safety Utilizes multiple development Utilizes multiple development

strategiesstrategies Based on simple structureBased on simple structure

4

System DevelopmentSystem Development Simple StructureSimple Structure

ControllingComputers

ExternalDevices

ALGORITHMSample InputsRun SoftwareUpdate Outputs

5

Testing ProblemsTesting Problems SizeSize

Input variables – sampled over timeInput variables – sampled over time Outputs variables – produced over timeOutputs variables – produced over time Sample time – variable or set frequencySample time – variable or set frequency

RequirementsRequirements Input file/matrixInput file/matrix Output file/matrixOutput file/matrix Analysis toolsAnalysis tools

Domain determinantsDomain determinants Input variable – minimum, maximum, and accuracyInput variable – minimum, maximum, and accuracy Output variable – minimum, maximum, and accuracyOutput variable – minimum, maximum, and accuracy

Test requirementsTest requirements Input file/matrix with all possible values for inputInput file/matrix with all possible values for input Output file/matrix much more complex problemOutput file/matrix much more complex problem

6

Solution ApproachSolution ApproachOverviewOverview

Automation to:Automation to: Generate large input matrices/filesGenerate large input matrices/files Perform simulation and/or test auto-Perform simulation and/or test auto-

generated codegenerated code Analyze output matrices/filesAnalyze output matrices/files

Methods to:Methods to: Evaluate domain coverageEvaluate domain coverage Aid debuggingAid debugging Evaluate resultsEvaluate results

7

Solution ApproachSolution Approach

Command and Control Algorithm

Generate Tests Simulate Model Test Auto-code Detect Faults Evaluate Results

MATLAB/Simulink Environment

Source Code

Executable Code

Verification and Validation Methodology

Suite of testing tools

8

Solution ApproachSolution Approach

Generate Tests

Simulate Model

Test Auto-code

Detect Faults

Evaluate Results

Suite of testing tools

Verification and Validation Methodology

Command and Control Algorithm

MATLAB/Simulink Environment

Model Information

Test Data

Test Results

ExecutableCode

Test Data

Test Results

9

MATLAB Automated MATLAB Automated Testing ToolTesting Tool

Creates test dataCreates test data Executes simulation or testExecutes simulation or test Captures resultsCaptures results Detects exceptionsDetects exceptions Saves test data or complete test Saves test data or complete test

resultsresults

10

MATT – Test SetupMATT – Test SetupInputsInputs

Set test running timeSet test running time Select and configure tests using 31 test Select and configure tests using 31 test

typestypes Now supports non-scalar inputs*Now supports non-scalar inputs*

11

MATT – Test SetupMATT – Test SetupStatesStates

Assign Starting values for state Assign Starting values for state blocks*blocks*

12

MATT – Test SetupMATT – Test SetupOutputsOutputs

Set output accuracy and exception Set output accuracy and exception rangesranges

Now supports non-scalar outputs*Now supports non-scalar outputs*

13

MATT – Test SetupMATT – Test SetupNew Exception TypesNew Exception Types

Percent change*Percent change* Allows exception detection if the output Allows exception detection if the output

value changes more than a specified value changes more than a specified percent over a specified number of stepspercent over a specified number of steps

Absolute change*Absolute change* Allows exception detection if the output Allows exception detection if the output

value changes more than a specified value changes more than a specified amount over a specified number of stepsamount over a specified number of steps

14

MATT – SimulateMATT – Simulate

Set up simulation Set up simulation and code and code generation optionsgeneration options

Simulate:Simulate: ModelModel Auto-generated Auto-generated

CodeCode Compare bothCompare both

15

MATT – View ResultsMATT – View Results

View input values View input values for every time-stepfor every time-step

View output values View output values for every time-stepfor every time-step Steps causing Steps causing

exceptions are exceptions are highlighted red*highlighted red*

View/edit output View/edit output exception info and exception info and settingssettings

16

MATT – Advanced MATT – Advanced Exceptions*Exceptions*

Advanced Exceptions allow exception Advanced Exceptions allow exception detection based on multiple exception detection based on multiple exception criteriacriteria A combination of output portsA combination of output ports Disjoint rangesDisjoint ranges

Create separate A.E definition for each range that needs to be Create separate A.E definition for each range that needs to be tested tested

Do a combinational process based on all the A.E definitionsDo a combinational process based on all the A.E definitions Overall system reliabilityOverall system reliability

Create multiple A.E definitions based on system specificationsCreate multiple A.E definitions based on system specifications Perform a combinational process based on all the defined Perform a combinational process based on all the defined

Advanced ExceptionsAdvanced Exceptions

17

MATT – Advanced MATT – Advanced Exceptions Exceptions Combination of output portsCombination of output ports

In the example shown below, an exception is said to have occurred In the example shown below, an exception is said to have occurred when:when:

Stick output is between 5 and 10Stick output is between 5 and 10 Alpha output is greater than 20 radsAlpha output is greater than 20 rads

After clicking the After clicking the processprocess button, one discovers: button, one discovers: 7 exceptions occurred from the given exception definition7 exceptions occurred from the given exception definition The time steps where the 7 exceptions occurredThe time steps where the 7 exceptions occurred The output values that produced the 7 exceptionsThe output values that produced the 7 exceptions

18

MATT – Advanced MATT – Advanced ExceptionsExceptions Disjoint RangesDisjoint Ranges In this example, two separate exception In this example, two separate exception

definitions are specified:definitions are specified: (1) Alpha producing values between 0 - 4 rads(1) Alpha producing values between 0 - 4 rads (2) Alpha producing values between 15 – 19 rads(2) Alpha producing values between 15 – 19 rads

These two exception definitions are combined, These two exception definitions are combined, and exception results are calculated in aggregate.and exception results are calculated in aggregate.

19

MATT – Advanced MATT – Advanced ExceptionsExceptions

Overall System ReliabilityOverall System Reliability By combining multiple A.E. By combining multiple A.E.

definitions, systems can be stress-definitions, systems can be stress-tested for accuracy and reliability.tested for accuracy and reliability.

20

MATT - StorageMATT - Storage

Save complete test setup for future useSave complete test setup for future use Input, State, and Output settingsInput, State, and Output settings Advanced Exception SettingAdvanced Exception Setting Input MatrixInput Matrix Output MatrixOutput Matrix

Import and export custom input Import and export custom input matricesmatrices Comma delimited files (*.csv)Comma delimited files (*.csv) MATLAB data files (*.mat)MATLAB data files (*.mat)

21

GISTGIST

Reads MATT test data filesReads MATT test data files Allows freehand specification of test Allows freehand specification of test

datadata Saves new MATT test data filesSaves new MATT test data files

22

GIST – Manage Input SetsGIST – Manage Input Sets

Add/Remove inputs Add/Remove inputs for altered modelsfor altered models

View graphs of any View graphs of any inputinput

Edit any inputEdit any input

23

GIST – Edit InputsGIST – Edit Inputs

Use simple click Use simple click and drag to create and drag to create new input valuesnew input values

View original input View original input values as drawingvalues as drawing

24

RATTRATT

Reads MATT test result filesReads MATT test result files Calculates reliability, probabilities, Calculates reliability, probabilities,

and completeness measuresand completeness measures Supports analysis of multiple test Supports analysis of multiple test

files (test suite)files (test suite) Exports to MS Excel and populates Exports to MS Excel and populates

analysis charts and graphsanalysis charts and graphs

25

RATT - AnalysisRATT - Analysis Load several MATT Load several MATT

test files at oncetest files at once Perform analysis Perform analysis

Input coverageInput coverage Output coverageOutput coverage MTTFMTTF Probability of Probability of

failure failure EtcEtc

View simple View simple results in RATTresults in RATT

26

RATT – ExcelRATT – Excel®® Tool Tool

Load RATT files Load RATT files into Excel®into Excel®

View graphs View graphs showing coverageshowing coverage

View coverage on View coverage on multiple multiple inputs/exceptions inputs/exceptions at the same timeat the same time

27

Current StatusCurrent Status

Windows and UNIX versions of Windows and UNIX versions of MATT readyMATT ready

Windows versions of GIST and RATT Windows versions of GIST and RATT readyready

Working with STEREO projectWorking with STEREO project Testing MATLAB/Simulink® models Testing MATLAB/Simulink® models

using MATTusing MATT

28

Upcoming FunctionalityUpcoming Functionality

MATTMATT View intermediate state values from simulationView intermediate state values from simulation Create and use custom test-typesCreate and use custom test-types Load models inside MATT Load models inside MATT

Allows MATT to launch outside of MATLAB®Allows MATT to launch outside of MATLAB®

GISTGIST Being integrated into MATT for more seamless Being integrated into MATT for more seamless

useuse RATTRATT

Being revised to accommodate Advanced Being revised to accommodate Advanced ExceptionsExceptions

29

Conclusion – Testing Conclusion – Testing EnhancedEnhanced

Using software tools to aid testing Using software tools to aid testing allowsallows More tests to be runMore tests to be run Each test to be run more quicklyEach test to be run more quickly Test results to be easily stored and Test results to be easily stored and

referencedreferenced

30

Conclusion – Tools Conclusion – Tools EnhancedEnhanced

Improved MATT capabilities allowImproved MATT capabilities allow Systems to start at any state with easeSystems to start at any state with ease More advanced exception catchingMore advanced exception catching

31

Questions and Contact InfoQuestions and Contact Info

Joel HenryJoel Henry henryj@cs.umt.eduhenryj@cs.umt.edu MATT, RATT, and GISTMATT, RATT, and GIST http://www.cs.umt.edu/RTSL/matt/http://www.cs.umt.edu/RTSL/matt/

MATLAB and Simulink users MATLAB and Simulink users