![Page 1: 1 Software Reliability Analysis Tools Joel Henry, Ph.D. University of Montana](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f4f5503460f94c71410/html5/thumbnails/1.jpg)
1
Software Reliability Software Reliability Analysis ToolsAnalysis Tools
Joel Henry, Ph.D.Joel Henry, Ph.D.University of MontanaUniversity of Montana
![Page 2: 1 Software Reliability Analysis Tools Joel Henry, Ph.D. University of Montana](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f4f5503460f94c71410/html5/thumbnails/2.jpg)
2
OutlineOutline
BackgroundBackground MATLAB Automated Testing ToolMATLAB Automated Testing Tool Graphical Input Specification ToolGraphical Input Specification Tool Real-Time Analysis Testing ToolReal-Time Analysis Testing Tool StatusStatus ConclusionConclusion
![Page 3: 1 Software Reliability Analysis Tools Joel Henry, Ph.D. University of Montana](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f4f5503460f94c71410/html5/thumbnails/3.jpg)
3
System DevelopmentSystem DevelopmentExample: Wind Tunnel SoftwareExample: Wind Tunnel Software
Controls devices that control:Controls devices that control: Wind generatorWind generator Model supportModel support Tunnel atmosphereTunnel atmosphere
Emphasizes reliability and safetyEmphasizes reliability and safety Utilizes multiple development Utilizes multiple development
strategiesstrategies Based on simple structureBased on simple structure
![Page 4: 1 Software Reliability Analysis Tools Joel Henry, Ph.D. University of Montana](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f4f5503460f94c71410/html5/thumbnails/4.jpg)
4
System DevelopmentSystem Development Simple StructureSimple Structure
ControllingComputers
ExternalDevices
ALGORITHMSample InputsRun SoftwareUpdate Outputs
![Page 5: 1 Software Reliability Analysis Tools Joel Henry, Ph.D. University of Montana](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f4f5503460f94c71410/html5/thumbnails/5.jpg)
5
Testing ProblemsTesting Problems SizeSize
Input variables – sampled over timeInput variables – sampled over time Outputs variables – produced over timeOutputs variables – produced over time Sample time – variable or set frequencySample time – variable or set frequency
RequirementsRequirements Input file/matrixInput file/matrix Output file/matrixOutput file/matrix Analysis toolsAnalysis tools
Domain determinantsDomain determinants Input variable – minimum, maximum, and accuracyInput variable – minimum, maximum, and accuracy Output variable – minimum, maximum, and accuracyOutput variable – minimum, maximum, and accuracy
Test requirementsTest requirements Input file/matrix with all possible values for inputInput file/matrix with all possible values for input Output file/matrix much more complex problemOutput file/matrix much more complex problem
![Page 6: 1 Software Reliability Analysis Tools Joel Henry, Ph.D. University of Montana](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f4f5503460f94c71410/html5/thumbnails/6.jpg)
6
Solution ApproachSolution ApproachOverviewOverview
Automation to:Automation to: Generate large input matrices/filesGenerate large input matrices/files Perform simulation and/or test auto-Perform simulation and/or test auto-
generated codegenerated code Analyze output matrices/filesAnalyze output matrices/files
Methods to:Methods to: Evaluate domain coverageEvaluate domain coverage Aid debuggingAid debugging Evaluate resultsEvaluate results
![Page 7: 1 Software Reliability Analysis Tools Joel Henry, Ph.D. University of Montana](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f4f5503460f94c71410/html5/thumbnails/7.jpg)
7
Solution ApproachSolution Approach
Command and Control Algorithm
Generate Tests Simulate Model Test Auto-code Detect Faults Evaluate Results
MATLAB/Simulink Environment
Source Code
Executable Code
Verification and Validation Methodology
Suite of testing tools
![Page 8: 1 Software Reliability Analysis Tools Joel Henry, Ph.D. University of Montana](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f4f5503460f94c71410/html5/thumbnails/8.jpg)
8
Solution ApproachSolution Approach
Generate Tests
Simulate Model
Test Auto-code
Detect Faults
Evaluate Results
Suite of testing tools
Verification and Validation Methodology
Command and Control Algorithm
MATLAB/Simulink Environment
Model Information
Test Data
Test Results
ExecutableCode
Test Data
Test Results
![Page 9: 1 Software Reliability Analysis Tools Joel Henry, Ph.D. University of Montana](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f4f5503460f94c71410/html5/thumbnails/9.jpg)
9
MATLAB Automated MATLAB Automated Testing ToolTesting Tool
Creates test dataCreates test data Executes simulation or testExecutes simulation or test Captures resultsCaptures results Detects exceptionsDetects exceptions Saves test data or complete test Saves test data or complete test
resultsresults
![Page 10: 1 Software Reliability Analysis Tools Joel Henry, Ph.D. University of Montana](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f4f5503460f94c71410/html5/thumbnails/10.jpg)
10
MATT – Test SetupMATT – Test SetupInputsInputs
Set test running timeSet test running time Select and configure tests using 31 test Select and configure tests using 31 test
typestypes Now supports non-scalar inputs*Now supports non-scalar inputs*
![Page 11: 1 Software Reliability Analysis Tools Joel Henry, Ph.D. University of Montana](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f4f5503460f94c71410/html5/thumbnails/11.jpg)
11
MATT – Test SetupMATT – Test SetupStatesStates
Assign Starting values for state Assign Starting values for state blocks*blocks*
![Page 12: 1 Software Reliability Analysis Tools Joel Henry, Ph.D. University of Montana](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f4f5503460f94c71410/html5/thumbnails/12.jpg)
12
MATT – Test SetupMATT – Test SetupOutputsOutputs
Set output accuracy and exception Set output accuracy and exception rangesranges
Now supports non-scalar outputs*Now supports non-scalar outputs*
![Page 13: 1 Software Reliability Analysis Tools Joel Henry, Ph.D. University of Montana](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f4f5503460f94c71410/html5/thumbnails/13.jpg)
13
MATT – Test SetupMATT – Test SetupNew Exception TypesNew Exception Types
Percent change*Percent change* Allows exception detection if the output Allows exception detection if the output
value changes more than a specified value changes more than a specified percent over a specified number of stepspercent over a specified number of steps
Absolute change*Absolute change* Allows exception detection if the output Allows exception detection if the output
value changes more than a specified value changes more than a specified amount over a specified number of stepsamount over a specified number of steps
![Page 14: 1 Software Reliability Analysis Tools Joel Henry, Ph.D. University of Montana](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f4f5503460f94c71410/html5/thumbnails/14.jpg)
14
MATT – SimulateMATT – Simulate
Set up simulation Set up simulation and code and code generation optionsgeneration options
Simulate:Simulate: ModelModel Auto-generated Auto-generated
CodeCode Compare bothCompare both
![Page 15: 1 Software Reliability Analysis Tools Joel Henry, Ph.D. University of Montana](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f4f5503460f94c71410/html5/thumbnails/15.jpg)
15
MATT – View ResultsMATT – View Results
View input values View input values for every time-stepfor every time-step
View output values View output values for every time-stepfor every time-step Steps causing Steps causing
exceptions are exceptions are highlighted red*highlighted red*
View/edit output View/edit output exception info and exception info and settingssettings
![Page 16: 1 Software Reliability Analysis Tools Joel Henry, Ph.D. University of Montana](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f4f5503460f94c71410/html5/thumbnails/16.jpg)
16
MATT – Advanced MATT – Advanced Exceptions*Exceptions*
Advanced Exceptions allow exception Advanced Exceptions allow exception detection based on multiple exception detection based on multiple exception criteriacriteria A combination of output portsA combination of output ports Disjoint rangesDisjoint ranges
Create separate A.E definition for each range that needs to be Create separate A.E definition for each range that needs to be tested tested
Do a combinational process based on all the A.E definitionsDo a combinational process based on all the A.E definitions Overall system reliabilityOverall system reliability
Create multiple A.E definitions based on system specificationsCreate multiple A.E definitions based on system specifications Perform a combinational process based on all the defined Perform a combinational process based on all the defined
Advanced ExceptionsAdvanced Exceptions
![Page 17: 1 Software Reliability Analysis Tools Joel Henry, Ph.D. University of Montana](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f4f5503460f94c71410/html5/thumbnails/17.jpg)
17
MATT – Advanced MATT – Advanced Exceptions Exceptions Combination of output portsCombination of output ports
In the example shown below, an exception is said to have occurred In the example shown below, an exception is said to have occurred when:when:
Stick output is between 5 and 10Stick output is between 5 and 10 Alpha output is greater than 20 radsAlpha output is greater than 20 rads
After clicking the After clicking the processprocess button, one discovers: button, one discovers: 7 exceptions occurred from the given exception definition7 exceptions occurred from the given exception definition The time steps where the 7 exceptions occurredThe time steps where the 7 exceptions occurred The output values that produced the 7 exceptionsThe output values that produced the 7 exceptions
![Page 18: 1 Software Reliability Analysis Tools Joel Henry, Ph.D. University of Montana](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f4f5503460f94c71410/html5/thumbnails/18.jpg)
18
MATT – Advanced MATT – Advanced ExceptionsExceptions Disjoint RangesDisjoint Ranges In this example, two separate exception In this example, two separate exception
definitions are specified:definitions are specified: (1) Alpha producing values between 0 - 4 rads(1) Alpha producing values between 0 - 4 rads (2) Alpha producing values between 15 – 19 rads(2) Alpha producing values between 15 – 19 rads
These two exception definitions are combined, These two exception definitions are combined, and exception results are calculated in aggregate.and exception results are calculated in aggregate.
![Page 19: 1 Software Reliability Analysis Tools Joel Henry, Ph.D. University of Montana](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f4f5503460f94c71410/html5/thumbnails/19.jpg)
19
MATT – Advanced MATT – Advanced ExceptionsExceptions
Overall System ReliabilityOverall System Reliability By combining multiple A.E. By combining multiple A.E.
definitions, systems can be stress-definitions, systems can be stress-tested for accuracy and reliability.tested for accuracy and reliability.
![Page 20: 1 Software Reliability Analysis Tools Joel Henry, Ph.D. University of Montana](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f4f5503460f94c71410/html5/thumbnails/20.jpg)
20
MATT - StorageMATT - Storage
Save complete test setup for future useSave complete test setup for future use Input, State, and Output settingsInput, State, and Output settings Advanced Exception SettingAdvanced Exception Setting Input MatrixInput Matrix Output MatrixOutput Matrix
Import and export custom input Import and export custom input matricesmatrices Comma delimited files (*.csv)Comma delimited files (*.csv) MATLAB data files (*.mat)MATLAB data files (*.mat)
![Page 21: 1 Software Reliability Analysis Tools Joel Henry, Ph.D. University of Montana](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f4f5503460f94c71410/html5/thumbnails/21.jpg)
21
GISTGIST
Reads MATT test data filesReads MATT test data files Allows freehand specification of test Allows freehand specification of test
datadata Saves new MATT test data filesSaves new MATT test data files
![Page 22: 1 Software Reliability Analysis Tools Joel Henry, Ph.D. University of Montana](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f4f5503460f94c71410/html5/thumbnails/22.jpg)
22
GIST – Manage Input SetsGIST – Manage Input Sets
Add/Remove inputs Add/Remove inputs for altered modelsfor altered models
View graphs of any View graphs of any inputinput
Edit any inputEdit any input
![Page 23: 1 Software Reliability Analysis Tools Joel Henry, Ph.D. University of Montana](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f4f5503460f94c71410/html5/thumbnails/23.jpg)
23
GIST – Edit InputsGIST – Edit Inputs
Use simple click Use simple click and drag to create and drag to create new input valuesnew input values
View original input View original input values as drawingvalues as drawing
![Page 24: 1 Software Reliability Analysis Tools Joel Henry, Ph.D. University of Montana](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f4f5503460f94c71410/html5/thumbnails/24.jpg)
24
RATTRATT
Reads MATT test result filesReads MATT test result files Calculates reliability, probabilities, Calculates reliability, probabilities,
and completeness measuresand completeness measures Supports analysis of multiple test Supports analysis of multiple test
files (test suite)files (test suite) Exports to MS Excel and populates Exports to MS Excel and populates
analysis charts and graphsanalysis charts and graphs
![Page 25: 1 Software Reliability Analysis Tools Joel Henry, Ph.D. University of Montana](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f4f5503460f94c71410/html5/thumbnails/25.jpg)
25
RATT - AnalysisRATT - Analysis Load several MATT Load several MATT
test files at oncetest files at once Perform analysis Perform analysis
Input coverageInput coverage Output coverageOutput coverage MTTFMTTF Probability of Probability of
failure failure EtcEtc
View simple View simple results in RATTresults in RATT
![Page 26: 1 Software Reliability Analysis Tools Joel Henry, Ph.D. University of Montana](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f4f5503460f94c71410/html5/thumbnails/26.jpg)
26
RATT – ExcelRATT – Excel®® Tool Tool
Load RATT files Load RATT files into Excel®into Excel®
View graphs View graphs showing coverageshowing coverage
View coverage on View coverage on multiple multiple inputs/exceptions inputs/exceptions at the same timeat the same time
![Page 27: 1 Software Reliability Analysis Tools Joel Henry, Ph.D. University of Montana](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f4f5503460f94c71410/html5/thumbnails/27.jpg)
27
Current StatusCurrent Status
Windows and UNIX versions of Windows and UNIX versions of MATT readyMATT ready
Windows versions of GIST and RATT Windows versions of GIST and RATT readyready
Working with STEREO projectWorking with STEREO project Testing MATLAB/Simulink® models Testing MATLAB/Simulink® models
using MATTusing MATT
![Page 28: 1 Software Reliability Analysis Tools Joel Henry, Ph.D. University of Montana](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f4f5503460f94c71410/html5/thumbnails/28.jpg)
28
Upcoming FunctionalityUpcoming Functionality
MATTMATT View intermediate state values from simulationView intermediate state values from simulation Create and use custom test-typesCreate and use custom test-types Load models inside MATT Load models inside MATT
Allows MATT to launch outside of MATLAB®Allows MATT to launch outside of MATLAB®
GISTGIST Being integrated into MATT for more seamless Being integrated into MATT for more seamless
useuse RATTRATT
Being revised to accommodate Advanced Being revised to accommodate Advanced ExceptionsExceptions
![Page 29: 1 Software Reliability Analysis Tools Joel Henry, Ph.D. University of Montana](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f4f5503460f94c71410/html5/thumbnails/29.jpg)
29
Conclusion – Testing Conclusion – Testing EnhancedEnhanced
Using software tools to aid testing Using software tools to aid testing allowsallows More tests to be runMore tests to be run Each test to be run more quicklyEach test to be run more quickly Test results to be easily stored and Test results to be easily stored and
referencedreferenced
![Page 30: 1 Software Reliability Analysis Tools Joel Henry, Ph.D. University of Montana](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f4f5503460f94c71410/html5/thumbnails/30.jpg)
30
Conclusion – Tools Conclusion – Tools EnhancedEnhanced
Improved MATT capabilities allowImproved MATT capabilities allow Systems to start at any state with easeSystems to start at any state with ease More advanced exception catchingMore advanced exception catching
![Page 31: 1 Software Reliability Analysis Tools Joel Henry, Ph.D. University of Montana](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f4f5503460f94c71410/html5/thumbnails/31.jpg)
31
Questions and Contact InfoQuestions and Contact Info
Joel HenryJoel Henry [email protected]@cs.umt.edu MATT, RATT, and GISTMATT, RATT, and GIST http://www.cs.umt.edu/RTSL/matt/http://www.cs.umt.edu/RTSL/matt/
MATLAB and Simulink users MATLAB and Simulink users