test and verification solutions
DESCRIPTION
Improved time to market through automated software testing Mike Bartley, TVS. Test and Verification Solutions. Agenda. The potential benefits & drawbacks of automation Compare with HW verif Can automation reduce TTM? Automating static & dynamic testing Deciding what to automate - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Test and Verification Solutions](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56813d45550346895da703b4/html5/thumbnails/1.jpg)
Test and Verification Solutions 128 October 2009
Test and Verification Solutions
Improved time to market through automated
software testing
Mike Bartley, TVS
![Page 2: Test and Verification Solutions](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56813d45550346895da703b4/html5/thumbnails/2.jpg)
2Test and Verification Solutions 228 October 2009
Agenda
• The potential benefits & drawbacks of automation• Compare with HW verif• Can automation reduce TTM?• Automating static & dynamic testing• Deciding what to automate• Choosing your automation tools• Case study
![Page 3: Test and Verification Solutions](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56813d45550346895da703b4/html5/thumbnails/3.jpg)
3Test and Verification Solutions 328 October 2009
The potential benefits of automation
• Reduced costs• Improved quality• Improved employee motivation• Improved TTM
![Page 4: Test and Verification Solutions](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56813d45550346895da703b4/html5/thumbnails/4.jpg)
4Test and Verification Solutions 428 October 2009
The potential drawbacks of automation
• Shelf-ware!• Maintenance• Skills base• Missing failures
![Page 5: Test and Verification Solutions](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56813d45550346895da703b4/html5/thumbnails/5.jpg)
5Test and Verification Solutions 528 October 2009
HW verification The mechanics of complex test bench
Driver Stimulusgenerator
Test ScoreboardFunctionalCoverage
Design
Under
Test
Monitor
assert
constraint
addr data
Assertions
Coverage
Coverage
![Page 6: Test and Verification Solutions](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56813d45550346895da703b4/html5/thumbnails/6.jpg)
6Test and Verification Solutions 628 October 2009
The mechanics of finding a bug
Design Under Test
PropagateStimulate
…..01010101
…..01001101
…..10011010
…..01001101
01100101…..
11110101…..
00010101…..
Actual
Results
Expected
Results
Compare
Observe
![Page 7: Test and Verification Solutions](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56813d45550346895da703b4/html5/thumbnails/7.jpg)
7Test and Verification Solutions 728 October 2009
Can automation reduce TTM?
• Getting test results sooner• Catching bugs earlier• Stabilising the software sooner• Assessing the stability of the software
![Page 8: Test and Verification Solutions](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56813d45550346895da703b4/html5/thumbnails/8.jpg)
8Test and Verification Solutions 828 October 2009
Automating static testing
• Types of bug found• A quick win• Beware false failures• Introduce filters slowly• Developers not testers (check in checks, IDE)
• No testing of intent
![Page 9: Test and Verification Solutions](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56813d45550346895da703b4/html5/thumbnails/9.jpg)
9Test and Verification Solutions 928 October 2009
Automating dynamic testing
• Test plan• Test design specification• Test case specification• Test procedure specification• Test execution• Test log• Test incident report• Test summary report
![Page 10: Test and Verification Solutions](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56813d45550346895da703b4/html5/thumbnails/10.jpg)
10Test and Verification Solutions 1028 October 2009
Deciding what to automate
• Complexity of the test environment• The test level • Frequency of execution• Ease of automating pass/fail• Test stability and repeatability
![Page 11: Test and Verification Solutions](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56813d45550346895da703b4/html5/thumbnails/11.jpg)
11Test and Verification Solutions 1128 October 2009
Choosing your automation tools
• Who writes test test?• Matching skill set• Separate orthogonal concerns• Portability• Ease of adding/banning tests• Integration into configuration/build system• Cost of the tool?
![Page 12: Test and Verification Solutions](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56813d45550346895da703b4/html5/thumbnails/12.jpg)
12Test and Verification Solutions 1228 October 2009
Case Study – Background
• Includes chips, accelerator cards, and software
• All work with industry-standard x86-based systems
![Page 13: Test and Verification Solutions](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56813d45550346895da703b4/html5/thumbnails/13.jpg)
13Test and Verification Solutions 1328 October 2009
Case Study – Background
• Low level host driver code– Code on host computer communicates with the chip on the board in
the PCI slot
• Libraries of code– Runtime library– Mathematics acceleration libraries (ACML, MKL, …)
• Application code– Linpack, Montecarlo, ….
• Code running on the chip– This had parallel C extensions not supported by our chosen tool
• Software models of the hardware• In-house compiler for the code running on the chip• Mainly C and C++, some assembler
![Page 14: Test and Verification Solutions](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56813d45550346895da703b4/html5/thumbnails/14.jpg)
14Test and Verification Solutions 1428 October 2009
Case study – automation details
• Integrated into configuration/build system• Automated mostly lower level tests• Developed internally by SW Engineers• Offshored manual tests• Offshored some automated test development• Integration of static analysis• Test maintenance
![Page 15: Test and Verification Solutions](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56813d45550346895da703b4/html5/thumbnails/15.jpg)
15Test and Verification Solutions 1528 October 2009
Case study - results of automation
• Improved quality• Shortened release schedules• Release schedules hit• Reduced TTM• ROI?
![Page 16: Test and Verification Solutions](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56813d45550346895da703b4/html5/thumbnails/16.jpg)
16Test and Verification Solutions 1628 October 2009
Questions?
• Mike Bartley
• 07796 307958