T24 Special Topics

5/2/2013 3:00:00 PM

New Testing Standards Are on the

Horizon: What Will Be Their Impact?

Presented by:

Claire Lohr

Lohr Systems

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

Claire Lohr

Claire Lohr has been an active professional in the computer field for thirty years, with the last twenty years emphasizing software process improvement and testing. Claire provides training - design, authoring, and instruction - and consulting services for a wide variety of both government and commercial clients. She is a Lloyd’s Register trained ISO 9000 Lead Auditor and has been trained to perform Software Capability Evaluations for the SW-CMM. Claire was the chair of the Working Group for the IEEE Std 829-2008 Software and System Test Documentation, and has served on both the IEEE Computer Society’s Software and Systems Engineering Standards Committee and the IEEE Computer Society’s Standards Advisory Board.

4/16/2013

1

1

New Testing Standards on the Horizon: What Will Be

Their Impact?

Claire L. LohrLohr SystemsP.O. Box 2998

Reston, VA 20195703.472.5457

clohr@computer.org

2

Topics

• Why bother with standards?

• Traditional objections

• What’s available (free first)

• Impact of ISO 29119

• Examples

• How to start

• Additional resources

• ?’s

4/16/2013

2

3

Why bother with standards?

• Common problems have common solutions

• Standards are reviewed and modified by groups of very experienced ($$$$) practitioners

• You can’t search the Internet for information unless you know what it is commonly called (or if you have never heard of it)

4

Traditional ObjectionsPast Present

Inconsistent and incomplete

Merging together

Overkill Includes high integrity and lower integrity choices

Just theoretical Includes examples

Expensive Increasingly free

Time consuming ROI of 1:5-6

4/16/2013

3

5

What’s available: free

1. nist.gov

Special Publication 800-115 Technical Guide to Information Security Testing

and Assessment

2. open-stand.org

Commitment for the future by IEEE, IETF, IAB,

Internet Society and W3C.

6

What’s available: free

3. Certification bodies of knowledge

• swebok.org Summary of testing (& rest of SW engineering)

• pmi.org The Software Extension to

the PMBOK® Guide

• istqb.org Glossary of testing terms & multiple tester certifications

• buildsecurityin.us-cert.gov Software Assurance (SwA) Competency Model

4/16/2013

4

7

4. Other free resources

• http://pascal.computer.org/sev_display/index.action SEVOCAB definitions

• http://standards.iso.org/ittf/PubliclyAvailableStandards/index.html free ISO standards

What’s available: free

8

What’s available: free (example)

SWEBOK testing section

Fundamentals

Levels

Techniques

Measures

Process

Tools

4/16/2013

5

9

What’s available: not free

7925-1 Vocabulary7925-2 Component testing

829 Doc’s1008 Unit test1012 V&V1028 Reviews1044 Defect categories

12207 Software life cycle15026 Integrity levels25010 Quality requirements25051 Quality reqm’ts for COTS

10

What’s coming

29119 Systems and software engineering—Software testing—• Part 1: Concepts and definitions

• Part 2: Test processes• Part 3: Test documentation• Part 4: Test techniques

4/16/2013

6

11

ISO 29119-1 Concepts and definitions

4. Definitions

6. Approaches

7. Automation

8. Defects

5. Concepts

12

ISO 29119-2 Test processes

6. Organizational

8. Dynamic8.1 Introduction8.2 Design & implementation8.3 Environment setup & maintenance8.4 Incident reporting

7. Management

4/16/2013

7

13

ISO 29119-3 Test documentation

5. Organizational5.2 Policy NEW5.3 Strategy

6. Management6.2 Plan6.3 Status Report6.4 Completion Report

14

ISO 29119-3 Test documentation

7. Dynamic7.2 Design7.3 Case7.4 Procedure7.5 Data requirements NEW7.6 Environment requirements NEW7.7 Data readiness report NEW7.8 Environment readiness report NEW7.9 Actual result7.10 Test Result7.11 Execution Log7.12 Incident report

4/16/2013

8

15

ISO 29119-4 Test techniques

• Equivalence partitioning• Classification tree• Boundary value analysis• Syntax testing• Combinatorial• Decision table• Cause-effect graphing• State transition• Use case• Scenario• Error guessing• Random

5.2

16

D

A B

E

ISO 29119-4 Test techniques

5.3 • Statement• Branch• Decision• Condition• Data flow

6. CoverageAnnex B Specification-based examplesAnnex C Structure-based examples

4/16/2013

9

17

Impact of ISO 29119Past Future

Inconsistent and incomplete

Consistent and complete in one place

Overkill Includes high integrity and lower integrity choices

Just theoretical Includes examples

Expensive Increasingly free

Time consuming ROI of 1:5-6

18

Example #1

I want to improve our testing processes

Test Manager

29119-3 5.3 Strategy (for focus)• Risk management• Test selection and prioritization• Test documentation and reporting• Test automation and tools• Configuration management• Incident management

4/16/2013

10

19

Example #1 (test manager)

29119-2 8.3 Incident reporting processa) Test results are analyzedb) New incidents are confirmed, if anyc) New incident report details are createdd) The status and details of previously-

raised incidents are determined

e) Previously-raised incident report

details are updated as appropriatef) New and/or updated incident reports are communicated to the relevant

stakeholders

20

29119-3 7.12 Incident report 1. Timing information2. Originator3. Context4. Description of the incident

5. Originator’s assessment of severity6. Originator’s assessment of priority7. Risk8. Status of the incident

Example #1 (test manager)

Add related

incidents

4/16/2013

11

21

Example #2

I want to document better so I can get another job

Tester

29119-2 Annex A Example Test Design*29119-3 Annex H Example Test Design* Chose this one – more

complete and more

summary

22

How to start

Suggested steps for transition

1. Survey your current baseline

2. Choose most valuable “next steps” (Kaizen)

3. Implement a pilot

4. Share what works

5. Continue to improve

4/16/2013

12

23

Additional Resources

www.iso.org

www.ieee.org

www. global.ihs.com

The Roadmap to Software Engineering

Standards: a Standards Based Guide, James W. Moore, 2006

24

Questions?

4/16/2013

13

25

The End