advanced software engineering 1 advanced software engineering: software testing comp 3705 (lecture1)...
TRANSCRIPT
Advanced Software Engineering 1
Advanced Software Engineering: Software
TestingCOMP 3705 (Lecture1)
Sada Narayanappa
Anneliese Andrews (Chair DU)Thomas Thelin
Carina Andersson
Advanced Software Engineering 2
Lectures Theory + discussions Cover the basic parts of software testing
1. Introduction
2. Black-box, Reliability, Usability
3. Inspections, white-box testing
4. Lifecycle, documentation
5. Organization, tools
6. Metrics, TMM
7. Research presentation
Technical
Overview
Technical / Manager
Managerial
Economic
Advanced Software Engineering 3
Examination
Written exam based on the book and lab sessions
Lab sessions (approved) Project/presentations are grades See class web site for Assignment policy
Advanced Software Engineering 4
This week
Read course program Project
• Read Projects in Software Testing/Pick research• Decide Research/subject• Discuss papers with me – describe why is it interesting
Lab• Prepare lab 1
Read Burnstein 1-3• Prepare Burnstein 4,12
Advanced Software Engineering 5
Schedule
Read
• Course program
• Projects in Software Testing Check homepage If needed we will have
• Extra Lab dates
• Ask TA with Lab work
Advanced Software Engineering 6
Facts about testing
System development: 1/3 planning 1/6 coding 1/4 component test 1/4 system test[Brooks75] Planning
Coding
Component test
System test
Advanced Software Engineering 7
Why use testing?
• Risk mitigation• Faults are found early• Faults can be prevented• Reduce lead-time• Deliverables can be reused• …
Advanced Software Engineering 8
Why do faults occur in software? Software is written by humans
• Who know something, but not everything
• Who have skills, but aren’t perfect
• Who don’t usually use rigorous methods
• Who do make mistakes (errors) Under increasing pressure to deliver to strict deadlines
• No time to check, assumptions may be wrong
• Systems may be incomplete Software is complex, abstract and invisible
• Hard to understand
• Hard to see if it is complete or working correctly
• No one person can fully understand large systems
• Numerous external interfaces and dependencies
Advanced Software Engineering 9
TMM – Test Maturity Model
Stages or levels to move from
• Chaotic, unstructured testing model
TO
• Systematic, process driven approach Software is an engineering discipline Requires
• Quality in Process
• Quality in Product
Are important
Advanced Software Engineering 10
TMM Test process Evaluation
• Guided by Capability Maturity model (CMM)• Stages or levels to evolve from and to• Each level, except Level 1, has structure• Level 2 and above have:
• Structure
• Goals
• Organization structure
Advanced Software Engineering 11
Internal structure of TMM
Level
Testing
Capability
indicateMaturity Goals
Maturity Sub goals
Activities/tasks/responsibilities
Implementation
and Organizational
adaptation 3 Critical views
Manager Developer User/Client
Supported By
Achieved By
Organized By
Addresses
Contains
Advanced Software Engineering 12
Test Maturity Model
Advanced Software Engineering 13
Advanced Software Engineering 14
TMM Levels• Level1- No process defined; debug/testing are
same• Level2- Test & Debug tools/Test plan/Basic test
process• Level3- Test Org/Technical Training/Lifecycle/
Control & Monitor, support V Model• Level4- Review process/test
measurement/Software quality Evaluation – Test Logging with severity
• Level5-Defect prevention/Quality/Process optimization
Advanced Software Engineering 15
Good enough quality
To claim that any given thing is good enough is to agree with all of the following propositions:
• It has sufficient benefits
• It has no critical problems
• The benefits sufficiently outweigh the problems
• In the present situation, and all things considered, further improvement would be more harmful than helpful
James Bach, IEEE Computer, 30(8):96-98, 1997.
Advanced Software Engineering 16
Quality attributes – ISO 9126
Advanced Software Engineering 17
Quality attributes
Advanced Software Engineering 18
Origins of defects
Defect sources
Lack of educationPoor communicationOversightTranscriptionImmature process
Impact of software artifactsErrors
Faults / Defects
FailuresImpact from user’s
viewPoor quality software
User dissatisfaction
Fault model
Advanced Software Engineering 19
Whoops, that’s my calculator
Advanced Software Engineering 20
Testing, Verification & Validation
Definition 1 Verification
• is the product right? Validation
• is it the right product?
Definition 2 Verification
• satisfies the conditions at the start of the phase
Validation
• satisfies the requirements
TestingThe process of evaluating a program or a system
Advanced Software Engineering 21
Definitions
Failure is an event, fault is a state of the software caused by an error
• Error – human mistake
• Fault / Defect – anomaly in the software
• Failure – inability to perform its required functions
• Debugging / Fault localization – localizing, repairing, retesting.
Advanced Software Engineering 22
• A TEST CASE consists of:• A set of inputs
• Execution conditions
• Expected outputs
• A Test is:• Group of related test cases
• Procedures needed to carry out the test case
Definitions
IEEE Definition
Organization may define additional attributes
Advanced Software Engineering 23
Scripted and non-scripted testing
• In scripted testing test cases are pre-documented in detailed, step-by-step descriptions
• Different levels of scripting possible
• Scripts can be manual or automated
• Non-scripted testing is usually manual testing without detailed test case descriptions
• Can be disciplined, planned, and well documented exploratory testing
• or ad-hoc testing
Advanced Software Engineering 24
Test oracle
• An oracle is the principle or mechanism by which you recognize a problem
• Test oracle provides the expected result for a test, for example• Specification document• Formula• Computer program• Person
• In many cases it is very hard to find an oracle• Even the customer and end user might not be able to tell
which is the correct behaviour
Advanced Software Engineering 25
Test Bed
• Environment contains
•all the hardware and software to test software component/system
• Examples:
•Simulators
•Emulators
•Memory checkers
Advanced Software Engineering 26
Other Definitions
• Important to understand the following definitions
•Quality– degree of meeting specified requirement
•Metric – quantitative measure
•Quality metric•Correctness – perform the function
•Reliability –perform under stated condition
•Usability – effort to use the system
•Integrity – withstand attacks
•Portability/maintainability/interoperability …
Advanced Software Engineering 27
Principle 1 – purpose of testing
Testing is the process of: exercising a software component using a
selected set of test cases, with the intent of
1.Revealing defects
2.Evaluating quality
Advanced Software Engineering 28
Principles
2: A good test case – When the test objective is to detect defects, then a good test case is one that has high probability of revealing a yet undetected defect(s)
3: Test result – The results should be inspected meticulously
4: Expected output – A test case must contain the expected output
Advanced Software Engineering 29
Principles
5: Input – Test cases should be developed for both valid and invalid input conditions
6: Fault content estimation – The probability of the existence of additional defects in a software component is proportional to the number of defects already detected in that component
7: Test organization – Testing should be carried out by a group that is independent of the development group
Advanced Software Engineering 30
Principles
8: Repeatable – Tests must be repeatable and reusable
9: Planned – Testing should be planned
10: Life cycle – Testing activities should be integrated into the software life cycle
11: Creative – Testing is a creative and challenging task
Advanced Software Engineering 31
Goals of the course
A test specialist - trained engineer- have knowledge of test-related
Principles/processes/measurements, standards, plans, tools, and methods, and
learn how to apply - testing tasks to be performed.
Knowledge
Skills
Attitudes
Advanced Software Engineering 32
www.swebok.org
Advanced Software Engineering 33
www.swebok.org
Advanced Software Engineering 34
Defect classes and Defect repository
Functional DescriptionFeatureFeature interactionInterface description
Requirement/SpecificationDefect Classes
Algorithmic and processingControl, Logic, and sequence DataModule interface descriptionExternal interface description
Design Defect Classes
Algorithmic and processingControl, Logic, and sequence typographical data flow DataModule interface Code documentationExternal hardware, software
CodingDefect Classes
Test HarnessTest designTest procedure
TestingDefect Classes
Defect ClassesSeverityOccurrences
Defect Repository
Defect reports/analysis
Defect reports/analysis
Advanced Software Engineering 35
Lab sessions
Preparation, Execution, Report
1.Black-box testing
2.Usage-based testing and reliability
3.White-box testing
4. Inspection and estimation
5.Software process simulation
Advanced Software Engineering 36
Project: Option 1
• Learn a specific area of software testing
• Collect and summarize research information
• Critical thinking beyond the written information
• Present information in a structured way
• Peer review
Advanced Software Engineering 37
Project: Option 1
Research: solve a research problem; survey the state-of-the-art and identify the research problems in some area; develop and justify an extension to an existing technique; etc.
Evaluation: apply and evaluate a technique or evaluate a commercial testing or, analysis tool.
Practical: Use an existing technique to test a system or design and implement a prototype for a system.
Advanced Software Engineering 38
Project: Option 1
Read Projects in Software Testing Divide in groups (2-3 persons)
Discuss with me
http://www.cs.du.edu/~snarayan/sada/teaching/COMP3705/FilesFromCD/Project/Project_SwTest.pdf
Advanced Software Engineering 39
Project: Option 2
Research paper presentation Find an interesting paper Many research papers we come across
during class – pick one for presentation Have four paper presentation Choose your team and prepare Reading paper takes time – start early