simula.simula.2992

Glimpses from the History of Software Testing

Sagar SenSoftware Engineering Dept., Simula Research

Laboratory

1

Wednesday 20 November 13

The Growth of Testing (Bill Hetzel)

Until 1956 - Debugging oriented 19571978 - Demonstration oriented 19791982 - Destruction oriented 19831987 - Evaluation oriented 19882000 - Prevention oriented

2


Bug in a Technical System

Quadraplex telegraphMangled Messages! The 26-year-old engineer combined diplex and duplex circuits to send two messages in each direction using changes in current direction and strength. The problem in this approach was the false break in a messages signal created by the changing polarity of the electromagnet in the diplex circuit when the current switched direction.

Edison worked around this by building what he later called a bug trap to isolate the unwanted break so that it wouldnt interfere with the meaning of the Morse-coded signal.

Thomas Alva Edison

3


Insomnia squad debugging with 3 hrs sleep

Bug in a Technical System

In 1889 the reporter for a British newspaper wrote, Mr. Edisonhad been up the two previous nights working on fixing a bug in his phonographan expression for solving a difficulty, and implying that some imaginary insect has secreted itself inside and is causing all the trouble. And in a 1916 article about Edisons insomnia squad, a reporter noted that they worked like fiends when they [were] fishing for a bug. That means that they are searching for some missing quality, quantity, or combination that will add something toward the perfect whole.

4


Talking Doll (1890) with Phonograph

Can you guess the tune from the sound?5


First Computer Bug

In 1947 Grace Hopper and her staff started using the word bug to describe technical glitches when they found a dead moth [above] in Relay 70 of Harvard's Mark II computer.

6


Checking a Large Routine (The idea of Assertions)(1949)

7




8


Difference between debugging and testing! (1957)

9

Make sure a program runs! Make sure a program solves the problem!

Debugging Testing

As computer applications increased in number, cost, and complexity, testing came to assume more significance because of greater economic risk

Charles Bakerreviewed

Dan McCrackens First Bookon Computer Programming


Gerald Weinberg Assembles First Test Team(1958)

A test team became suddenly important for the first human-flight mission

10


Chess Players, Music Lovers, and Mathematicians (1969)

Chess player, dude with IQ 172 (self-taught pianist at 10), crystallographer from Oxford, PhD. in English (Columbia University), ex-fashion model, proto-hippie, and more chess players...

Applicants

11


Testing Software Modeled by Finite-State MachinesTsun S. Chow (1978)

1. Tsun Chow derived test sequences from a finite-state model of a program (comment printer for instance)

2. This played a key-role in the origins of model-based testing of software

12




13


Glen Myers Definition of Testing (1979)

14

The process of executing a program with the intent of finding errors

Demonstration model aimed to show that program had no faults. But Myers wanted to demonstrate that the program had faults.


Software Engineering Economics (1980)

15

Barry Boehmstated that the cost to fix error increases

exponentially the later you detect it!




16


Federal Information Processing Systems(FIPS) Methodology

17

No single VV&T technique can guarantee correct, error-free software. However, a carefully chosen set of techniques for a specific project can help to ensure the development and maintenance of quality software for that project.

Product Evaluation

Analysis Review Test


Growth of Professionalism in Testing

18

Testing takes off!

JobsNational Meetings

PublicationsStandards

Test managerLead testerTest analyst

Test technician

ANSI/IEEE Standard for Test Documentation

(1983)Standard for Unit

Testing (1987)

Testing Conference (Bill Hetzel)

Industrial Testing Conf. (1984)Academic Testing Workshops

Testing books at average rate of 2/year

since Myers (1979)


Information Systems Manifesto (1984)

19

Studies distribution of defects, showing that over half of the defects have their root cause in poorly defined requirements.




20


Exploratory Testing (1988)

21

Exploratory testing is a style of testing in which we learn about the behavior of a system by designing a small test, executing it immediately,

using the information gleamed from the last test to inform the next.

Simultaneously learning about the system while designing and executing tests, using feedback from the last to inform the next.

Explore areawith resourcesto discover information

Explore editing profileswith the authentication featureto discover surprises


Fuzz/Random Testing (1988)

22

Automated or semi-automated, that involves providing invalid, unexpected, or random data to the inputs of a computer program. The program is then monitored for exceptions such as crashes, or failing built-in code assertions or for finding potential memory leaks.

Shortcomings...1. Usually finds very simple faults

2. Poor code coverage

3. Hard to catch boundary value conditions

"The Monkey" was a Macintosh application developed by Steve Capps prior to 1983. It used journaling hooks to feed random events into Mac programs, and was used to test for bugs in MacPaint.


Classification Tree Method (Grimm, Grotchman 1993)

23

Any system under test can be described by a set of classifications, holding both input and output parameters. (Input parameters can also include environments states, pre-conditions and other, rather uncommon parameters).


Pairwise Test Strategy (1996)

24

Pairwise testing (Cohen)In their paper The Combinatorial Design Approach to Automatic Test Generation, authors David Cohen, Siddhartha Dalal, Jesse Parelius and Gardner Patton introduce pairwise testing. They base their combinatorial design theory to generate tests on the usage of the Automatic Efficient Test Generator (AETG), during projects at Bellcore.

The AETG system uses new combinatorial algorithms to generate test sets that cover all valid n-way parameter combinations. The size of an AETG test set grows logarithmically in the number of test parameters.

Test Every OS with Browser, Every Browser with Resolution, and Every OS with Resolution. How?


25

Cem Kaner, James Bach, Brian Marick and Bret Pettichord start the Context-Driven school of testing based on the principle that 'The value of any practice depends on its context'.

Context-driven Testing (1999)

Best practices coming from academia didnt work so well when you tried to apply them in industry; that best practices published in books by people who worked in the telecom business didnt work so well for people in Silicon Valley;

Bachs Schools of Testing

Routine School Math School Quality School Context-driven School


Testing in Recent Times

26


Continuous Integration (Martin Fowler, 2000)

27

Never comment out failing tests.Helps avoid one developer's work in progress breaking another developer's copy.

Battle royale between coders (Feature Content) and testers (Feature Working)!


Test-Driven Development (2002)

28

Test-Driven Development (Beck)In the book Test Driven Development: By Example Kent Beck publishes the software development technique in which a test is written before the functionality is coded.


How to Break Your Software: James Whittaker (2002)

29

A recipe book of 23 attacks to break softwareExample: Generating System Faultsa) Monitor your systemb) Inject tests of your choiceFaults to look for...- Out of Memory

- Locked Memory

- Out of disk space

- CRC errors


Testing Experience

30

Testing Experience magazine launched in 2008The magazine Testing Experience is launched by the German company Daz & Hilterscheid. A magazine to share experiences and opinions of testers from the industry.

After all humans are the weakest linkin causes for software failure!


Types of Testing Today: A Plethora!

31


The Future of Testing?

32


What next?

33

1. Journey of Software Testing: We journeyed from the debugging era (before 1956) to test-driven development where testing comes first.

2. Testing tools? The only tools that made an impact in the industry are (a) test automation tools such as continuos integration (b) Coverage analysis tools (Peter Russell, Testing Experience in Future of Testing)

3. Glimpses from the future of testing? (a) Design for testability (b) Lean is hot. Lean is Every. Test Management should be Lean (c) Agile Testing (d) Test data management (rejuvenation of test data) (e) Automatic test generation with human-in-the-loop (f) effective testing system of systems.

4. 22nd Century Testing? Software that is completely verified automatically. What will happen to testers?


simula.simula.2992

Documents