expanding our testing horizons

Expanding our Testing Horizons

Mark MicallefPEST Research Lab, MaltaWednesday, May 3, 2023

Two Worlds

• Understanding• Clean Solutions• Complete• Perfect

• Value• Pragmatic• Working

Idea: Catching problems earlier makes them easier to fix

Static Analysis

ParserSource Code

Analyser

IntermediateRepresentation

Analysis

Static Analysis

Source Code

The Problem Static analysis tools can identify thousands of alerts This leads to cognitive overload… … which in turn leads to… … the technique being abandoned

Concept: Actionable Alert Identification Techniques (AAIT)

Some discoveries

Priority of alert

Size/Length of the

method

Size/Length of the class

Age of alert Date of last modification

Code churn Complexity of a method

Nesting of a line of code

Code (line) coverage

0

1

2

3

4

5

Very useful Somewhat useful Slightly useful Irrelevant Missing

Num

ber

of In

terv

iew

ees

A context-specific AAIT

Expert SystemAlerts

ContextFiltered and Ranked

Alerts

Idea: We should be aware of the effectiveness of our test suites

Statement coverage can be misleading

public int multiply(int x, int y){

return (x y); }

@Test public void testMultiply(){ assertEquals(5, multiply(5, 1));}

*/

Passesx

Mutation Testing

Mutation Testing

Test Suite

Mutant 1 Mutant n

Program

Problems with Mutation Testing

Generating mutants is expensive Executing tests is expensive Susceptible to equivalent mutants Manual investigation of unkilled mutants

Applying context

Version n

δ

δ

δ

δ

Version n+1

δ

δ

Version n+1 changes

δ δ

δ

δ

δ

δ

The results

Code Churn Affected LOC

Unchanged code

Decrease in mutants

Decrease in execution time

Low 12 99.8% 91% 91%Medium 60 98.8% 62% 88%High 720 85.6% 46% 89%

Insight: Testing will never provide guarantees that bugs do not exists

Testing Activities

Test Planning Test Case Design Execution Reporting &

Management

Testing Activities

Test Planning Test Case Design Execution Reporting &

ManagementRuntime Testing

Testing(Multiple Paths) Runtime Testing

(Single user-generated Path)

Runtime Testing

Problems with Runtime Testing

Defining mathematical properties may not come naturally to people

Performance Overheads How do you fit this into your

development process?

Applying Context

Given I am a bank teller processing a transactionAnd account 67 has $100 and account 113 has $50When I transfer $50 from account 67 to account 113Then account 67 will have $50 And account 113 will have $100And this should happen within 3 secondsAnd the transfer should be loggedAnd a receipt should be printedAnd the respective clients should be notified

Relevant to feature

Cross-Cutting Criteria

Applying Context

Cross-Cutting Criteria

Feature 1 Feature 2 Feature 3

What can I do?

You can… Periodically scan academic journals for new ideas

Try Google Scholar, set up alerts for your favourite topics Sell the benefits of research to yourself and

colleagues Actively seek out collaborations with research groups,

local or otherwise Consider pursuing research for academic credit

yourself or sponsor and employee to do so

Some collaboration models

Provide a case study Sponsor an intern Hand off a problem to a research group Contribute to a research trust

Thank you

http://www.um.edu.mt/ict/cs/pest@markmicallefmmicallef@gmail.com