03.03.2014, copenhagen

© author(s) of these slides 2014 including research results of the research network ES and TU Darmstadt otherwise as specified at the respective slide

Malte Lochau

TU Darmstadt

[email protected]

ES Real-Time Systems Lab

Prof. Dr. rer. nat. Andy Schürr

Dept. of Electrical Engineering and Information Technology

Dept. of Computer Science (adjunct Professor)

www.es.tu-darmstadt.de

Multi-Objective Test Suite Optimization for Incremental Product Family Testing

(joint work with Hauke Baller, Sascha Lity and Ina Schaefer)

03.03.2014, Copenhagen

ES – Real-Time Systems Lab

2 | 03.03.2014 | PLE

Contents „What is a good ordering for testing the members of a product

family?“

Extension of the classical Test Suite Minimization Problem Weights Bounds Variability

Approx. optimal solution(s) for Multi-Objectives via ILP Solving

Incremental heuristic solution based on dynamic programming to appear at ICST‘2014


3 | 03.03.2014 | PLE

Test Suite Minimization Problem [Harrold et al., 1993]

t1

r1 r2 r3 r4 r5 r6

t2 t3 t4…

Test Cases

Test Requirements

Complete Test

Suite


4 | 03.03.2014 | PLE

All-Transitions Coverage R = {r1,r2, …, r8}

Complete Test Suite TS (4 Test Cases)Minimal Complete Test Suite TS‘ (1 Test Case)

Example: Model-Based Testing

r1

r2

r3

r4 r5

r6

r7

r8

test

model

Test Suites

TS = {(r1r2),(r1r4),(r3r7r8),(r3r5r6)}

TS

‘ = {(r1r2r3r5r4r7r8r5r6)}


5 | 03.03.2014 | PLE

Minimum Set Cover Problem

Given:

Set of Requirements

Set of Test Cases

Sub sets satisfying Requirement

Find:

Minimum sub set such that

NP-Complete

Assumptions:

All Test Cases are relevant:

Each Requirement is satisfiable:

ii TtRrTt ::

Ø: ii TRr


6 | 03.03.2014 | PLE

Cost/Profit Test Suite Optimization [Malishevsky et al. 2006, Xu et al. 2012, Rout et al. 2013]

t1

r1 r2 r3 r4 r5 r6

t2 t3 t4…

Test Cases

Test Requirements

Adequate Test

Suite

2 1 4 3 1 2 Testing Profits pTS

4 2 1 4 Testing Costs cTS

+ + + + + =

+ + + =

≥ Profit Goal kp

≤ Cost Bound kc

∑=7Minimal Test Suite ∑=8


7 | 03.03.2014 | PLE

Test Suite Profits pTS

TS = {(r1r2),(r1r4),(r3r7r8),(r3r5r6)}

pTS=(2+1)+(2+1)+(2+2+4)+(2+2+1)=15

TS

‘= {(r1r2r3r5r4r7r8r5r6)}

pTS=(2+1+2+2+1+2+4+2+1)=15

TS

‘‘ = {(r1r4r5r2),(r3r7)}

pTS=(2+1+2+1)+(2+2)=10

TS‘‘

is Adequate Test Suite for kp = 10

Example: Model-Based Testing with Costs/Profits

Test Suite Costs cTS

TS = {(r1r2),(r1r4),(r3r7r8),(r3r5r6)} cTS = 2+2+3+3 = 10

TS‘= {(r1r2r3r5r4r7r8r5r6)} cTS = 9

TS‘‘ = {(r1r4r5r2),(r3r7)} cTS = 5+2 = 7

r1(2)

r2(1)

r3(2)

r4(1) r5(2)

r6(1)

r7(2)

r8(4)

Profits are counted once

kp=10


8 | 03.03.2014 | PLE

Minimum Partial Weighted Set Cover Problem [Könemann et al., 2011]

Given:

Set of Weighted Requirements R

Set of Weighted Test Cases T

Profit goal kp

Find:

Sub set with minimal such that

pTS ≥ kp

where and

TSt

TTS twc

ØTST

RriRTS

i

i

rwp


9 | 03.03.2014 | PLE

Product Line Analysis


10 | 03.03.2014 | PLE

Sample-based Product Line Analysis

[Cohen et al. 2006], [Gustafsson 2007], [Oster et al., 2010], [Perrouin et al, 2010], [Kim et al. 2011], [Johansen et al., 2012], [Henard et al., 2013], [Haslinger et al., 2013],

….


11 | 03.03.2014 | PLE

Family-based Product Line Analysis

[Krishnamurti et al., 2006], [Gruler et al., 2010], [Classen et al. 2010], [Gnesi et al. 2011], [Cordy et al., 2013], …

𝑂 (?∗𝜑 )


12 | 03.03.2014 | PLE

Incremental Product Line Analysis

∅

[Batory et al. 2007], [Engström et al. 2011], [Lochau et al. 2012], …


13 | 03.03.2014 | PLE

T={t1, …, t15}

t7=(r1r6r8r7)

t8=(r1r4r5r2)

t11=(r1r4r7)

t12=(r9r10r8)

t13=(r3r11)

t14=(r1r4r11)

t15=(r3r7)}

Family-based Product Line Test Suite Design

4 Product Variants

r1(2)

r2(1)

r3(2)

r4(1) r5(2)

r6(1)

r7(2)

r8(4)

r9(2)

r10(2)

r11(8)


14 | 03.03.2014 | PLE

Complete Minimal Product Line Test Suite [Cichos et al. 2011]

Assumptionsif then satisfies on

Given:

Set of Requirements R

Set of Test Cases T

Set of Product Variants P

Sub sets satisfying Requirement

Sub sets executable on product variant

Sub sets relevant for product variant

Find:

Minimum sub set such that


15 | 03.03.2014 | PLE

Constraint Minimum Set Cover Problem

t1

r1 r2 r3 r4 r5 r6

t2 t3 t4…

Test Cases

Test Requirements

p1

Product Variants

p2 p3

5 2 3Product Costs/Profits ppTS

2 1 4 3 1 2

4 2 4 1 Testing Costs

Testing Profits

+ + =≥ Profit Goal kpp

pTS ≥ kp

rpTS ≥ krp


16 | 03.03.2014 | PLE

Constraint (Minimum) Partial Weighted Set Cover Problem

Given:

Set of Weighted Requirements

Set of Weighted Test Cases

Set of Weighted Product Variants

Profit Goals

(Cost Bound

Find:

Sub set (with minimal such that

and and is an Adequat Product Line Test Suite

Ø:, TSTTRrPp

RriRTS

ijj

ij

i

rwrp

Ø: TSTTRr

PpjpTS

ijj

i

j

pwpp


17 | 03.03.2014 | PLE

Integer Linear Programming (ILP)

(Integer) Decision variables Linear objective function Linear constraints


18 | 03.03.2014 | PLE

ILP for Product Line Test Suite Optimization

Product pj selected xj=1

Test Case t selected into TS yt=1

Requirement ri satisfied zi=1

Binary Variables

ppTS

Product selected Requirements must be satisfied

pj select Test Case from Test Case Set

NOT pj OR One Test Case from Test Case Set

tp

For all Requirements

(of a Product)

Requirement satisfied For all Products, a Test Case must be selected

Again (ri Set) (NOT ri OR Set)

r t

Test Cases must be applied to a Product

tp

Priotization

ppTS ≥ kpp ppTS - Δpp = kpp

ppTS ≥ kpp

cTS ≤ kc

rpTS ≥ krp

Differences must not be negativ

Normalization Value


19 | 03.03.2014 | PLE

Re-Formulation as Product Selection Problem

ILP finds Adequate Test Suite (if exists), but Problem is NP-hardAnother Point-of-View: Incremental Product Selection

t1

r1 r2 r3 r4 r5 r6

t2 t3 t4

p1 p2 p3

rpTS

cTS

ppTS


20 | 03.03.2014 | PLE

Adequate Product Sub Set

Given:

Set of Weighted Requirements

Set of Weighted Test Cases

Set of Weighted Product Variants

Profit Goals

(Cost Bound

Find:

Sub set P such that an adequate test suite exists with


21 | 03.03.2014 | PLE

Incremental Product Selection

p1

p2p3

p4

𝑃𝑆𝑖 𝑃𝑆𝑖+1

p1

p2p3

p4

p5

P∖𝑃𝑆𝑖

p7

p6p8

p9

p5

is adequate

is adequate

𝑇 𝑆𝑖

+

𝑇 𝑆𝑖

+

𝑇

profit

goals reached?

select next product

cost bounds

exceeded?

no adequate solution

found


22 | 03.03.2014 | PLE

Product Selection Heuristic1. Are Profit Goals reached? (rpTS ≥ krp and ppTS ≥ kpp)

2. No:a. For each unselected Product

i. While unsatisfied Requirements exist, select one: With high profit and uniqueness/rareness Select a satisfying Test Case :

With the ability to satisfy many other open Requirements for many other Products and low cost

b. Rank Products and Select By: Product profit, resulting Test Case costs, and Requirement profit (weighted)

c. Recalculate ppTS, rpTS and cTS

d. Is Test budget excelled (cTS > kc)

i. Yes: “Error” Ende. Goto 1.

3. Yes: “Success” End

r

n

?

?

tn

?

?

?

cf. [Harrold et al., 1993]

[Harrold et al., 1993]

+ Products


23 | 03.03.2014 | PLE

Experimental ResultsVarying: Number of: Requirements, Test Cases, and Products Mapping: Requirements/Test Cases and Test Cases/Products Weights

In Total: 51 generated Data SetsTime Limit: 24h

t1

r1 r2 r3 r4 r5 r6

t2 t3 t4

p1 p2 p3

50, 100, and 500 Test Cases

50, 100, and 500 Products

50, 100, and 500 Requirements

Mean and Variance

Mean and Variance

-/o/+

-/o/+

Heuristic is 21.5 times faster

68.6% less

accuracy

Efficency improvement factor: 127.48


24 | 03.03.2014 | PLE

Discussion & Future Work

Summary Heuristic produces useful results and scales Feasible tradeoff between accuracy and efficiency

Ongoing and Future Work Backtracking if bounds are exceeded Search-based optimization techniques Methods and criteria to find appropriate weights and bounds Beyond ILP: Dynamic weight function, On-the-fly generation of test artifacts, …

Application to Real-World Case Studies


25 | 03.03.2014 | PLE

Industrial Case Study: Heidelberg Ion-Beam Therapy Center

DCU Case Study forBeam Cycle Creation

Complex Automation Engineering Software

Highly-Configurableand Runtime Adaptive

Re-Engineered as (D)SPL ~ 200 Features (Boolean

and Non-Boolean) 150% Test Model

~ 640 Transitions SPL Implementation in C

~ 175.000 LOCTreatment

Rooms

Gantry

DCU-Z DCU-R DCU-R DCU-P


26 | 03.03.2014 | PLE

Model-Driven Development of DCU Code

Parameters

+

User Data.h

Hand-written

ECA Code

.c .h+

States

Tra

nsiti

on T

rigge

r

.hAutomata

Stubs.c

insert

deploy

generate

SFT


27 | 03.03.2014 | PLE

DCU SPL Test Model Specification

Device Control Unit

Z

SD T

TS

R

RB

RK

Device ControllerManual

Experiment

Quality Assurance

Adjustment

Therapy

Standby

Idle

Version

Checksum

Timing Master Parameter

EnableSyncSupplyFrequence

EnableGenRTBMasterClock

Device Control Parameter

EnableSwitchOn

EnableShutdown

BeamRequestTakeOverRefValues

DelayTimeActivatedelay: [0,100]

DelayTimeFeedbackdelay: [0,100]

TimeoutFeedbacktimeout: [0,100]

RefValueTHoldMaxdelay [0,100]

AddRtbDelayTimedelay : [0,100]

ErrorMonEn

LowCurrentTimertimer: [0, 100]

Mode 0

Mode 1

Mode 2

Mode 3

Mode 4

DCU Type Parameter

DCU Master Parameter

Type

bDevice Control

b

P

Ramped

bbbbb

Operation Mode

MeasuringMode

RtbTimingEn

150% SFT

+


28 | 03.03.2014 | PLE

Thank Your For Your Attention!

Integrated Model-based Testing

of Continuously Evolving Software Product

Lines

Multi-Mechanismen Adaption für das

künftige Internet

03.03.2014, copenhagen

Documents