SIUTATION TESTIXG OF HERiRCHIC.4L FJNTE STATE 4LACHI'(TES JS THE RATIO^.^ ROSE REAL TDIE TOOLSET

A thesis submitted to the Department of Compiiting k Information Science in conformity mith the requirements

for the degree of Master of Science

Queen's Cniversity

Iiingston. Ontario Canada

September '200 1

Copyright @ Bradley Michael Graham. 3001

Bibliothèque nationale du Canada

Acquisitions and Acquisitians et Bibliographie Services senrices bibliographiques

395 Wellington Street 395, rue WdKingtOn Otfawa ON KlAOM OaawaON KlAON4 Canada Canada

The author has granted a non- exciusive licence allowing îhe National Libraq of Canada to reproduce, ioa., distriiute or seil copies of this thesis in miçrofom, paper or electronic formal.

The author retains ownership of the copyright in this thesis. Neitha the thesis nor substantiai extracts fiom it may be printed or otherwise reproduced without the author's permission.

L'auteur a accordé une licence non exclusive permettant a la Bibliothèque nationale du Canada de reproduire, prêter, distribuer ou vendre des copies de cette thèse sous la forme de microfiche/iihn, de reproduction sur papier ou sur format électronique.

L'auteur conserve la propriété du droit d'auteur qui protège cette thèse. Ni la thèse ni des extraits substantiels de celle-ci ne doivent être imprim6s ou autrement reproduits sans son autorisation.

Abstract

Reactim systems are a class of systems characterized by the maintenance of a continuous and timely interaction with their environment. Testing reactive systems cliffers from testing sequential systems: instead of checking simple p a i s of inptits and outputs. seqiiences of inputs and outptits have tn btx checked. Testing techniques cannot be basecl strictly on a functional view of the .ioftware rather they musc be baseci on sorne history preserving view such as one of the finite state machine formalisms.

Mirtacion testing is a heuristic technique originally clevelopetl to encourage the construction of tiigh qiiality test mites for seqtiential software. [t offers a potentially highly automatable. hiit also c.oniptitationally rspensive way of creating thorough test suites. The success of mutation testing is largely dependent on a set of miitation operators definecl for the language clescribing the software.

This thesis ewliiates the feasibility effectivcness of perfoming mutation testing of rextive systenis rnorlelerl hy active nbjccts tvhose behavioiirs are definecl hy hierarchicat finite srntp ma- chines (HFSM) as implenientecl in the Rational Rose Real Time toolsct (RoseRT). This is nchierecl by proposing a set of mutation operators. building tools to perforni mutation testina. pcrformirig esperiments on a sniall set of HFSMs. and analyzing the results and obserntions nintle rliiririg the eperiments.

This thesis presents preliniinary eviclence indicating that mutation testing of RoseRT HFSSIs is botii feasible (n-ith tool support) and can resiilt in the creation of faiilt revealing test secluences-

Acknowledgement s

I would like to thank my thesis supenisor Dr-Terry Shepartl for his Friendship. ideas. giii<lnnc~. and iissistance in cotripleting this thesis.

t tvoultl Iih CO rleclicatc th& thesis to my children Benjamin ancl Selena nncl to rny wife Iinchleeri without wtiose love. support, and encouragement it n-oiiId not have heen possible.

Contents

1 Introduction 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1 Motivation L

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Problem 1 9 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Objectives -

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4 Contribtitions 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5 Synopsis 4

3 Software Testing Background 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1 Prcliniiriiiries 5

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Test Ornclcs 6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 Test .4 cleqtiacy 6

2.4 Classification of Test Data .i rfequacy Criterion . . . . . . . . . . . . . . . . . . . . . 8 2.4.1 L'sage Baseti: Test cases corne From the espectetl usage profile . . . . . . . . . S

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.2 Black Bos Techniques S . . . . . . . . . . . . . . . 2.4.3 M i t e Box: Test caes corne from the source c:o<le 10

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5 EvaluatingCriterion 14 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . '1.5.1 Summary LS

A Detailed Look At Mutation Analysis 16 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Mutation Testing 1G . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Nutation Theory 17

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Applying Operators 18 3.4 Mutation Operators: Siruulating Progrmimer 1Iistakes . . . . . . . . . . . . . . . . . 20 3.5 \Veaknesses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5.1 Selective Mutation 23 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6 A Xotion of Fauit Size '1-1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6.1 Weak Mutation 26

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.7 Summq- 27

4 Testing State Machine Based Software 39 4.1 Finite State Machines (FSBI) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Protocol Testing 31 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 EFSM nntl HFSM 3'2

4.4 Testing Estendeci hierarchicai FSSI . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 4.5 Statecfiart Slutation Operacors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 4.6 Surnrnq . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

5 The RoseRT Toolset 39 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 Overview 39 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.7 Structure .. . 41 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 Behaviour 43

3.4 Transitions and Triggers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 - - 3.2 Active Object issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

5.5.1 Pnonty inversion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-32 Spontaneous Events 46 3.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

6 Mutation Testing of RoseRT Models 48 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1 hliitation Operators For RoseRT 48

6.2 h Proposed RoseRT CSM Slucation Operator Set . . . . . . . . . . . . . . . . . . . -50 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.1 FSSI Operators Xi

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.2 Coverage Operacors 51 6.2.3 C'SLL-RT/RoseRT Featrires hased Operator Set . . . . . . . . . . . . . . . . . 52

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3 Sumrnary 53

7 A Mutation Testing Systern for RoseRT [MTS-RTJ Using TXL 5 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7. 1 Orenîew of the hplementation -M

. . . . - - . . . . . . . . . . . . . . . . . . . . . . . . . 7.2 Performing Mutations .... 56 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.21 TSL ar

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2 . 2 TSL Grammnr 58 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.3 T-U Transformations 59

. . . . . . . . . . . . . . . . . . . . . . 7.3 Caps~~le File Format .... . . . . . . . . . . . 60 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.4 Basic Slutatiori Process 61 - . . . . . . . . . . . . . . . . . . . . . . . r . ù Esample Uutation Operator Program . T-IT 63

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.6 S u r t i m q 64

8 Creating and Running Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Testing Prowss and Automation

. . . . . . . . . . . . . . . * . . . . . . . . . . . . . . . . 5.2 Testing RoseRT Models ... . . . . . . . . . . . . . . . . . . 8.3 Thc RoseRT Generic Test Hrirness (RT-TcstHarness)

. . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.4 The RT-TestCaseSlanager Progam 5.3 RT-SescRunner . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.6 Data Xnalysis: RT-SttperDiff . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Y.7 Hunting for 11i1t.mts

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . $.Y Finciing Faiilts in the Original . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.9 Surnrriary

9 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.1 Capsules Siimmarq-

. . . . . . . . . . . . . . . . . . . . . . . . . . 9.1.1 Expianation of Table Elements

. . . . . . . . . . . . . . . . . . . . . . . . . . 9 2 E-xperiment 1: CeU Phone Component . . . . . . . . . . . . . . . . . . . . . 9.3 Luperitrient 2: CeU from a Garne of Life Sfodel

. . . . . . . . . . . . . . . . . . . . . . . . . . 9.4 kperiment 3: TratFic Light Controiier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3 Esperirnent 4: Cririse Control

. . . . . . . . . . . . . . . . . . . . . . . . 9.6 Expecimcnt a: Radar Control Cnit (RCG) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.7 Ek~eriInent 6: Pop Machine

. . . . . . . . . . . . . . . . . . . . . . . . . 9.8 E.qeriment 7: Electronic \Varfare Jammcr . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.9 Other C a p d e s Esatnined

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.9.1 Radar module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.10 Ind idu id Operacor Discussion

. . . . . . . . . . . . . . . . . . . . . . 9.10.1 Covenge Operators, CV T. CSE . CSS 9.10.2 Action DeIete Operators . .WT . .AD E' ADX . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.10.3 TRD: Transition DeIete . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.10.4 SDL: State DeLete

9.10.5 internai Transition Hiitory Operators ES (Internai To e l temal) . STI (es - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ternd To internai)

. . . . . . . . . . . . . . . . . . . . . . . . . . . 9.10.6 TAT Transition Alter Target

. . . . . . . . . . . . . . . . . . . . . . . . . . . 9.10.7 SCP: exchange Choice Point

. . . . . . . . . . . . . . . . . . . . . . . . . . . 9.10.8 TCP: Transition Choice Point

. . . . . . . . . . . . . . . . . . . . . . . . . . . 9.10.9 TGT: Trmit ion Guard True

9.LO.lOTCk Transition Change Action . . . . . . . . . . . . . . . . . . . . . . . . . 93 . . . . . . . . . . . . . . . . . . . . . . . 9.10.1 1 TTG: Transicior: TriGger ruchangeci 94

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 .IL Operator Siimmary 95 9.12 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.13 Initial Test Sequences 9s . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.1-1 Killing mutants 94

9.15 Equivalent SIutants . . . . . . . . . . . . . . . . . . . . . . . . ... . . . . . . . . . 100 9.16 Finding Fades . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . LUC) 9.17 Strong vs Firm SIiitation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . LOI 9-18 Oracling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 9-19 Feasibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103

9 . 19.L Xiimber of SItitnnts . . . . . . . . . . . . . . . . . . . . . . . . . ... . . .. 103 9.192 CPC Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 9.19.3 Friiilts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-4

9.2U PreambIcs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 9.21 Testing yoiir owm code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . L05 922 Siimniary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105

10 Conclusions 107 10.1 Siunmary of Resiilts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 IO.2Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . LOI1 10.3 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 10.4 Concliitling Remnrks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112

10.4.1 Design .\ Iiitation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . LL2 10.42 Operator Rationale . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . LI2

A Testing Glossary 117

B Example Capsule Text File 118 3.1 Bandling C++ Within Capsule Fies . . . . . . . . . . . . . . . . . . . . . . . . . . . LlS B.2 Capsule File for Cruise Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119

C TXL Grammar Files Used For Parsing RoseRT Capsule Files L28 C.l Grammar file: SIarkC'pCapsuIe-grm . . . . . . . . . . . . . . . . . . . . . . . . . . . 128

. . . . . . . . . . . . . . . . . . . . . . . . . . . C 2 Grammar file CapGrnmmarVien.grm 13-t C.3 Granimar file: protocol.grni . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133

D RoseRT Capsule Statemachine Behaviour 140 . . . . . . . D.1 Top state . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ... 140

D.2 Actions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . L-LO D.3 Transitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141

D.3.1 Kinds of transitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 . . . . . . . . . . . . . . . . . . . . . . . . . D.3.2 initiai point and initiai transition 141

. . . . . . . . . . . . . . . . . . . . . D.3.3 Transitions to and fiom the initiai state !41 D.3.4 Xo initial transition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 D.3.5 Group transitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142

D.4 Choice points . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . L42 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D.41 Choice Point Example I-12

D.5 Aistory hierarchical state machines . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-E . . . . . . . . . D.5.L Continuation kinds s h d o w histo- deep h i s t o - and defauit 143

D.52 B i t o ry Esample . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243 D.6 Transition se,- ents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143

D.6.1 dunction points . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Z U D.6.2 Joining transition segments . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144

. . . . . . . . . . . . . . . . . . . . . . . . . . . . D.6.3 Continuation Kind Property 144 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D.7 Tran5ition SeIection Rules 145

D.Ï.1 Transition Selection Rides &ample . . . . . . . . . . . . . . . . . . . . . . . 146

E Capsule Statemachine Diagrams 148 . . . . . . . . . . . . . . . . . . . . . . . . . . E.1 Experiment 1: Ce11 Phoue Component 143 . . . . . . . . . . . . . . . . . . . . . . . . . . € 2 Experinient 2: Cc11 from Came of Life 149 . . . . . . . . . . . . . . . . . . . . . . . . . . E.3 Experiment 3: Traftic Light ConcroUer 1.50

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . E.4 Esperiment 4: Cruise Control l5-L . . . . . . . . . . . . . . . . . . . . . . . . . . . . E.5 Experiment 5: Radar ConcroI Cnit 156

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . E.6 Experiment 6: Pop Machine 157 . . . . . . . . . . . . . . . . . . . . . . . . E.7 Experiment ï: Electronic Warfare . larnmer 123

List of Tables

4.1 Proteiim/FSM Mutation Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Proteum/ST Mutation Operators

8.1 Data hnalysis Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.1 E'cperimenc 1: PBX Ce11 Phone . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2 Experïment 2: Ce11 from Grune of Life . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3 Experirnent 3: Traflïc Light Controller

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.4 Esperiment 4: Cruise Coritrol . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.5 E'cpcriment 5: Radar Control L'nit

9.6 E~periment 6: Pop &tachine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.7 Experiment E EWS Jammer

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.8 Euperiment 8: Radar . . . . . . . . . . . . . . . . . . 9.9 Percentage of test sequences required to kill mutants

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.10 Firm and Strong mutation results

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D.1 Continuing Junctions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D.2 Terminating Junctions (to h i s t o ~ )

List of Figures

2.1 Criteria Subsuniptiori . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . L:!

5.1 RoseRT SIotlel Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . -41 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2 RoseRT Class Diagram -E

5.3 Stnicture Diagram for SomeCapsuIe . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 .. .4 C S ~ I . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5

7.1 Directon Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 . . . i 2 RoseRT Sliitation System:Essentiai Workfiows . . . . . . . . . . . . . . . . . . . . . . a i

7.3 RoseRT SIiitation System:Essential Workfiotvs . . . . . . . . . . . . . . . . . . . . . . 61 7.4 TAT example state machine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

8.1 Skeleton Test Harness Class Diagam RoseRT SIutation System:Essential Worlcfiow . GS 5.2 RT-TestHarness Class Diagrani . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . fi9 8.3 RT-TestCasehIanager . . . . . . . . . . . . . . . . . . . . . . . . .... . . . . 71

9.1 Continuation Transition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Y0

0.1 Xo Initial Transition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . L-1:' D.2 Choice Point . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-13 D.3 An Esaniple Cse of History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14-4 D.4 . Junction Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14s D.5 Transition Selection Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147

E.1 Ce11 Plione TOP state . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-18 E.7 CeIl Phone Conncctccl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149 E.3 CeII From Game of LXc TOP state . . . . . . . . . . . . . . . . . . . . . . . . . . . . . i-19

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . E.4 Traffic Lights TOP State 1.50 E.3 T r f i c Lights initializirig . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . E.6 T r a c Lights FIashingAmber 131 E.7 Traffic Lights Working . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132 E.S Traffic Lights Transitioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . L.53

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . E.9 Traffic Lights Green 153 E.10 Cruise Control TOP state . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 E.11 Cniise Control On . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5-1 E.13 Cruise Control Active . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.55 E.13 Radar Control Unit TOP State . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.56 E.14 Pop Machine TOP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157 E.L5 Pop Machine initialize . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1% E.16 Pop Slachine Operating . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.58 E.17 EWS Jammer TOP State . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158 E.18 EWS Jammer working . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159

Chapter 1

Introduction

This thesis d l evaliiate the feasibility and effectirencss of performing mutation testing of iictive

objects modeled by hierarchical finite state machines (HFSbIl as implementetl in the Rational Rose

Red Time toolset ( RoseRT). This will be achievcrl by proposing a set of mutation operators. building

tools to perform miitation testing, performing esperiments on n sniall sec of HFSMs. ancl anaiyzing

the results ancl obsenations made during the experiments.

1.1 Motivation

Reactive çysterns iire a class of systenis whose main feaciirc is a tirnely interaction with ttic c r i 6

rotiriient. Ititcraçtion occttrs when the sgstem respoods to an environmental stimulus by performing

sonie output action. Subway control. patient hospitai monitoring, communication protocols. ancl

proccss controllers are esamples of such systems. The software cotitroiiiig reactive systems is often

niodeled by some finite stace machine formalism. Testing and d ida t ion of finite state machine basecl

specifications is done in generai by reachability andjsis. many t g e s of simulation. and by several

test sequence generation methods [Fabbrit 981. Yewer CASE tooIs such as the Rationai Rose R e d

Time tool can be used to design reactive systerns as coIlections of collaborating active objects n-hose

behavïours are motleled by HFSSIs. These CASE toois dso pronde automatic source code gener-

ation Gom the graphic design. HFSM have ex~ended scate variables. guarded transitions. choice

points. hierarchy. and may use timers to initiate events. Tt does not appear that conventional state

machine testing techniques are su5cient to handle the increased complexitj- of EtFSM PIÜiierburg

Testing's usual purpose is to reveal the existence of errors [Myers 791. From thi point of rien: n

successful test is one that reveals an error in the software. Generating successful tests is a rMcuit

problem that is impracticd in most cases. It usuaUy requires that a tester assume a <les!srructîve

attitude toward the software and then try to find or guess al1 of the w q s in which ~ h e software can

be made to fail. However it is impossible. in practice. to think of al1 of the ways a piece of non-tri~iaI

software rnay fail. Thcrefore systematic methods of testing software have been developed to guide

testers.

Nutation testing is a heuristic testing technique that offers a potentinlly higiily nucomatable. hrit

d so compiitationally espensive may of generating fault revealing test cases. In practice, mutatiuri

testing appears to find manu of the ivays that a piece of software can fail. Much rcsearch lins heen

donc (since 1978) on source code mutation tcsting. However the large compiitationd costs. iirriong

d i e r factors. liave preveritecl its wiciespreatl use.

The major rnotimion for this rcsearch is to investigate chc cost and effectiveness of iisirig rnu-

tatiun techniques to tcst active objects modeled by KFSMs.

1.2 Problem

The rr.sc!arch prablem acidressed bu this thesis is describecl as foIlows: 1s mutation testing a IeasibIe

iitid effective' nnay of testing reactive software modeled by hierarchicai finite state machine (HFSSII

formalisrns and created through gaphical eciiting tools stich as the RoseRT toolset? Siore speciti-

cally are the weaknesses of HFSM mutation testing as prohibitive as the weaknesses OC source code

mutation testing? And are the strengths of source code mutation testing as strong ivhen they are

applied to HFSM testing?

1.3 Objectives

The objectives of this thesis are as fo1Iows:

Provide a brief background on the general problem of softnare testing.

' EBecrive at revealing errors in a pmgram.

O Provide a thorough background on conventional source code bascd mutation testing.

Provide a brief background on testing finite scate machine software.

O Propose a set of mutation operators for use on HFSMs as implemented aith the RoseRT

toolset.

O Builcl tools to automate the process of applying mutation operators to RoseRT HFSJIs.

O Biiilcl a tool to facilitate the creacion of test sequenceç for RoseRT HFSSIs.

O BiiiId a generic test harness to reduce the cost and t h e reqiiirccl to test RoseRT BFSSIs

O Builcl i001s to help in the analysis of miicacion testing resiilts.

O Pmforni preliniinary esperiments CO evaluate each of the proposecl mutation operators and CO

wxiuate rriutatiori testing of RoseRT HFS3Is.

1.4 Contributions

The original contribiitions of this ttiesis inc1ucic the foIlowing:

O Critical analpis of the background Literature on mutation testing.

O Proposed mutation operators for RoseRT capsules and a prelirninary analysis of eadi operator.

O CEG for RoseRT capsuIe text files.

O 5Iutant creation toob basecl on TSL.

O Test sequence creation tool (RT-TestCaseS1anager)-

O Generic test Iiarness capsule ( RT-TestHames).

O Test results ,malpï~ tools.

O Development of an efficient procedure for performing mutation testing on capdes.

O Xnaiysis of the feasibility of mutation testing of RoseRT capsules.

O Identification of some areas of future rvork.

1.5 Synopsis

This thesis is rlivided into 10 chapters as foiiows:

This chapter introtluces the problem. defines the objectives. and lists the contributions of the

t hesk .

Chapters TNO. Three. and Four present background material relatecl to software test in^. niu-

cation analusis. and finite state machine testing.

CIiapter Five presents the RoseRT toolset.

Chapcer Six presents the proposed RoseElT mutation operators and their rationde.

a Chapter Seven contains an overview of the TSL based mutarion systcm and clisciisses the

application of mutation operacors to RoseRT models.

Chapter Eight covers creating test cases. running thern against the mutants, and iuidyzing

che results.

Cliapter Xine prcsents the resiilts of niucation cesting experinients concliicted in order to t m i -

iiate the operators and the feasibility of niucation cescing in RoseRT.

Chapter T m contains the conclusion. and discusses future work.

Chapter 2

Software Test ing Background

This chapter provitles a brief background on the generd subject of software testing. It provicles some

clefinitions and cliscusses the rneaning of correctness befoce t iming to the subject of test adequacy.

classification of testing techniques and how they cem be cornparecl. [t provides the context for

Chapter 3's cliscussion of mutation testing.

2.1 Preliminaries

In this work the terni prograni is tised CO refer to any individuaiIy testable piece of software. wtiicti

ma>* take the forn~ of a single function. a module. or a coniplete program. -4 test case is a single t-rihie

(possibly a vector) of progam input, which enables a single execution of n progrruu. X test set or

test suite is a finite collection of test cases. Testing definitions ÏmplicitIy assume an unrcalistic but

simple "functiond"program architecture: a batch program which is given a single input. cornputes

a single result and terminates. Each cornputation is a function stnctIy of the program's ciment

input. not on prior program States. i.e. there are no sequences of inputs. .Uthough this assumption

is iinreaiiitic in generd. it has proven to be a uselur simplification in esploring and communicating

ideas in testing theory [Hamiet 941. Testing softs-xe in whkh the current output is a Function of

both the curtent input and of previous inputs (often described by h i t e state machine formalisms)

requires test cases that are time sequences of input vectors. Lesting finite state basecl software

is ciiscussed in chapter 4. rlppendk A contains Further definitions and additional terms used and

abused in the testing iiterature.

2.2 Test Oracles

A fundamental assumption of sohware testing iç thac there esists sonie mechanism. separate €rom

the program under test (PUT). for cletermining whether or not the resiilts of a test eseciition are

correct. Siicti a niechariisrn is referrecl to as an oracle. K e c m consitler some process in n-hich test

cascs are taken as inputs hy both the PCT and the oracle. computations are performed. ancl resriits

are comparecl.

Itleally a e Kant an automateci oracle which always gives the cnrrcct ansaer. Often howewr.

oracles are people. who mostly compute by hanrl thc expectecl results slioidcl he. This is n

tlifficiilt and error pronp process, thtis hiinian oracles can make many rnistakes (e.g. Xmmatin et

al. [.4rnrnann 9-11). -4s n resiilt rvheii rhere is n iliscrcparicy bectvceii clic oracle and clic prugrarii.

chc rcsult reportcc1 bu c h ornclc rnust often be verificd before clecicling thnt the PCT is in error.

Hiiman oracles often use whatever specifications exkt for the PCT. However ic is often the case thac

specifications (if the' esist) contain errors hr ther conipouncling the oracle problem.

2.3 Test Adequacy

Testing is plaguetl bu problenis of complcsity. Tt is impossible to completeIy test a prognni n-ich

evcn moclerate complesity. One could. in theory. show program correctness by riinning the program

on e v e n possible input (exhaustive testing) and chccking the resiilts. tiowever. for any non-tririal

progam the set oFpossib1e inputs is so large. possibly infinite. that exhaustive testing is not feasible'.

For example. if a progarn ha three 16 bit integer inputs. (ranging over 63336) exhaitscive testing

woultl reqtiire running the program over lOL.' times. ifeach run took one microseconcl. this still takc

over S years to test. Red software is much more cornplex WC can never be sure that any morlerrrteiy

sizeci piece of software is correct. The best that can be clone is to perforrn lenough- testing to gain

coufitience that the program works and d l continue to work as specifietl. ï h e question becomes

%on- much testing is enough'!" The a m e r is bound up in the notion of test ndepacy.

Since a program cannot be e.xhaustivelj- tested. much of the aork in testing theory involves

findimg subsets of a program's input (Le. test case selection) that na choroughIy test a pro.pni.

The questions to be annrered are [Goodt 751:

':\ pmgram that could be euhaustively cested could be replaceci by a finite table that contains dl of the .ansivers!

a 1s there a way to test a program which is equivalent to e?diaustive testing?

O Is there a practical approsimation to e.xhaustive testing and if so how gootl is the approxirna-

cion:'

A Test selectiori cntenorr (or test criterion) is the bnsis for test case selection. A tcst cricerion

specities the set of conditions chat must be satisfiecl by a set of test cases. in [Good+ 731 the nuthors

rlefined an ideol test set as a test set that succeedsJ only wtien a program concains no mors. A

critenon for an ideal test is both reliuble ancl uolid. ,-\ criterion is reliuble if al1 the sets of r t s L vases

chat satisFy thc criterioti clecect the siune errors. In other worrls it (lofs not mattcr which satisfying

set is selected. every set will detect the sarile errors. .-\ critcrion is u d i d if for any wror in t h

prograni there is some set sacisfjing the criterion chat will reveal the error. [C;ood+Ï3] prowtl tliat

if a testing criterioti is both txlirl and reliable. and if a tcst set satistjing that criterion fin& no faults

in a program. chen the prograni contains no errors. Homevcr it has been shon-n that no dgorichm

exists that rcill dctcrmine a valid criterion for an arbicrary progani. Thcrefore the answer to the tirst

question is no: there is no way to test a non-trivial program chat is equident to eshaustive tescing.

Further the reliability requirement is too strong to be itiecl to clefine practical tcst ncleqiincy rriteria,

Findiig a criterion that is both reliable and valid and chat can also be satisfieti bu i l nianageable

number of test cases is usuaiiy not possible. Therefore the focus of test arbqitat~ rritcrin researrh

is on fiuding practicd lipprosiniatiotis (heuristics) to the icled criteria.

In practice a test atleqiiacy criterion can be used as a stopping nile to cletermine whether or not

enough testing has been done. For esample staternent coverage states chat enoiigh testing Iias h t m

done when every statement in the prograni under test h'as been esecutcd at Ieast once. Wiether

o r not this is a good criteria is a separate issue. however it is a criterion (i.e. it's a Lieuristic). A

criterion c m also be use CO measure the quality of testing. In the case of statement coverage the

percentage of statements covered by a test set gives a measure of the degree to nhich a test set

h a satisfied the statement coverage criteria. in industry chi measure couid be used as a stopping

criteria. For esample. stop testing when 85% statement coverage has been achieved. The term

adequacy criterion is often used to mean testing method.

-hocher role of a test critenon concerns the observations to be made during testing. For esarnpIe.

statement coverage requires chat the tester sornehow determine if each statement in the program

Lias been esecuted. This usualIy requires instrumenthg the program under test.

''ln this sense succe& means that z test euecures without revealing an emr .

2.4 Classification of Test Data Adequacy Criterion

The goal of a test criterion is to find a stibset ai the possible inputs to ri program that rviren feri to

the program as test cases arc to rewal fauics in the program. Since no algorithm exists co

fitid chese subsers. testers ~ e l y a n a number of techniques and heuristics that work wel1 in practice.

but largely Iack an undcriying theary The testing literaturc ciiscusses many ways EU cI;issih. these

techniques. h comrnon way of organizing testing technicps is to cntcgorize tiicni iu being risage

b a e d techniques. black box techniques. or white h m techniques.

2.4.1 Usage Based: Test cases corne from the expected usage profile

-1 usage or aperatiand baser14 criterion cari be based on stocfia.çciç infriruiriticiti about hon- t l ~

software id be used in the field. An input probability ciensity futictiori is neetlctl. cxprcssing

the probability chat somc input will actualIy occur in tise. Operational testing then proccerls bu

seIecting independent randorn points [rom the input cfomain accorcling tc, this ilistribrition. Ysrially.

an accurate operationai profile is far too detailed to obtain. and even a crude apprnxiniatioii retptires

considerable effort [Hamiet Nb]. Operational profiies ate often trsetl s i t h the grid of qtiancifying

the reliability of the PUT.

2.4.2 Black Box Techniques

Black box testing is also knorvn as functionai analysis. It is baser1 on the idea that a progam imple-

ments a function that maps d u e s from its input ciornain to d u e s in irs outpiit tfomain. W t h black

box testing the source code of the program under test is not lookecl at, rncher test cases iire tisually

i leri~ed in tetms of the program's mapping as described in the program's specificncion. Boaever

some bIack bos techniques can be nsed in cases where a program does not have a speçification or

where the specification is incomplete.

The folIon-ing techniques use characreristics of the input domain in attempting w hi1 represen-

cative sets of test cases-

4Cssge based is &a d l e d operatiooril testing and is sornctimes called mdum testing Rudom t a t i n g c m also be IUMI in the tvider context of statisticd testing from any disriburion. rarher chan the nne considermi tiere whicli is based on opentional use [Hamiet N b ] .

Equivalence partitioning

The idea behind equiwlence pnrtitioning is to partition the input domain into a small number of

.-eqiiimIence'- classes. Testing a single element from an equivalence class is incended co be equi\-alen~

to tiaving testeci al1 elenients of that class. In a sense it attempts EO b d reBabIeS test cases. This

heuristic is often b s e d on the system design description. The clomain of each input ~wïabIe that

p l u s a role in one of the components is clivided into 2 or more sets of values. These sets of wIiies rire

the equivdence partitions that are further classifiecl as being either d i d or inmiid. -4 specification

Iikc riach hook has u 6-digit number coidd resiilt in 3 equimlence classes: one vaiid. and 2 invalicl [iliet

931. .-\€ter finding the e q u i d m c e classes. test cases are constriictecl siich that as many. (possibty

d l ) ec~i i i~~~lence classes as possible are -coverecI".

Equi\dence partitioning suffers from some obvious weciknesses:

i. the assumption that the program beliaves equivdently for al1 data from a given dass is not

crue in general[Vliet 931. This is because no algorithm exists to finrl the reliabk partitions for

an? non-trivial program.

2. Testing c.orribinations of ecpiitdence class members is not considered. If a coniponent is a

fiirictioti of two twiables. x anci y. ancl if botti s and y can be partitiritied into three eqiii~tlence

p,xtitions each then there are 9 possible mys to combine exemplars from the 6 partitions.

Same of these cornLinations will probably reveal errors that are not cletectable by just itsirig

the minimum 3 test cases that are adequate for eqtridence partitioning adequacy.

3. Without iooking a t the structure of the PUT. it is probabiy impossible to dctennine dl of the

eqiiivaience classes [[Jalote 971: equivalence class boundaries rire tlefined by the structnrc of the

code.

FinalIq- it is often worthxvhiie to use the equitdence classes of the output. For an eqiiitalent

output cIass. the goal is to generate test cases such that outputs Lie in each of the output equi~aience

ciasses. This is more difficult than finding input partition classes. however output classes have b e n

found to reveai errors chat are missed by simpiy considering the input [.Jaiote 971.

:Reliabie in the sense OF [Cood- 751.

Boundary Value Analysis

Boiindary d u e anaiysis is a variation of equinlence class testing and is basecl on the observation

that software often fails on boundan: \dues (eg . off by one errors). Rather thcm testing some

rantlom element from each equivalence class. boundary d u e anaiysis concentrates on the extreme

d u e s from each class. Cornmon boundary idues are I. 0. 1. nidl. ancl Iists witti 0. 1. or r r w x

elernent(s). Extreme values from the output classes c,m also be tested.

Botinciary mlue analpis suffers from the same weahesscs as cqtiivdencc partitionirig. L i t i i s ~

~ffeçt graphing [JIeyers Ï91[.Talote 971 seeks to ansxer chc combinatorid tleficiency of etpi~alence

nntl boitnrlary value anaiysis bu tcsting cambinations of cquivalence class elements. honever it does

not scnlc well. As the software g o m in size the number of test conibinations grows exponentidly.

Thc combinationai problem is not rcstriccetl to cause-effect graphingl racher it is another facet II€

thc intractability problem pensive in tcsting.

BlackBox Testing Observations

Sirice I h c k bos testing does not reIy on the implementation. test cases CIO not become invalid if and

nhen the iniplenientation is changed (as long ris the specification cloes not change). [n ncldition.

test cases can be devetoped beforc or in parrailel n'itli the progr~un. On the other Iianti specifications

are often incorrect or incomplete *ut1 are iisiially wi t ten in natiirnl langiiage n-herc nrnhigtiity is

iiievitable. Even wlien a forrtid description Ianguage is usetl. it is likely that sonie pottioti t i f ctit*

specification has been missed or neglected: incompleteness is always a prolkri . Fiirttier. :r sp~citi-

cation itself c m becorne intractable because it is not possible to specik every possible n+ay in hi ch

software can be tisecl or abtised. Finall-. even when specifications are the resdt of a collaborative

rer~tiircttietits elicitation phase involring the customer. completeness is elusive. Ciistomers c m usii-

aliy tell whether a prototype is. or is not. n-hat they m n t . but they are unable to spec- exactly

what they n-ant until it has been buiit. Specification problems contribute approximately 30 percent

of dl bugs in software.

2.4.3 White Box: Test cases corne Çom the source code

Khïte box testing criteria. aisa h o m as g l a s box or cle,u box testing, can be dividecl into m o

classes: structurai and fauit-based criteria. Fauit based criteria are further ciassiiied as b e h g eicher

error seeding or mutation testing.

Structural Testing

A large portion of the testing literarure focuses on structural testing techniqiies [Zhu 9ïIFliet

931[.laIote 971. As a result there arc many ccchniques known. With stnrcti~rd testing. test cacç ;ire

generated based on the actual code O€ the PCT. The tester c m then observe the extent cn which

die test cases esercise n prograni's structure. The underlyirig prer~iise of strucciirat testing is chat it

is riecwisary to krinw ttiar id1 thts parts of sorne stnictural aspect (for naniple statcrnents, branches)

of 3 progrnni have bwri tuercisecf at l e s t orice. 111 practice this rnay bc tlifficiilt or impossible to

achieve.

Codc cowrage cesting is b ~ ~ c t l on the numbcc of i~istrrictions. stntements. hrnnrhes. or pnt1is

that are c~eciiteci at mn time. Code coverage c~an be cIisctrssec1 on the hasis of il prngnni's control

gaph. In clic concrol graph. nodcs rienoce actions (staterncna. and stntenicnt btocks). wIiilc the

directed edges connect actions with siibsequent actions. -1 path is a sequcncr of nodcs connected

ttiroiigh etlgs. Cyr-lcs in thc grapti correspond to Ioops in the program. Prograni esectition reqirires

following n certain path ttirough the p iph . When a node has muItipIc otrtgoing cclgcs. test cases

are designed to select among them (these are br,mches).

in the ided case. test cases traverse al1 possible paths. This is cdletl di-paths coverage and

k eqiiident to eshaiistively testing the progam [kliet 931. In general ail- pattis coverage is not

possible for the fol1owing reasons:

1. A loop often resuIts in an infinite number of paths.

2. For a branching progiïm. even in the absence of Ioops. the number of possible paths increases

eqonentially with the number of branching instructions.

3. There rnay exist some infeasible paths. manifesteci as branches that never euecute. The inka-

sible path problem is undecideable .and is the main probtem with d path anaiy5s methotls

@amiet 941.

The best that can be done in the presence of these problems is to find a coverage that espresses

the degree to which test data appro-xhates t h ideal coverage. Xany such coverage metrics hat-e

b e n studied. The most obvïous is to sMply count the number of nodes traversed. This is c d e d

statement coverage and is often used in industry (possibly because of its ease of comprehension, use.

ancl measurement). However statement coverage is rather weak as it is easy to h d test sets hi ch

give 100% statement coverage yet leave undetected faults in the program.

X srronger coverage constructs test cases that ensure that every br'anch in the control vaph is

tnken at least once. This is ail-edges or more commonly branch coverage. Again 100% coverage does

not gitarantee the absence of faults.

Another category of coverage is cdecl data-fion; analysis in wtiich the control graph is irsecl to

esamine the clefinition 'anci uses of variables dong esecution paths. X mriable gets clefinecl at ;i

certain statenient if it is assignecl a new vnliie at that statement. \:ririables are then siibsecpmtlj-

itsed in one of tno wqs: P-uses are predicate uses such 'as those in the conditionai part (if if

statcrnents. rvhile al1 other uses are C-uses (computation uses). A path from a "definition" to a

.'iise" in which the variable is not redefinecl is calleci a definition clear path.

In the dl-u.ve.5 .vtrritegy, test cases are constructecl that traverse definition clear paths between

cach clefinition of a wriable and each subseqiient use. This ensures thnt cach possible use of n

clefinition has been tested [Rapps+ 351. There are several other wiriants of rhta Hom- strate+ siich

as d l P-uses. dI C-uses etc. that c m be seen in Figure 2.1.

Figure 2.1: Criteria Subsumption.

in practice coverage methods may not be cost effective. Some code staternents' such icj ~ r r a r

banders. may be ctificttlt to execute and also be uniiiely to occiir in use6. Aclding test cases to

cover such statements may not be worth their expense.

Fault Based Criteria

Faiilt b;setl adequacy criteria measures test adequacy accortling to the ability of a test set to cletect

known faults in a program. Error seeding and mutation testing are examples of t h i cnterion.

Error Seeding

Error sceding is a technique for estimating the number of faiilts in the software. It \vas origirinted

by CIarlan Slills in 1972 [I<night+ $51. The original iclea ans to estimate the reliability of ii piece of

software basetl on an estimate of the number of faults remaining in it. Artifcial faults are seetled into

the progam. <and the testing is etduated based on the number of seeclecl faiilts found. Error seecling

is based on the assumption that artificial errors are as ciifficuit to detcct as naturally occirrring

errors. Cnfortiinately artificial faults are not necessarily representative of natural faults ancl this

assumption has proven to be untme in general [Gardiner 991.

Mutation Testing

Sliitation testing is the process of systeniatically constriicting small syntnctic variations. called

mutants. of a progam and then determining if they c m be clistingished by esisting test cases. An

example r~iutation wouid be to change a greater than or eqiid operator to ;i grcater than optmtor.

Each mutant that is distinguishecl by the test set is termed -killedU. the rerriairiirig niiitarits are eithcr

equitdent to the original program or ineqiiitdent but rindistinguished by the test set. Ec~iii~lilent

mutants are d isc~rfed while trndistinguished mutants require the addition of new test cases. The

funclanientd premise of mutation testing is that in practice. if a program contains a real fault. there

miIl tisually be a set of mutcuits (representing artificiai faults) that can only be M e d by a test

set that also detects the reai fault (Offuttt001 [Geist-t-921. Kowemr the fauit reveaiing ability of

mutation testing is highly dependant on the set of mutation operators used to create the mutants-

Slutation testing has remained an academic endeavow. however it is experiencing a great deai of

interest because it provides a directo quantitative memure of the quaiity OF test data For a program.

"or example creating a test case to invoke an enor handler for a rnemory allocation may be difficutt to do.

Mutation testing approximates boundaq testing: bounclaries are the edges oF regions on nhich

mutants are killecl [Hamlet 94. For example in boundary analysis a pretlicate e-qression may define

a botinclary on the input data. Killing al1 mutants of the pretlicatc espression implies thac the test

cases have arleqiiately testecl the botindaries cfefined by the predicate.

2.5 Evaluating Criterion

Hon do we compare one test criterion to another? One ml1 kriown n.ay CO compare test criterion is

through the subsumption relation. A test criterion Cl subsumes another test criterion C2 if wery

set of test cases that sacisfies criterion Cl also satisfies Ci?. For erample. if a test set ha covercct

every branch in a provani (satisfied the branch criterion). then that test case is giiarnriteeci to have

also coveretl cvcry statement. One s q s that hranch coverage subsumes statement coverage. The

subsiimption criteria can be clisplayetl in a hierarchical chart stich as figure 2.1 [Gardiner 991.

Only some mcthods. those n-ith a common basis can be itsefiilly coniparetl iising siibsiinies. For

example. variants of path coverage can be compared. Hosever. niariy methocls are incomparable -

neither sirbstimes the other. Further the siibsumption relationship has a serioiis Han.: siibsurriptiori

is only valid under LOO% coverage. Branch coverage subsumes statement coverage. in other ~ o r c l s

if every branch in a progam is covered then every statement is also covered. Howe~er if some test

set Ilas a brruich coverage of 99% subsumption s q s nothing about statement coverage. The 1% of

itncoveretl branches rnay lead to a miich I'uger proportion of rincovered statements. However. in

generd. a high branch coverage iisudly implies a high statenient coverage and therefore a stroriger

test set.

Regardess of a criterions position in the subsumes hierarchy no single testing technique is guar-

anteed to expose di possible failures. Eventually d l of the ciefects a methocl can fintl are forrntf

ancl the methods usefuiness (for esposuig faults) is enclerf [Hamlet 941, It is generdly agreecf chat

an effective testing strate= musc include several clïüferent test methocls. Different techniques tent1

to reveai clifïerent types of faults. Therefore the iwe of multiple techniques usualIy results in the

discovecy of more faults.

Endiy some tvork indicates that human factors are more important than any theoreticnl factors.

Sonie esperiments [Hamiet 581 show that the most important factor in software testing ma- be

the ski11 of the tester. This fiuther clmages the usefulness of iising the subsumes relationship. IF

a technically more powerful criteria (Le a criteria -4 that that subsumes criteria B) is harrler co

use. then an rinskillecl tester using it rnay not test a program as well as n skilletl tester tvho hases

his testing on the subsumed cnteria [Hamiet 891. Siormver the attitude of the tester may be an

important factor. It appears chat most successful testes (i.e testers who create tests that reveal

rnany faults) adopt a destructive attitude tomcl the PCT LJalote 971. It is argued that the cleveloper

of a program d l find it difficult to adopt a clestructive attitude towxrd the program that he has

created. therefore a cleveloper shoultl not test their onm work. Fiaily Hamlet has proposecl that

perhaps it is the process of iising a methodical test technique rather than the techniqiie itself thnt

rnakes testing siicccssful. He ~heorizes that bu forcing testers to pay systetriatic attvntiori co die

software. testers are tnorc successful ac finding ktults [Haxilet SS].

2.5.1 Summary

Testing is a costly business. Testing dong with clcbttgging and fixing the cocle consiinlei on nwrngp

about 50% of the time and money budget of a typicai software cleveloprnent projcct [SheparcI+ 011.

Much of the cost is attributed to the person-hours spent in gcnerating test cases. riinning clic tests

and comparing the results. Most test adequacy criteria are untlecicleable. therefore no algorithm can

be usecl to generate appropriate test cases. Most often the generation of test cases is carried orit

rnanually bu the tester(s). hspecting the resuIes of test nins is aiso tiine consiiming.

With the rising cost of personnel and the decreasing cost of CPC cycles there is a strong moti-

vation to automate as rnuch of the testing process as is possibleÏ. Mutation testing (to be ftirther

discussed in Chapter 3) has several features that are anienable ta autoniation. It is also consiciered

to be among the strongest testing techniques with respect to its ability to reved faiilts in a prrigram

[Offut+ 96b!.

'However. automating the entire testing pmcess is neither feasible. cost effective. or an effective way of revealing faults

Chapter 3

A Detailed Look At Mutation

Analysis

This chapter takes a detailed Iook at mutation testing bu tliscussing hon- niutation testing is prr-

lormed. the theory supporting it. and its strengths anci weaknesses.

3.1 Mutation Testing

Mutation testing is a systeniatic method of obtaining a thorougli test set. During mutatiou testing

mutant versions of the ptogram under test (called mutants) are created bu seeding the origina1

program with faults. one faiilt per mutant in accordance uith a predefined lauguage specific mutant

operator set. Test riata is then used on the mutants mith the goal of causing each mutateci version

of the prograrri to fail ( a mutant fails if its output differs from that of the original program). This

is called kiiüng a mutant. W e n a mutant is killed it is considered dead and is not iiseci in hrther

tests. A test case that c m diïtingtikh ( H l ) a mutant W considered useltrl. others are not.

Mutants that are not kiiied by any of the evisting test cases fdI into one of three cntegories.

First. the mutant couid be functionaily equident to the original progrmi. in wtiich case it c:annot

be cListin,@hed from the original. Such mutants are discarded. Second, the mutant is killabIe but

the test set is inadequate. In thii case the test sr: must be augmented by the addition of a nen-

test case. F i d i y the ori,@al program may be fauity in mhich case the mutant may represent the

correct version or may reveal the fauit.

A live mutant directs a tester to the exact statement chat musc be reached. The tester then uses

inforniarion about the Iive mutant to generate the nea test. In the esample of substituting -! =* for

'<'. these t ao operators a g e e on the ahole range where the '<' espression is TRL'E. If test clatn

only falls in thï range then it cannot distinguish the mutant. The Live mutant implicitly poses the

question: "the test data given so far says it doesn't matter whether or not this operator is .! =' or

.<y - . why is chat?'' The ansn-er d l guide the creation of a ne%- test case or possibly reveal an error

in the original program [DeSIilIo 781.

Once enough ncw test crises have heen aclcled. ail of the live niutants are esecuted against al1 of

the ne%- test cases (an? test case may. coinciclentalIy, MI other live mutants). The iterative cyrle

tif acltling new test cases and esecriting them against the live mutants continues iintil either no live

mmants remain or the mutation score is high enough to satisfy the tester. The mutation score for n

test set is the ratio of cleaci mutants over the total number of non-eqiiivdent mutants. -1 mutation

score of 100% is cdled mutation-atlequate and satisfies the mutation test aclequncy criterion. In

practice miitation scores of less than LOO% are often consicleretf goocl enough. However. assumin::

100% mutation aciequacy can tic achievecl. Fhat cloes this say about the correctness or qualit? of

the progrni so testeci'! The answer depencls on several assiunptions.

3.2 Mutation Theory

l h t a t ion analysis is preriicatecl on the existence of a class F of faults chat can he irijectetl into

n prograrn. It provides a set O of operators. knom as mutant operators cach of wtiich mocltiIs

one or more of the faults in F. Each proqamrning langiage has ics own set of mutant opcrntors.

ln tirircltvare. testing is baset1 on fault modeis that closely model actuai hardware fauits (such as

stuck at 1. stuck a t O faults). The hu l t model provides a set of simple faults that can be combined

to mode1 any fauIt in the hardware. However. since software is more comples thaa hardware. no

hardware inspired fault model exists for software [Cai 981. Testing techniques like mutation must

use heuristics to establish the fault class F and its corresponding mutant operator set 0. Hoaewr.

if rnutation testing can Hi di of the mutants of a progam. then that program c m be asured to

be Eree of the errors represeated bu the mutant operator set 0. Hence the effectiveness of mutation

testing ciepends on the accuracy and compieteness of the fault model as modeiied by the mutation

operator set. in principle. mutation testing based on an exhaustive fault mode1 (Le. ail possible

faults considered) would constitute a perfect test method. Kiiiing al1 the mutants would show that

no other program agrees with the program under test on the test set. hence the program under

test must be correct if any prograrn is [Hamlet941 [Budd+ 821. Bon-e~er. it is impossible to predict

al1 of the ways in which software may fail. therefore sohware faiiIt classes as modeled by mutation

operator sets are based on Desses (heiiristics) about what kind of faults can he introcliicecl into a

program.

Mutation testing assumes that faults in n program are the result of mistakes matle bu program-

mers. .A mistake made by a programmer may leacl co one or more faults locatecl in the tcxt of a

pïogram. Since a fault in the text of a program can take many forms. mutation testinp further as-

sumes that proganniers protliice nearly perfect programs in which mistakes are syntactically small

tleviations of the intended program. This is called the competent programmer hypothesis [DemilIo

YS]. In reality this hypothesis does not hold. Although a fault may be a syntactically small devin-

tion in a sin& stacement of a program it could just as n d l j im be spreacl ovcr several LinesS. Tu

overcome this problem mutation testing proposes the coilpling eflect hllpothesis. which States thar

simple and comples errors arc coupled. This implies that a test that kills a simple non-equitalent

mutant will aho kill a complex non- ecpivaIent miiram. The coupling effect hypothesis has been

srrpporterl both empiriçally [.-\cree+ 791. [DeiIillo-t 791. [Offut 921 .and theoretically [IIorY-L]. Baseti

un these hypotheses rrwstg niiitation uperators rire selecteri €or tlieir irbility to niotlel srnall syntactic

faults chat rnight bc the result of a mistake made by a programmer. This leacls to the probtern of

discovering what kind of smdl syntactic faults a programmer is most likely to niake.

3.3 Applying Operators

XppIying a mutation operator to an original source code stacement results in what is called a first

order mutant. in general one application of a mutant operacor can generate more than one tirst

order mutant. For esample repIacing a logical operator niith every other possibIe logical operator

resdts in severd distinct mutants for each logical operator in the on$d progam. It is &O

possible to appIy a mutation operator to a mutant program. These -mutants of a mutant" are

ralled second order. Because the number of mutants groas expanentially it is iisiiaiiy not practicai

to use anything but k t order mutants- This decision is justified by the competent programmer

-1t could even be a fault of omission. 3Stuch of the mutorion Iiterature implies that mutation operators are designeci strictly to simulate programmer

crrors. Kowever, this is not the case; mutation operators are amïved at thmugh various means.

hypothesis in n-hich it is argued that first order mutants simulate cedi t ic programmer errors %hile

higher order mutants do not [DeMillo 781. More importantly empiricril m r k by Budd and Ltathirr

support thii restriction to first order mutants. Budd sho=ed that higher ortler mutants do not

provide significant advantage with respect to the construction of a better test suite [Buclcl 801.

S[athiir cites experirnents in which test data chat ~+,?s able EO ciistinguish mutants created bu simple

operators. was also able to diitinguish iip to 99% of the liigher order mutants [Mat+ 911. It appears

that tests that cletect simple faults collaterally detect any large faults.

The theoretical foundation of mutation testing ultimately rests on the vdidity o i the campecent

programmer hypothesis and the soundness of modeling faults by simple syntactic changes to a

program. These appear to be weak foundations: the conipetent programmer hypothesis does not. iri

generril. holtl while simplicity means that operators niay fail to yieltl hiilts that are represctitatiw

of iritligerioiis faiilts. Sonie stiidies indicace that simple niutarits can exhibit bdiavioiirs sirnilar to

the behaviours of trrie faiilts [Darad9951 however there are other stutlies that indicace the opposite.

Fiinher. while achieving a mutation score of 100% shows that the program cfoes not have .an? of the

faults modeled by the mutation operator set the meaningfulness of this vaiile is tlepenclent on the

cornpleteness of thc fauit model.

.Uthoirgh the iintlerlying theory of mutation testing appears to be weak. in practice mutation test-

ing h a turned out ro be one of the strongest known atiecliiacy criteria [Butlrl SO][Offtitt 36b][Fr'rankl

- 971. Xlthough a direct comparison via the stibstimes relationship is noc possible. mutation cestuig

appears to be as good a t revealing errors as data How testing. [Franklf 971 [Offrit+ 96bI. Uutatiori

cesting approsirnates other cesting hcuristics: in Wong et al's [Wongt 971 study of T l C operacors.

sonie operators aere identified as being able to:

1- force a tester Co select test cases horn different parts of the input doniain

2. select test cases From nearby parts of the input clomain of each scaiar reterenco

3. construct test cases to esamine points on or near predicate borders

4. review the logic of the program under test by mutating di Iogicd operators

5. obtain statement coverage

6. force each arithmetic expression to take on a negative, zero, and positive \-due

Mutation testing encourages the construction of test data that distinguiçhes mutants, iiihen test

data fails to distioguish the mutant. and when the tester cannot show that the mutant is ecluitxlent

to the original. then the e-xistence of an error becomes highly probable [Wong +97]. A difficiilt to hiIl

mutant tiirects the progarnrner's attention to poorly covered and possibiy poorly nritten sections

of the code. -1 mutant that is difticult to kill may in facc indicate untestable code, Le. code that is

tao cornplex to test. Such code shoidd in most cases be ren-ritten.

3.4 Mutation Operators: Simulating Programmer Nlistakes

The mutation operators of current mutation systems were cleveloped primarily iising the researcher's

on-n programming esperience and consiclcration of typical programmer errors. For cxrimple die C

mutation operators rvcre *hc result of -long delibcrations aniongst the authors in which ieveral

aspects of the C langiage and program developrnent of C were examined" [=\grawal+ 391. Thc

hlothra system @lehIiüo 881 uses '11 operators that were derivctl from sturiics of -tupical crrors"

made by Fortran progr,ammers. [t has been arguecl that esisting mutation operacor sets are itnlikdy

to be sound. as thcy ~ e l y totally an the authors cspcricncc tvhich rnq be hiasecl and insotriplctc"

,anci they may have tliscarclecl certain types of mutation operacors or else simply not (:onsideret1

certain types if ttiey had no experieiice tvitti them [Iiini+ 991. Kim et al. argue that a so~uid

operntor sec must be generaterl throttgh some systematic mechotl such as the rhemical incliistry's

HXZOP (Hazard and Operability Studies). They have used HAZOP to generate "rigorous" mucation

operators for the Java language and are currently building a Java mutation system to validate tlieir

daim. However the HAZOP-Java case is the esception and most operator secs are the procfuct of

the researcher's intuition.

Some mutation systems also inclircle operators that enforce statement coverage. branch coverage.

domain coverage. and extreme values criterion v o s + 981. Although these latter operators do not

simulate programmer mistakes they are included in order to enforce other testing heuristics and

have the goal of improving the fault revealiig ability of mutation testing. ;Uthough the rationale

behind some operatos is global (i.e. mutating reiational operators). mutation operators are generdly

Ianguage dependant. Operators for two Ianguages can differ due to the differences in their respective

syntas. and due to features that are intrinsic to a particular language. For esample some of the

M d 3 operators deal 6 t h that Ianguage's unique tasliing feature. The foilotving paragraphs describe

some esisting operator sets.

The Mothra system from Purdue University and Georgia Tech. is probably the most well knom

mutation systern. It automates the process of mutation testïng Fortran 7'7 programs by creating and

executing mutants. managing test cases. and computing the mutation score. Each operntor is rep-

resented by a 3-letter acronqm. For euample. che -array reference for array reference replacement-

(.AAR) mutant operator causes each am-. reference in a progam to be replacecl by each other clis-

tinct am- reference in a program. Similar naming conventions are used for other mutation systems

(Proteum's C operators have 4 letter acronyrns). Slnchra's 22 operators are goupecl into S classes.

which are further grouped into 3 levels:

1. Statement analysis: This level corresponds to craditional statement aniilysis tests. i.e. trsting

thnt al1 statenients are executed. i d that every stntement has an effect.

( a ) replace each statement by TRAP (execiiting the TRAP statement niIl kill the rnrrtant).

ttiis ensiire statcmcnt coverage:

(h ) replace ench statement by CO-iTISC'E:

(c) replace each statenient in ii subprogrm by return:

(cl) replace the targer label in each GOTO:

(e) replace the label in each DO statement.

2. Predicate .an<[ domain anaipis: Thii IeveI c m be icjecl to create test data to ~xercise al1

preclicate houndaries ancl data domains.

(a) take the absolute d u e or negatke absohte value of an e-xpression:

(b) replace one arithmetic operator by another:

(c) replace one relational operator by another:

(ri) replace one logical operator by another:

(e) insert a un- operator precehg CU expression:

(f) alter the talue of a constant: alter a DATA statement.

3. Coincidental correctness: This IeVd provides test data that e l î a t e s coincidental correctness

enors (smdl errors in the original program that have gone undetected because. by the nature

of the test data. the program h a ~ coincidentdy produced correct resuits).

(a) replace a scalar variable. array reference. or constant by another scalar variable. ar-

reference, or constant:

(b) replace a reference to an array name by the name of another may.

Fortran is an old language and lacks many of the features found in later and syntactically richer

lnnguages siich as C and Ada. -4s a resuit the 4Iothra system uses on- these 22 operators that were

tlevelopetl ,and refined over a 10 vear period poas+ 981.

CSas et al. propo~ecl mutation operators for -1daû3 [Voast 981. The Ath language hris many

features not found in Fortran. This required studying how to miicate programs that iise:

srrong typirig and information hiding,

0 dynartiic tlnta structures and rtatic long lived data stornge striictures.

cornplex control structures sucb as exception handling and tasking,

0 conciirrcnt processing

As a result of this study 65 operators were clefinet1 in 5 categories [\*oas+ 981:

0 operanil ceptacernent operators whicli cause each operand to be replacetl by each other syn-

tacticdly Iegal operand

0 statement operators which modify entire statements and modik the cantrol structures of the

Ianguage

0 expression operators which niodify matheniatical operaton and entire espressions

0 Coverage operators: Decision coverage is nchievecl by replacing cach dccision by TRLE and

FALSE. Condicion Coverage is ncliievetl bu replacing each condition bu TRLT and FALSE. Tt

is possible to iise both decision and condition operators to obtain rnuitiple condition-decision

covcrngc. mhich requires that the entire tmth table for a decision be covered.

0 tasking operators which are specific to the lda53 tasking constructs

Since the h d a mutation operators have never been irnplemented by a tooI the d i d i - of this

operator set has not been empiricaiiy evaiuated. Mutation operators have d s o been proposed for

C[Xgawal+ 891 p e l a m a r o t 9Tfl COBOL [Dehlillot SSII and .Tava Kim-+ 991-

3.5 Weaknesses

The niain problems of mutation testing are the large cornputationai and human resources required

to carry it out. Mutation testing is expensive in terms O€ both cime .and space complesit. BuclcL

anaiyzed the number of mutants generated for a program and found it to be roughly proportional co

the product O€ the number of data references tirnes the number of data objects p u d d SOI. This is a

large number for even sniaiI progarns. For esample a 4 line program usecl in [DcSIillo 781 gcneratps

40 nuitants. Each mixant is tquiwlent in cornplcxity ;incl size to the orignal program ancl musc

h c riin ngainst eacti test in the test set tintil it is kllecl. Sonie of these n;iit.mts tvill noc teminace.

some are eqiiivalent. and some are iindistinguishable from the ociginai program by the curent tcst

set. Eqiiklent iiiiitants are another major probleni. Sorne empirical data indicatcs that aboiit 10%

of al1 mutants are equiixient [DeMidIo 891. The eqiiidence problem is undeciclable and is donc

in practice by the tester making a decision based on looking at the cocle. If the mutant is jurigerf

ferror prone) to he non-equivalent then the tester niust create ii new test case to kiil it. Sonir aork

tins been clone on nutomated equidency checking with a succcss rate of about 50%. The miication

liternturc contains severai ideas for rediicing the prohibitive computntional costs of mutation tcsting.

The riext section discusses severai of them.

3.5.1 Selective mutation

The cost of esecuting mutants can be recfuceci by retfucing the nuniber of mutants t h nwcl to h ~ f

cseciiteci. Researchers have proposecl random mutation and seIective mucation'" [Offitt+ 937 as t~vo

techniclues in ahicli only a subset of the possible mutants are esecuted.

Acree [Acree 801 and Budd [Budd 801 evamined the idea of using O&- a s m d randomIy selectecl

subset of dl possibIe mutants generated in Mothra. Both experiments shoned chat a test set that

is adequate with respect to a randomiy seIected 10% of al1 mutants in Mothra has a mutation

score of over 99% n-ith respect to d mutants. However. randody sekcting mutants ignores the

fault cletecting capability of individual mutant types. Biidtl's M t cletection e-xperimcnts found

that mutants generated n<th respect to one mutant operacor may be more effective in detecting

certain 5-pes of fauIts chan mutants generated %<th respect to another operator. This suggests that

n-hile mutants are selected for esamination. they shodd be weighted dXerentIy dependiig on their

respective fault detection capability This has become knom as selective mutation.

'"This a h appears in the lirerature as constrained mutation.

Selective mutation origindly proposed by 4Iathur [Mat 911 is an approximation technique that

cries to select only mutants that are truly distinct from other mutants. The main advanrage of

this approach is that it does not requùe the generation of al1 mutants. The problem is in Gnding

a n effective smdl set of operators. X good selection should not o n - recluce the eseciition cost but

should d s o maintain good fault detection capabilit.

In one analpis O f i t et al. showed that test chta that kilis al1 mutants created by only 6 of the

5Iothra operators niIl usually kill almost ail of the mutants created by d l 22 of the original operators

[OMrti 961. LSing the obscnation that different mutant operators generate mutants a t clifferent

rates they decideci to perform e.qerinients in which the S most industrious operators were not iisec1.

This is called thii X-selective mutation. Tliey then comparecl selective to strong mutation. They

found that test sets that were 100% selective mutation acieqiiate were almost 100% strong niiitation

adequate (8 out of 10 test sets were 100% iitleqiiatc). The exact results are:

a 2-selective mutation yielded about 20% compiitational savings with 99.99% niirtation nricquacy

-1-seIective mutation yielded 40% compucational savings with 99.5% strong rnttcation arIeqitacy

r 6-seIective mutation yielded over 60% computationd savings with 99.2% strong mutation acl-

cquac.

This early data indicates that selective mutation can Save up to GO% of the computational cost airh

a negiigible loss in mutation acIequacy. To tinderstanrl why this occiirs the idea of fault size xi11

bc considereci in ttie next section. Both 4Iathiir [Mat 911 and Offut et al. [Offut+ 931 stiggmecl

ttie ttvo tnutacion operators to be oniitted are SVR (scalar variable replacement) .and XSR (an-

for scdar replacement). These operators generated the most mutants because SVR replaces evwy

scalar in a progarn with every other compatible scalar founci in the progarn. .UR replaces eve-

scdar variable in a program nith every array reference of corripatitile type. Adclitinnally they fotind

that the S%R and ASR operators produce mutants that are easily kiiietl by tests that kilt d e r

types of mutants.

3.6 A Notion of Fault Size

X mistake (not necessanly a coding error) made bj- a deveIoper m a - lead to one o r more fauits

located in the text of a program. A fauit may be an isolated mistake in one statement of a program

or it may be spread over several lines. One could Iook a t a fauit as being the dierence between an

incorrect prograrn and some correct program. -4 fault has both a syntactic .and semantic measure".

The following section is based on [Offut+ 961.

The syntactic nature of a fault can be described by the number of changes requiretl to correct ic.

-1 fault also has a semantic aspect: every program P can he viewetl as having a specification S thrit

clefines a sec D (the input domain). a sec R (the output range). and a mapping from D to R. The

program gives results basetl on a superset of D. If the input is not in D. P may protluce output

chat is not in R . The semantic characcerization of a fault vicws the faulty prograni as containing ri

computation that produces incorrect oiitput mec some siibset of the input clornain,

The size of a fault is the clifference between a correct and incorrect version of the progrmi. The

sjntactic size of a fault is the minimum number of statements or tokens that neecl to he changecl to

get a correct progratn. The semantic size of a Fault is the relative size of the sub-clomain of D for

ahich the output mapping is incorrect. hlthough the icleal use of sem'ancic size would be basetl on

a usage clistribution that assigns a non-uniform probability to each input. it can bc approximacecl

by a uniform distribution (each inpuc having equd probability). With respect to size the syntactic

and semantic characterizations of a fauit are very clifferent.

For a syritacticdly sn~all f d t . one ~oken or one statement m.?'- be incorrect. For n scnianticaIIy

small fault. Pb behaviour on a very smdl portion of D is incorrect. Clearly a small syntactic fiiitlc

cati protluce a large scrnaritic faiilt. Sirnilady a tnajor syntactic fault rnay affcct only a smnll portion

of the input doniain: a sniall sernantic fault. O l course a large syntactic fault c m result in a large

seniantic fault. Hoaever. the interesting class is ahere a srnaII syntactic fault procIuccs a srrial1

semantic fadt.

SIiitants created by mutation operators that produce Large semantic faults n-iU be e'q- to ki11.

[Budd+ 521 discussed the concept of program neighbourhoods. A neighbourhood of a progtam P is

a set of programs chat are semanticaliy -closeE to P. We aan t to use syntactic mutant aperntors

to produce mutants chat are dose ta P. since such sernanticaiiy small mutants shotdd be harder

to !dl and thus have the potentid to lead co higher qudity tests. SIoreover mutants that are

s m d syntactically but large sernanticaIly generate noise: the? add difficulty to the mutation process

without increasing the testing t d u c of the resdtiog test cases. This is sliown by the fact that aiany

mutants are trivialIy Wied by almost j - test case that reaches the mutated statement.

l l~here are other ways to define fault size, for a m p l e the consequence of a fault occurring.

In ternis of semantic fault size it appears chat selective mutation is ming the mutation operators

that tend to generate small semantic faults. Zi'nfortunately it foil~ws that such operacors niII aIso

produce a proportionally higher number of equivalent mutants as indicated in [Offut 461. i7iewcd

in terms of semantic fault size. hard to kiü mutants are mutants with very smal1 fault size vhiie

equitalent mutants have semantic fault size zero.

3.6.1 Weak Mutation

.As tiefinetl so far. rlistingriishing a mutant requires that the oiitpiit of the mutant differ From the

output of the original program. This requires nmning the mutant to completion (nsstiming oiitpiits

only occur on completion which nlthough untnre in generd is the consemtive xsumption iised in ~ h i s

section). Weak mutation is proposer1 as a way ro reduce the coniputadoniil cost of rii1it;itiori tcstiiig

(iioa referrecl to ris strong iiwtacion) by suggesting that a nriicruit be considetecl ciiitinguislieci if the

data state of the mutanc cliffers focni the data scate of tht. origind irnmediatdy nfter the esecrition

of the mutatetl stacement ~ o w r i e n $21. The iclea is chat whether or not the niiitant is n-eakly

killeri. execution of clle mutant can be terminated immediateiy after the data state conparison thtis

reciucing the computation tinie. With ttie introduction of weak niutation cesting, original niutation

testing became known as strong mutation. Weak mutation leads to somc intercsting observations

about wliat constituces a fault.

if a mutant is Mlecl in the strong sense then it will be killed in the veak sense bu the s.me

test (stroug ~nmation subsumes ive& mutation). However the opposite is not necessarily trile. a

niutation may alter an internai Stace. but subsequent execution mq- ignore. correct. or in some other

n.q mask the error state. For esample a mutation may have the effect of crttising zrn extra iteration

of a loop. Thii ail1 be detected under weak mutation. as the internai data states wiII rliffer. hnwever

the mutant may be equivalent. The foiiowing two C Fragments iLiustrate this case:

for (x=lx<lO;xf+) count = x*cost[xl:

r e t t m count:

for (s=Os<'>O:s++) counc = X*COS~[S]:

r e t m count:

Origind kërsion Mutant Version

C l e q both versions d yield the same result, hence ne have a ~ct ioaa i ly equiialent mutant.

-Uthough t h codd be cIassi6ed as a programmer e m r . in general Ke are not inte~ested in finding

this kind of "faultn". On the ocher hand embedding a simple print statement n i th i i the loop n-ïii

cause an extra line to be printed. In this case the mutant would be ciistinguished under strong

mutation testing (assiirning printed oiitput is compared).

Strong and weak mutation testing are evtreme ends of a continuum. in PVoodn-arcl+ 881 Wooci-

ward and Halewood introduce the idert of firm mutation as an extension to we,& mutation. Xith

weak mutation the ciata state is compared a t the end of the rnmatecl statement. nith strong rriii-

tacion the cornparison is made after program termination. Firm mutation compares the program's

data state (and possibly ruiy intemediate oiitputs) at some intermecliate time after eseciition of the

mitant. Cornparison poincs are user seIectable and muiilly corrie iiftcr the eseciition of a coniplete

progrnm block such as a for Ioop body. Corisider a mutation operator applieci to a stacement in ttir!

bot[! of a ioop where the loop is part of a proceclure calIed by the body of a progam. Then the data

Stace coulrl bc conipareci immediately after esecution of the rnutatecl statement (weak rniitation).

immediately after tennination of the loop. irnmediately after the rettirn frorn the procediirc. or after

program termination (strong mutation).

The original Slothra -stem did noc nui compiled versions of the mutants. Rather it usecl a

piirpose biiilt interpreter riinning on a slowi3 computer. Rit t i 'an interpreter weak mutation ti

rcIativeIy e a d y to perform. Programs can be nin to an arbitrary line. haited and the tinta stnte

examineci. This is also possible with a compiled progam howver it is not ,as easy ancl is firrther

complicated by code optirnizations. The real computation savings cornes by recliicing the riirinirig

tirne of a relütively slow interpreter. With a cornpiled prograrri running on liigh speecl conipriters

ninning tinic may become less important than compiling tirne and the possible benefits of wriik

miitation testing are diminished.

3.7 Summary

This chapter has discussed the process of performing mutation testing, its theory. strengths ancl

weakness. and some ideas for overcoming its deficiencies.

Mutation adequacy appears to be a strong test criteria in that mutation adequate test suites are

good a t reveaiing faidts in a program. Aithough computationallj+ expensive to perform. research

"In an extreme case the program could be sensitive ta the performance penalty of an extra loop iteration. in which case the tester is interested in this kind of problem.

L 3 Ç I o ~ by tadafs standard.

inclicates that by using weak niutation a large increase in efficiency can be obtainecl withoiit a large

sacrifice in the adeqiiacy of the test set. One of the objectives of this thesis is to decemine if the

liuman and compucation costs that are the major neaknesses of mutation test are as significant

when mutation testing is applied to the textual descriptions of finite state machine.

Chapter 4

Testing State Machine Based

Software

Ciiapter 2 stated ttiat the problcni of testing software couId be approached in a varietu of w a p . d i c h

tvere categorized as being usage bueci techniques. black box techniques. and white hox techniqttes.

These are general prrrpose rnethods. in the sense that tliey arc (lesigrictl to work for jiisc abolit a-

type of soltmre [Bernhard 941. There dso e?ust sotne spccidied testing eechniqites tnrgetetl at

jpecific typrui O€ systerris siich as ttiose based on h i t e state tnachines. This chapter tlisciisscs sorric

[if die craclitionai inethods usecl to test software specified by tinite state tmdiinti furrnaiisrris. It

concludes xitti a look at recent work in ivhich mutrrtiori testirig has Lieen propusecl iw a rnetiind of

testing HFS31 specifications.

4.1 Finite State Machines (FSWI)

F S K s rnodei systems chat can be described by stimuli and operations on state. Stimuli (ohen caiied

evencs) are inputs from the oucside n-orkd. wtiiie operations are the FSSiS responçes to the stimuliL".

FSM based systems are ohen cdled reactive or erent drit-en ystems. With Eunctionai software the

output of a compucation is a htnction of its current input. Qïth FSM bbased softme the output of

a computation is a function of both the current input and the current state a€ the -stem. where the

LJSome of the literature refers to stimuli as r e c e i d messages or in bound signais. ivhile resporises are olten cdlei generated messages or out bound signjls.

curen t state of the system depends on the historj- of the software's inputs. The input to a FSM is

a possibly infinite sequence of input events, each including a certain number (possibly zero) of input

parameters.

FSM software can fail in al1 of the ways that functional software cm. Additionaiiy F S X s c m fail

because of sequencing errors. in [Chow 781 sequencing m o r s are clivicied into three classes. Assume

X ancl A' are tno minimal15 FSbIs with the same input alphabet. A' is the correct version. The

error classes are:

1. Operation errors: -4 is said to have operation errors (also callecl output errors or outpiit fa~ilts).

if h is not eqiiivalent to -4' and A c m be modified to be equi~alent to A' bu changing the

output function of A (without adding or cleleting states in A). Output errors are relacively

ensy to cletect. hriy test suite based on a transition tour of A (trnversing cadi trimsition r>f -1

nt l e m once) is guaranteed to detect dl outpiit errors [Bach+ 941.

'1. Transfcr Errors: A is sait1 to have transfer errors if X is not cquivrilent to A' and X XI^ tie-

tnociifetf to be ec~uiwlent to A' by ch'wging oniy the nest state Function of X (withoiit acicling

or deleting states).

3. Extra states (missing states): A is said to have extra (missing) States if in order to nitde A

eq"\alent to A' the number of states in h must bc reducccl (increasecl). Since X and A' are

mininial. ati tinecpnl number of states implies that A and A' are not eqiiivalent.

By making some restrictions on the structure of a FSJI Chow provecl that it is possible to gerierate

test chta paranteed to cletect the above sequencing errors [Chon- 781. His method. noa calleil

the wrnethod, is both vdid and reliable in the sense of [Good+ 751 and provides an aigorichni for

generating a test suite. The method is based on resuIts of automata theory and is limited 6- the

foliowing assumptions:

I. the FSM is completely specified.

2. the FSM is minimal.

3. the FSM starts in 6xed initial state,

4. and every state is reachable [Chow 781.

lSA minimal Stace machine is a state machine whose behaviour coiild not he matched by another scate machine that h& a fewer number of states.

Chow's assurnptions restrict the method to deaiing strictly ~ 5 t h the possibIy abstract controI struc-

ture ofa FS41 design (without these assumptions the generai problem is undecicfeable [GilI 621). This

impks that there are no counters or control mriables manipulatetf in the operatians that intiueucc

the secpencing olaperations. Yote that these assurnptions do not in gcneral hold for more powerfril

moclcling cechniques basecl on estended FSMs. hierarchical FSlls. or Statechiirts.

Chow's rv-methocl although tdid and reliable c m require a large number of test sequences.

Researchers working in the field of protoc01 tcsting have developed other test derivation niethods

For FShls airned at reducing the size of the test suites. Among these niethociç. the T-method. D-

method. and L-rnethod. providc algorithms for gcnerating test suites that are tisndly sniailer t11;in

those produced by the w-method. Both the D and C method are guarantecrl to finrt nll oiitprtt nntl

transfer errors [Bernhard 94. However the T-method. although proclucing the srnailest test suites.

is orily gtiarnntced CO detect dl transfer errors. Cnfortiinately al1 three of these methods are limitetl

by the same asstiniptions as the wnicthotl. Although the resdting suites are smalIer than those

gcneraterl by the w-mcthod they can stil1 become large.

4.2 Protocol Testing

Generally spe'aking. protocols: specify the rides useri by a coilection of computer syscenis that arc!

cnrnmunicating on sorne network [Bernhard 941, The protocol (conformanceL~ testing problem is

the problem of decermining if an implementation correctly impkments a FSM specification. Con-

formance testing is based on the idea that many dierent implernentations wilI he tested against a

singIe specification. This is expected when an organization defines a standard (e.g. the OS1 protoco!

stack specification [IS7.5961) chat is used by ma- different irnplementers. Thuç the mettiods iised in

conformance testing are highly dependent on the esistence oEa precise specification [Bocht94]. Since

the cornpleteness of the FSJI Eest suite generation methods ciiscussed above is limiteci bu Chorrfs

assumptions and since comp1eu protocols can vioIate these assumptions other tescing methods for

protocols have been eqIored [Bocht !Ml. Among them are some fault-based methods.

As with hardware testing, the fadt-based testing of protocols is bxed on a fauit model that

defines di of the fauIts that are to be detected. Chow's sequencing errors are a nattud basis for

çuch a protocol ladt model. Each Fadt is rnodeIed by a mutant O€ the specification. A mutant is

t6Protacol testing concentrates o n testing €or confamance of a protocol to its specificarion. This is i Iittle diffecenr €rom testing to reveal raults.

a FSM obtained by applying the following four types of operators on the specification. They are

basecl on ChoK% sequencing errors [Petrenkof 961:

T'pe 1: alter the tail state of a transition (a transfer fatilt)

Type 2: alter the output of a transition (an output faiilt)

Type 3: atltl a transition: and

Type 4: atltl an cutra state or delete a state.

These operations model possible programmer mistakes macle during the trnnslation of an FSSI basecl

specification into a program. Fault-basetl identification of mutants is most naturd in the contcst of

cliagnostic testing where the goal is not only to detect erroneoiis behaviour. but also cleterniine hon-

the iniplemeritation coulcl be correctecl in orcler to make it coriform to the specification. To date

tliert! lias been no enipiricd support cletermining whether the assiimptions of this or ottier protocol

faiilt models are realistic [Boch+ 9-41.

4.3 EFSM and HFSNI

The common interpretation of '.state" for softn-are systems is that each state represents one rlistinct

set of al1 ~zlicl ~zltics of al1 program variables. Even with simple progams containing few primitive

mriables. this interpretntion leads to an unmanageably large numbcrs of states". One way to retluce

this complexity is to use an estended FSM (EFSB[) model. in which the specification of a system

module. represented as a F S X includes the foilowing additions:

O .\clditional state variabirs: States becorne abstract view of the object that tlefine its qualitative

hehaviour. mhile mriables pertain to quantitative aspects. In this case a change in a variable

d u e does trot rilways imply a state change [Selic 94.

O Both the input and output interactions mal* include parameters.

Transitions may have enabling predicates and transition actions.

- The enabling predicate (often called a p a r d ) must be satisfied for the transition to be

taken. It depends on the present state (FSill and evtended variables) and possibly the

input pairameters of the stimuli.

LÏFor rxarnple. a sirilgel6 bit integer has over 65000 states.

- The action defines the output producecl. including the effective oiitpiit parameters. antl

possibly sorne updates of the additionai state variables. This iç ciifferent from the simple

FSII mode1 in mhich outputs clo not have parameters antl do noc tipdate state nriables.

Figure 4.1: HFSX

.ki extension to the representation of EFSIls and FSJIs is to introduce hierarchy. Wcti a

liierarchical FSM (AFSM) .any FSM state can have sub-states. Figure -4.1 shows a HFSM ahich tias

3 top level staces (Sl. S3. S3). The top-level States S3 and S3 dso contains three stibstates (S2L.

STL. S23 in S2 and S31. S32 in S3). Assume the current statc is SI. LVhen transition t l is taken

the nelT top-level state becomes S2. Hoaever nithin state S2 siib-state S22 becomes the ciment

sub-state. if transition t3 is taken then regardless of the mhich siib-state of S2 that the machine

is in the machine ni11 move to top-level state 53 and within state S3 the current state n-ill be S31.

An HFSM cati tie converted to an ecpitdent FSM. Figure 4.2 sLotvs the -tiattened' version of the

HFSSI of Figure -4.1. Xote t h che flattened version required the addition of 11 transitions.

Rierarchicd. concurrenti" state machines, cailed Statecharts. nere first introduced .mcl system-

aticaily studied bu Hare1 to mode1 reactive systems parel 871. Hierarchy recluces risual complesïty

whereas concurrency increases the rnodeling poaer. EFSJL HFSM. and Stateciiwts al1 increase the

representational power of FSWs. Honwer this increase in representational power makes the tescing

activity more ciiicult. ,Mi of these representations can be viened as compact notations for FS-Yfs.

'JConcunency in Sratechms is represented by ";uidn stares. rvhich are nor used in the UML-RT/RosenT and thus are not covered in this rwrk.

Figure 4.2: "Flattenecl' HFSSI.

hlthough it is sometimes possible to unfold an EFSWs into a Hat FSMs (possibly for sottie rrsciiig

activity). it is seldom done in practice.

4.4 Test ing Extended hierarchical FSNI

If certain lirniting assurnptions are made about the forrn of the prerlicates nncl actions in EFSlIs.

thcn the anaiysis of the behaviour of the specification and systetnatic test selection remains decid-

able [HigaSJI. In this case one of the test derivacion methocls for FSXs (which guacantee cornplece

sequencing ercor fadt coverage) can be used to generate test sequences. Ln generaI Liowever. partic-

ulariy vhen actions may incliide Ioops, the question of deciding n-hich input parameters shouIt1 be

iised in order to force the selection of a particuiw transition beconies undecideable (similar to the

question of deciding the executability of a given branch of a program). Fu-ther. specifications .anci

designs based on EFSM/HE'SM models are seIdom minimal or complete. Since the test sequence

generacion methods that are idid and rdiable for FSM are neither vdïd nor reIiable when applied

to HFSMs other test criteria have been suggested. Offut has presented four test criteria chat can be

used in EFSM specification based testing [Offut+951:

1. Transition coverage: the test set musc satisfy every transition in the specification graph. This

is analogous CO branch coverage.

2. Fi111 preclicate coverage: for each predicate on each transition. the test set must include tests

chat cause each clause in the predicate to result in a pair of outcornes where the talue of

the predicate is directly correlated with the d u e of the clause This relies on clehitions Eiom

DolfSB [RTCA 921 and is sirnilar to .CIC/DC. This subsumes transition covernge.

3. Transition-pair coverage: for each pair of adjacent transitions in the spccification graph. tlic

test set musc traverse the pair of transitions in secluence. This is meant to check the interfaces

among States. This subsunies transition coverage but is meant CO test the software differencly.

4. Complete sequence criterion: the test set must contain tests that traverse .-meaningfiil se-

quences" of transitions on the specification graph. where these sequenees are diosen bu the

test engineer based on e.xperience. domain knowleclge. anri other hun1.m-basecl kno\vicctge.

Thesc criceria are btack-box in nature. clepending only on the specifications. buc are partly niorinteci

by notions of stnictural coverage.

4.5 Statechart Mutation Operators

Fabbri et ai. have studied mutation testing of both FSSI and Statechart basecl software speciiications

[Fabbri+ 981 [Fabbri+ 98bl. in [Fabbri+ 981 an operator set for FSLI hasecl specifications m a

deriwcI anci a cool calied Proteurn/FSMLS w u built. Their mutation operators are basecl on the

error classes deEned by Chom- and on heuristics (guesses) clevised. by the authors. about typicai

programmer miscakes made during the creation of FSMs from specifications. The operators. Listecl

in table 4.1. aIso guarantee other minima1 testing requirenients such as covering ail transicions.

Table 4.1: Proteum/FSM Mutation Operators

LgProtetirn stands for Program Testing wvith ~Iutation.

The authors state that an empirical study was to be condiicted whose piirpose woultl be to

e d u a t e the cost of applying mutation testing to FS& and to tietermine an essenciai rniitntion

operator set. however results have yet to be publishecl.

In [Fabbri+ SSb] the FSSI operators of table 4.1 were espancled to cover Statechart basecl spec-

itications. The new operator set is based on their earlier work aicl on the intrinsic featiires of

Stntccharts. The Statechart opcrators are dividetl into three cacegones:

1. FSM mutation operators. These are the sarne operators shown in table 4.1.

2. EFSM mutation operators. These operators are basecl on sonie of the mutation operacors for

the C language and for boolean expressions.

3. Statecharts-fcaturcs-bûsecf opcrators, These opcrators consicler intrinsic Statechart featiires

siicfi as history. broaclcasting and parallelism. The definition of these operators n-as also

inspirecl by C h ~ n - ~ s error classes. interpretetl accorduig to the Statecharts concepts.

2. 1 arc missing 3. 1 event miçsing 4. 1 event extra

S. 1 output exchanged

.5. 6. 7.

9. 1 state missing EFSSI operator set

1. 1 ex~reçsion deletion

event exchangecl destination eschanged output misshg

2. 1 t3oole.n expression negsrion 3. term associrrtivity shift 4. 1 arithrnetic operator by arïthmetic operator 5. / relational operator by relational operator

logical operator by logical operator

9. 1 variable by constant replacement 10. 1 constant by requiretl constant rcplaccment Il. 1 constant bv scaIar variable rwlacernent

Statecharts feature based opcrator set 1 transition's history tIeIeted

transition witli Liistory bu transition replacenierit history missing h by h* replacement tl* by h replacement 11 extra li* extra in(s) conclit ion missing in(s) condition state replacement not yet(e) condition niissing not yet(e) conditiori event replacerrierit exit(s) event missing exît(s) event state replacement entered(s) event missing entered(s)event state replacement broadcasting origin transit ion replacement broadcasting destination transition replacement

Table 4.2: Proteum/ST Mutation Operators

-4s mith the Proteum/FSSI workan operator set %as detined and a tool prodiiced (Proteum/ST).

Table -12 lists the Proteum/ST operators. &?th this tool mutations are applied to textiiai repre-

sentations of Statecharts m-ritten in SSL langage (Statecharts Speciiïcation Langage). As in the

Proteum/FSM case no empirical results are yet a d a b l e .

4.6 Summary

The traclicional mechods of testing FSSI presentetl in this chapter were for the most part clesised

a i t h the intention of testing the correctness (conformance) of the specification to implemencation

translation performed by programrners. Testing a RoseRT generated program is somewhat different

in that RoseRT is iffied to develop a specification and then to autoniaticaiiy generate the source

code for the application. Effectively the specification becomes the implementation. This diapter

11;s presented material that is useci as the basis of performing mutation testing of RoseRT HFSMs.

However before clisciissing mutation testing of RoseRT progranis it is n e c e s s q co c,Ae n more

dctailecl look aE the RoseRT tool, This is the subject of Chapter 5.

Chapter 5

The RoseRT Toolset

This chapter gives a brief introduction to the RoseRT toolset. [t then hriefly cliscitsses bot11 the

stniccural and betiaviotiral motleling of active objects as implemented bu the toolset. liiitation

testing of active objects requires a detailed understanding of how HFSMs are iised to define ru1

accive object's hehaviour. This cliaptcr provitles those cletails.

5.1 Overview

RoseRT is ii rnodeling tool riesigned for specifying, visualizing, tloçur~ientùig. aritl aiiturri;rting th!

construction of coniplcx. evcnt-driven. and potentially clistribirtecl r d - t i m e systems [Lyons 981. tt

is nlso an exploration tool in chat it c ~ m be iised to explore the design o l a project. Tiie rnocleüng

langiiage iisecl iri RoseRT is bascd on Lnified Siotleling Language (LXL) frir R e d Tmc CSIL-RT)

and the R 0 0 5 1 (Reai-ïime Object-Oriented Blodeling langage, originaiiy developed by O b j e c T i e

[SeIicf 941). One can rnake a distinction between two kinds of reaI time systems: time-&ken anri

event-<iriven. in time-driven systems events and outputs occur a t periodic rates b a e d on reaI-time

clocks. In event-driven r e d - t h e systems an output is a response to an event that must occrrr wichin

a specified bounded time. RoseRT is targeted a t real time systems of the event-*en End- RoseRT

appiications are constmctetl nfth the aid of class. object. sequence. collaboration, Stace and ocher

diagrams. most of thern based on L . L .

RoseRT supports the iterative development of r e d - t h e systems at hro Ievels of modehg. At

the schematic level. ciesigns are expressed in the C'hIL-RT visilal langiiage (formerly R0051). At the

tletail level C or Ci-+ source statements are addecl at various points in the model. For example each

state transition can have an associated code segment (siibsequently cailed an action) that is esecutetl

ahenever the transition is taken. Similarly. states can have associated entry and exit actions that

are executed when the state is first entered (entry action) and ahcn the state is eventually left (exit

action).

-1 RoseRT mode1 is esecutable: it is possible to compile. run. and clebug the mode1 on the host

rlevdoprrient machine. L-sin:: the obsenability ~ p t i o n ' ~ of the toolset. it is possible to graptiically

observe the rtiotlel 'as it nioves througli states. hlthough RoseRT is primarily a modeiing and

sinruiation tool. it Li possible to generate a complete and deliverable application strictly tising the

toolset.

Figure 5.1 shows the laycrs involvetl in runtiing a RoseRT modcl. At the top is the RuscRT

mode1 consisting of Lioth schematic and detail level elenients. The Services library is a Iraniework

siippliecl by RoseRT. It is siniiiar to a virtiid niachitic and provides the following services:

O 11echanisms that support the iniplementntion of rlynarnically creatctl concurrent comm~inicat-

ing state machines.

O Thread management and concurrency control.

O Timing,

O inter-thread 'and inter-process communication.

O ObsertabiIity and debugging of a running model.

At compile t h e C f + code is generated. based on the model. then cornpiled. The compiled object

code is then linked with the RTS iibrary to produce an esecutable program. L-nder W i d o m L T

this is an .ese file. Other formats are possible and depend on the target real-time operating syscem

(siich as VsWorks or Q 3 ï ) .

"'Oboermbility is carriecf out by using an expandeci version of the RoseEtT services l i b r i q in n-hich communication betwveen the taolset and the wecuting application is performrd via a TCP\iP connection.

I Rose RealTi nie Model !

LI Rame ULo<Ju Timing U F [ _ l State / 1 Madiine 1

Servi ces Li brary

Figure .5.1: RoseRT Mode1 Environment.

5.2 Structure

The structure of a L..iL-RT mode1 iç clescribecl in ternis of Capsules. Ports. and Connectors. Tliese

arc extensions added to the base CML Inngtage through its scereocype rnechnnism. F i g m 3.2 sho~vs

a class diagram for a simple LXL-RT system wtiilc figure 5.2 shows ;L stnicture cliagrani containing

these three core stntctirral cornponents. Base L X L elements such as cIasses c m also be usecl in n

LXL-RT design (DataCIass in figure . 5 2 ) .

Capsules are an extension to a Gl lL class. The- are highly encapsulated active objects. possibly

n-ith rnethods and data. Each capsule c m be considered a logicaiiy concurrent process chat interacts

with the environment through one or more b o u n d q objects called ports. Capsules have no public

methods or data- MI manipulation of a capsule's data (which is protected or prixtte) occius as a

response to messages arriving on the capsule's ports. Collaboration diagrams are used to describe

the structurai decomposition of a Capsuie class. The capsde .'SomeCapsulë in figure 5.2 has 1 chta

actribute (of type -DataCIass"). one method ("CaldateTieToGo'}, and three ports. Ports are

aiso inclicated by an icon in the lowest cornpartinent of the capsule box. The Log and T ' d g ports

are supplied by the services library. A log port provides the abiity to m i c e to the standard output

device while the timing port d o m a capsule to request timeout e\-ents. The -CamniandCorunis-

R + C l o g : Log a+ I NewPcRf : Erning .+ I CommandComnis: ComrnandProtoçol

W ~ P ~ O ~ O C O ~ S

CommandProtocol

+aStaItUp O cast : int = 100

--

Figure .5.1: RoseRT CIass Diagram

protocol role is a port objeçt that instautiates the interface clefinecl by the ~~CornrnanclProtoct~l"

protocol. As shown in figure 5.3. capsules can dso contain other (sub) capsiiles as part of tlieir

structure. Such nestirig can Lie carriecl to a r b i t r a l depth.

A port is a physical part of the implernentation of a capsuie that mediates the interaction of

the capsule with the outside world-it is ;ui object that implernents a specific interface [Lyons 9SI.

Ports reaIize protocols. which define the Row of information (sibgds) between connecteci ports of

capsuless. Ports dIom capsules to export multiple different intedaces: each possibly tIesignet1 for a

specific roIe. Figure .5.2 contains a CommandProtocol protocol which SomeCapsuIe fias instantiated

as its CommandComms port. The class diagram for the CommandProtocoI shows the signals that it

clefines. There are four -in' signals (arrows entering the box icon) and 1 out signal (arrows leaving

the bos icon). Ports are either conjugated or not. For a conjugated port the in signais become out

signais. while the o.ut signals become in signais. Two ports are compatible ruid c m be çonnected

together when the in signals of one port are the out signais of the other.

Connectors are used to join the ports ~f coiiaborating capsules. They represent the channels

through which messages (signais) Bow. Conneccious c m be made staticaiiy a t design time or cly-

Figure 3.3: Structure Diagram for ÇorneCiipsiile

naniically at riln tirrie. -4 RoseRT nioclel consists of a netaork of collaborating capsules (ancl classcs)

communicating wÎth each other strict- through messages sent nnd received on their ports. Figure 5.3

shows a single connector which represents a connection becween a containirig capsde's Cornmiuirl-

Cornms Port and a contained capsule's SewPortl port.

5.3 Behaviour

The behwiorir of a simple capsule is defined by a capsrile's RFSN (CSM). CSMs .are basetl on

ROOM Charts [Selicf 941. ahich in turn were based on Hareik Statecharts parel 871. The niain

difference berneen Harel's Statecharcs and capsule state machines is that capsule state t~iiicliirie~ ilo

not include and stace2'. X CSSI specifies the behaviour of a capside in teaction to cvents receivctl

frorn its ports. In sipals deline the stimuli that cause transitions in the CSSI tu be taken. Out

signals define the responses that cm emitted by the capsule. In the context of capsules the term

signal is sjnonymoris ~ i t t i event and message.

Complex capsules contain an internai network of coiiaborating subcapsules joined by connec-

tors. These subcapsuIes are capsdes in their ovn right. each n-ith its own optionai CSM. and c m

themselves be decornposed into subcapsules. This type of decomposition c m be carriecl to ahat-

ever depth Ïs n e c e s s a aüowing the modeling of arbitrariiy cornplex hierarchicd stnxctures and

behaviour,

21Concurrency in Stiitecharts is modeleci by and States. ROObt md UhIL-Kï rnodel concurrency via capsules.

5.4 Transit ions and Triggers

Wien a capsule is rcsting in a state. it is ready to proces events. .In event (&O cded a message

or signai) arriring at one of the capsde's ports ma!* trigger a transition. in the RoscRT toolset

twents rn- be assignai one of 5 priorities. An event wiI1 on1y trigger a transition if it satisfies the

triger specification of any of the outgÔing transitions attached to the ciment state or any of its

parent states. If multiple transitions are enabIed then a scoping ruIe is iiçed to rIecidc which nf the

canclitIate transitions d i be taken (when a transition is caken it consumes the cvenc: therefore n

single eveuc triggers a single transition). The sc~ping rule specifics chat transitions be t.vnluatec1 iit

the lowst possible ~ubstate first. if none of these transitions are enabled for the ciirrcnt ment then

die triuil;itioris of che subs~ate's itnrirediace pareiit arc evaluarcd. IF norie of ttiese are etiabletl tiieu

t!vdiiation concitiues uuti1 the top level state is reached. IF none of its trarisiciotis are enahlecl tken

clip ewnt il; cliucar(lerl and there is no effect on the CSM.

A transition iwuaiiy takes the CSBI fiom one state to another. Hon-cvcr transitions can s t m

and enci on the same stilte. Each transition may have an action abject associatecl nit11 it. An

action object is a C or Ci+ code bIock similar to the body OF n c las methoci (chiring the code

generation process. actions are transIated into cIass methods). -411 signais sent by a CShI (Le. the

statc machine's outputs) must be codcd in the rietai1 Ievd in an action object. X transition code

segnietit tia access to ail uf its capsule's vxiables (n-hich are the CShI's estendecl variables) and

any ai~totriatic mriabIes instmtiaterl in the transition's code block. .4dditionally the niessage that

caused the transicion is stored in a predefined extencled state m-iabIe (a void pointer rianiecl rtdatn)

so chat it can be acçessed by the transition code ifriecessary (for ex'uariiple ic cari be tticd to ilcttw~iiric

the source of the ruessage and to recover the parameters. if a- p a s 4 in ttie message).

Figure -5.4 depicts a tien- of a HFSM in which composite scates are shom dong n-ith tbeir

decompositions (trie toolset does not show KFSMs this say: each composite state is shon-n in its

o m window). Whcn a capsule is created. it is in the TOP state. ï h e TOP state cannot have sibling

states: it is the root of the state machine hierarchy. h e d i a t e l y foiiowing instantiacion a capsule

receives a meation" went that tnggers an initia1 transition. In figure 5.4 t h k is the transition

iabeled -Tln. The initid transition is the start of a new concurrent thread of execution. From n

design point of viexv al1 capsdes c m be treated as concurrent objects each running on their onn

Iogical thread. in Iater stages of dewloprnent logical ttrreads can be assigned to separate physicai

threads or d t e m a t i d y grouped into a smder number of physicd threads (for efficiency remns).

Figure 5.4: CS11

Tlie initiai transition is triggered by the "creation" event. With the excepcion of thc initial

transition al1 other activity is triggered by the arrital of events at one of the interface coniponencs

(ri port). -4 simple crigger spccification consists of thrce parts:

The name of the triggering signal of the event chat causes the transition.

The name of ttie port on tvhich the event is espesteci to arrive,

An optiond giiard condition. -4 giard is a Boole,m expression that is evaliinteti rlynarnicaIIy

when the event is scheduletl for processing. ft is entercd irito the tooiset i is ;i C i - rettmr

statement. If thc gciard ecdiiates to TRUE the crarisition is taken. r)cliertvise t h rwrit is

ignored. if no grard code is present the giartl rlefairlts to TRLT.

X single transition cari have more than 1 trigger. The semantics of these cornpound tr isers are

sirch that satisfying any one of the simple triggers d l cause the transition to occur. Final- there

is a universal trigger chat specifies a shorthand way of sa-yÏng that any incoming signa1 event ni11

cause the transition. h universal trigger may aiso have a guard associated n-itii it.

In the toolset each Ievel of the HFSM is shonn separately. each composite state in a leveI is then

displayable as a state machine in its onn windon-. This is camed on to arbitraq- depth. Each state

c m have a code segment associated nith state entry and state exit. For eYLLrnp1e ahen transition

TI in figure 5.4 &es state SI is entered. This causes the entry code for S I to be euecuted. SimiIarLy

vhen transition T2 is triggered? the exit code for SI wi l i be euecuted. Transitions t int nit across

multiple States are partitioned into mdtiple transition semgmem. each of which may have it's on-n

code block.

5.5 Active Object Issues

5.5.1 Priority Inversion

-1 capsule is an active object &ose behaviour is detined by a HFShI. Cspsiiles eshibit riin to

completion sernantics. This means that once a transition chain h a been s tmed another transition

c h i n for chat capsule d l noc be starter1 imtil the current one is coniplete. 1 transition chain is

coniprisecl of al1 of thc actions tiiken in response to the arrivai of a single event. In figure 5.-L when

an event triggers the transition T2 a transition chah consisting of four actions is initiated. Firçt the

exit action for state S1 is executed. folIowed bu the action code for transition T L transition T3. and

findly by the entry code for state 521. Once this chain h a been startecl it cannot be internipcecl

by the a m d of a new evcnt at one of the capsrrlc's ports. This may lead to priority inversion in

the case where a high priority message arrives whik a capsuIe is in the rniddle of a transitioii ctiaiti

triggerecl by a Lower priririty et.cnt. The benefit of nia to completion sernantics is that within a

capsiilc there is no concrirrency. This avoids having to protect a capsule's data structures against

race conditions.

5.5.2 Spontaneous Events

Passive objects modeled by FSBls onIy reacc to stimuli from the environnient. in addition to this

hehai-ioiir. active objects modeled by HFSJI. such as RoseRT capsules. m q initiate actions. in-

clitding self-actions. in RoseRT motieis che latter causes apparently spontaneous behaviour by a

capsule which is iisudly the resuit of that capsuieS t h e r e-upiring. and less ohen due to some other

intemipt. e.g. an device hidden by the capsule. Obvîously the addition of spontaneous interna1

events adds to the compIexity of adequately testing a capsule.

5.6 Summary

This chapter has presented a brÏeF overcie~ of RoseRT motleIs in terrns of their structure <ancl

hehmioiir. It has also introduced the idea of active objects. A thorough discussion of a cool as

compler as RoseRT is beyond the scope of this chesiç. X more detailed description of RoseRT statc

rnactiine behaviour is prcsentecl in .Appendi. D.

C hapt er 6

Mutation Testing of RoseRT

Models

Ctiapters 2.3. -1. and 5 presenced ctie backg~und information necessary to consider mutation cestirig

of RoseRT CS& This Chapter proposes and discusses an initial set of 16 rriutation opcrntors for

cssts.

6.1 Mutation Operators For RoseRT

The work by Fabbri cc al. is concernecf wich mutation testing of FSM and Statechart specificntions.

However. with RoseRT the mode1 entered into the toolset is more chan a specEcation. With the

addition of detail level code and the code generation facilities of the toolset the specification becomes

the implementacion.

In the n e z section mutation operators that operate miinly on the schematic levei elemencs of

RoseRT CSXs are defiaed. Schematic eietnents are those elements that are presented graphicdlj-

b - the tooiset. They roughiy correspond CO elements of the state machine contror structure (Le.

States. transitions. guards. stimuli. and responses). A t the detail Ievel any mutation operator that

is reasonable to use on C or Ctt cm be applied to a RoseRï Model. Hoivever this thesis o d y

looked at schematic level mutation spedic to the b?uiL-EU? language as impiemenced in che Rose

Real-Time tooiset. Therefore dthougb entire code blacks such as transition actions zrc mutated

(cleletecl or exchanged), no mutations are applied a t the source code token level of the detail code.

Further no mutations of structure diagrams. protocols. or optionai capsules are performed. Whereas

some operators (Fortran and C) are partially based on studies of typical programmer errnrs. ~hese

RoseRT operators are not (no studies are arailabIe). instead the RoseRT operators are baset[ on:

a Chow's error classes.

a Proteimi/FShi/ST operators.

a uperators useil in tradicional progranunhg Ianguage mutation syxemi-'.

- niainly iriterested in coverage operators

intrinsic featrires ancl weahcsses of the RoscRT toolset that can Icad to rnistakes.

a discussions nith iiscrs of thc tool:

- stiiclent uscrs of the cool (3)

- An R\iC faculty member who taught the cool as part of two offerings of a course. He

WLS able to provide information on obsenations of common sttitient errors.

- in the cnd these ciiscussions tfirl not influericc the choice of operators presented in this

thesis.

a personal experience in using the tooI over an 18 month period for courses nncl other projects

O The itlea chat mutant operators shouId mimic semmticall~ srnail faults as disctrssecl in chapter

3. Operators chat result in semanticaiiy large faults aiIl be of little use'" as such errors n+out(l.

in practice. be foimd during development.

Althoiigh studeut users arc signüicantly cfifferent than professioual users of a- tool, the rcarcity

of both professionaiiy cleveloped models and professiond cievelopers meant that only t a o of the

capsdes used in this thesis were professional1- developed. This is acceptable in the contcst of this

thesis. ,as the main intention is to obtain some preliminary resuIts n-hile evduating the feasibity

of implementing mutation cesting wïth RoseRT. Future mrk. if deemed worthwhile. ivouid require

information on errors committed by experienced developers and access to RoseRT modek developed

by them.

'-Specificdly Fortan, C, and Ma83 operaton. "In fact they wouId be detrimentai to mutation testing as they would increase the cornputationd burden on the

mutation systern.

6.2 A Proposed RoseRT CSM Mutation Operator Set

The operator set proposed in this tliesis is dividecl inco three catgories: FSN operators. covcragc

operators. and RoseRT incrinsic operators. %te ~hat ci& is not a rigd classification as sonie

operators could qualify for rnembcrship in two of thc classes. Wiereas mutation testing researchers

often imply that mutation operators mimic programmer errors no such daim iç made here. Sonie of

~hese operators are basecl on C h a t s secpencing errors. sonie are designed to obcain cuverage. soine

are intended ta siniu1at.e programmer mistakes. and some are clesigpci rio simulate subtle rnistakes

iinintentionaliy encuurnged by the RriscRT tooisct.

6.2.1 FShl Operators

The definition of mutation operators for this category &-as baseci on the error classes i ~ s i x l in Pru-

teum/FSS[ ~ h i c h in turn are hasecl an Choa's error classes arid on tieuristics about cypiçnl titrcirs

iriarie by prograinniers rful-ing chc crciltion of FSXls.

1, TRD: Transicion Deletc. Detete each transition in the CSM. This operator produccoi I miicmt

for eve- transition in the rnodel.

1. TTG: Transition Trigger exchangecl. Replace a transition's trigger n-ith cve. nthcr pnssibk

trigger. The RoseRT literature uses the tenu triggcr to refer to the stirnuhi~ everit r:;itrsing a

transition to be taken. Consider a CS11 with T s i rnpFi triggeretl transitions tuid E possible

triagering ec-ents. This operator ni11 produce on the orcler of (E-1) ' T tiiutarits.

3. TAT: Transition Alter Txget. Alter the target stace of a transition. Siïth a CS11 the target

mets are: of a transition is defined in tcrms of the transition's source state. The possible ta,

O The parent of trie source state.

O Anp sibling of the source state.

O The source state itseif. This is cded a transition CO self in the RoseM Iiteratirre.

Ttie TAT operator changes the target state of a transition to be each of the possible alternatives.

It is impossible to give a formuIa for the nurnber of TAT mutants generated for an a r h i t r q

capsule. The number of mutants depends on the hierarchical structure of the capside.

"A simpIy triggered transition responds to a single event. X cornpiex tnggered transition responds to muhipie diEerent eveots.

One of the Proteurn errors is wrong dort state- RoseRT models do not have a start state per

se. rather there is an initial transition that originates From a pseud+state. With RoseRT the

wrong stort state error is performed using the T-IT operator to chauge the target state of the

initial transition to each of the originai target's sibling states and to the parent if the parent

is a substate with an exiting transition.

4. ADT: Action Delete Transition. For every transition with an action. delete the action. This

operator protluces 1 mutant for every transition in the CSSI that has an action associatetl a i t h

it.

.5. ADE: Action Delete state Entry. For every state in the CShI with an entry action. ticlete the

action. This operator produces 1 mutant for every state in the CSM chat has an entry action

associateci with it.

6 . ADX: Action Delete state exit. For every state in the CSSI rvitli an esit action. clclccc the

action. This operator protiuces 1 mutant [or every state in the CSM that has an exit action

nssocirited with it. The ADT. XDE, and AD-Y operators correspond to the o n t p t missirig

error or Table 4.1 and are riesigned ro End action (output) errors.

7. TCA: Transition Chauge Action. For every transition action objecc in the CShI replace the

action aith every other trarrsition action in the CSSI. In this thesis each action in che CSM

is assumed to be unique. This assumption hoIds in the rnajority of cases. however there coultl

be a few exceptions. This asstmption 1s macle in order CO sirnpli- the iniplementation of

this operator. The cost of iising this tvrong assuniption is chat more mutants rire generntcti.

Liowever thcse Fe%- excra mutants are copies of t'xisting mutants and rio not neetl cstrrr test

sequences to be killed.

S. SDL: State Delete. Delete each state in the CSM. This operator produces 1 niiitant for ewry

state. Deleting a state in a HFSM also deletes aIL of that states sub-states and al1 of the

transitions to and from the deleted state. [ntuitively thii will ofken be a large sem,antic error.

6.2.2 Coverage Operators

These operators are inspired by the Mothra trap operator in which each statement of a Fortran

program is replaceci with a Trap c d terminating the program. in 4Iothra this enforces statement

coverage. With RoseRT these operators are based on the belief that di transitions should be

esecutetf at l e s t once, al1 states shoultl be reachable. and that there shodd be no terminal scates

in the niodeI. The necessitu of guaranteeing no terminal states is application dependent. Although

il continuoiisIy running reactive system uually would riot have terminal states. the existence of a

terminal state is not always an error. However it is probably tvorthtvhile to identify 'an- terminal

states in the software then clecide whether they are intentional.

1. CVT: CoVerage Transition. For every transition ir i the CS31 change the tra.~sition's (pcissibIy

empty) action to output a terminate message. iiilling al1 of thcse mutants ensures transition

roveragc. This operntor protliices 1 mut'mt for eveq transition in the CSSI.

2. CSE: Coverage State Entry. For evecy state in the CSSI change the state's (possibk empty)

cnt- action to output a terminate message. Killing ail of these mutants ensitres ttiat al1 States

in the CSSI are reachable. This operator produccs 1 mutant for evc- state in the CSSI.

3 . CSX: Coïerage State Exit. For evecy state in the CS>[ change the state's (possibIy ~rnpty)

action to outpiit a terminate message. Iiilling al1 of thcse mutants ensures that there are no

rwrninnl statps in the CSM. This operator prodiir~s L miitant for evecy transition in the CSM.

6.2.3 U-vIL-RT/RoseRT Features based Operator Set

TIiese operators adtlress the intrinsic features of CSIL-RT <anci the RoseRT todset. Sote that sanie

of the operators listed in the other classes (such as CSE ancl CSS) ancl coiilrI be incliitied here

tion.e\.er tEey fit more naturally into othcr classifications.

1. TGT: Transitiori Guard True: For every guarded transition in the CSJI. change the guarcl ta

true (i-e. the transition is always enabled). One could also change each guaril to aiways return

falçe. however for guarded trarisitions. this has the same effect as deleting the transition ahich

is perfomed by the TRD operator. This operator generates one mutant for every ginrded

transition in the CSM.

2. TCP: Transition Choice Point: For each choice point in the CSM. change the expression

controiiing the choice to dm-ays be True and always be false. Choice points alivqs emiuate

a booIean expression in order to seIect one of m o possible outgoing transitions. Boolean

expressions are entered as Cf+ code. This is the only operator considered in this thesis

in ahich detail Ievel code is mutated. However the mutation is simple in Ehat source code

tokens are not mutated. rather the choice point's code segment is completely replaced by the

statement retuni TRUE: or retum FI4LSE:. This operator generates two mutants for each

choice point in the CSII.

3. XCP: exchange Choice Point Transitions: For every choice point in the CSAI eschange the

triie and false transitions. This lias the effect of negating the boolean expression controlling the

choice point. This operator generates one mutant for every choice point in the CSAI. Althoiigh

this appears to he a cletail level mutant the operator is actually applied a t the schematic level.

4. XTI: eSternal To Internai: For c iwy externd self transition in the CS11 change it CO an

interna1 transition to seIf. Transitions to self haw an intcrnal-eutcrnal Hag associatecl n-ith

them. ff esternal is selected then any scate esit actions within scope are eseciitecl. followeti

by the internai transition's action. foilotvett by any state etitry actions within scopc. if the

internal flag is selecterl then only the internai cransition'ç action is euccurcri.

$3. ITX: Intemal To eXternai: For every incemal cransition to self in the CS51 charige it to ;in

ctsternal transition to self. This is the cornpienient of STI. 00th the STI and ITS opcracors

rriirriic subtle rnistakes niade bu iisers of the toolset. The toolset encourages this niistdx iri

tliat seleçtirig bctwetln internal or esternd is accornplishecl by selecting a check box on one

pane of a thrce pane dialog box. It is easy for a developer to miss or neglect ttiis sritting.

if none of the states n-ithin the scope OF the transition have entry or exit actions then the

internai-estemal selection cloes not atfect the behariour of the CSM.

6.3 Summary

These operators can also be divided into two categories depending on the number of mutants they

generate. Linear operators generate mutants in ünear proportion to the number of states anci

transitions in a CSLI. For example the TRD operacor is Iinear. It produces L mutant for ench

transition in the CSbI. Yon-liiear operators generate m u t a m at a geater rate than the linear

operators. For example the TTG operator generates mutants in the orcler of T f h > . where T is the

number of transitions in a CSM and tr ig is the number of possible triggering signais ( in sibaals)

that cari trigger a transition in a CS3I.

There are possibly an infinite number of 1~x4s CO mutate a program whether it is state hasecl

or not. The operators proposed in thiç chapter do not constitute an eshaustive set rather they are

exploratory and intended strictly as an initial investigation. The nevt chapter discusses hon- these

mutation operators arc appiied to the teutuai representation of CSMs.

C hapt er 7

A Mutation Testing System for

RoseRT [MTS-RT] Using TXL

This chapter begins with an overview of the entire mutation testing system for RoseRT [MTS-RT]

chat as implemented as part of this thesis. The rest of this chapter is concerned witti explaininj;

what TSL is and how it was useù CO apply mutation opcracors CO RoseRT niodcls.

7.1 Overview of the Implementation

The developnient of tooIs to support anything but explaratory mutation testing is mandaton since

the number of mutants quickiy becomes large for even small program. The tools deveioped for

this thesis correspond to the qpical mutation testing activicies of mutant generation. test case (test

sequence) creation. program execution. and resuit analysis. Testing rvas perfomed at the capsule

Iex.el. i.e. each test session was targeted at testing the state machine of a single capsule. The correct

operation of the SITS-RT is dependent upon the diiectory structure s i d a r to the one shonm in

[igue 7.1. The actuai directory structure is defined in an initialization 6ie. The basic workfiow of the

system is shonn in tigure 7.2. Several intermediate steps have not been shown in order to Ïmprove

cIxity. inputs to the system are the RoseRT mode[ and an initial populatiou of test sequences-

The initial test sequences rnay be baed on transition tours. snz-itch covers. or any ather heuristic.

In practice, home~er. they usualIy corne Erom the tester is boking at the speciücation or the

capsule to be tested. Subsequent test sequences are gneratetl in order to kill the remaining live

mutants. The overdl proceçs can be broken into three phases. in the first phase mutants are creaced.

in the second phase test cases are created in the thirrl phase the mutants esecrited. and che results

'his chapter describes the fint phase. Chapter S describes the remainder.

Figure 7.1: Directon. Stnictrire.

7.2 Performing Mutations

h11itation operators were applied to the textual representations O€ RoseRT capsules tising TSL (to hr?

described later). Rose RT modeIs are saved in plaintevt disk files callecl petd files (.rtmciI estension).

An esample RoseRT capsule file is includecl a s Appendiu B. These Files .are formatteci as a linear

sequence of sections. Each section describes one of the major eiements of the mode1 sitch irs a

capsule or protocol. Petal files are writcen in accordance with a Rationd p r o p n e t - context free

grarnmar (CFG) that c m be described n-ith BNF or TIL. B3T' Grammars are adable for the

earlier ObjecTime R00M models from which RoseRT is derived. however no RoseRT gramman

werc made avaïiable by Rational. X working T'CL granimar wns developed for the RoseRT capsule

section through the e~amïnatiou of severd empIes , The complete RoseRT capsule gamniar as

tieveloped for thk thesis is inchdeci as Appendiu C.

P h n n Mutahm 1 MTS-RT

Mutant Files P est Sequeme P Note th& the onginai capsuie ir nui through the system just like an^ athér

'. \

\ \

~nfm&hfmmthe resuitr is fedback &O the test scquence 3neaJ\ation pmcess in ohs,to kiii remaUruig L w *tant5

w 1 4

_ Rwdt Files R e d Cornpaison Mutatisaa Score

- RT-SuperDa

Figure 7.2: RoseRT SIutation System:Esseritial \Vorkffows.

7.2.1 TXL

TSL (Tree ïransforrn Language) is a generd piirpose programming Ianguage clesigned to support

transformationai programming [Cordu+ UDI. TSL is useful for transforming an input. describeri by

a CFG. into an output. described by the s m e or possibly clifferent CFC;. X TSL progarn consists

of a collectiori of one or more rules that describe by c-ample how tto transforrn the input inco EIIC

output. TSL prograrns can aIso be used to extract various pieces of information frorn riny source

file describable by a CFG.

A TSL program operates in three phases- F h t the entire input is tokenized and pused iri

accordance with a supplied CFG. The output of this phase is a parse tree. in the second phase

transformation niles are applied to the parse tree to produce a new (transformed) parse tree which

m q confom to a different grammar. -Us0 during this phase intermediate files can be nritten in

order to extract interesting data. The finai phase unparses the transformed tree by perforrning an

inorder d k of the transformed passe tree and mithg the Leaves to produce the output [Cordyf

001.

7.2.2 TXL Grammar

T-YL g r m a r s are specified in a uocation simiIar to Eutendeci Backus-Nauer For~ii (B3T) [ C u r d ~ t

001. The basic unit of a TSL grarnmar is the define statement. h d e h e statemerit specifies an

ordered set of alternative forms for one non-terminal type. corresponding roughly to the set of

prodrrctions for n single non-terminal in a BXF pamrnar. In deîïnitions. literds and user definecl

keywortls mrist he prefked with an apostrophe in order to resolve conflicts with TSL's keyn-arcls

and prerlefined non- cerniinals. Each alternative Form of a definition is specified as a secluence oE

~erminai syrnbois and non-terminais. :\Iternatiws are scparatccl bu rhc vertical bar synhal. Non-

cerminaIs appearing in the body of a definition are encIoserl in scjuarc hrackets [ ]. For exaniplc one

definition for a state maciiinc transition could be:

def ine transition ' ( 'object 'Trans [stringlit]

'source [stringlit] ' target [stringlit] [opt aaioa-objectf [opt event-guards-List]

'1 end def ine

This definicion would be referenccd in other definicions or TSL rides as the non-tcrtni~ial sytiibol

[transition]. The follun-ing tes% corresponds to a transitioa object taken from a RoseRT capsoie that

matches this ctcfinition of a transition:

(object Trans "t-scan2jam1' source " :TOP :s-operating:s-scanning : Junctionl" target " :TOP: s,operating:s,jamming: Junctioni"

1

[stringhl ù predefined in TSL see beIori-. Xocice chat the esample docs not h a ~ e an instance

of either the [opt actionabject) or [opt event@ardslist] non-terminais. These non-terminais are

declareci optionaI by the opt keyword.

The foilotring List shotvs some of TU'S predehed non- teda ls :

id . h y identifier be,@ming nith a letter or underscore and contuiuuig with any number of Ietters. digits. and underscores.

[number ] An>- unsigned integer or real nurnber.

[stringlit Xny double quotecl string beginning and ending ~ 5 t h a ciouble quote and consisting of a- number of characters in between. For example ~:TOP:saperating:sscanning:.Junctionl~ is a legal [stringlit] non-terminal.

[charlit ] Like [stringlit] but beginning and entling with single quotes. For example -+'

Xon-terminais can be modSed by prefkuing them with opt. repeat. or list as follows:

[opt X ideclares tliat the presence of the non-terminal S is optional. [ri the transition clefinition given above both the [actionabject] 'and [eventguarclslistl nonterminals are optional.

[repeat X 1 States that zero or more repetitions of nonterminal S will be niatched.

[repeat X+ 1 states that one or more repetitions of nonterminal S d l be niatched.

[list X ] States that zero or more repetitions of S separated by a comma will be matchecl,

[list X+ 1 states that one or more repetitions of S separatecl bu a comma will be matchcd.

7.2.3 TXL Transformations

In the second phase the input parse tree is transformecl into the ciesirecl riiitpitc parse tree bu the

application of a set of transformations (specified by TSL functions arirl riiles in the TSL progan~) .

The basic TSL rule defiiiition has the following forrn:

rule name argument [argument,typel replace Ctypel

pattern by

replacement end function

R-here name is the mle name. and [type] is the non-terminal target of the rule. The tgpe cletermincs

the scope of the rule, Le. which nodes of the parse tree the nile will match. The pattern clefines

the particdar type and shape of the sub-trees that must be matched for the nile to proceetl. The

replacement represents the subtree that will be used to replace the subtree that n*as matched in

pattern. The d e d l perform the replacement o d y if the pattern match succeeds othernise the

pane tree is left unchanged. -1 d e may &O take an argument of ~ p e [argumentzype].

The folowing is an example of a function that matches a transition object and repIaces it bu

a ne%- transition object in &-hich the target of the transition has been changed to that supplied by

the NewTarget argument. Narne, Src, and Tgt are variable names bound to the elements of the

hct ion 's pattern. The convention of lead capitais has been used for variable names.

59

function alterTarget NeuTarget Cstringlitl replace [trans-oh j ect]

' ( 'object 'Trans Name [stringlit] 'source Src [stringlit] 'target T g t [stringlit]

'1 b Y

' ( 'object 'Trans Name 'source Src 'target NewTarget

'1 end function

Rules are applied reciirsively. if a rule's pattern matches a sub-tree of type [SI anct the replace-

nient is a new sirb-tree that is also matched by the rule's pattern then the nile mil1 be applietl to the

replacement. This continues until the new stib-tree (alwqs of type [SI) does not match the riile's

pactern. Fiinctions arc rules that are not applied recursirely-

Rules and functions r n q also contain variables. constructors. and cleconstriictors. Vnrinbles are

names boiind to sub-trecs of a particiilar type when the variable is introducetl. Constnictors are

useif insicic rules to builcl intermecliate subtrees for later use in the ride. Deconstructors are usecl

insicie d e s to break apart (match) large subtrees into their constituent parts (in accorclance with

the grammar).

It is aIso possible to write rules that do not perform a replacement. rather they n-rite sorne

portion of their scope to a file for later use or analysis. TSL is a miich richer progamming langiage

than çan be tlescribecl in this section. For an in ciepth description see [Corclyt 001.

7.3 Capsule File Format

The telaual form of a capsule file is organizecl as a Linear secluence of sections. :i complete capside

test fde is included as Appendk B. The behaviour of the capsule is ciescribed in the stateniacfinc

section. The statemachine section contains a single top IeveI state object which is the parent of

al1 States in the capsule's FiFSSL State objects contain a k t of their contained state and pseudo

state objects foUoved by a List of the transitions bemeen the states and pseudo states in the List.

State objects may &O contain a section describing their optionai entry and exit actions. Pseudo

state objects correspond to the states. junction points. initial points. and choice points founci in the

behaviourai level of a RoseRT capsule. Junction points are an implementation detail chat permit

transitions crossing state boundaries to be comected. - b y state object may contain its own pseudo

0 TXL Markup and Info Extraction: TSL programs are tised to perform a marlrvp on the

cripsule Ne. Slarkups are used to iiniqueiy iclentifr. each pseuclo state object and transition

object in the capsule. X rnarkup consistc of inçerting a '<' symbol lollowed bu a unique

identifier before the nonterminai and inserting a '>- symbol after the nonterminal. -1 second

TSL program parses the markedup capsule file in order to produce several ciifkrerir. l i t s of

information. These lists are stored in separate files and include:

- Iist of transitions

- Iist of States

- l i t of al1 pseudo state objects

- list of protocols and their signais used bu the capsule

- l i t of guards

- list of choice points

- list of transition action objects. which are the code boclies associated n-ith a transition.

0 Mutat ion Programs: For each mutation operator there is a mal1 control program Iivritten

in C++) ancl a TSL program. Each control program invokes its correspontlin,~ TSL program

iisirig information from one or more of the information files as parameters. A single invocation

id the çoritrol prograrn will execiite its TXL prograiri multiple tirries, once for eacli nultarit co

be gnerated. Somc concrol progams also mite a log mhich correlates the rntitated elernent

with the file rianie of the mutant.

TXL Remove Markups: EacIi mutant capsule is processerl bu another TSL prograni n-hic11

rernovcs the rnarkiips.

Coucatenation: For each miitatetf capsule a coniplete mutatecl RoseRT rnodel file is creacerl

by concatenating the pretiu. specitic mutant. and SUEY filris. The resuiting mode1 files are non*

ready for use by the RoseRT code generation program.

The program for each mutation operator is i n ~ k e d Gorn the comniand Line. A more sophisticated

mutation system wouid ailom seIection of desired mutation operators via check boxes on a G U form.

however in this system individual operators are selected by ntnning or not ninning their associated

control programs.

7.5 Example Mutation Operator Program - TAT

The TAT operator alters the target state for a transition. The control program tat.exe follonis the

following procedure when invoked:

1. BuiId a list of al1 of the transitions in the capsule by reaclig the transition list file.

2, biiilcl a list of al1 states in the capsule by reading che state L i t füe.

3. For each transition in the capsule:

(a ) BiiiLl a lisc of al1 Icgal targets for the nirrent transition. Legal transitions incIiirle al1 sibling states, the transition's source statc .mcl possibly the parent scate. The parent state is only possible if there is an explicit transition frotn one of the sibling states to its parent. This requirement is necessary because chaiiging the target state of s traiisitic~n requires that the target state aiready have <an entry junction point. Transition c3 in tigrre 7.3 is stich a transition. The original target state is not includecl in this list.

(b) For each entry in the target list:

i. The control program invokes the TAT TSL program to find the crirrent transition and replace its carget with the target supplied. The unique identifier for the ciirrent transition. the new target. and the fiIe name into which TXL niIl wi te its oiitpttt are passed to TSL as command Iine parameters. The file name d l be T.-\TSS.cap. The 'i corresponds to the number of the unique identifier of the current transition. the Y corresponds to the list indes of the new target. TAT-4-7.cap inclicates that transition number 4 has a new target which is the rth entry in the L i t .

ii. Log the mutation in the TAT log Ne. This fik is Iielpful in creating test seqtiences to kill live TAT mutants: the tester does not have to look at the capsule file which is considerably larger.

Figure 7.4 shows both Icwls of a two level statemachine. Suppose the current transition is t'll whose

source is S27 and tiirgct is S21. The IegJ target states for the TAT mutation operator are S23. S21

(this sould be a transition to self which is Iegal), and S1 via the jtinction labellet1 t l .

It is usualiy not possible to view the mutant with the m e n t version oE the PITS-RTI toolset.

One of the sections of a capsule file is the stateview section. Ki section recorcls the physical Iayoiit

of the state machine elements as they are entered bu a programmer. if the mutant file is loaded

into the toolset the ciifference between the statemachine and stateview sections may either cause the

coolset to crash. or be ignored in whicti case the stateview incorrectiy shows the tinmutated siate

machine. in order for the tooiset to correctly display the mutated state machine it is necessq

to alter this section or remove it compIeteIy. Removing it results in the toolsec creating a ne&-

graphical vie=- based on the state machine section. However aii formatting is lost and the resutt

is an incomprehensible mess that mus be reformatted mmua.iiy- The alternative is to dter the

stateview section. This wouid require the development of anodier grauuuür and more TXL progünis

#'

4" fi,, , , t ' " !fl * - - ,... ?.!<*':.'

- .. , ., - , - . . . " , .. < .,. ,,.;;;,. . . .. -*.

., :;>..: -2. '. -. , , . , . . " . ..b'.,?p.. : '

Figiire 7.4: T-AT example çtate machine.

effectivelu doubIing the work effort. It w s decided that havïng correct stateriews n;u- not

the effort requircd.

The architecture of the MTS-RT system is essentiallu that of a batch sequentid systern. AIthough

not particularly user friendly, a batch architecture is neil suitecl for prototyping a çustern consisting

of many s m d components. Each component cati be dereloped and tested in isalatian. Cade changes

when necessary are isolated. and the products of intermediate stages c m be easiiy esaminecl.

This chapter has provided an overview of the entire LITS-RT mutation system. and an e+uplma-

tion of how mutation operators can be applied to the textual version of CSMs using TX.

Chapter 8

Creating and Running Tests

This chapter describes the process of creating test cases. nmning test cases against the mutants.

analyzing the results of execution. and tliscusses what to do when a fault is found in the original

program.

8.1 Testing Process and Automation

Regarclless of the testing strategy (test adequacy criteria) usetl. the basic testing process consists ol:

Creating test cases based on the testing criteria.

Running the program under test and Leding it the tesr; cases.

Observing the mn-time effect the test cases have on the program. such as rneasuring coverage.

Comparing the actuai program outpiits against the expected outputs.

Recording the results in test logs and anaiq-ting the Iogs.

The feasibility and d u e of automating some of these steps depends on rnany factors inciuding the

type of software being testeci. the necessity of being able to repeat tests in the hrture (perhaps

regression testing). and the avaïiabiiity of an autornated oracle. For esample the value gaïned from

automating the testing of a singIe release web apptïcation is probabIy not wonh the cost. On the

other hand it may be weil m r t h the c o s of automating the testing process for a long livect muhi-

version system such as telephone mitchhg software. It is usudiy possible to automate the processes

of nmning the program against each test case, and recordmg the results. A program that cloes this

is a test driver. A driver cdls the program being tested and keeps crack of hon- it performs. It

shoulcl:

O Set rip the environrnent needcd to cal1 the iinit being tested. This m q involve creating and initidizing global variables.

O Invoke the prograrn under test against each test case or sequence of test cases. The test cases for these invocations could be read from files or be ernbericletl in the driver% code.

O Record the results and possibly check their correctness against the expectetl residts. and n ~ i t c the outcornes to a log.

in addition to tlrivers. testing often involves the use of stubs. -4 driver simulates the pan of the

program that c a b the iinit iuitler test. Stubs sinidate parts of the program c d e d by the unit onder

test. -4 stub can:

Check the reasonableness of the environment provicled bu the d e r .

O Check the reasonableness of the arguments passed by the calIer.

O !dodi& the arguments and the environnient. followed by returning values in siich ;i rvay tliat the caller can proceeri. It is best (obviously) if these effects are consistent with specification of tlie unit the stiib is siniulating. A goai of tcsting rtianagcrncrit is to scticcltilc testing so ;ü: to retliice the niimber of stubs reqiiired.

Testing reactive software mocieled by active objects such as RoseRT capsules aften requires chat

the timing of messages bc controlled. For a passive prograrn morieled by a FSM. timing is nor

controlIed by the FSM. rather it is controlled by the tcst harncss. With an rictivc object timing is

controlIed both bg the active object and the test driver. Active objects may generate their ottn et-encs

in response to iriterrupts and timers. In the case of a capsule. some transitions ni- be triggered

by an internai event such as a timeout. an intemipt, or a message Gom the Game service. In swh

cases it mq- be necessary that the test driver evplicitly create a situation Ï n which the internai event

c m occur. This requires that the time sequence of inputs to the capsule be controiIed bu the test

driver- The ganularity (events occur at intervals measured in miHisecon&. seconds. minutes etc.)

and precision (the acnrracy of timing measurements is measured in milliseconds. seconds. minutes

etc.) of the timing depends on the specification or requirements of the application.

8.2 Testing RoseRT Models

Often a RoseRT capsule is testecl by using the obsenability feature of the toolset. This feature allon-s

a cleveloper or tester to manualIy inject s iga l s into the ports of a capsule and observe ancl recorcl

the responses. This methocl. while good for developmerit. is very labour intensive ancl tests c-aiinot

be ensily nor aiitomatically repeated. I n oit-iised alternative is to cnnçtnict a clriver capsii i~ char

esercises the production capsule nith a limiteri nrimber (ohen one long sequence) of -harcl-cotlecL"

stimuli. This methocls weaknesses are the tediormess o l conscnicting many correct driver capsules

and the lirnitccl numher of twts ttiat cari be performeci hy a single ststic ciriver capsiile. To siipport

mutation tcsting a clrivcr is necderi that can:

be easilg (minimal progamming) used to test most capsiiles.

a execute the capsule iinder test (CapL'T) over a large and tlynamic set of test seqiiences.

a control the temporal seqiicncing of messages sent to the Cap UT.

a record the stimulus-response history of the CapCT for siibsequent analysis.

\Vith these criteria in niincl a test sequence creation tao1 (narned RT-TestCase.CIanager) and test

driver (ricmed RT-TestHarness) were deveIoped in tandern. RT-TestCaseManager is iisccl to create

test sequences that are eventrially read by RT-TestHarness.

8.3 The RoseRT Generic Test Harness (RT-TestHarness)

The RT-TestfIarness is a RoscRT moclel into which a capsule to be tested is imported'" (the protocoIs

instantiated as ports by the CapUT are also required). X skeleton version of the RT-TestHarness

mode1 is used as the basis for each capsule to be tested. The class diagram for the skeleton is s h o m

in figure 8.1.

;\fier importing the capsule CO be tested and any required protocok into the skeleton. a minimal

amotuit of nork m u t be performed by the tester via the RoseRllT toolset (about 10 minutes each

for the capsirles used in this thesis)- Figure 8.1 k an esample of the encl resirlt the class d iab~am

for a completed RT-TestHarness. in this esample RT-Testffarness instantiates tn-O classes and two

capsules:

"The tooiser dlowvs capsules and protocois ro be exporred and irnporred ro and Gom diik files. These featiires support software engineering activities such revision cantml, muse. and t e m cieveloprnenc.

*Capsule> B

Driver

.# ( TestComms

.+ i l a g + i timer # i TestComms

+ l hame <<Pottz> + i aboniïmer

Figure 8.1: Skeleton Test Harness Claçs Diagam RoseRT Mutation System:Essential WorkHows.

O JarnrnerDriwr: .A siibclass of the generic Driver capsule which is minimaiiy modifieci to inter- face &$th thc specific rapsule under test. The code for the sendEvent methocl is generated bu the RT-TestCaseManager application to be described in section 8.4.

CnpLT: an instance of the Jammer capsde which is imported from sonie development motlel. Two intemal self transitions are added as test instrumentacion to this class. This instrrrmen- tation is usecl to return information about uncxpected events. the CapGT's rnirrent stnte. and the Capu'T's temination.

O TestReatler: a class xhich reads test sequence files.

O DataLogger: a class used to record the resuhs of test mns.

Xlso shonn rire two protocol classes- TestProt is a protocol used to perform state queries of the

CapCT. Jarnmer Protocol is imported and cornes with the CaptiT. In this example the CapCT onl-

has one port. Some capsules have rnuitipie ports. In such cases the protocol for each port must be

imported. Yotice that the attributes and operations for the DataLogger. Test Rearler. and Driver

capsules have been eiided.

The test harness nuis according to the foUoning sequence:

Figure 8.2: RT-TestHarness Class Diagram.

Datahgger +databgger

1. Rcatl in etivironiriertt data from ari initialkation file.

restCOm*s

<<port>>

>

2. Determine the name of the current niutant by parsing commancl line arguments. RT-TestHarness is irivokerl bu RT-TestRunner (to be dicussed in section 5.3). Communication betn-een RT- TestRunner and RT-TestFiarness is via command line arguments.

un *Prntoc0la TestPrnt

+OstateReply O +otrap O +ounexpected 0

a +statcRequest O

3. For each test sequence file in the tests riïrectory

(a) Sec if this test sequence was run during a pretious session. This is tlone by checking to see if there is a corresponding results file for this test sequence file.

dapsule~ Jammerûriver

endEvent0 ~Capsule- 0

rackData0 CapUT

m# I JxComms- JammerPrniocol

+DstopJam O +njamStatusRequast O 4

(b) if the test vas run pre~iousIy then restart step 3

+ogoOpwate O +og~5hurdown O +oreset O O+jamStatusReply O

(c) instantiate the CapCT. Instantiating a ne-- instance of the capsule for each test ensures that the CapCT starts in a knam state.

(cl) initiate the data logger for a new test nui.

+ i Jwcornrnr

(e) For each signa1 in che test sequence file:

i. Send the signal (via the sendEt-ent method supplied by RT-TestCaseSIanger to be described later).

ü. if the signal &O specifies a dekiy then wait the specified duration before initiating another event. The deIay feature can be used to test events such as timeouts that are initiated by the CapCT.

K. Othemke mit for and log my responses.

I + i TçstComm:- + i log + i JxCornms + j IookThmughTimer + 1 BITTimcr

iv. dfter a short intend (default is 1 second) send a state request to the CapGT. This del- d o m the CapCT to respond to the most recent test input bu powibly rwving to a new state or sending a message CO one of its ports.

( f ) Close the results file for this test sequence.

(g) Destroy the CapUT.

4. Terminate the RT-TestHarness application.

Note that RT-TestHarness is just another RoseRT moclel from nhich C++ code is generaced.

The generated code is ultimately compiled and Iinked with the RoseRT senrices library ta procluce

a stancl-donc executable. One of ttiese test hamess models is created for each mutant to be tested.

The actual code generation. compilation, and mnning of each mutant test harness is iintler th^

control of the RT-TestRunner program.

Depentling on the CapLT it rnay be riecessaru to further mociify the driver siib-class. For esatriplc

sonie CapL'Ts rnay Iiavc otic or iriore periodic events that pnll the environment (siniiilatetl hy RT-

TcstHarness) for claca. Rather chan requiring the tester to creace test cases chat explicitly respond

to mch polletf recluest it niay be more efficient to add one or more internal self transitions CO the

RT-TestHarness wtiose purpose is to respond to the periociic reqiiests. This wns rarried out in sortie

of the capsiiles testecl in this cspcriment.

8.4 The RT-TestCaseManager Program

RT-TestCaseJIanager is a G U tool deveIoped using the Visual Cs+ and thc JIicrosoft Foiindation

Classes. Tt facilitates the rapid and easy creation of test sequences. Its main window is shon-n

on figure 8.3. Test sequences are stored in separate files in a specifietl clirectory (see fig. 7.1).

The leftmost pane of RT-TestCaseManager display the contents of this directory. Selectiug one

of these tests causes it to be displayet1 in the rightmost pane. One of the outputs Gorn the TSL

JIarhwp ancl Info Extraction phase (see chapter 7) is a file containing a List of al1 of the siqals

the CapLT can accept as inputs. These are dehed by the protocols that the CapCT instantiates.

RT-TestCaseManager reads in this information and displays it in the center pane. X signai can be

added to the end of the m e n t test sequence by selecting it Gom the center pane. If the seIected

signal takes an integer parameter then the value to be sent rvith the signal is <ais0 entercd ( ~ h e

default is void). Other data types are possible but are not included in this version. -ln optional

%-ait time can also be speciiied for each signai. A tirne entered in the vait column instnrcts the

RT-TestHarness to wait that amount of cime before sending the nevt message. Each test case aIso

bas a 6eld for comments. In the test case shom the comment indicates thnt the purpose OC the

curent test sequence is to try and kiii the mutant icientified as T-AT-ilo.

RT-TestCaseMariager ais0 generates the code body for the seniiEvenc method i~serl by the srib-

classed driver cIass (JamrnerDriver in figure 8.2).

in t sendEvent ( int event Index, int sendilata, int uaitTime1 { svitch (eventIndex1 €

case(JXC0MMS-STARTJAM) : { datalogger . appendlog ("sending JXCOMMS-STARTJAM\nW ; JxComms . startJam(sendûata) .sendo ; break; >

... more case statements ... )//end svitch return uaitTime;

1

The sendEvent method correlates the signal data in the test sequence üies (encoded as integer

constants) nith the actual RoseRT library cidi required to send the event. This code is generated

basecl on the contents of the center pane as descnbed above.

8.5 RT-Test Runner

Once the mutants have been created and an initial population of test sequences createcl the ac tud

ninning of the mutants ngainst the test seqiiences c m hegin. Running the mutants is performecl

11'. the RT-TestRunner prograni. RT-TestRunncr is a cornm~md line program that execnites the

following nlgorithrn:

1. Reaci in environment ctata from an initiaiization file.

2. Build a list of al1 mutants by reatling d l of the mutant file nanies in the mutcuits ciirectory.

3 . For each mut~ant in the iiit:

fa) Check whether the current niutanc was huiIt tliiring a previous test session. This is performed by looking for the current mutant's esecutable in the esecmables tiirectory

(b) if the esccutable nlreatiy exists then:

i. itivoke it and pass iti its name (the name of the current mutant) ris a comrnancl line argiiment. The progam being invokecl is the esecutahle version of the RT- TestHarness mode1 which concains a mutatecl CapGT.

ii. Go to step cl.

(c) If the executabte does not esist then concatenate the current mutant capsule fiIe a i th the p r e K ~ and suAiu files ttientioned in chapter 7.

i. invoke the RoseRT comma~id line tools whicti will generate C+t code basetl on the current mutant.

ii. invoke the C++ compiler which compiles the Cf+ source cocle =and links it with the RoseRT services Iibrary. These activities represent the largest computational biircien of the mutation syscem.

iii. Place a copy of the new emcutable into the executables clirecton for future itse.

il-. invoke the executable passing It the name of the current mutant as a command iine argument. The program being invoked is the ~ ~ e c u t a b l e version of the RT- TestHarness which concains a mutated CapCT. (Same as step b.i above).

(d) The executable (RT-TestHarness) mites test resuit files to the resuIts/currentMutant t i i r ec to - currentAIutant is a dieccory name identi-ng the tnutmt. An esample is TATJ2.

(e) Log the run in RT-TestRumer Iog me (this is a single text fiie that keeps a record of dl mutants nia by RT-TestRumer.

4. Close the RT-TeçtRunner log fiie and terminate.

8.6 Data Analysis: RT-SuperDiff

Before running RT-TestRunner for the first tirne the non-mutated version of the capsule file (caiied

cleanCapsuIe) is put into the mutants directory. This is the firçt %nutant" run. Its resitlts are

the results against nhich al1 of the mutant's results are comparecl. At m y tirne (even when RT-

TestRumer is ninning) the tester c m invoke RT-SuperDB. RT-SuperDiff is a program thnt pe r iom

a file cornparison between each mutant test results file and the results of clennCapsule. -4s each

cornparison is petformecf a log fiIe is written. Cpoii completion of al1 comparisons the Log fiIe is

prncessetl aricl convcrtecl into a table which c m be read tlirectly or importctl into a sprendshcet

progran1 as in table 8.1.

killeti Test effectiveness total mutants total mutants run niiitation score

Table 8.1: Data JLndysis Results.

Thii table shows .5 tcst cases executed against 18 mutants. -ln 'L* in the body oE the table

nieans -Live-. The row labeled "kilted" shows how many mutants were detected (Mled) bu che

çolumn's test case. The column labeted -hiIed" s h o ~ s hon- man'- test cases were able to detect the

row's mutant. h zero in the kiiled column indicates a mutant that has not been detected bu any of

the existing test sequences: a iive mutant. The row labeled test effectiveness indicates ho^ mm>-

mutants were kiiietl by the correspondig coIumn's test setluence. The colunm labeIetl proportion

killecl indicates the relative ease with nhich a mutant m s killed by d l of the test sequences. Tlie

mutation score is tlisplayed in the last row. In mutation systenis like hIothra. once a mutent h.as

hcen tletected. it is niarkecl as dead and never nui again. However in thii system we are interescecl

in analyzing the:

0 effectiveness (how many mutants ic c,an rlistinguiçh) of pruticular test cases against al1 miitants. and

0 the scrcngth (how hart1 chcy are ro kill) of mutanc operators against d l test cirses

Therefore deid mutants are not removed €rom the systcm .and crery mutant k eventiially riin against

Ç'VE'Cy tCSt Cirse.

8.7 Hunting for Mutants

By looking a t cliarts like table 8.1 the tester can see which mutants are live. Tlie next stcp is to

ii&l test seqiiencrs niniecl at killing these rernaining mutants. Creating new test secpiences to kill

rriiitants is the niost labour intensive p m of the entire mutation testing process. Testers can use

rxrioiis pieces of inforniation to create new test sequences:

0 The name of the mutant tells which mutation operator has beeri appliecl to ivhicti elcrnent the capsule. For example TFtD 3-1 for says that the mutation operator ti TRD: TRransitiori Delete. the 5 signifies that the transition object with unique identifier 5 was deletecl. The trailing L is alaays a L For TRD since there is oniy one way to delete a transition. Other operators can be applied severd times. For exatnplc TAT. Transition Alter Target. cari be applictl as many timcs as chere are legal aiternative target stntes. In this case the trailing number will inclicate n-hich alternative target is selectetl. Lleally the spreadsheet presentation shoitlcI Iink to this information attached ta a RoseRT CS31 diagram. As this would reqiure considerable fievelopment effort this Eeaturc was not developeci ditring chis thesis. It is left for future work.

a Some mutation operator control programs produce Iog Mes chat show the original and mutaterl capsule element. The TAT operator. for e.xampIe. Iogs the original and mutatecl target state for each transition chat it mutates.

0 The marked irp capsule file c m also be consulted.

&*hile trying to create new test sequences the tester may decide that a mutant is equitalent to the

original capsule. The tester may aiso decide that a certain mutant is too difficult to distingukh.

This may indicate chat the code is untestable which in turn may imply that the code shoulrt be

returnecl to the developers for improvement (design for testability was poor).

Xfter new tests have been added to the system RT-TestRunner is re-invoked. RT-TestRumer

and RT-TestHarncss ni11 only nin new test sequences. Sote that the original program miist be run as

well in order to obtain the expected resuIts for the new test sequences. A strength of mutation testing

is chat while adding test cases to kiU live mutants the correctness of the results of the compiitation

are not important: n e are only interestecf in whether the mutant's resiilts ciiffer froni the original's.

The correct behaviour of the original progrmi should only be checkecl after clie tnutant tins b ~ e n

killetl. This allows the possibiiity ofcuüing redundmt test cases from the test siutc. There is also the

possibility thnt wtiilts crrnting a killer tesr secpiencc a f d t is foiind in thr, original capsula (possibly

heciiiise the iiiutarit is the correct version). The next section disciisses this eventuality.

8.8 Finding Faults in the Original

The mutation literature is poor when it cornes to tdilking about mhat to clo n-hen the original progrnm

ha a faiilt. It is irnpliecl that fndts are fount1 tlitrïng the process of creating a new test to kiH lire

rriutarit. Wien any test is run on the original capside. the result ni- reueal incorrect hehvioiir.

Hoivever this recluires an oracle. a role often played by the tester. Faults may ;&O be rer-ealetl

through the process of exarnining the mutant code in order to generate the new test. This can be

though of as a directed inspection in that the rntit~uit points to the exact program statemcnt to

examine. This is supportecl by Hanilet3 suggestion that systematic testing is sticccsshil het::iiise it

forces a tester to pay systernatic attention to the software [Harnlet 881.

Wien a fault is found anci kxerf in the origind capsule (mhich is part of the RT-TestRarness

nioclel) al1 of the prerïous testing may Iie invaIidated. It is possible that a test case that MIS a

rnutnnt version of the faulty progam may not kir1 the new mutants created alter the fatth has bcen

correcteci.

With this system n-hen a f a d t is repaired. the corrected capsule must be extracteci Gom the

RT-TestBamess mode1 and the mutant generation ptocess must be cepeated as described in chapter

7. The tester must decide whether it is necessary to rerun any of the dead mutants against the Luecl

capsule since. as discussed above. a change to the CapU'T may affect the validity of pretious tests.

i.e. n Lest seqtience that killed a mutant may no longer 'di tbat same mutant bmed on the corrected

version of the capsule. This iç an important decision because generating and compiling mutants is

a major computationai burden of the system.

8.9 Summary

Onc of the problems of automateti testing in generai is that of excessive log files. Sonie of the log

files gcncratcd bu the MTS-RT systetn rire only consulteci when there is n problern with the systemi.

Others are consultetl in order to find information neccssary to kilI livc mutants. [n practice only a

m i d l perceritage of the data lrigged during the mutatiou process is ever consulted. Tliereforc orle

of the problems to be addressecl in future work is how CO efficiently extract the sniall mioiint of

information that is actuaily usehl to the tester.

This chaptcr introtlricetl some arrtomated testing icleas anci describeci the progams iisecl to create

mutants. control the execution of mutcuits against the test sequences. and anaiyze the test restilts.

After cfescribing the programs the proceçs of killing niutanst cwtl finding faults was coveretf. SIany

implernentation details of thc various prograrns have been Ieft out in order to iniprove clarity of the

presentation.

Chapter 9

Results

This chapter presents the result of performing euperirnents on seven capsules ancl cliscrisses the

performance of each of the miitacion operators proposcd in chaptcr 6. The chapter conclucfes trith a

tlisciission of the rnany observations made during the performance of the cxperinients. The niimher

of cxperiments performetl in this thesis is smalI. Thereforc the observations concerning rtiirc;icion

ciperator cffectivencss arc pre l imina . Frirthcr eqerirnents are requirecl to make firm any conclnsioris

c h v t i frorii this w r k .

9.1 Capsules Sumrnary

Eleven capsules were usetl in this thesis. CompIete mutation testing was perforrnecf on seven of them

whiIe the others were either rejected due to poor quality or aere only iised to investigate other aspects

of niutation such as niunber of niucants generaeed, Of the capsules fully testetl. one taken Gom

a moclel provitlecl t>y Rational. tn-O were from examples inclucled nith the toolset. tn-o were Frorn

personaf projects. ancf tmo were ffom graduate student projects. The foilowing sections describe

each capsule and the observations made whie performing mutation testing on them. Each capsuie

description includes a table summarizing botb capsule and mutation details. The statemachine

tliagrarns for the seven tested capsuies are inchded as hppencliv E. Table 9.1 shons the s t i m m a -

of the results for the k s t mperiment. The foilowing section esplains the table.

! Causule Details !I Mutation Details I i transitions states pseiitlestates levels of hierarctiy ports in signals out signals actions

Miscellaneous Yumber of Test Sequences iengths of test sequences longest test sequence

compile time (al1 mutants)

3 32.2

26 secs

14 minutes

Number of Mutants 6

coverage ITS/'CTI TRD TCP SCP TGT

TCh TTG SDL Testing Details mutants generated mutants compilecl cqiiimlent invalid 11 valid 9.5 tlcacl. firm 9 1 mutation score, firm 96% ~Ieatlstrong Y9 mutation score, strong 94%

Tablc 9.1: Experiment I: PBS CeIl Phonc

9.1.1 Explanation of Table Elements

Capsule Details

a Transitions: This is the total number OF transitions in the CSM. including internai transitions

and initial transitions.

a States: This number includes al1 states in the CSM including the TOP statc ahicti is the root

of the CSBI hierarchy.

Pseudo States: This is the total number of elements that are foruid in the RoseRT capsuie Éile

as elements of state lists. It incrudes states. initia1 points. jiuiction points. aricl choice points.

See section 7.2.4.

a Levels of hierarchy: this indicates the depth of the deepest containeci state machine. This

includes the TOP level root state object-

a Ports: This is the number of ports used by the CS31 and therefore the number of ports

through which the RT-TestHarnes communicates wïth the CSM. If the CSAI contains ciniers-

the number of timers is aiso indicated.

in sïpnls: This is the total number of in signols entering the CSJI. This is the nrimber of

tlifferent messages that can be sent by the RT-TestHarnes (ic does noc include timer messages)

as stimuli to the CSM. in signals are defined as p a n of a protocoL

out siqnafs: This is the total number of out signafs chat can be emitterl by the C'SU. This is

the niunber of messages thac can be received by the RT-TestHarness. mit signafs are definet1

as parc of a protocol.

Transition .ktions: this is the nimber of transitions chat have action objects associateci mith

theni. Action objeccs are C++ code segniencs.

0 Siinibcr of Slritants Protlucctl: This section lists each operator. or group of operators if they ' r~

sirriilcar. followetl I)y die riurnber of mutants generated bu the tnuration system.

Testing Details

LIutants generated: This is the total of the mutants listecl iu the criutation cletail section.

hIiitants compile& Tt& is the t o t d nurnber of mutants that successfully conipiletl. Sorne of

the ~riutants generated by the SITS-RT systern are syntacticaily inlaiid and therefore da not

compilc.

Equivalent: this is the nurnber of mutants thac were judged to be equimient to the original.

un-mutated. capsule. Equitdency was determineci througti the course of mutation tcsting

while trying to kiiI Iive mutants.

Intaiid: this is the tocd number of mutants that were either qntacticaliy inraiid (for the code

generator or the compiler). or chat upon inspection were not ali id RoseRT files. The latter

form of ind id i ty occurred if. for example. the SITS-RT system created a mutant by adding

a trigger to a transition that shouId not have a trigger. For example a continuation transition

is not triggered. it is a continuation of a transition that enters a state kom a superstate and

continues to a substate. Figure 9.1 shows such a case. Transition t l l is a continuation of

tr.uisition tl. It is not possible to add a trigger to t21 using the graphitai tooiset however it

cm be done bu the SUS-RT system (the TTG operator). Such a murant is not syntacticalIy

intalid. rather the trigger is just ignored by the mutant. Such a mutants are equitalent.

However since they should not be generated in the first place. they have been classifieri as

invalid rather than equivdent. -4 Future version of the TTG operator roulti look for and

esclude continuation transitions however this was not clone in this version.

I 1

Figurc 9.1: Continuation Transition.

0 Valicl: This is the total number of miid mutants.

killetf to obtain a 100% mutation aclcquncy score

mutants minus the i n d i d and equivaIent mutants.

i.e- the nurnber of mutants that musc bc

. It is eqtral to the number of generated

0 Dcatl. firm: This is the number of mlid rntitants that were eventually killecl by the test

seqirences. -1s riiscussed in chapter S some instrumentation n-as atlderl ta the CapC'T in

order to allow obsen-ation of the CapCT's current state and ariy uiic.spcctet1 messages tIirit it

rcceived. This estra information sas part of the output used to clerermine whether or n a ;i

mutant had been hlled by ri test seqtrence.

SIiitation score. firm: This is the firm mutation score which was arriveci üt by d iv id ig the

number of firrnly clead mutants (e-upressed as a percentage). by the number of d i d mutants.

Dead. strong: This is the nurnber of mlid mutants that nere eventiiaiiy killed by the test

sequences. The instrumentation information cotlected as part of the mutanté outprit n-as not

used in the determination of mutant currectness. O* the observable behat-iour of the CapLT

R-as used to compare the mutant's output to the original's output.

O Mutation score, strong: This is the strong mutation score which aas arrived at by tlividing the

number of strongly dead mutants. by the nimber of mlid mutants (exprcssed as a perccncage).

Miscellaneous

O Sumber of test sequences: This is thc number of test sequences iisetl during the niiitation

tcsting.

O Lenghs of test seqiiences: this is a comma-separated list of the lengths of each of the test

seqiiences iisetl. Lcngtii refers to the number of indigenous messages sent to the CapCT. It

does not iriclude iristriiriieritatiori messages (the only one iisetl in this thesis is the state recliiesr.

message) senc to the CapLT.

Longesc test sequence: This is the length of tirne in secontls it takes to execiite the longest

test sequence on the origilia1 (noti-mutateci) version of the CapCT. This nuniber is s ip i t icmt

n-hcn a capsitlc uses tiniers.

O Proccssor: Threc compiiters n-ere merl in this thcsis. a Pentirtm II '266hIHz. a Celcron 3661iHi.

;iricl n Pentiiim iiI Ï33itIHz. This colurnn inclicates which one ivas iisetl to test this capsule.

O Compile time (ail uiiitants): this is the total time reqirired to generate ,mtl cornpile (not nrn

'any test sequences) al1 of the mutants for the e.xperiment.

9.2 Experiment 1: Cell Phone Component

This niocle1 was siipplied by Rational in respome CO +mail comniunications, The P B S systeni is a

mode1 ttsecl cfuring adwnced rnodeling courses taught by Rational. However. the Ce11 phone capsrile

is a relatively simple capsule extracted lrom the model. Table 9.1 shows the sumrnary of the resiilts.

The first test sequence is ii transition tour that takes ïnto account looping in order to increment

a counter used in a choice point. The transition tour Mled S4 of the 89 kiiiable uiutants. W d e

constructing the second test sequence cn-r, issues were found. a reduntlant transition and some clead

code.

The dead code r i the result of an i f statement in a state exit action n-hose predicate alnays

e d u a t e d to false. The dead code fauit n'as found wtiiie constructing a test sequence to Hi an -ADS

(Action Delete state exit) mutant. The question poçed by the live A D S mutant ms -ïvc cIeleteci

this code segmenl but it doesn't seem to matter, why is that?" The answer was chat it tiidn't matter

because the code does noching.

The redundant transition m s a result of the dead code and was found while trying to kill one of

the TRD (TRansition Delete) rntitants, FVMe constructing a aew test sequence it nas not possibIe

CO kill one of the TRD mutants. This memt chat after deleting the transition rione of the test

sequences were able to tletect a change in the capsule's observable behaviour. On the other liant1

thr coverngp iiperator (C\*T) For this transition w u kiileti iriciicacing tliat the transition is cnkm in

i~tic of the test scquences. The Eact cliat chis transition lias an obsenxblc oiitpirt action malces the

t ~ o obsenztions appcar contriirlictorp. Hrirvever. i~ was iound t t i a~ the transition $vas rctlundarit.

The capsule has two transitions that respond to the s m e event. One is in the top state whilr! the

other is in one of the substatm. When the capauIe is in the sub-statc the inner cransitiori responds

to the event and transitions to a target state %?thin the substiite. When the ctirrent state is the

top state the top statc's trruisition responds to the event. Xlthough che two criuisitious origiriiite in

clifferent Ievels of the statc machine tiierarchy they boch have the same target statc and the snrrie

outprit action. Thereforc by curitroIling wtiidi state ttie niachine is in the tester çan musc either

one of tiic transitions to be taken. But with the inner transition deletecl only the auter transition

can be taken. Howwer since both transitions have the same output action ancl carget state the

absenable behaviow is unchanged. if the dead code in the state's exit action not been tleacl but

rather had an obsenab1e effect on the CShI's behaviour then the rectundant transition woltl no

Ionger be redunclant.

Since hoch of these issues dicl not affect the obsenable behatiour of the capsule, i.e. they (tic1 not

restrIt in incorrect behaviour. tlien by sonie definitions they n i - not be classilied as fadts. However

bot& tfead code and ceclundant code (transitions) can Iead to difficuities in maintenance.

This mode[ had an interna1 timer that used IO second tirneouts. in order to kill sorne of the

mutants it would have been necessary to traverse thii transition 10 times, each tirne requiring a

del- of 10 seconds to don- the che r to expire, Due to the number of mutants it was decided to

change the constant cuntroiiing the number of iterations h m 10 to 2.

9.3 Experiment 2: Ce11 from a Game of Life Mode1

Tiiis niode1 sas iriclucled as an example nith the toolset. This is a single level state machine

sirniilating the life cycle of a single ce11 creatiire in a g m e of life simulation. This state machine hac!

two internai self-transitions. Triggering these transitions either increments or decrements a mpsiile

variable but does riot chcange the -gaphical- scate of the capstile. Whcn one of the ocher cransicions

is taken. the capsule changes state based on the value of the affectetl variable. Killing mutants for

this capsuIe requirecl interleming ~xeciition of these internai transitions with the %are çlianging-

trirn~itions. The tao test seqwrices iisetl tvere bascd on triuisition tours. A single transition tour was

riot possible. Table 9 2 shows the siimrnary of the resiilts. So issiies were foiincl with this capsiiIe.

transitions states pseudo-states levels of hierarchy ports in signds oiir. signals actions

MisceUaneous Yumber of Test Seqiiences Lengths of test seqiiences longcst test sequence

compile cime (al1 mutants)

processor

37 minutes

Pentium U '266 MHz

Mutation Details Number of Mutants ADS/E/T coverage tTS/STI TRD TCP SCP TGT TAT TCX TTG SDL Testine Details mutants generatecl mutants corripilecl cqui\alent intxlirl valid dead. firm mutation score, k m deac1.strong mutation score, strong

Table 9.2: Experiment '2: CeU from Chme of Life

9.4 Experiment 3: Traffic Light Controller

This mode1 was included as an e m p l e with the toolset. This ras the first capsule testeci in rhicii

di top IeveI states have sub-states. This capsule also has a terminal state that is part of the design.

so it n-as not possible or necessq to obtain 100% state exit coverage (CSX). One rather large

semantic faiilt mas found. While atlding a test case to kill a CVT mutant it n-as found that the

timer tnggering the uncovered transition wiis never set. This hult was repaired nhich requirecl the

mutacian cesting procedure to be restarted. Table 9.3 shows a summary of the rcsiilts.

Capsule Details transitions States pseud~scaces leveIs of hierarchy ports in signais out sipals actions

Miscellaneous Yamber of Test Sequences lengths of test sequences longest test sequence

compile time (al1 mutants)

processor

23 10 52 4

2 - 1 timer -1 - n 9

a - - , .a,12.7?7 23 secs

4-1 minutes

Pentium iI 266 5IIh

Mutat ion Details Number of Mutants XDX/E/T cowrage ITS/STI TRD TCP SCP TGT TAT TC.4 TTG SDL Testing Details mutants generatctl mutants compileci

cleacl. firni mutation score, firm cieacl.strong mutation score. s t rong

Table 9.3: Experiment 3: Traffic Light Controller

Because this capsule iisecl multiplc interna1 timer events test sequences took a relatively large

tirne to esecrite nith the Iongest being 23 seconds. This means that ninning a single test sequence

against the original 343 mutants took over 3 hours. The total time required to compile al1 of the

mutants tn-ice (the second time aas after the fault fk) cmuitl run a11 .3 test cases against the 343

mutants n-as in excess of 14 hours.

9.5 Experiment 4: Cruise Control

This capsule was the main controuer for a cruise control rnodel. [t n-as developed severd months

ago by the author of this thesis. This capsule has a periodic event that is triggered once per second.

This event sends a message requesting the vehicles m e n t speed. .Uthough this coiild be viewetl its

just another input-output sequence, because of its periodic nature it decided to motfi&- the RT-

TestHarnes The RT-TestBamess dnwr capsule used in this experiment n-as modified bu addirig an

incernai sclf-transition that responds to the data request from the CapIiT by sending the speecl back

to the CapCT. This 6ees the tester from having to respond to each request individudly. in orcler

to alIow the tester to update the speed value returnetl by the RT-TestRarness a new message was

added to the RT-TestCaseManager. Previous to chis d l of the messages (signds) seIectabIe through

RT-TestCaseManager corresponded to the signals that the CapLT coiilcI receix-e on its ports. The

new message. however. cioes not correspond to signais sent to the CapCT. it just npdates the speecl

ixrinble.

Ttic procedure follon-ed iir testing this capsule was to generate. compile and rtrn n strbset of the

mutants, Ench subsct consisced of al1 of the mutants prociuced by cither a single or n relatecl qoirp of

operators. There were three relatecl groups sets: the action clelete operator set ( A D S . ADE. .\DT).

th^! coverage operator set (CVT. CSE. CSS), and the internal-esternalseIf transition set (ITS. STI).

The other operators werc arltled singly. The first set risetl wiis the coverage set. Aftcr ohcaining

100% rniitation caverage the ADS/E/T mirtnnts were added. hl1 of these mutants were kiled bj-

th^ ~sisting test sequences. Sext the TRD mutants were atlcled. Again al1 of these n-cre killetl by

the esisting test sequerices. Sext the TAT mutants n-ere atldetl followed by the remaining mutants.

In eacli case al1 of the nen- mutants were killecl by the esisting test seqtiences. In surnmary .Ifter

ohcaining 100% miitation covernge no new test cxes were requircd to kill any nf thc siihç~qtrentIy

generated mutants. Table 9.4 shows the summary of the resiilts. Xo issues aerc idcntifierl witti ~ h i s

c;rpsiile.

9.6 Experiment 5: Radar Control Unit (RCU)

This capsule w u estracted [rom an electronic warfare simulation developed by gaciiiate stuclerits

at RSIC. The same procedure used in rsperixnent 4 was followed with this experiment. Kitli L0[1%

mutation coverage (n-hich required 2 test sequences) the -DS/E/T mutants were nrided. -4nalyzhg

tiie live ADT mutants lead to the ciiscovery of tn-O dead code segments. Adding the TRD and TXT

mutants did not prodiice any live mutants. hdding the TC.1 mutants resuited in a single Iir-e mutant

which %-as kiIled by the addition of a thiid test sequence. Adding the TTG and other remaining

mutants did not require the addition of hirther test sequences. Table 9.5 shows the sumrnary of the

resdts.

scates pseudo-states levels of hierarchy ports in signds cirit signalç transition actions

Miscellaneous Srirnbcr of Test Sec~uences letigths of test sequences longest test sequencc

cornpite rime (al1 mutants) 33 minutes

11 Mutation DetaiIs 1 Il Nurnber of Mutants I ! -

hDS/E/T coverage ITS/STI TRD TCP XCP TGT T - T TCA TTG SDL Testing Details - mntants generated 1 .LU0 niirtants compiler1 364 cquidenc intxlitl 33 ! valid j 965 rIeatl. firm SG5 mutation score. finn 1 ~ 0 % r1earl.s~rong 363 mutation score! strong / LOO%

Table 9.4: Experiment 4: Cruise ContmI

9.7 Experiment 6: Pop Machine

This motle1 was developed and tested by the author as part of this chais. Thii capsule tias 3 chice

points nrid a single tirrier. This çapsuk was incIuded in order to inwstigace the ability of a capstile's

tlesiger to test the capsule. As mentionecl in chapter 2. sorne cesting literattire suggests tbat an

arithor of a program crinnot cffectiveIy test his o m work because testing to rereal firuits is basically

a destructive process. It is feIt chat the author of a program is biased torvard riot destroying bis

onn creation and nîil therefore not be as effective at breaking it. intuitive. hon.ercr. mutation

testing stiouId be inunune to auy such bias because the niincisec of =any tester (inclutlmg the capsuleS

author) is to kill mutants. Table 9.6 shows the s u m m q OC the reçults.

8 faults n-ere revealed through mutation testing of this capsule. The CbT. TBD. ml TXT

operators each ret-eded a single fault. The ADT operator rewded tn-o. and the TCA operator

revealed three- EWng mutants resulted in the creation of test sequences that esercisetI non-intuitive

control paths. For esampIe when adding a test seguence to Hi a TC.4 mutant it T;LS riiscovered that

it rvas posçibIe to bu- a second pop for haIf price: X single cm of pop costs one douar. It was found

that by imerting 6 quarters into the machine foverpaying) and then p u s h g the pop select button

Capsule Details ' transitions States pseudo-states levels of hierarchy ports in signals out signals transition actions

Misceiianeous Yumber of Test Sequences lenghs of test sequences longest test seqiience

cotiipile cime (;dl itiiitants)

processor

13.6,3 3 secs

3-1 minutes

. - .

Number of Mutants ADS/E/T coverage ITS/STI TRD TCP SCP TGT T-IT TCA TTG SDL -- -

Testing Detriils mutants generatecl mutants compiIec1

intalid d i c i cleaci. firm mutation score. firm cleadstrong mutation score. strona

Tablc 95: Esperirnent 5: Radar Control Cnit

the ciistorner ~votilcl receivc a can of pop and two quarters in change. Then the customcr only had

to insert two more quarters in order to receim a second can of pop. This type of faiilt is typicaI of

niistrikes niade by software clevelopers n-ho assume chat their software wilI be risecl in cxpected ways

and it is typical of the types of fauhs that a code's developer ail1 have trouble in fincIing plyers

Ï9][.Jaiote 971. dlthough this is onIy a single example. the Fact that such an interesting fauit n-as

founcl (the half pnce faiilt) indicates that mutation testing may mitigate the psychological argiiment

chat an aiithor cannot effectively test his mn software. This is an interesting subject for fiiture

research.

9.8 Experiment 7: Electronic Warfare Jarnmer

This capsule was e-nracted from an electronic d a r e simulation developed by graduace students

at RCIC.

The same procedure used in experiment 4 n-as foUowd with this eqmiment. Xith 100% mu-

tation coverage (which required 3 test sequences) the JLDS/E/T mutants were added. X fourth

,, t a t i o n Details I

I l Number of Mutants 1 4 states pseudo-states hreIs of Liierarchy ports iri i;ignnls oiit signnls transition actions

Misceilaneous Sumber of Test Sequences lengths of test sequences longest Lest sequence

cortipile tinie (al1 niiitants)

Table 9.6: Experiment 6: Pop Machine

15 43 6

27 6 3 1

104 310 LI11 14

6x2 574 25

1-4 ADS/E/T $8 coverage 4 ITX/STI

3 4- 1 timer TRD w

I TCP S XCP 13 TGT

T-AT 1 TCX

processor

test scquence n;ns aclded to HI the 8~ ADS/E/T. mimnts. .kithg the remaining rnutanls cliti not

reqiiire the aridition of Iiirther test sequences. Table 9.7 shoxs the surnmary of the restdts. So

issiies werc fottnd wich chis capsule.

inralirt valid

9 25.26.2.15.10.3.15.2,4

3 secs

36 minutes

9.9 Other Capsules Examined

TTC SDL Testing Details mutarits generateri ~iiiitants compiIed erluisalent

Pericium LI1 ï33h[Hz

9.9.1 Radar module

This c a p d e riras taken fiom a mode1 deveLoped by some gaduate students. Due to the poor qudity

of the code tliiç experiment n-aç not completed. This capstde indicates that mutation testing shouid

not be used on s o f m e that is not at some minimal Ievel olquality. Each issue Eound requires çome

corrective action be made ta the code. foliowed by regeueration and rerunning of aii of the mutants.

This is estrernety expensive in terms of computationd and human resources. Table 9.8 shons a

summ- of the initial results.

deatl. firrn mutation score, 6rm

.540 99.63'2

rleatl.strong 532 mutation score, strang / 9%

Capsule Detaiis tr.msitioiis stntes pseuclo-states levcls of hiernrchy ports in signais out signals transition actions

~Miscellaneous Xumber of Test Seqiiences lengths of test sequences longest test sequence

corripile tirrie (al1 mutants)

Mutation Details , I 1

-

- Y

Y

13 6 39 4

3 -i 5 cimers 6

Number of Mutants

coverage 23

Table 9 . 2 Eqcrinienc 7: E\VS .Jamtner

,

ITS/STT TRD TCP SCP TGT T.kT TCA TTG SDL Testing Details I

9.10 Individual Operat or Discussion

- J

4

4 6.9.15.9 -

I secs

65 minutes

Celcruu 3GGUHz

-L j 13 O O .>

26 4s

-5 I

mutants generated mutants compilecl eqii ident invalici

Ttie tiest sectioti clisctisses obscnations made concerning intiiviciiial operators.

254 230

O 26

9.10.1 Coverage Operators, CVT, CSE, CSX

mlirl WQ

clcatl. finri E S mutation score, firm LOU% cIcntl.strorig

l -- .,sg --

mutation score, strong LUO'X

Thcse opcracors are designed to show chat dl transitions are necessary. to show chat al1 states are

reachable. and to iclentify an!- terminal states in a capsule. Creating test cases to Hl the coverage

operators is relatively e a s . Test sequences that brirr ail O€ the coverage operators also kill man>- of

the other mutants. in some cases .di of them. In the capsuies tested 2 faults nere revealerl by the

CVT operator.

The importance of statement coverage has been discussed in chapter 2. Xthough 100% stacemenc

coverage is not often targeted in industry. it is deemed important in studies of acfec~uacy criteria

and for cr i~ica l"~ softmre for the reason that in order to have confidence in critical software it is

'fiCritical softwue is usually IabeIed as safety, mission, or Iife critical. In generai it refers ro any softrmrc in which the consequences of failure are high. i.e. I o s of life, hedth. money, resources.

pseudo-states levels of hierarchy ports in signals out signais transition actions

Capsule Details

co\'erage rrs/sn TRD TCP SCP TGT T.T

1

lcngths of test seqiiences longest test seqiirnce

Muta t ion Deta i l s I

Miscellaneous Yumber of Test Sequences

compile tinie (al1 niiitants)

Number of Mutants 1 -U~X/E/T 1 9

transitions

proccssor

9

1 [ -4

i secs

States I 4

T C I TTC

34 minutes

Celcron 36GXIHz

Table 9.8: Espcriment fi: Raclar

SDL Test ing Details mutants gencrarecl

necessary to know, tiiirt as i l rriinir~itirri, every fiiriction of the softxare has beeti escrc:istid nt l e s t

once. The rtiiwon for not obtainirig 100% statement covcrag in industriai code baseci testing is ttiat

the importance of iincowrecl staternents rnay not bc mortti the additional cost of covering thern. For

example (deacl) ctebiig code may not be rvorth testing. Often a piece of software wiII have sonie code

that is intencletl only for use cIunng deveIopment. Such instrumentation code may be -deactiva?itecl"

in the procluction version thorough a compile time snitch (Le. #ifclef DEBCG statenient in C).

Aclcling test cases to obtain coverage for such code segments is probably not worth the cost.

4

1-11

In ternis of CSJIs. however 100% coverage appears to be more important. The preliminary

residts of the esperirnents described in this thesis suggest that test sequences that kill ail coverage

operators also kill the majority of the other mutants. By concentrating on killing coverage mutants

first. the number of subsequent Live mutants is greatly reduced.

C'sing mutation to achieve coverage can be inefficienc becaiue each CSBl rcsuits in 2S'T niutants.

=here S represents the number of states and T the number of transitions in the C S X One mutant

is protluced for each transition and two mutants are produced for each state (state e n t - state esït)

in the CSM. Altematively a singIe instrumented version of the CSM couId be produced in n-hich

each transition and state entry-exit has code segment h a Instrumentation code added that recorcls

the fact that the code segment has been euecuted. However this has implications For thnimg. The

mutants compiled 1 124 equi\aIent 1 ..' inidici valirl deacl, ürm mutation score, firm

17 124 74 60%

instrumentation adds a run time cost to every transition cmcl state entry-exit code segment. This

riin time cosc cloes not occiir with mutation operators: the capsule execuces norrnnlIy (Le. there is

rio instrumentation penalty) until the mutated transition or state is execiiteri at which cime cnpsuIe

esecmion stops. In order to avoid the tirne lags produced by instnimentation it m- be n-orth the

cost of creating 2S*T mutants.

9.10.2 Action Delete Operators, ADT, ADE, ADX

Tlic action clelete uperators arc clesigneci to show chat al1 code segments in the CS)[ have an effect

on the behaviour of the capsule. If after deleting an action from a CSM the potentid1i2' ohsenable

behavioiir of the capsule does not change then the action is unnecessary or is deacl code.

In the capsules testecl in this thesis 5 issues were fountl bu these operators. Aniotig these issues

= test cases to were dead code segments and preclicates that never etailiate to mie. While creatiti,

kill Iiw action clelete rnutants. intcresting and non-intuitive paths through the state machine are

tlcvelopcct. The prelirninq resiilts observeci in this thesis indicate that the action rielete operators

;irp hcttrr at revealing subtle (srnall semantic size) friults than any of the other operators used in

tliis thesis. The valiie of these operators is also enhanced becanse they gncrate very few mmutants.

one for each action in the capsule. The iiurnber of actions in a CS&[ is at niost 2S+T (eacti state

can have tn-O actions, one for encry, one for exit. while each transition c m !lave a single action).

however the total number is usually much less.

9.10.3 TRD: Transition DeIete

At tirst glance the TRD operator appears to have the s m e cffect as the C t T operator in thnt a

test secluence that kiki a CVT mutant siiould kill the corresponding TRD mutant. However th-

are not the same as was shown in the first experïrnent.

Assuming that 100% CVT coverage has been obtained. a live TRD mutant ah-s indicates

an issue with the CSM. The C\T coverage indicates that the transition has been cakeri. but the

Iiw TRD mutant indicates that rernovïng the transition does not affect the capsde's obsenable

behavioiir. This is a contradiction. On the other hand issues reveded by the TRD operator are not

necessarii- fadts. As shown in e-qeriment 1 a Live TRD mutant may indicate redundant code.

"It may be necessq- to observe the CapbT for sorne time after the execution of the rnutated elernent in oder for the effect of mutation to become observable-

Although TRD selclom indicates something not s h o w by CVT it can be useful. in chi thesis

two issues nere founcl bu the TRD operator.

9.10.4 SDL: State DeLete

t\lniost al1 of the mutaiits produced by th operator either clici noc compile or ctirew a riin tirne

exception. Dcleting a statc with a contained scate machine dwq-s prorluced a run time crror from

ttic RoseRT code gerierator engirie. Therefore only IeaF states could be rlcletcd. Howevcr rniitants

gcricratetl by tliis operator w r e eil~iily kiUe11 and tlid not reveal any faiilts iirriong the six çiipstlles

tisecl in chis thesis. tt appears chat deleting x Stace is a large seniantic fault. An iniprovetl version

of tliis operator coulti be wricteti in orrier EO reduce the nt~rnber of conipilatiori tmcirs atirl riin titric

exceptions that occur. Such an improveti operator would onIy clelete leaf states. where i i leaf st;itcb

is a state that does not Iiave a contained statemachine.

9.10.5 Internai Transition History Operators ITX (Internai To extemai),

XTI (externa1 To Internai)

These operators do not appear to be very usefii1. They do not generate many mutants and the unes

generatecf are often equivalent to thc original. Eqi~i~alent mutants are protlucetl ahen the states

tvithin the scope of the internal transition being mmated do noc Iinve entcy or exit actions. in siich

cases there is no behat-ioural clifference becween an internal and external transition. S o issues tvere

rcvealccl by chese operators during this thesis.

9.10.6 TAT: Transition Alter Target

This operator rnodels the transfer errors rnentioned in Choa's error classes [Chow 781. It apperirs

that most T-IT mutants are easily Wled by test sequences that kilI aU of the coverage mutants. On

average the T-IT operator generates more mutants than aU of the coverage operators combined but

l e s than the number o l mutants generated by TCX or TTG. During the eqecïments this operator

revealed a single issue. W d e t u h g to kill a TCX mutant the question posed by the mutant n;as

-1t doesn't matter nhether this transition goes to state .A or to this choice poiut. a - is that?

The answer was that by going to the choice point instead of state h the condition e d u a t e d by the

choice point always evaiuated to F-USE. However the FALSE transition leaving the choice point

also nent to state A. Therefore the behaviour of the mutant waç equivalent CO chat of the original

capside: the result of taking either the rnutated or original transition was to arrive a t state A.

9.10.7 XCP: exchange Choice Point

This operator generates few mutants. one mutant for each choice point in a capsule. It has the effect

of reversing the boolean espression controlling the choice point. No issues nere itlentified by this

operator cluring this thesis. However. since the nimber of choice points in the collection of testeci

capsi~les was sniall. a nieaningful analysis of this operator coiilci not be niacle.

9.10.8 TCP: Transition Choice Point

This operator generates two mutants for every choice point in a capsule. It ensiires that t d i brandi

(if ii rhoice point is trriversed. It appears chat TCP mutants are easily killeti anri therefore represent

Inrgc sernaitic faults. They ttierefore do not appear to be very goocl a t rereaiing faults in n capsule.

Hoaever not enough choice points were obsewed in the capsules of this thesis to permit a nieaningful

anaiysis.

9.10.9 TGT: Transition Guard True

Tliis operator niutates the expression bgmditig a transition by Iiaving it alw-s return tnic. Gttnrdetl

transitions seem tu be rare. only one &-as found in al1 of tlie capstllcs examiner1 (luring this thesis.

S o analysis of this operator can be offered on the basis of this thesis.

9.10.10 TCA: Transit ion Change Action

This operator generates on the orcler of T*A mutants per capsule. where il is the riumber of transition

action objects present in a CSM. This is usrially niore than an>- of the other operators used in this

thesis. However rnost of the TCX mutants are triviallq- killed by test cases that kiii the other

mumnts. However the remaining ones can reveai Fauits. in this thesis three issues were reveaIed by

TCX mutants. X good scrategy appears to be to kiii the TCA mutants aFter di of other mutants

have been kiiied. The idea being that the test sequences used to kilI the other mutants niil also Hi

niost of the TCX mutants. Then creating test sequences to kiii the remaining TC.1 mutants may

reveal faults.

TC.-\ also generates many syntacticdiy ilkgal mutants. When the action object for a parameter-

izecl transition uses the parameter then the parameter type of the message must match the parameter

type of the action object. Khen a mutation is performecl in which an action object e-xpecting a pa-

rameter ol type T is ,~ssocintetl with a transition triggered by a message with a clifferent parameter

type the resiilt is syntas error. A transition's parcameter is always accesseci by casting a voict pointer

namcti rtdata to the rlesirecl type. Since the identifier rtdatn is aiways present in the code segment

of an action object that accesses ii parameter it is possible to automate tlie elimination of this type

of syntacticaliy intalid mutants at the mutant generation stage. Hoaever this is Ieft for f~itiire work.

TCX also tends to gcnerate a fen- equivaIent mutants. When tao or more action objects arc the

sanie. which is often thc case if the action is a simple message send. ttien TC:! sri11 generate a pair of

tvpivalerit niiitants for racti pair of eqirident action objects. Hoaever some automatecl eqiiiralency

checking is possible. TSL [Cordyt 001. for example. coiilcl Lie tisecl to compare action objects bcfore

the mutants are generatetl (by parsing the action object and performing non-terminal comparisons).

9.10.11 TTG: Transition TriGger exchanged

Scst to TCA die TTC operator generates the most niritants per CSAi. in a aue wliere a CSM uses

aiany clifferent triggers the ritmber of TTG mutants can beconie dominant. However despite the

large nimiber of TTG mutants. niost are trivially killed by test cases createti to kill other mutants.

-4s with the TC.-\ operator a good strategy is to not use the TTC operators untiI most of the other

operator\ mutants have been killed. The TTG operator did not reveal any issues among the capsules

usecl in this thesis.

Originaily it n;1s decided that TCA would be applied to ail transitions. even those tiiat dicl not

have a tcrigger in the original. The rationale n-as that triggeriess transitions n-ere probably the kinds

of faults a e were Iooking for. This iJ an inefficient po t c - There are many transition segments that

correctIy clo not have triggers. These are continuation se,pents and se,gnents that Ieave choice

points and go to States. XppIjing the TTG operator to a continuation segment d l result in one

syntacticaiiy legai but invalid mutant for each signal that can be appiied to the capside. ThÏs

can represent a large nurnber of inmiid mutants. Tt appears that untriggered trans-itions are rare

in practice. represent easily found faults, and muld be found by the CVT operator in an>- case.

Therefore any future work should not a p p l the TTG operator to transitions that rio not aIready

have a crigger.

9.11 Operator Summary

Typicdy rvhen perforniing mutation testing, killetl mutants are removed from hr ther testing as soon

as they are killetl. However as this thesis was investigating the effectiveness of mutant operators. it

~r-as neccssary to run al1 tnutarits against ail test seqiiences even after ri. test secluence liacl killed a

mutant. -1 mutCant that is distinguishecl by mnny tests is easily killable ancl probnbly rqresents ii

large serriantic fault. X mutant killcci by a lotv tiuniber of tests is probably r i stuall setnaritic fault.

tvhile orle ttiat is (leclarerl eqriivrilent is a size zero semantic faiilt. The intention here wns rn see

dietlier a particiilar operator gencrated rnucarits ttiat tvere on average eitiier tu kill tlian tiiose of

;inother opcrator.

hl1 of the operators produced mutants that were killeri by al1 of the test secluences (inrlicating

large semantic faiilts), mut~ants chat were killed by a single test sequcnce (semantically smaLler

faults). .mil mutants that were killetl by multiple test seqiiencw. Table 9.9 shows tht. riiinilic~r of

test scqiiences (expressetl as a percentage of al1 of the test sequences run against a given mutant)

reciiiiretl to kill the givcn mutants. For esample the table s h o w chat for the pop machine rapsrilc.

tlic 45 (-overage rniitants were killed by an average of 70.3% of the 8 test sequences. The top rnn-

.ihow the nnmc of cach capsule tested in this thesis. In the same ceIl but undermath the namc is

the tiuniber of test sequerices that were run against the mutants. The first column shows the nanie

of the mutant operator(s) and the number of mutants protluced by that operator(s) for pach i,apsule

testerf. Sot ai1 of the operators proposed and used in this thesis are inclurlecl in this table. Oiily

the operators that produced a large number of mutants are includecl. Yotice that the action rleIete

(XDT. ADE. XDS) and coverage (CVT. CSE. CSS) operators are grouped.

On average mutants were killed bu 68% of the test sequences. The game of Iife ceil capsule was

relatively simple which accounts for the relative case with n-hich its mutants were kilIed. There are

a ïea results that are far from the 68% average. these are marked in bold. The average column

summarizes the data for each mutant operator. This coiumn allovs the operators to be ranked

accordiig to the number of test sequences required to kilI a mutant. lccording to the table the

TCX operators produce mutants that are the easiest to kill. R-hile the coverage operators produce

mutants that are the hardest to Hi (in terms of the number of test sequences required). However

the tlifference between the average for any particular operator and the total average of 68% is not

significant. Further. dile to the small sample size. no statisticdly significant conclusion cnn be made.

Thercfore baseci on these results it is not possible to say that one operator produces mutants that

are on average easier to kill chan the miiteants producecl bu any other operator.

Table 9.9: Percentage of test sequences required to kill mutants

Capsule

So.Tests ADT/E/'i Mutants Coverage .CIutnnts

1 T.T Mutants

' lfiitants TTC Mutants

Hoivever these preI iminq results indicate that some operators ni- be more effective a t reveding

faiilts than others. The operators appearing to be the rriost effective a t reveding faults arc tbc

çoverage rint1 action delete operators. Moreover these operators gencrace reIatively small niimbers

of mutants. The TCX operator. althotigh generating many mutants. also appears to bc goocl at

revealing faults.

P B S

3 66.7 6 --

r .S 1s 76.0 26

Traffic Light a 73.3 12 70.5 40 54.8 U 62.6

9.12 Procedure

TRD lii~tarits , 1::*3 1 4 4 TCA 1 75.0 1 60.2

27 7-1.0 16 103

Tt is difficult to say that in generaI one operator is better than another a t revealing faults. Ho%-ever

the coverage operators generate a srna11 number of mutants (linear in S and T) and are relatively

easy to kill; just ûnd a test scquence that traverses the state or transition. Test sequences usetl to

kil1 cwerage mutants tend to kiii large numbers of mutants generated by other operators such as

TCX. TAT, and TTG. These non-linear operatos can generate hundreds of mutants each for even

relatively srnail state machines. most of which are noise. By using the coverage test sequences to

Cniise

4 60.0 13 34.4 34

total average total mutants

161 70.0 36 337

Ce11

9

78.9 7 4 . 1 17

18 47.9

68.1

27 60.0

1944 i

62.0

RCC

3 72.2 S 83.3 16

11 85.0

136 1 70

61.4 104 67.0

13 32.2

S 61.0 2s 52.0 40 116

66.8 131 380

95.0

Pop

8 42.3 1s 70.3 45

60.3 1 I

S5.S 35 163

66.7 71.2 26 62.2

48 1 LU 1 16

310 70.3 101 602

69.6 I

68.9

.Iammer

4 48.4 12 1 . 7 23

62.5

48 54.1 121 243

Average

63.1

91.0

Total 5Imants

66.6

71.8

1 73 74.6 / 183

780

483

ki11 the majority of these noisy mutants, the tester c m focus on hilling the remaining non-1ine.v

mucants which seem to mode1 semantic&- smdl faults. This reduces the burden on the tester-

In the first 3 experiments a.ü O€ the mutants werc gnerated and nui agaiast ail of the test

secltiences bcfore any analvsis ivas periormed. During analpis live mucants were examinecl in ari

arbitra- order. new test çecpences were aclded. and ~ h e live mutants were rim against al1 of the

ncldeci test sequences. Adding a new test sequence CO Hl a particuIar mutant often haç the side

effect of killing other live muttmts. For this reason tn-ing to Ml di the rernaining nititants in ri single

iterntion n-as ncver atternptcd. ïnsccatl iisudy only a single nen- test sequcnce ainiccl at killing 2 or

3 live mutants rn added during cach iteration.

The proccss of riinning a11 of the mutants each tirne is inefficient. If a fniilt is foitncl r.hcn the

cnpsiile mtist he repurctl, the mutants regenerated. and the tests r e m . \$%en a fatilt is foiinrl earIy

in the analysis the hitlk of the results are not iised. W t h small capsides the nuniber uf mutants

genera~ctl was on the order of 100. In these cases thc inefficiency of this methocl is tolerable.

However when the number of mutants is Iarger a more efficient process is neecietl. Starting ai th the

4th experirnent the follo~\lng process was folloued:

L. Cienerate the roverage mutants (CVT. CSE- CSX) and add test sequences. (itcratively) imtil

LOO% coverage is obt'zined.

2. Generateaction mutants (XDT. ADE. XDS) and add test sequences to El1 al1 the nea mutants.

3. Generate the TRD mutants and add test sequences ta H l ai1 of the neiv mutants. Since TRD

is almmt the same as CVT the number of live mutcuits produced bu this operator is mai1

(tisualIy zero).

4. Generate the rernaining linear operator mutants (Sn. ITS. SCP.TCP. SDL. TGT) and acld

test sequences CO kir1 ail the new mutants.

.5. Generate che TAT mutants and add test sequences to kiii a11 ne%- mutants. TRD produces

manj- mutants, on che order of T*S.

6. Generate the TCX mutants and add test sequences to d nev mutants. This operator

produces man>- mutants on the order of T f A . where A is the number of action objects in the

capsde. In the fen- capsdes studied in this thesis TCh encourageci the creation of subtle but

legal sequences.

7. Generate the TTG mutants and add test sequences to Ml al1 new mut~uts . TTG protluces

many miitnnts. on ~ h e ortler of T*mumber 01 in signnls.

3Lny tirne a faiilt Li found the procetlure is stoppetl. the capsule repaired. and the mutation

proccctiire starter1 over. Ki th this procediire the noisy rniitants producecl hy siibseqiient operators

;ire likcly tri bc killed by the existing Lest sequenccs. Therefore racher chan facing a large nuniber of

mostly noisy live mutants. the tester only deals with a smallcr number of progressively hirrcler to kill

mut.urts nt e c h iteration. Further the total number of mutants tbac musc be çorripiled cliiritig t h t h

cntirc testing phase may be recliicerl, Ench tirne a Fairlt is foiintl ancl correctcri nrily the tniitants that

Iirn-t! n1re;irly bceri tested rieetl to be regeneratetl ancl cecompilecl. By killing cbe coverage niiitarirs

Hrst. che mniher of other. l e s intuitive. rniitants to be killec! is recliiced. Thiis both the effort of the

lester itiid t.he cunipiitatiorial biirden arc rediiced. It is intcresting thnt a siniilar strategy thas not

been proposetl elsewhere in the mutation literature.

9.13 Initial Test Sequences

Since ;il1 niiirants ;ire live prior to the running of the first test sequerice, initial test setpiences are

not tieçtissarity crentetl for the express piirpose of killing porticular livc mutants. Rathcr they may

reprt'sent espcctetl uses of the capsule. transition toius. or possibly are randoni. They riiay be

basecl on sperifications. Hontever most of the capsules testetl in this thesis ntere not accornpnnierl

by specifications. The initial test sequences used in this thesis were baser1 on trarisitiori totiI+".

~ i t h o u t regard for whether the resulting test sequence representecl a reasonable itse of the capside.

Trnnsition tours are usually easy CO create liowever complications a i s e tvlieri transitions have girards.

dien choice points are present. and if internai events (timers. intermpts) are possible. After rtuining

the mutants against the initiai test sequences other "mutant kiUingFtest sequences are addeti to !dl

an'. remaining l i ~ e mutants.

Test pIanning issues were not a significant factor in t h k thesis due to the smalI size of the capsules

tested. Bowever in larger projects test planning rnq- be important. Good test plCanning requires

some structure amongst the tests. Sets of unrelated test sequences generally are poor choices. since

there rvilI be a lot of diiorganized repetition in the sequences. Good test planning recognizes this

and es structure to the test sequences [Shepard OL1. The transition tours usetl to kill coverage

"A single transition tour is not dways possible. In this case a set of partiai transition tours is needed.

98

mutants are a gooti bais for test planning: their segments can becorne preambles for other test

scquences that need to be at some -pseiido-initial' state.

Tu ki11 a- nliltant requireç that the mutant be reached by the test sequence. Conçider the A D S

(Action Delete state exit) mutant. To observe a Jiaerence UL behaviour it is nccessary chat, a a

minimuni. the state chat has had ics exit action tleleteci hy this operator be exiterl. This reqiiires a

test seqirence whoçe prefix pirts the CSM in the mutated scate. Then the tester ndds more signais to

the -pre.arnble" test sequence in order to produce an absenable clifference in the mirtant's hehavioiir.

ctius killitig the ADS rnutant. Segnients of transition cour resr seqiiciir:es cari lit! I I S E ~ i f i clie prwrr~hle

tu piit the CSSI in the "psendo-initial" state.

9.14 Killing mutants

Crcating test secluences to MI niutants rva the rnost labour intensive part of perfornii~i: ~tiritation

testing during this thesis. [t reqrrims matching the lire mutant name to the stnte clingram eierncnt

ttiat h a been mutated and cletermining what the mitcation il;. For linpar operators the nnnie i iF thc

rriiitant tells what the mutation is. For non-linear nperators siich ns T.T the operator's log file musc

t)e rromiulteci. Log files correlate the rliucant nuniber with the details of the mutation. For esample

the T..U log file wili show the origind and aew t a rg t state [or a transition. After cletermining vhat

the mutation is the cester miist constmct a test sequence that will kill the miitanE. For operators

stich as the coverage operatos this is relatit-eIy E . However finduig test secluences EO HI niutants

produced bu operator such as TCA and TTG rcquires a more creative effort.

Better tools aoiild eliniinate che tediousness of tuatchi~g a live mutant to the iicttral stiitc rliugram

elcnients thrit have heen mutated. Hovever the general probIeni of test sequence generncion is

irndecideabte and must be done rnanualIy. Rhile in the process of creating a mutant kiIler one

nnswers -hard questions' such as T v e deleted thîs action object. but it doesn't rnatter. why is

that:'" Xnswerïng this question leads to a new test sequence that should Ul che rnucant or reveai

an issue wïch ~ h e originai capsde.

hocher decision to be made is whether to create a single minimai test sequence to WI each

mutant. or whether it is better CO creace larger test sequences designed to kiU severd live mutants

(Le. combine short sequences into longer ones). From an understandabilis point of rien. short

minimal test sequences are preierred. These are easier to understand and easier to cun through

an oracle. On the other hand longer sequences generate more interesting paths through the çtatc

machine that rnay kill other as yet untargeted Iive mutants.

On çewral occasions a test sequence did not Ml the targeted mutant. .hnlysis revcded that

propagation of the fault caused by the mutation required that the capsule be obsemed for somr

tirne after it had received it's 1 s t input. This was the case with action rlelete operators in which

an action cariceling a timer was tleleted. Since the .'deleteci' timer was still active the capsuic hatl

co he obsemecl for. in one case. 5 seconds. before the timer expired causing a tirneoirt event which

in ciirn caused the capsule to emit an observable response. In test sequences chat rlid not waic long

cnough the capsule was terminated before the timeout could occur with the result that the incorrect

behaviour (an unespected response and possible state change) m not obsemcd.

9.15 Equivalent Mutants

One of the tIisatlvantages of procedural mutation is the hurnan cost of identifying eqiiivdent mutants.

This tloes nct seem tu be as large a problem with capsules. In the capsules testet1 in this thesis it rvnç

round that in several cases two or more transitions had the same action object (i.e. the C+L code

bodies tvere functiotially equivalerit. These were often simple functioii calls to cancel a tirrier 1. Khen

these capsides are rnutateti with the TCA operator each pair af eqirit;rlent action objccts generatcs

2 eqiiivdent rnut~ants. A visual inspection quickly and reliably reveals die eqtiitxlence. This type of

equim1ence checkng could also be ,zutornatecl using a tool such as TSL that is nblc to comparp tn-O

tion-terminais ahile ignoring both white-space and comments. Other cquimlent mutancs loiinrl in

these capsulcs incluclcd:

0 a transition chat could target one of two target statcs a i th no chnnge in behnvioirr.

0 repeated ~ar iable assignments (Le. a variable assignment made in one segment of a transition

ctiain is added to another transition on the same transition c h a h (TCA operator).

9.16 Finding Faults

The mutation fiterature seldom discusses what to do when a fault is found in a program umler test.

Should ali of the mutants be regenerated based on the new version or oniy the ones that nwe stil1

live? in this thesis when a fault found in a capsule the fault was fked. d l of the mutants

were regenerated. recompiled. and d l of the mutants aere run against al1 of the test sequences.

Basicaily the mutation testing was re-started from the beginning, the only riifference being rhat the

test sequences created in the previous iteration Rere present a t the start of retesting. This is an

extremely time consurning step. On a slow cornputer such a s two of the three iisccl in this thesis.

reconipilation can take 10 to 20 seconds per mutant. To \didate the necessity of rerunning formcrly

dearl mutants the results of mnning the existing test sequences or1 the buggy version were cornparecl

n-ith the results of running the same test sequences on the -repaireci" niiitants. In most cases one

or niore of the forrnerly deari rriutmts w x a live mutant when briset1 on the correcterl capsiik.

9.17 Strong vs Firm Mutation

\Vith strong niutation only the outputs of the progarn uncler test are comparecl. internal r l a ~ a scates

;ire not examinecl. Witii both n e d i and firm mutation ttie internal data state is examinecl. [fat any

time the data state of the orignal ancl a mutant differ then the mutants is weakly (firmly) killccl. in

this thesis a form of firm mutation testing was performed. Minimal instriinientation was arlrietl cn

cadi mutant that allows the TestHarness to que. the ciment state of the CnpCT. and also co infonn

the TestHarness whenever the CapCT received an iinexpecteri message. An ttne.upectet1 messagr is

a message that does not match any of the t r igers of any ciurently enabled transitions. ivithorrr.

adcling the instnimentation, unelrpected messages are tiiscarded by the CapCT. tF the otisen-ability

option is in use then an unexpected" message is sent to stdout. However if obsen-abilit- is not

enabletl theri the uneqected message cannoc be observed. X typicd result tiIe for an instrunicntccl

mutant is s h o m in the Ieft column of TabIe 9.LO. .Ili of the testing performed in ttiis thesis tisecl

instrumentation.

in order to caIcuIate the strong mutation score. the results file sxs fiIterec1 to remove ar- Lies

generated by the instrumentation. The right coIumn of Table 9.10 shows the results of filtering the

left colurrin. in al1 seven experïrnents the test sequences that yielded 100% firm mutation yieIded

aIrriost 100% strong mutation. Xctuai resuits ranged Gom 97% to 100%.

Weak and firm mutation testing vare orighaiiy proposed as methods of improting the efficien- of

mutation testing. Efficiency was improved by terminating the executing mutant as soon as a mutant

was found to be weakly killed. in the mutation testing of passive capsuies the potentid swings

to be realized is less since the major computationai burden is compiling the mutants. Koaever

wdïing TESTCOSISISST-4TEREQCEST CapCT in state:S3 sencling TCCCO!dlKFIRE paiweTime :-'LOO0 Driver:Catcli?ill Ru CapCT mg: firc Driver:CacchAU RK CapLT m g : n-atchDogReset ?TXL msg, timwut: Xo rcsponsc GomCapCT scntling TESTCOSIIIISSTATEREQC'EST CapCT in scace:S3 sending WSiSCOM S1SSE'I'DRJJ)ARJIRRAY ?;ILL msg, timcoiitr Si1 response fromCapCT CnpCT in stat~:S3 scnrling ~V5ISCO.\ISISSE?iDR.~DARARR.4\* iinexpccterl Oitrcr rnsg: timeoiic S L Z L nisg. timeoiit: So response froniCapCT ~cricling TESTCOSIJ[SST.TEREQL'EçT CapKT iu state:S3 mirling ~VlISCOSf~ISSE?;DE~VSS.\RRRR4Y iinexpected Oiitcr m g : timeoiic XCLL mg, timmuc: ';O rcspowe fromCapCT sending TESTC03IMSST.-\TEREQLTST CapCT in state:S3

sending TCCC0MMSEiR.E paiweTimc :-?O00 Drivcr:Cat&411 Ru CapLT msg: h c

Drivcr:CatdiAiI RK CapUT mg: w~tchDogRcset 'UTLL m g , timmut: 30 rcçponse fromCapCT

scnding iVlISCOXiSISSESDR4DARW.iY ?iTT.L m g , tirneout: Xo rcsponsc FromCapCT

scntling iVlfSCOXIliSSE3?)RiDARR4RRRlY

X L L msg, tirneout: So response FrornCapCT

X Z L rnsg, tirncoi~c: So responsc frnniCapCT

Table 9.10: Firm and Strong mutacion restilts

~ w a k mutation testing of ciynamic capsules has A grenter potential for tirne savings as therc is the

possibility of nvoicling lang timeout cvents. in either case it a p p e m that test secpierices chat firnily

kill mutants are d s o sufficient to strongly kill the same mutants. Sirice the purpose of niutation

testing is to develop strong test cases the question of we~i / f i rm vctsiis strong mily h~ ninot in ehcl

contcsc of capside testing. On the other hancl the estra state information collected ditring chc firm

rriutation testing pefiormed in ttiis thesis can provicle information that tieIps the tester create niutant

killing test sequences. This may be its greatest value.

Al1 testing methods require an oracIe against nhich the result of each test case is checked. As

mentioncd in Chapter 2 oraciiig crut require extensive and expensive humm effort. Mutation testing

offers a way to reduce this cast. Diiring mutation testing test cases are adclerl to the test suite in

order to kir1 live mutants. T'picaüy during mutation testing the resuIt of each neu- test case must be

vecified. via the oracle to ensure that the original program is correct. Hofiever it is possible to add

new mutant kiiiing test cases n-ithout first going to the oracie. It is Iikety thac some of the nem test

cases may not rwrk as pIanned and not kiU m y Live mutants (This occurred several cimes riuriag

CI& thesis). The results of these useless test cases do noc nced ta be verified. rather the test c x e

simiil(1 be tliscarcled. Oniy when s test case has shom itself to he uselul (Le. it has killed mutants)

shouId it be checked with the orade.

Only one of the capsules tested in this thesis was accompanied by specifications. The correct

behaviour of the other capsules &-as decided by the tester. Correctness of each test n-as rietertnined

on the liasis of thc results file that wx the result of execiiting the original capsde againsr the test

squcnce. Thcsc files contain a Iot O€ information that is usecl in firm mutation cornparison but is

rioc needed to clecirie correcmess. Since correctness is juclgecl strictiy on the basis of the ubservabIe

input-output sequence the results files were filtered to remove the instrumeritation data. Corrcctncss

of ttie original çapsiile nas determinect basecl on the fikered files.

9.19.1 Number of Mutants

In general i~ appenrs that state machines rio not generate as uiany mutants ns source cotte mutation.

[Oîfiic+ 931 contains statistia indicating that Fortran progruns of 30 lines generate From 1000 to

3000 mutants. With the current set of operators che number of mutants generatetl is maini? iI

function of the number of states. transitions. actions. triggers. and Ievels of hierdiy. Capsules I I S P ~

iri this thesis generateti froni 100 ta over 3000 mutants. The rnost nriable Factors are the nuniber

of triggers and the nntmber of action objects. Wherens a HFSM with 10 states and 10 transitions

wiii d w a c generate 30 coverage mutants and 10 TRD mutants. the number of TTG. TCX. TAT

mutants karies with the number of action objects. triggers. and leveIs of hierarchy in the capsule.

9.19.2 CPU Time

Dependiig on the nature of the test sequences cornpihg mutants c m be the &est cornputarional

espense. LU years ago such eupense &-as prohibitive. however gii-en contempocar?; computers such as

the Pentium DI 733,CiFIz used in this thesis. 3400 mutants can be compiled in about 7 hours. ic-hich

is not prohibitive- Sharing the Ioad amongsc severai computers can further reduce the t h e . The

other computationa1 e-qense is esecutbq each mutant against the test sequences.

-ifter sending a message to the Capu'T, the test harness used in this thesis can be instmcted

to wait for a response before sending the next message. A clelay between sends is usually needeti if

the capsule has 'an interna1 timer and the test sequence requires that the timer be aiiowetl to expire.

However. if the capsde responds immediately then there is no appreciable clel- between test harness

sentls. If the CapCT does not responci then the t e s hamess waits until the tester selected time period

tins clapseti. This means that the lower bound execution tirne for a test seqiience c'an be e-xtremely

short. much less than a second for the passive capsules (capsules that do not use interna1 timers

o r iriternal everits) iised in this thesis. However some of the dyriarnic capsules uset1 in this thesis

required rriultiple waits of up to LO scconcls for a tinicr co expire. Orie of the capsules rcqiiirrd a

test setlucrice chat uscd 10, 10 second clelays for a test scqueoce ext'ctttion tinic of alniost 2 riiinutes

( tliis tvas rctliicctl).

Running huntlrctls or even a feu-- thoiisand passive mutants against a hantlful of test secluences

is feasible #ven eonternporary coniputers. In the exampie citctf above compilation of 3400 niiitants

took 7 hours. whereas running 4 test setpiences against each mutant requiretl only 34 miniitcs.

However consider a test seqiience that has a singIe timecf event of 10 seconcfs. The cime increase

woiilci be 34000 seconds or about 10 hours. E v e n time that a simiiar delay is acitled to a test

sequence 10 hours is added to the total testing tirne. Testing these types of capsules may reqiiire a

tester to motlify the capsule by reducing the time in t end . if modifications are not possible then it

rnay not be cost effective to use mutation testing for these capsules.

9.19.3 Faults

As mentionecl eulier when a fault is fountl in a capsule the capsde niiist be rep;iiretl. the rriiitiirits

regenerated. reconipileci. ancl al1 of the mutants execrited against al1 of the test sequerices iigiiitl,

For n capsule chat generates maq- mutants the repeated cost of compiling mutants can hecome

prohibitive. K i th the capsule that generated 3400 mutant each time a fault is found and repaired Ï

hours are required to recompile al1 of the mutants, This indicates that mutation testing is probably

better suited to testing capsdes that are of high qualitt. when they are @en to the tester. Capsuies

chat appear to be buggy or of Ion- quaiitj- shodd be rejected.

9.20 Preambles

Often several test secluences are required that start From a common non-initial state. Siich ueqirences

consist of two pieces. The first piece. caiied the preamble. is used to get the capsule into the common

scate. The reuiaincler of the test sequence performs the -new" testing chat is unique to each test

scquence. If preambles are short then simply repeating them in each test sequence is acceptable.

However long preambles are a problem and present the opportunity for eEciency improvements. X

possible way CO eliminate re-executing common preambles is to record the capsule's total statc once

che preamble Iias b e n execiitetl. Then each test sequerice that relies on the preiimble. cile r~cordeci

state can be restorecl ancl the test seqiience run from there.

IKth most of tlie capsules testecl in this thesis the (tinie) length of common pre-nmbles r a s noc

liirgt. t*noiigii tri n-arriint iisirig a state recortling fiinction. Ho\vcvcr thcre arc sorne c;ises where Iong

diirarion timeoiit events constitiite a large part of the riin time. Avoirling the repetition of tt~ese

timeouts coiiIcl be worthwhile. This has been left for future work.

9.21 Testing your own code

Convericiritial tvisdoni says chat clevelopers should riot Lest their own CO& (as disciissed in section

2.5). SIutatirin tcsting triay mitigate this argument. The exercise behinc1 mutation testing is to kiIl

mntants. the programmer-tcster knom that the nititant is a faiilty version of his o c n progarn (his

creacion tvhich he tvants to protecc). It behooves the tester to kilt che mutant in orrier CO stion. that

k s program (the progenitor of the mutants) is the correct versioti by finrii~ig a t e s case that kilb

ttic mutant. if he cannot kill the mutant then either the mutant is equimient (this is the weakn~ess

of the argument. the programnier-tester rnay riot be entireiy objective if objecti\-ity means that his

iwacion is faufty) or the programmer-tester must face up to the fact that his version is fadty-

9-22 Summary

A live mutant is a signpost to a trouble spot in the code. Possibly the utiIity of mutation testing is

that it points the tester to potential fauits in the code by asking hard questions of the forrn: -the

test data is t e h g me that it doesn't matter that 1 have changed this element of the program. Why

is that?" The ansver is that the mutant is equivalent. or the test data is not adequate enough to

müke the ciifference apparent. or the fault is rnaskecl by subsequent actions of the program, or ttiere

ia ;i fault in the original progam.

This chapter h'as presented the results of and rficussed the e'cperiments performeri to e d i i a t e

mutation testing of RoseRT capsules. It h s presented a prelirninary nnalysis of the miiracion

operators proposed in chapter 6. Fin& the chapter conclirded with n discussion of chc niain

observations made during the performance of the experimcnts.

Chapter 10

Conclusions

Tliis ttiesis t i u met al1 of the objectives ot~tlincti in Chapter L: hackgoiind information on software

testing. mutation testing. and finite state machine testing was presented: a set of niutntion riperators

ivas proposcd: testing toois were biiiIt: and experirnents were performed in order to e~diiare mucation

testing ciE RoseRT HFSMs.

10.1 Summary of Results

Ctiripter 9 riiscussetl the esperirrients carried out u d the obscnations made. The foIIowing section

presents a concise surnmary of the important obsemtions. Tt is impartant to note ehnt clrcst* ;inv

preliminan obsenations. and are based on a srnall number of eqwriments. These conclusions are

cticreforc noc well supported bu ernpiricai data and some of thern may in f x t be mong.

O Firiding a fault effeccively requires restarting the cesting process. This impiies that mutation

testing is not suitabte for testing capsules that are likely to have many fauits.

rn If we expect to tind o d y a few fadts then oniy a few of the hundreds or thousands of mutants

wiII actuaiiy Iead co the creation oia test sequence that reveals a FauIt. The remaining mutants

do not coutribute anl-thiog to the testing procesS. Tt is therefore usehi to kiII as rnaq of ~hese

niutrints as possible sith the ûrst test sequences. Initial test sequences based on transition

tours seem to be effecti~e at doing just th%.

Proceduce iç an important aspect in etficient mutation tesctng. Working ~ - i t h subsets of the

total mutant population rather than d g ail of the mutants every time reciiices the iiniount

of CPC time reqtiired and potentially rvasted. Further the tester is not beleaguered by the

large number of live mutants that may e-xist after the initial test scqiiences are executed. An

effective strategy appears to be LO add mutants in the following order and to kill d l of them

before adcling the next set of mutants:

- Coverage operators (CVT. CSE. CS'i)- Test sequences that kiU coverage mutants tend

to kiU many other mutants

- Action delete operators (.mS/E/X) - iiilling these operators seenis to lead to faiilt

revealing test sequences.

- TRD - llost TRD mutants are killed by the C3.T test sequences. Any live TRD mutants

indirate an issue tvith the capsule.

- Lon- volurrie operators: internal/external. choice point. state clelete. ancl trigger (TGT)

- Since the effectiveness of these operatos is ciirrently iinknonn thc only rcasnn tii use

thern at this stage is becatise they do not generate manu mitcants.

- T.\T - This operator and the ne- two produce mutants that are more than linear in the

numbcr of statps and transitions. However the TAT operator protluces fewr miitanls

t h r i eitticr TC.4 or TTG.

- TC'X - This opcrator proctttccs many mutants niosc of ntiich do not contribrite nny thin;:

to the testing process. By adding these mutants nt this point it is Iikely that many of them

ail1 be killed by the esisting test sequences. However the remainitig Live TCX trititarits

seerxi to be effective at revealing faults.

- TTG - This operator does not seem to produce semantically srnnll fatrlts. hon- ver i t

generntes rnany mutants. If it is going to be uscct at dl . it should be usecl last in the hopc

that niost of its triiitants n-il1 be MIed bu existing test sequences.

E~amining [ive mutants in order to deciare them equitalent or in order to create a nem test

sequence is the most labour intensive part of the mutation testing process. Improring the

evisting tooIs is possible and has the potentiaI to greatiy reduce the difncuity =and time reqtùred

to examine live mutants and to create nem test sequences.

The ~Lifference betn-een weak, hm. and scrong mutation does not seem to be as reletant with

RoseRT KFSNs as it is n-ïth source code mutation. The time and cost sat-ïngs realized bu

perforrning weak mutation tating on the capsules in this thesis is not significant. This m -

be due ta the frict that the mutated elements of a HFSM are qualitativeiy more significant

structures than many of the elements mutated in source code (Le. states and transitions are a

larger granularity than source code statements? they represent a higher IeveI of abstraction).

Executing a WSM mutant that results in a weakIy/Fimly obsenable difference alrnost aiways

leads to a strongly- observable dierence.

0 h1iitation cesting tloes not micigate the oracle probleni. A srnall gain c m be macle in that

ineffective test sequences (ix. ones chat do not !dl mutants) can be identifiecl arici ~liscnrcled

hcfore havirig to determine tvhecher the original capsule's behaviour is correct d i e n eseciiting

the test sequence.

0 Capsules Stace maclùnes ancl possibly KFSM's in generd do nut seem to nenerate ri?; rtiary

miitants as source codt? miitation. The lowcr nimber of rniitnnts retiiices the niairi disatl~.iint;ijic

of convenrional source code mutation testing which is the necessity of runuhg thousnticls of

miitants ngainst miiItipIe test cases.

0 The major cornpucational espense of testing passive capsules is the cime reqiiired to compile

al1 of the mutants: execittion time for the test seqiienccs is insignificant comparecl to the

compilation time.

For dynmic capsules the major computational expense m e be the r~xecircic~ri t i n i ~ of test

sequences. Internalty timed events of Iong duration (on the order of several secancls) iri;iy

niake mutation testing ECO inefficient to use. It is possibIc ro retiuce cimcoiit intenals. however

if a capsule uses niiiltiple interacting timers then coordination of the tinied events becomes a

problern.

0 Mutation testuig may aeaken the claim chat programmers shoiild noc test their own code.

S[utation testing encourages the construction of tests chat kiU mutants. It does not seem to

reIy on the abZty of the tester to assume a destructive attitude tomrd the program being

tested.

0 in some of the capsules tested in this thesis. mutation testing guided the construction of test

sequences that traverseci non-intuitive fauit revediig paths.

10.2 Future Work

Although the o b j e c t k s of this thesis were met there is much more ~ o r k that c m be done.

The main weakneçs of this thesis is the limited number of capsules that were used in experiments.

In order to strengthen the conclusions drawn frorn this thesis it is necessacy to perform mutation

testing on a much larger and more diverse population of capsules. SIodels developed For inclustrial

use by professional developers are needed. however this would require greater cooperation from

industry.

The set of operators proposed in this thesis is a subset of possible mutation operators. in

addition to a d d i g other behavioural operators it is possible to use mutation operators that mutate

the structure of one or more capsules. For evample capsule roles can have multiplicities peater

than 1. A possible mutation is to change the ma?cirnurn number of containecl capsules ttiat can

be instantiated by a containiig capsule. Also mutation of toolset elements cleaiing ivith message

prioritics ancl threads coiild be considerecl. Finally the RoseRT toolset supports inlieritance. Wlicn

a capsiile is siib-cliissetl the capsiile inherits the siiperclass's CSSI. The (levcloper is then ihle c»

atld states ancf transitions to the inherited CSSI. Future research could look at performing mutation

trsting of the entirc superclass-~ubclass CSSI or just thc siib-clnssecl CSLI.

Theoperators propased in this thcsis are di first-ciegree operators in that cach mutant is the rcsulr.

of niaking a sirigle syntactic change to the testual reprwencation of the state machine. In source

code mutation second and higher degee mutations are not used becaiisc they prorluce iiri espcitieiitiril

increase in the number of mutants generatecl %?thout an ensiiing improvement in thc fault revealing

powcr of the testing process [Offut+ 70001. This result may not lioltl in scat@ machine testing: aitii

a graph a tirst-degree mutation may be an nth degree mutation when \-ieverl clifferently. For exarriple

(leleting n state is a tirst-clegree mutation from the point of view of tieleting states. Iiowever it is

similar to an nLh degee mutation from the point of \<en- of deleting transitions [Dean 011. In uther

words when a state with nvo incoming and t a o outgoing transitions is cleletecl this is eqii ident to

deleting the four transitions. This is not exactly the same as a fourth clegree mutation since rather

than deleting four a r b i t r w transitions. the deIetecL transitions are related by their association n-ich

a common state- Another =ample is the second-degree mutation in which the target states of m o

transitions originating a t a common state are swapped. When a state has more than tn.0 outgoing

transitions this "paincise" operator could he appIied severai times. once for each unique pair of

transitions Ieaving the state.

hproving the tools used in this thesis would increase the efficïency with which mutation testing

could be performed. Some automated equivaiency checkhg could be integrated using TSL. integrat-

ing the tools such that they presented a çingie consistent user interface to the tester woiild reduce

some of the cediousness euperiencecl in using the current prototypical system. For esample the staric

charts produced bu RT-SuperDiff could be changed such that clicking on a live niutant on the chnrt

results in the presentation of al1 information concerning the mutant.

The cime required to compile each mutant from the generated code is sipiticam (10-'20 seconds).

Future work could look at applying mutation operators to the p ~ m e tree c l h g compilation rather

than CO the capsule text file.

It is possible CO automatically generate certain types of test sequences. .A TSL prognm coiilcl

be useci to trnnslate capsule state machine diagrams into a probleni description for a planner sirdi

as BlackBox [Kautz; 981[BIackboxI[FrohIich+ 001. The pl.anncr can then search for ancl prodocc

transition tours 'anci statc tours.

Finally nuitatiori of the dctail level C++ source code coiild be the subject of future research.

10.3 Contributions

This thesis ilas presentcd pre l iminq evidence indicating chat mutation testing of RoseRT FiFSlIs

is both feasible (n-ith tool support) ancl c m result in the creation of fault revealing test secluences.

The contributions of this thesis incliide the following:

i Critical anaiysis of the background literature on mutation testing,

O Proposed mutation operators for RoseRT capsules and a p re l iminq analysis of each operator.

O CFG for RoseRT capsule test files.

a Mutant creation tools based on TXL.

i Test sequence creation tool ( RT-Test CaseManager).

O Generic test harness capsule (RT-TestHamess).

a Test results analpis tools.

Development of an efficient procedure for performing mutation testing on capsules.

Analysis of the feasibility of mutation testing of RoseRT capsules.

a Identification of some areas of future work.

Ill

10.4 Concluding Rernarks

10.4.1 Design Mutation

h t a t i o n testing of source code is a verification activity: ;did we build the t h b g right" Mutation

tcsting of specifications is like ~.alidacion:"are we buiIding the right thing' However n i th CSSI

mutation testing, the design is being tested. X Rose Sfodel n-ithout detail level code is a specification.

SIutating this is similar to specification mutation. However it is not n-orth doing because al1 oritpi~ts

of the CSSI are the result of detail I e d code: there is nothing to observe. With the atltlition of

detail levcl code. mutation testing of CSMs is doser to. but is not the same as source code mutation

(the source code is not being tiirectly mucatcci). Uowcvcr. one coultl argue chat mutation testing of

CS!& is nctiially tl~sigri nuit ation.

10.4.2 Operator Rationale

The mutation litcrature often implies that miitation operators are tlesignerl to mimic small syntnctic

niistakes niade by programmers. The competent programmer hypothesis and coupling effect hy-

pothesis arc then citecl as being the theoretka1 foundation iipon which this tiecision is baseri. Thii

cloiirls the real piirpose of mutation testing rvhich is to encourage the creation of high -qirality-"tests.

Whether mutation operators are designed to mimic typical programmer errors (whaterer they are)

or not is irrelevant. h t a t i o n operators shouid be designed to mco~lrage the coristnrction of f i d t

reuealing tests (this is what is done in practice: Mothra has statement coverage operators. Proteiun

has transition coverage operators). On the other hand lion- does one design an operator that is es-

pected to-encourage the construction of a fault revealing testr This is where mimicking programmer

m o r s beconies usefiil. It is a heiiristic usecl to attack the ambiguous problem of tlesigting mutation

operators that will encourage the construccion of f a d t revealing tests.

Bibliography

[Acree+ 791 AT. Acree. T.=\. Butld. R.;\. DeSIillo. R.J. Lipton. F.G. Saynartl. Mutation AnaI- pis. Technical ReportGIT-[CS-79/08. School of Information and Cornputer Science. Gcorgia Institute of Technology. 1979.

[Acrce 801 A.T. Acree. On Mutation. PhD thesis. School of Information and Cornpucer Science. Beorgia Institute of Technology. Atlanta. GA. hugiist 1980.

[:lgrawal+ 891 H. Agawal et ai.. Design of Mutant Operators for the C Progamming Lan~qiage. SERC-TR-41-P. Software Engineering Research Centre. Pirrduc University. 1989

[.Ammarin 9-41 P.E. Ammann. S.S. Brilliant. .J.C. Knight. The EfFect of imperfecc Error Deteccion on Rcliability .Assessrnent vin Life Testing. IEEE Transactions i i t ~ Software Erigineering, Co1 '10-2. Feb 1994.

[Bernhard 941 P.J. Bernhard. X Reduced Test Suite For Protocol Conformance Testing. Comniuni- cations of the AC'II. 1994.

[BIackbosj http://cs.washington.edu/homes/kautz/blackbo+/incles.html

[Boch+ 941 G. Bochrnann. A Petrenko. Protocol Testing: Rei-iew of Methotls and Relevnnce for Software Testing, Proceedings of ISST.4 94, AC11 1994.

[Rucid SOI T.X. Butld. Mutation hnalysis OF Prograrn Test Data. PhD thesis. Yak Cnivcrsicy. Scn- Haven CT. 1980.

[Budti + $21 T..LBudd. D. Angluin. Two Sotions of Correctness and Their ReIation to T~sririg. Acta Inforniatica. LS(l). pp3145.Sov 1982.

[Butler - 931 R.W.Butler. G.B. Fineiii. The infeasibiiicy of ex~erimental quantification of Liie critical softnare reiiabiIity. IEEE Transactions on Software Engineering, Vol. 19. So. 1. 1993.

[Cai 981 Kai-fian Cait Software Defect and Operational Profile hIodehg, Klumr. Boston L99S.

[Chow 781 Tsun S. Chow. Testing SoFtware Design Modeled by Finite-State SIacitines. [EEE T r w actions on Softn-are Engineering, May 1978.

[Cordyt 001 .J.R. Cordh I.H. Carnichael. R. Haiiiday. The TSL Progamniing Lmguage \-ersion 10. http://~~-~t2d.ça/inrIe~.html

[Dean U l I Communication niith Tom Dean May 2001.

[Deharo+ 971 11. Delmaro et ai.. Integration Testing using interface LIutation. iEEE Transactions on Software Engineering, h l . 27. Yo. 3.4Iarch 2001.

[DeSIilIo+ ÏS] R. DeMillo. R. Lipton. F. Sayward, Hints on Test Data Selection: Help for the Practicing Programmer. Computer 11. p 34-43, April 1978

[DeSIillo+ 791 R. DcMillo. R.J Lipton. F.G. Sa-n-ard. Program SIutation: a ne&- appronch to pro- gram testing. infotech State of the A r t Report. Software Testing and Edriation. Vol 2 Imqtecl Papers. Infotech Lnternational. 1979.

[DeSIilIo+ SS] R. DeMillo. D.S.Guindi. \V.;.41.;.4lcCracken. A..J.Offut. K.Y.King. An Estended Ovewiew of the SIothra Software Testing Environment. in Proceedmgs Second Workshop on Software Testing \-enfication and Analysis. IEEE July 1988.

[DeSIilIo 891 R. DeMillo. Test Xdeqiiacy and Program Miitation. ICSE S9, 1989.

[Fabbri+ 981 S.C.P.F Fabbri. S[.E.Delrnarro. J.C.~Ialdonado. P.C. SInsiero. Proteum/FS&i: A Tool to Support Finite State Machine Widation Basecl on Sfiitation Testing. Proceecling of chc 19th International Confercnce of the Chilean Compiirer Science Society. 1998.

[Fiibliri-k 9Sbl S.C.P.F Fahbri. .I.C.SIaldonado. SIutation Testing Applied to Validate Specifications Bascd on Stntecharts. Proceedings of the 10th [riternational Symposium on Software Reliabiiicy Engineering. 1995.

[Frankl+ 971 P.G Frankl. S.S. Weiss. C. Hu, .U-&es versus SIutation Testing: .ln Experinientd Coniparison of Effectiveness. .Journal of Systerns and Softwarc. Vol. 38. 1997.

[Frohlich+ 001 P. Frohlich. .J Link. .4utornatecl Test Case Generation Froni Dynnrnic JIoticIs. ECOOP ZOO0 LSCS 1SSO. pp.472491.2000.

[Garrliner 991 S. Gardiner (Ed.). Testing Snfet-Relatecl Software: A PracticaI Hancibook. Springm. London. 1999

[Gili 621 Introc11iction to the Theory of Finite State 5Iachines. .\IcC;rnw-Hill Book Compati- hic. 19G2.

[Good+ÏJI .I. Goodenough. S. Gerhart. Toward a Theory of Test Data Selection. CEEE Transactions on Software Engineering, .lune 1975.

[Gest+S'L] R. Geist. X..J.Offutt. and F. Harris. Estimation and enhancement of real-tirne softn-rire re1iabiIit.y through mutation analysis. E E E Tr.ansactiotis on Cornputers. vol. 41. pp. -550-558. 41- 1992. Speciai issue on FauIt-Tolerant Computing.

[HamIet S8i D. Harnlet Special Section Software Testing, Comrnunicatioris of the XCM. ikI 31. Xo 6. June 1988.

(Harnlet+ 901 D. Hamlet. R. Taylor. Partition Testing Does Yot inspire Confidence. KEE Trnnsac- tions on Software Engineering, 16(12): 1402-1411. Dec. 1990.

[HamIet 941 D. Hamlet. Software Quality. Software Process. and Software Testing.

[Hamiet 9431 D. Hamlet. Foundations of Software Testing: Dependability T h e o n SIGSOFT 94. ACSI. 1994.

fHarel 871 D. Harel. Statecharts: a visual formalism for complex systems. Science of Computer Programming S. pp231-274. 1987.

[Kowden 92) W.E. Howden, Weak Mutation Testing and Cornpleteness of Program Test Sets. EEE Transactions on Software Engineering. Vol. Se08. ?io.-l. .Jdy 1982.

[ET4961 Reference Slodel for OSI.

[Jalote 971 PanIsJij ..Talote. An integrated Approach to Software Engineering, Springer-\ërlag7 xew York. 1997.

[Jorg 9.51 P. Jorgençen, Software Testing: X Crdtsmans Approach. CRC Press. Boca Raton. 1993.

[I<niitz+ 981 H. Kaucz, B. Selman. BLACKBOS: A New Approach to che Application of Theorem Preting to Problem Solt-ing. Working notes of the Workshop on Planning as Combinatorid Search, held in conjunction with .OS-98. Pittsburgh, PA, 1998.

[Kini-i 991 S. Kim. J .b . Clark. .J..L IIcDerrnid. The Rigorous Generacion of .Java .CIirtation Oper- ators Using HAZOP. Technical Report. The University of kmk. 1999.

[Koo 961 1. Koo. Slutation Testing and Three Variations.

[Iinighti 851 J.C. Knight. P.E. Animann. An Experimental Evaluation of Simple SIethods For Seecling Progam Errors. Proceedings of the Eighth International Conference on S o f t ~ x r e En- gineering. pages 337-3-12. August 1985.

[Maci 911 A. SIathirr. R.DeMil10. On The Usc of Software Artifacts to Evaliiate Effectiveness of SIiitation Analusis for Detecting Errors in Production Software. Technical report. Software Engineering Research Center. Purclue University . SERC-TR-92-P. SIarch 1991.

[Liskov 011 B. Liskov with .J.Giittag. Progam Development in .rat;r. Xtlclison-Wesley. Boston. 2001.

Lyons 981 -4ndrew Lyons. CSIL for Real Time Overview.

[Mat 911 X.P. SIathur. Performance. Effectiveness. ancl Reliability issues in Softm-are Tcsting. LEE€ Proceecling of 15th Xnnriai international Cornputer Software ÿ; Application Conference. 1991.

[SIycrs 791 G. Lieuers. Thc Art of Software Testing, A \Vile-interscience Publication. 1979.

[Sior S4j L..I. SIorel. X Teon of Error-Based Testing, PhD thesis. Cniversicy of LIarylcantl. ColIege Park MD. 19%. Technical Report TR-1395.

[Sliillcrbtirg 951 M. SIiillerburg, Systematic Testirig: a Means for Valitlating Reactivt! Systonis. Eu- roST.4R 94 Special Issue of Software Testing, Verification and Reliability. \QI .?. S o 3. 199.5.

[Offuc 921 A . J . Offut. hvestigations of the Software testing coiipling effect. ACSI Trans:ictions on Softwarirc Engincering 5fcthodology. 1(1):3-15. Jaiiuary 1992.

[Offut+ 931 X.J. Offut. G. Rothermel. C. Zapf. An Lqeriniental Etduation of Seleetive SIiitation. 15th Annual Conference on Software Engineering. EEE. May 1993.

[Offutt 951 A. .J. Offutt. A Practical System for Slutation Testing: Help for the Common Progam- mer. I2eh International Conference on Testing Cornputer Software. p99-109. 199.5.

[ O f i t i 061 A.J. Offut. A. Lee, G. Rothermel, R.C'ntch, and C.Zapf. An Experirnental Detennina- tion of Sufiicient mutation operators. AC41 Transactions on Softmre Engineering Slethotlologu. :(2):99-)-tls. +ri1 1996.

[ O f i t f 96bj X.J. Ofhic. .J. Pan. K. T e w ~ T. Zhang, An Esperimental Ernluation of Data Flow ,and JCutation Testing, S o h a r e Practice and Experience. 26(2):1763-176. February- 1996.

[ORut+ 961 A.J. Offut. J . Hufffiman Hayes. A Semantic Mode1 of Progam Faults. ISSTA 96. L996.

[Offutt+00] A. .J. Ofhm. R.H- tintch. Mutation 2000: Gniting the Orthogonal. Mutation 2000: 1Iutation Testing in the Twentieth and Twenty First Centuries. p-15-55. 2000.

[Petredmt 961 A. Petrenko. G. Bochmann. On Fault Coverage of Tests for Finite Stace SpecSca- tions. h t t p : / / ~ ~ 7 ~ . i r o . u m o n t r e a I . c a / p u b / t e l e ~ 1996.

[Rapps+ 351 S. Rapps. E.J. Weyuker, Selectïng Software Test Data Using Data Flow information. iEEE Transactions on Software Engineering, SE-11. April 1935. 367-375.

[RoseRT 991 Rational Rose ReaiT'ie 6.1 Oniine HeIp: Jlodelùig Language Guide. 1999.

[RTCh 921 RTC.4. hc . SoEtmre Considerations in Airborne Systems and Equipment Certification. Document So. RTC.4/DO-liSB. Dec. 1992.

[Shepard+ 011 T. Shepard. SLLamb. D-Kelly, More Testing Should Be Taught. Cornrniinications of the ACM. (Vol. 44. ?JO. 6), dune 2001.

[Sheparrl 011 Communication with Te- Shepard Aug. 2001.

b?iet 931 .J-C. van Lliet. Software Engineering: Principies and Practice. Wiley. Great Bricain. 1993.

[\:os + 981 J.11. Voi~as. G. SIcCraw. Software Fault Injection: Inoculating Progrmis Againsc Errors. \Viley. Xew kbrk. 1997.

PVeyrktrr YS1 E..J. Kcyuker. .-ln Empiricai Stuciy of the Complexity of Daca Flow Testing, Proceed- ings of the Second Workshop on Software Testing, Verification. ancl Analysis, Banff. Canacla. .Jiily 1988.

[Wang+ 971 W. Eric Wong, J-C. hidonado. X E . Delrnaro. S.R.S. Souza. Cse of Proteurn to Accelerate Mutacion testing in C Programs. Proccedings 3rd ISST.4 International Conference on Reliability and Qudity in Design. h r c h 1997.

[Wootiwartl- 881 S[.R.Woodtvard. K. Hdemood. From tT;Cak to Strong, Dead or Alive:' An Anal- p i s of Some hlutation Testing Issues.Proceedings of the 2nd Korkhop on Software Testing, Qrificacion. and Analysis. pp. 152- 138, Jiily 1988.

[Zhu 371 Fi. Zhu. Softn-are Cnit Test Coverage and Adeqiiacy. .\Ch[ Computing Siineys. Der LBDÏ.

Appendix A

Test ing Glossary

error - An error is clefined as the human activity that results in software containing a fault. Errors

;irr iisiidIy niistakes niade by the programmer. Honcver not riIl m o r s leati t o failrircs.

failure - A hilure occiirs nhen a hiilt eseciites. X failiire is an es t~rnal . inr'rirr~c't 11oti;~vioiir of ;i

program (an incorrect output or run time failure)

fault - -4 fault is the result (manifestation) of an error. Also known <as clefects or brigs. The terni

faiilt can d s o refer to missing behavioitr. i.e. functionality that nns not irnplementeri (dso

called n fauh of omission).

test - The act of esercising software with test cases. A test can fincl a failure or clemonstrate

cornpliance.

test case - A single input vector to a program.

test adequacy criterion - Some measure or mle useci to establish a firm requirement for hoa

much testïng is necesan;.

test sequence - X t h e series of vectors input to a program.

Appendix B

Example Capsule Text File

This appenrlix shows a RoseRT capsule file. This capsule is From the criiise control esperinient. It Las

a structure. classattribt~tes. and statemachine section. Only the statemachine section nns stlk>jp(T

to mutation. The seccions describing the graphiral placemctit of the eleirients arc rioc iriclircled iri

this description.

B.1 Handling C++ Within Capsule Files

The [action-objectl non-terminal contains C f + code. Therefore capside files ciln be parsect by risirig

two separate grammars. one for the CSM schematic lever. and one For the C++ rletail level. How~t-er

each Lie of C+-. s h e n encoded nithin the capsule file. is prefked s i t h a vertical bar symbol. Thc

Folloning capsrile file Frahment shows an instance of an [action-object] non-terminal. Yotice the tn-O

Lines O € Ctt rode.

entryAction (object Action quid "385F19800276" body

I ControlCamms . l ightProblem0 . sendo ; I EngineComms . disengage0 .sendO ;

1

Because of the Ieatüng vertical bar SI-mbol it is oot possible to parse the C++ se,gnents wÏth a

standarci C++ garnmar. The presence of the veaicd bar symbol requires that the Cf+ gammar

be changed to ailox- a vertical bar s p b o l to be option&- present anpühere in a Ctt program

chat a tine break is le&. Rather than changing the C f + grammar it waç decidecl to eliminate

the C++ grammar and pre-process the original capsule file before parsing it nith T U . The pre-

processor encapsulated each C++ l i e nithin double quotes. Double quotes chat were present in the

original C i + line were escaped (prekred n-ith the backslash s-bol). The capsule grarnmar r a s

then changecl so chat action objects consist of [stringIitI non-terminais rather than C++ fragments.

After mutants are generated the doubIe quotes are rernowd bu a post-processor progam. These

pre-process/post-process steps are not discussed in Chapter T as they were not consiclerccl relevant

to the discussion OF the core mutation system.

The reniaiticlcr of this appendix shows the capsirle file for one of the capsdes testetl in this chrsis.

clic criiise cotitrol capsille.

B.2 Capsule File for Cruise Control

(object Capsule "CruiseControl" quid "3A6400CFOOD6" structure (ob j ect CapsuleCollaboration "Structure"

quid "3A6400CFOOE4" ports (list portsList

(object Port "ControlComms" classifier "Logical Viev::ControlU quidu '13A6411800149" isRelay FALSE)

(object Port "EngineCommsN classifier "Logical Viev::engiueM quidu "3A6412510149" isRelay FALSE)

(object Port "timer" classifier "Logical Vieu::RTClasses::Timing" quidu "37E9292COl8C" isRelay FiILSE isWired FALSE)

(ob ject Port "oatchDogTimer" classifier "Logicaï Vieu::RTClasses::Timing" quidu "3ïE9292CO 18C" isRelay FALSE iswired FALSE)

(ob ject Port "periodicTimer" classifier "Logical Vieu::RTClasses::Timing" quidu "37E9292COl8C" isRelay FALSE isWired FALSE)

(object Port "log" classifier "Logical Vieu::RTClasses::Log"

quidu 1137E92926019B11 isRelay FALSE isWired FALSE)

(ob ject Port "TestComms" c l a s s i f i e r "Logicd Vieu::TestProt" quidu "3A549694028D" isconjugated TRUE isRelay FALSE) ) )

class-at t r ibutes ( l i s t c lass -a t t r ibu te - l i s t (object ClassAttribute "setSpeedn quid "3A64124301F3"

type " int") (object ClassAttribute "DELTA" quid "3A6412AF036B"

i n i t v "5" type " int" changeable "Frozen")

(object ClassAttribute "actualSpeedtt quid "3A64136002BBB1 type " int")

(ob j e c t ClassAttribute " t imer ID" quid "3A64151B013Cn type "RTTimerId")

(object ClassAttribute " th ro t t l e" quid "3A641A5E0360m type " in t " )

(object ClassAttribute "~atchDogTimerID~~ quid "3A641BFDOlA4" type "RTTimerId" 1

(object ClassAttribute "periodicTimerIDM quid "3A70910100A2" type "RTTimerId" 11

sçatemachine (object State-Machine quid "3A6400CFOOEOv S t a t e (object S t a t e "TOP"

s t a t e s ( l i s t S t a t e s (object I n i t i a i P t " In i t i a i1T) (object S t a t e I1error"

entryAction (object Action quid "3B5F19800276" body

I ControlComms . l ightProblem0 .sendo ; 1 EngineComms .disengage (1 .sead() ;

1 s t a t e s ( l i s t S ta tes

(object f u i t i d P t " I n f t i a l * ) (object JuncPt "Junctionl"

isEntry TRüE emernal lyVisible TFLUE)

(ob j e c t JuncPt " Junctionl" external lgVisible TRUE) ) )

(ob j e c t JuncPt " Junction4" isEntry T U E )

(object S t a t e "on" entryAction (ob j e c t Action

quid "3AT096E103BC" body

I timerID = tiner. informEvery(RTTimespec(2~0, O) ) ; //ask f o r speed every 2 seconds

e x i t Action (object Action

quid "3A70970902SB" body

I i f (timerTD.isValid()) timer.caacelTimer(timerID); I i f (watchDogTimerID.isValid0) watchDogTimer . cancelTimer (vatchDogTimerID1;

1 s t a t e s ( l i s t S t a t e s

(object I n i t i a l P t " I n i t i a l " ) (object S t a t e "active"

entryAction (object Action quid "3A70911EOSOC" body

IControlComms .lightOnO + send0 ; I periodicTimerfD= periodicTimer- informEvery(RTTimespec(1 .O ,O) ) ; I

1 exitAction (object Action

quid "3A7090FDOOED" body

I ControlComms . l ight0f f (1 . send0 ; I EngineComms . disengage0 . sendo ; I 1 periodicTimer . cancelTimer(periodicTimer1D) ;

1 s t a t e s ( l i s t S t a t e s

(object I n i t i a i P t " I n i t i a l " ) (object JuncPt " Junctionl"

ex te rna l lyv i s ib le TRUE) ( o b j e a S t a t e " justRighttl

s t a t e s (list S t a t e s (object I n i t i a l P t " I n i t i a l t t ) (ob j e c t JuncPt " Jiuictionl"

isEntry TRüE ex te rna l lyv i s ib le TRUEI 1)

(object S t a t e "tooSlou" s t a t e s ( E s t S ta tes

( o b j e a Tni t i a iP t " I n i t i a l " ) (object JuncPt "Junctionl"

isEntry TRüE ex te rna l lyv i s ib le TFtüE) ) 1

(object S t a t e "tooFastn s t a t e s ( E s t S t a t e s

(object I n i t i a l P t " i n i t i a l " ] (object JuncPt "Junctionl"

isEntry TFlüE e a e r n a l l y V i s i b l e TRITE) 1)

(ob ject CholcePt "CP2" guard (object Guard

expression "return setSpeed>actualSpeed; "

1) (object ChoicePt trCP1"

guard (object Guard expression

[ i n t deltaSpeed = çetspeed-actualspeed;

(object JuncPt "Junction4" ishtry TRüE)

lob ject JuncPt "JunctionSV isEntry TRUE externallyvisible TRUE)

(object JuacPt "Junction6" isEntry TRUE externallyYisible TRUE)

(abject JuncPt "Junction3" isbtry TRüE externailyVisible ïFLiJE)

(ob ject JuncPt "Junct ion2'' i sEntry TRITE1

lob ject JuncPt " Junction7"l (object JuncPt "Juaaion8"

isEntry TI1iE) (object JuncPt "Junction9"))

transitioas (List transition,list (abject Tram "True"

sourceRegion 1 source " :TOP:on:active:CPl" target

" :TOP:on:active: justRigbt: Junctioal") (object Trans "False"

sourceRegion O source ":TOP:on:active:CPl" target " :TOP :on:active:CP2")

(object Trans "True" sourceRegion 1 source " :TOP :on:active:CP2" targe t

" :TOP:on:active: tooSlow: Junccionl" action (object Action

quid "3A708FT60004" body

"EngineComms .setThrottle(throttle+5) .sendo ;" 1)

(object Trans "False" sourceRegion O source " :TOP : on:active : CP2" target

" :TOP :on:active:tooFast : Junctionl" action [ob ject Action

quid "3A708Fïi3015F" body

"EngineComms -setThrottle(throttle-5) .send() ;" I I

( o b j e a Trans "timeout" source " :TOP : on:aaive : Junction4" target ":TOP :on:active:CPl" eventGuards (List event Guardslist

(ob ject EventCuard Event (ab j ect PortEvent

p o r t s ( l i s t p a n s L i s t ports

signals (l ist

(object Trans "accel" is In terna1 TRUE source ":TOP:on:active:Junction2" t a r g e t ":TOP:on:active: Junctioa7" a c t i o n [object Action

quid "3AfOA4090093" body "setSpeed+=DELTA; " 1

eventGuards ( l i s t eventGuardsList (obj e c t EventGuard

Event (ob jec t PortEvent p o r t s (list p o r t s L i s t

po r t s

s i g n a l s (list

s ignals

(ob jec t Trans "decel" is1nterna.i TRUE source " :TOP:on:active:function8" t a r g e t ":TOP:on:active:function9" ac t ion (ob j e c t Action

quid "3A70A40E032FM

eveatGuards ( l i s t eventGuardsList (obj e c t EventGuard

Event (object PortEvent ports ( L i s t portsList

ports " :Structure :Contro1CommsV)

signals (list signaLsList

s ignals " :base:deccel l ' ) ) ) ) ) ) )

( s b j e c t S t a t e "paused" s t a t e s (list S t a t e s

(objem I n i t i a l P t " I n i t i a l t 1 ) (ob ject JnncPt "Junctionl"

isEntry 'CRUE excernàl lyVis ib le TRüE)

lob j e c t JuacPt " Junction2" e x t e r n a l l y v i s i b l e TRüE)

l o b j e n JuncPt "Junction3" e r t e rna ï lyVis ib le TRUE)))

(objec t JrrncPt " J m c t i o n l " isEntry TRUE

externallyvisible TRUEl (abject JuncPt " Junction2"

extedlyvisible TRITE) (object JuncPt "Junction3"

isEntry TRUE) (obj ect JuncPt "Junction4") (ob j ect JuncPt "JurictionS"

isEatry TRIE) (ob j ect JuncPt " Junction6") (object JuncPt "Junction8"

isEntry TRITE) 1 transitions (list transition-list

(object Trans "disengage" source " :TOP:on:active: Juactionl" target " :TOP :on:paused: Junctionl" eventGuards (list aventGuardsList

(ob j ect EventCuard Event (ob ject PortEvent

ports (lis portsList ports

" :Stnicture:ControlCommsfl) signals (list signalstist

signals ":base:coastM signals ":base:brake"I3)1)

(ob ject Tram "resume" source " :TOP :on:paused: J ~ n a i o n 2 ~ ~ target 'q:TOP:on:active: Junction3" eventGuards (list eventCuardsList

(ob ject EventGuard Event (object PortEvent

ports (list portsList

ports '' :Structure : ControlCommsl')

signals (list signalsList signals ":base:resume")) 1 ) )

(ob j e n Trans "getSpeedm islnternal TRüE source " :TOP : on:Junctioa3" target ":TOP:on:Junction4" action (object Action

quid "3A70973900DD" body

IactuaiSpeed=(intl~rtdata; I I//reset the watchdog timer tif (watchDogTimerID.isVdid0) vatchDogTfmer . cancelTimer(vatchDogTimerID) ;

1 uatchDogTimerID = uat chilogTimer. inf ormIn(RTTimespec(5 - O, O) ) ; 1

eventGuards (list eventGuardsList (ob j ect EventGuard

Event (ob ject PortEvent ports (list portsList

ports

" :Structure : EagineComms "1 s igaa l s (list s igaa l sLis t

s ignals " :base: speed") ) ) 1 ) (object Trans "requestSpeed"

i s h t e r n a l TRUE source " :TOP : on: Junct ion5" t a r g e t ":TOP:on:Junction6" ac t ion (object Act ion

quid "3A70904700AE" body "EngineComs .getSpeedO .sendo ; " 1

eventGuaràs ( l i s t eventGuardsList (object EventGuard

Event lcsbject PartEveat ports ( l i s t por tsLis t

por ts " :Structure : timer") s igna l s ( l i s t s igna l sLis t

s igna l s " :base:tirneout"))))l

(ob ject Tram "setPushed" source ":TOP:on:JunctionS" ta rge t ":TOP:on:active:JunctionS" ac t ion (object Action

quid "3A7094CFO229" body "setspeed = actualspeed;

eventGuards ( l i s t eventCuardsList (ob j e c t EventGuard

Event (ob j e c t PortEvent por ts ( l i s t por t sLis t

por ts " :Stnctue:ControLComs")

s igna l s ( E s t signaZsList s igna l s " :base:set t i ) ) ) ) )

(object Trans "coastReleasedtt source " :TOP :on: paused: Junaion3" t a r g e t " :TOP :on: active: Junction6" ac t ion (object Action

quid "3A7094CB002Em body "setspeed = actudSpeed;

(ob j ec t EventGuard Event (object PortEvent

ports ( l i s t p o r t s l i s t por ts

s igna l s (list s igna l sLis t s igna l s

tt:base:coastReLeasedit) 1)) 1 )) Cobject S ta te

S t a t e s (list S t a t e s (ob j e c t InitialPt " I n i t i a i " ) (object JrrncPt tlJunctionl"

isEntry 'RüE

external lyl l is ible TRUE) (ob j e c t JuncPt " Junction2"

ex te rna l lyv i s ib le TRUE) (object JuncPt "Junction3"

isEntry TRUE ex te rna l lyv i s ib le TFlUE)

(ob j e c t JuncPt "Junction4" isEntry TRE exte rna l lyv i s ib le TRUE) ) )

(ob j e c t JuncPt " Junctionl" isEntry TRUE)

(object JuncPt "Junction2") (object JuncPt "Junction3"

isEntry TRUE) (object JuncPt " Juaction5") )

t r a n s i t i o n s ( l i s t t r a n s i t i o n - l i s t (object Trans "watchDog"

source ":TOP:Junction4" t a r g e t :TOP : e r r o r : Junct ionl" eventCuards ( l i s t eveatGuardsList

(ob j ec t EventGuard Event (ob j ect PortEvent

ports ( l i s t p o n s L i s t por ts :Structure : uatchDogTimer")

s igna l s ( l i s t s ignalsLis t s igna l s ":base:timeout")))))

(ob ject Trans " I n i t i a l " source ":TOP:1nitialw t a r g e t " :TOP :off : Junctionl")

(object Tram "turnOnM source " :TOP :off : Junction2" t a r g e t ":TOP:on:Juxctionl" eventGuards ( l i s t eventGuardsList

(ob j e c t EventCuard Event (object PortEvent

por ts ( l i s t por tsLis t pofts 'l :St~cture:ControlComms~)

s igna l s ( l i s t signalslist s igna l s " : base:onU) ) ) ) )

(oh j e c t Trans "turn(lff If source " :TOP :on : Junctian2" t a r g e t " :TOP :off : Junction3" eventGuards ( l i s t eventGuardsList

(ob ject EventGuard Event (ob ject PortEvent

por ts ( l i s t por tsLis t por ts ":Stnicture:ControlComms")

s igna l s [List s i g n a i s t i s t s igna l s ":base :offw) ) ) ))

(object Trans "tl" source " :TOP : error : Jrrnction2" t a r g e t " :TOP : of f : Junction4" eventGuards (Lis t eventGuardsList

(ob ject EventGuud Event Iobj e n PoaEven t

ports (list porcsList por ts " :St~cture:ControlComms")

signals (Iist signalsList signals ":base:offU))))))

Appendix C

TXL Grammar Files Used For

Parsing RoseRT Capsule Files

This appcnclk shows the T U gamrnar files usecl in the MTS-RT system. The cornplete grnriirriiir

is rtistribiitetl ncross chree files.

C. 1 Grammar file: MarkUpCapsule.grm

%File: MarkUpCapsule.gm %B.M. Graham, Queen's University, 15 May 2001,

include "CapGrammarVieu-p" include " protocol .grm8'

keys 'State 'abject 'states 'statemachine 'State-Machine 'saurce 'target 'Trans 'transitions 'action 'Action 'body 'eventCuards seventCuardsList 'Event 'PortEvent 'Capsule 'PetalRT 'language rcondition 'entryAction 'exitAction 'documentation 'ChoicePt 'sourceRegion

end keys

end compounds

def i n e program [repeat petal-section]

end def i n e

def ine left-mark l < Cid1

end def i n e

def ine right-mark ' >

end def i n e

def ine petal -sect ion [opt petal-header] I [capsule-ob j e c t l I lprotocol,obj ec t l

end def ine

def ine capsule-object [NL] ' ( 'object 'Capsule [ s t r i n g l i t ] [IN]

[NLl [unique-id1 [NL] Crepeat section]

'1 CEXI end def i n e

def i n e s e c t i o n [statemachine] I [ s tmct t l re] I [language-specif i e r ] i [stmcture-diagram] I Cstate,diagraml I [c lass-at t r ibutes] I [col laborat ions- l is t ] I ~ p r o t o c o l ~ r o l e s l I [class,operations] I [uses-relationship-list] I [documentation-blockl I [associat ion- l is t ]

end def i n e

def ine assoc ia t ion- l i s t [NL] 'assocsOwned '( ' list 'assocsOunedList[IN]

[repeat association-ob ject] ' 1 CEXI

end def i n e

def ine association-object CNLI l C'object 'Assoc btringlitl [LUI

CNLI [unique-id1 [NL] [ends-List]

' > CEXI end def ine

def ine uses-relationship-List [NL] 'used-nodes ' ( ' l i s t 7uses-re la t ionship- l is t [IN]

[repeat re la t ionships] ' ) EEXI

end def i n e

def ine re la t ionsh ips CNLI ' ( 'ob j e c t 'Uses-Relationship CINl

Crepeat option1 ' 1 CEXI

end def i n e

dei ine p o r t s - l i s t [NL] ' por t s ' ( ' list *por t sL is t [IN]

Crepeat port-ob ject] 1 CEXI end def ine

define port-object [ml ( lobject 'Por t [ s t r i n g l i t ] [NL] [IN]

[id] [ s t r i n g l i t ] CNL] runique- id1 [repeat option]

' 1 [EX] end def iae

\ textbf{to here u i t h tab)

def ine statemachine INLl 'statemachine ' ( ' ob j e c t ' State-Machine CNL1 CINI

[unique-id] 'S ta te [repeat pseudo-state-obj ec t ] [NL)

% there is an lg one but repeat helps vri t i r ig n i l e s ' 1 CExl

end def ine

de£ i n e state-ob j e c t [Ni.] ' ( ' ab jec t 'S ta te [ s t r i n g l i t ] [IN]

[opt documentat ion-block] [repeat action-ab jea;] [opt s t a t e s - l i s t ] Copt t r a n s - l i s t l

' 1 CEXI end d e i ine

def ine documentation-block CNLI do cumentat ion Cs t r ing l i t l

end def i n e

def ine s t a t e s - l i s t ENLl s t a t e s ' ( ' l i s t 'S ta tes CINI

Crepeat pseudo-state-ob j ec t l ' 1 CExl

end def ine

define object Cpseudo-state-ob ject]

end def ine

define pseudo-state-object [old-pseudo-state-abject] I (Nt] [left-mark] [old-pseudo-state-ob ject] [right-mark]

end define

define old-pseudo-state-object Estate-ob jectl I ~ j u c t i o n ~ o b j e c t l I Cinit id -po in t -abject 1 I [choice-point-ob j ec t l

end def ine

def ine choice-point-object [KL] ( 'abject 'ChoicePt [ s t r i ng l i t ] [IN]

Copt guardl '1 Cwrl

end define

def ine guard [NL] 'guard ' ( lob j ec t 'Guard (IN] CNLl 'expression Cexprsssion-body] CNL] '1 CEXI

end def ine

define expression-body [s t r ingl i t ]

end def ine

def ine junction-ob ject CNL'I ' ( 'object ' JuacPt [stringLit] [IN]

[repeat option] ' 1 CEXI

end def ine

def ine initiai-point-ob jec t [Nt] ' ( 'abject ' In i t i a lP t Cstringlitl [IN] ' 1 CEXI

end define

define t rans- l i s t [NL] ' t ransi t ions ' ( 'list ' t rans i t ion- l i s t [IN]

[repeat trans-ob j ect] '1 cm

end def ine

def ine trans-ob ject [old-trans-ob jectl I Cm1 Clef t - m a r k 3 [o:ald-trans-ob ject] Cright-mark]

end define

define old-trans-abject [NLI ' ( 'object 'Trans Cs t r ing l i t l [ I N ]

[opt source-region] [opt predicate] CNLI 'source Cs t r ing l i t l CNLl ' t a r g e t Cs t r ing l i t l [opt action-ob ject] [opt event-guards-listl

' 1 CEXI end def ine

def ine source-region [NL] ' sourceRegion [number]

end def ine

def ine event-guards-list [NL] ' eventGuards ' ( ' l ist ' eventGuardsList [IN]

Crepeat event-guard-ob j ec t l '1 CEXI

end def ine

def ine event-guard-abject [NL] ' ( ' object ' EventGuard (1 NI

[opt code-condition] [repeat event]

'1 CEXI end def ine

def ine code-condition [ml ' condition Cs t r ing l i t l

end def ine

def i n e event [NL] 'Event ' (job j e c t 'PortEvent [IN]

[repeat ports-and-s ignêls l ' 1 CEXI

end def ine

def ine ports,and,signals Cportsl I Csignalsl

end def ine

define por ts [ml ' por t s ' ( 'list ' por t sLis t CINI

[ml Crepeat single-port1 ' 1 CEXl

end def ine

de£ ine single-port [NL] 'por ts [ s t r i n g l i t ]

end def i n e

define c lass -a t t r ibu tes CNLI ' c iass -a t t r ibu tes ' ( ' l i s t ' c l ass -a t t r ibu te - l i s t CINI

[repeat class-attribute-object J ' 1 CEXI

end def ine

def ine class-attribute-object [NL] ( ' ob j e c t 'ClassAttribute [ s t r i n g l i t ] [IN]

Copt unique-id1 Copt documentation1 Crepeat opt ion1

' 1 CEXI end def ine

def ine class-operations [NL] 'operations ' ( ' l ist ' Operations [ IN]

Crepeat operation-ob ject] '1 CEXI

end def ine

def ine operation-object [NLI ' ( 'object 'Operation [ s t r i n g l i t ] [ IN]

Copt unique-id1 [code-section] [repeat operation-option]

' 1 [EX] end def ine

def ine operation-option [NLl ' v i r t u a l Cbooleanl 1 [NL] ' r e s u l t Cstr ingl i t l 1 CNLI ' v i s i b i l i t y Cs t r ing l i t l I [NL] [parameters-list]

end def ine

define parameters-list [ml 'parameters ' ( 'list 'Parameters [IN]

Crepeat parameter] ' 1 CEXI

end def i n e

def ine parameter [NLl ' ( lob j e c t ' Parameter Cs t r ing l i t l [IN]

[ml ' type Cstr ingl i t l ' 1 CEXI

end def ine

def ine documentation [NLl documentation [ s t r ing l i t ]

end def ine

def ine signals [NI.] ' s igna l s '( 'list 'sigaïsList [IN]

Crepeat signal] ' 1 CEXI

end def ine

def ine s i g n a i [NL] 'sipals [ s t r i n g l i t ]

end def i n e

def ine action-type ' ac t ion I ' entryAct ion I 'exitAct ion

end def i n e

def ine action-ob jec t [NL] [action-type] ' ( 'ob j ec t ' Action [IN]

CNW [unique,idl Copt code-section]

INLI ' 1 CEXI end def i n e

define code-section [ N t ] ' body [ s t r i n g l i t l I CNLl 'body [repeat barJinel

end def ine

def i n e bar-line [NL] [TAB-11 ' I [ s t r i n g l i t ]

end def i n e

def ine doc-bits [ s t r i n g l i t l l Cid1 I J:

1 ' . end def i n e

C.2 Grammar file CapGrammarView.grm

%File: CapCrammarView.grm %B .M. Graham, queenJ s University, April 2001.

def ine petal-header ' ( l o b j e a 'PetaïRT [NLI CINI ' version [number] CNL] ' -u r i t t en Estr ingl i t ] [NI.] 'charSet [number] ' 1 CEXl

end def i n e

def ine language-specifier [NL] 'language [ s t r i n g l i t ]

end def ine

def i n e structure-diagram [NL] ' structureDiagram ' C J ob j e c t Cid1 [ s t r i n g l i t ] [ IN]

[NL] [unique-id] [i tems,list l [opt f i l t e r - f l ags l

'1 CEXI end def i n e

def ine f i l t e r - f l a g s [NL] 'f i l t e r F l a g s [number]

end def ine

def ine i t ems- l i s t ' items ' ( ' l ist Cidl CNLI CINI

[repeat view-ob j e c t l '1 CEXI

end def i n e

def ine view,object [opt id] ' ( 'object Cid1 [opt s t r i n g l i t ] [opt s t r i n g l i t l [unique-id] [NL] [IN]

[repeat viewable-objeml [Nt] 1 [mi CEXI

end def i n e

def ine viewable-object [position] 1 [option] 1 [unique-id1 1 [bound-icon-list] 1 [label-ob jec t l 1 [state-diagram] 1 [vieu-ob j e c t l 1 Cview-tgt-tagl 1 [ver t i ces - l i s t l 1 [id] [coordinatesl

end def ine

def ine v e r t i c e s - l i s t ' ve r t i ces ' ( ' l ist [id] [IN]

[repeat coordinates] ' 1 CEXI

end def ine

def ine view-tgt-tag [NLl Cid1 B Cnumberl

end def ine

def ine label-object

[repeat vieuable-ob j ect] [NL] ' 1 Ex1

end def ine

define bouud-icon-list ' bouadIcons ( ' Es t [id] [NL] [IN]

[repeat viev-ob ject] ' 1 CExl

end def ine

def ine s tructure ENLl ' structure ' ( ' ob jert * CapsuleCollaboration

[ s t r i ng l i t l CNL] [IN] [unique-id1 [opt classifier,roles] [opt ports-List] %mutation change [opt connectors-list]

' 1 CExl end def ine

define connectors-list [NLl ' connectors ' ( ' l is t ' c o ~ e c t o r s L i s t [IN]

[repeat connectorj ' 1 CEXl

end def ine

def ine connector [NL] ' ( lob ject ' Connecter [ s t r ingl i t ] [IN]

[fi] ' connectorCardinality ' constVaZue [str ingl i t ] [ends-list]

' 1 CEXI end def ine

def ine ends-l is t [ml ' ends ' ( ' L i s t 'endsList [ IN]

Cm1 l ends Cst rmgl i t l ENLI ' ends [s t r ingl i t ]

' 1 [EX] end def ine

define classif ier-roles [NL] c lassif ierRoles ( 'List 'classif ierRolesList [IN]

Crepeat capsale,rolel '1 CEXI

end define

define capsule-role [NL] * ( 'ob j ec t ' CapsuleRole [s t r ingl i t ] [ IN]

[NL] ' c l a s s i f i e r [ s t r ingl i t ] CNLl [unique-id1 Crepeat option]

' 1 CExl end def ine

define option Cpredicatel I [labelled-thing] I [set t ingl I [a t t r ibute-set] I [value]

end def ine

def i n e value [NL] 'value ' ( [ s t r i n g l i t ] [numberl ' )

end def ine

def ine a t t r ibu te - se t [NL] ' a t t r i b u t e s ' ( ' l is t ' Attribute-Set [IN]

[repeat a t t r i b u t e ] 1 CEXI

end def ine

def ine a t t r i b u t e [NL] ' ( 'abject ' Attr ibute [ I N ]

[repeat option] ' 1 CEXI

end def ine

def ine labelled-thing [NL] [id] [ s t r i n g l i t ] l [NL] [id] [number] ! [HL] Cid] [id] [ s t r i n g l i t ]

end def ine

def ine s e t t i n g [NLI Cnumberl

end def ine

def ine unique-id 'quid Cs t r ing l i t l I 'quidu [ s t r i n g l i t ] I ' O [number]

end def i n e

def ine coordinates ' ( [number] , [number] )

end def i n e

def i n e pointer-to-id Cidl [unique-id]

end def uie

def ine option-setting Cidl [nnmber]

end define

def ine language-id ' language Cs t r ing l i t l

end def ine

def i n e boolean 'TRUE I IFALSE

end def ine

def i n e posi t ion

POS ( [number] ' , [numberl ' ) end d e i ine

def ine predicate [NL] [id] [boolean]

end d e i ine

def ine named-string [NL] [id] [ s t r i n g l i t ]

end def ine

def ine state-diagram [NL] lstatediagram ' ( 'object 'State-Diagram [ s t r i n g l i t ] [IN]

Diagram" [unique-id1 [i tems-listl [opt f i l t e r - f lagsl

'1 CEXI end de£ ine

C.3 Grarnmar file: protocol.grm

%File: p r o t o c o 1 . p % B.M. Graham, Queen's University, 15 May 2001,

de f ine protocol-abject [ N t ] ' ( 'object ' Protocol [ s t r i n g l i t ] CINI

CNLl [unique-id] [Nt] [repeat section]

' 1 CEXI end def ine

def ine col laborat ions- l is t [NLllcollaborations ' ( ' l i s t lcol laborat ionsLis t

[IN] ( ob j e c t 'Collaboration [ s t r i n g l i t ] [IN] [tinique-id] '1 ' 1 [EX] CEXI

end def ine

def ine protocol-roles [NL] 'protocolRoles I ( 'list JprotocolRolesList

[Nt] [IN] Crepeat protocol-role] '1 CEXI

end def ine

de£ ine protocol-role [NL] ' ( 'object 'ProtocolRole Cs t r ing l i t l

[Nt] CINI Crepeat prot-role-section] ' 1 CEXI

end def i n e

def ine prot-role-section [in-signalsl 1 [out-signalsl

end def i n e

def i n e in-signais CNLl ' i n S i p a l s ' ( ' list ' inSignalsList [ IN]

Crepeat protocol-signal] CNLI ' 1 CEXI

end def ine

def ine out-signals [NL] 'outsignals ' ( list loutSignaisList [IN]

Crepeat protocol,signal] [HL1 ' 1 CEXI

end def ine

def ine protocol,signal [NL] ' ( 'object ' Signai [ s t r i n g l i t ] [IN]

Copt data-type1 CNLI ' 1 [EX3

end def i n e

def ine data-type 'dataType Cstr ingl i t l

end def i n e

Appendix D

RoseRT Capsule Statemachine

Behaviour

This appenciiu contains dctailed <lescriptions of capsule staternachine behaviour as implementeci by thc RoseRT toolset. It is extrncced and adapted from [RoseRT 991.

D.l Top state

Every capsule state machine has a unique cap Ievcl State ttiat is the root of the stnte machine hierarchy. The top state cannot have exit and entry actions. or outer self- transitions.

D.2 Actions

Actions are the things the behavior does ahen a transition is taken. They represent esecutabIe atomic computations that are witten as statements in a det'aii-Ievel prograrnming Iangriage and incorporated into a state machine. They are simiIar to a procedure cd1 in a procedural language- Actions are atomic. in the sense that the- cannot be intempted bu the a m d of a higher priority event. An action therefore runs to cornpietion.

Actions may be composed of nny number of operation caiis. creation or ~Iestruction of otber objects. or the sending of messages to other objects- An action may be attachecl to the foiiowing parts of a state machine:

1. A transition (including a transition to an initiai state)

2. A state. as an eut- action

3. 1 state. as an exit action

D .3 Transitions

,k transition is a relationship betaeen two states: a source state and a clestination state. It specifies thnt wtien an object in the source state receives a specified event ancl certain conditions are met. the behavior will move from the source state CO the destination state.

D.3.1 Kinds of transitions

O Yormal transitions -4 transitions that originates anci terminates on different states.

O Three kincls of self-transitions chat are characterizcd by originating ancl terminating on the same state. having no continuing segments. and not encling on a continuing jiinction point.

1. inner self transitions Where the exit and entry code is not esecutetl for the state on which it originates ancl terminates.

2. Inner interna1 self transitions \Thece the transition esecutes withouc exiting or rcentcring the state in which it is defined. And in addition the exit <and entry actions of iitl states which where esited aricl re-entered are not esecuted. These kincls of transitions are similar to having global operations defincd on a state machine: shen t;iken do iiot chiinge the Stace of the system.

3. Extemal self transitions Where the esit and entry code is esecutctl for the statc on wtiich it origmates and tcnninatcs.

D.3.2 Initial point and initial transition

An initial point is a specid point which e~plicitly shows the beginning of the state triachitie. YCN connect the initial point to a start state. Where the start state ~ i11 be the tirst active state in che nhjects state machine. The transition from the initial point to the start state. initial transition. is

rlL 1 StiLtC ctie Brst transition taken before a y other transition. Only one initial state is allotvctl in P. 4 tliagrani.

:Vote: The trrlnsition jrom fin initial pnint to the sturt state clin have rm wtirm: tiu.u~-

tuer. the other trm.sition /eutirres. inchrrling R pmnl condition rrnd triqger event. rire nnot ulloweh

D.3.3 Transitions to and from the initial state

Only one outgoing transition can be placecl korn the initial point. There can be several incoming transitions to the initiai state. in this case the initiai state acts like a junction point n-hicti forces the behavïor back through the initiai transition. If the initiai transition is usetl to completeiy initialize an object, then any incoming transition to the initiai state d i effectively reset the behavior of an object without haiing to destroy then re-create it-

D.3.4 No initial transition

A state machine does not require an initiai transition. In Bgure D.1. mhen the state machine is created we are in the Tcp (parent) state until the triggeriag event for tL is received. :Vote: The initial state icon is a srnaIl fiiled n'rcle.

l

Figure DA: -\O initial Transition.

D.3.5 Group transitions

Group transitions are transitions from hierarchical states that arc cornmon to al1 the stibstates aitliin that state. Thus conimon behaviour that is normally representecl by equivalent trcansitions tiom every state. can be reprcsented by a single transition originating frorn the containing hierarchical state.

D.4 Choice points

Choice points allow a single transition to be split into two outgoing transition segments. each of whicti c m terminate on a different state. The decision of mhich branch to take is made after thc transition is taken.

Choice points are motiVatcd mainly by practical considerations: it often happeris that the clecisioti on which state to tetminate a transition can only be made following certain calcii1ntions. Each choice point has an associated boolean predicate that is etaiuated after the incoming transition action is esecuterl. Depending on the truth d u c of this predicate. one or the other branch is taken.

D.4.1 Choice Point Example

A very cornmon use of choice points is to count events. That is when the decision of transferring €rom one state to the next depends on the number of events that have occurred. For example. in figure D.?, if a player can on& draw car& once he has received 3 car& from a cIeder. ?ou could mode1 this bchavior the foiiowing way:

The pIayer would keep track of how man? car& he has received and evety time a new carcl is received would test if he has enough cards.

D.5 History hierarchical state machines

The history of a state is dehed as the substate that was the Iast current substate the Iast time the state nias active. In the case of simple states, they are alnays the Iast active state.

Initial .

Draw

AIICards

I

Figure D.2: Choice Point

History is iiseful u-hen dealing Fvith situations where an event cakes control away frorn the ctirrent Stace iuid initiates ir separate behavior sequenee for handling the new event. The netv seqiience can involve nex States and transitions. Hoivever, once cornpleted. ne often want to resunie froni the point hefore the interruption occurred.

D.5.1 Continuation Ends shallow history, deep history and default

iVhen a transition terminates on shallow history, the active substate becornes the niost rec:encly active substate prior to this e n t . Whereas. deep hiscor?; implies retttrning CO Iiistory at al1 state liierarchy levels.

D.5.2 History Example

X comnion use of histor?; is shown in the state machine in figure D.3. In this case transition ce is a self-transition that has a trigger for an event that none of the substates can handle. When thar event occurs the self-transition %<II 6re then go to history. meaning that it uiii revert to the Iast active substate. The effect is to perform event handling n-ithout chan,& the state of the system.

in figure D.3 if the curent active state is 52. then when ee is triggered, the exit action for S2 tt-ill be taken, then the actions for ee d l e-ecute. and ha l ly the entry action for S2 d l he eseciited.

D .6 Transition segments

Transitions that span muitiple hierarchies. thuç cross state boundaries. change conte\% on the frorn the source to the destination state. Therefore they must be partitioned into different segments. Each transition segment has a distinct name' and only the originating segment lias a trigger deherl.

The sum of al1 transition segments is called the transition chain. Transition chains are excciitecl in one single riin-to-completion step.

D.6.1 Junction points

Junction points provide a way of correlating differcnt segments of a transition that sp,m multiple hierarchical state machines. Jiinction points are locatetl on the boundary of statcs. and reprsent cither the source or the destination of a transition segrlent. .Jttnction points are split into those ttiat terminate on the state boundary. history junctions. and those mhich are notations t h continiie rvithin the state. SimiIarIy transitions outgoing from a hierarchicdiy nesteci state are rlivirled into those that terminate on the enclosing state and those that continue Gom the state b o u n d a ~ - to ;L t a r g t state. Every transition, whetlier composecl of multiple segments or of a single se,ment. cventually terminates to a junction point.

D.6.2 Joining transition segments

'in-O or more transition segments can converge on to a single jtinction point. This allon-s multiple trxisitiotis to be definecl. whkh perform the same action in response to an event. and share the s m e destination state. Iu figure D.4 t4 and t9 both terminate on the sanie junctioti point. OnIy the originating transition segments can have cnggers defined. For example. dthough t6 can have actions. it camot have a trigger.

D.6.3 Continuation Kind Property

When a transition terminates on a composite stat e (there are no other transition s continuing from the junction point). the behalor of the state machine a t this point is determineci bu the Continuation kind propecty of the terminating junction point.

This selection specifies the semantics for how the state &tory will be used when there is no continuing transition. There are three options:

O Defaultspecifies that the default (initiai) tr'msition shodtf be taken.

O Historyspecifies rhat the state shouid return to shdow history.

Dcep Historyspccifics that the state should return to tleep history.

menning that ai1 substates also return to histor. This is the behavior for al1 capsule Stace machines. so ic is autornaticalIy selected. The defauit for capsule statc machines is to always go [O deep h i s to r~ so deep history niII bc automatically selected for capsule States. and the selections !vil1 Lie grqerl ririt.

.Junction points are clispla~cd differentlj- depending ahether a scate is shown from an abstracr view. as a substate uE anotlier containing state. or whether stiotvn frotn a cletailet1 vica. the insicte of n composite state. Junction points viewetl from the nbstrnct state view are almays shown .~s a ~aliti dot. Hon*cver. from a detail view üke figure DA. different junctions are shonn with gnphicd clues. Ti~bles D.1 ;uid D.2 describe in cletail the cypes of junctioris shown in figure D.4.

Figure DA: Junction Types

D .7 Transition Selection Rules

Wùen an event is ready to be processed by a state machine. a search for a candidate transition cakes place to determine which one wilI be taken. A transition is said to be euabIed if its trigger is satisfied bu the current event. meaning that the transition has the same event and incerface specified as the current event . and the guard condition evaluates to crue.

The searcli nrder is definecl hy the folloshg algorithrn:

Type of Junction internal

external

external in- corning

external oiit- going

Table D.1: Continuing .Jtinctions.

Example t8 originates fiom an internai junction. The junction is noc visible fiom the abstract x-iem of this state. tT originates from an extemal junction. This junction is visible from the extemai view of the state s'o that it c m eventually be connected to. Cntil an extemai junction hiis been connectecl it behaves like an internal junction. t6 originates From an estemal incoming junc- tion. The label of the transition segment that is attached to che junction is shonm as e6. t6 cannot have n trigger. t.5 teminates on an extemal outgoing junc- tion. The next transition segment is shosn as CS. t5 has ii trigger.

1. The search begins in the innerrnost ciment active state.

Shown as... a small circle on the state bound- a-y

a solid circle connected s i t h a solid line to the container state's b o u n d a .

a solicl circle connected u-irh ;i solicl line to ttic container statc's bniinclary with an a r ro r nt ch^ circle. a solid circle connectecl nich n solicl line to the container state's baiindan with an arrow ac the container state boiinciary.

2. n'ithin the scope of the innermost current active çtate. transitions are cnluatect seqiientiall- if a transition is enabled. the search terminates and the corresponding transition is tnken.

3. If rio transition is enablecl in the current scope. the search in step 2 is repeated for the next tiigher scope. one level iip in the state hierarchy.

4. If the top-Ievel state has heen reachert and no transitions are enablecl. then the current ewnt is ciiscarcled and the state of the behaviour remains unchangeci.

Wiieii a transition is enabletl the algorithni continues with the follonring:

1. If thc enablecl transition is not an internal transition. theii eseciite tlit! esit iictiotis (if al1 substates starting with the deepest history up to the cricrerit scope.

2. If the enabled transition is an internat transition, then none of the substates exit and entry actions are executed.

3. Euecittc the enablecl transition (this includes ail the transition segments) actions. This chairi ultimateiy terminaces on a simple state. Yote that =ecuting the transition may inclutle esecuting other state entcy and exit actions as meii.

4. The termiriating simple state becomes the current active state.

D .7.1 Transit ion Select ion Rules Example

in figure D.5 tL is a group transition outside of SI. IFS11 is the current active state vhen the currenc event enabled transition tl. then the foilowing actions would be evecuted as part of the transition c h a h

L. Exit action for SI1

external r t8 terminates on an internai tenninating junc- tioa. This junction is not visible kom the eu- temal rie%- of chc state.

t7 terminates on an extenid tenninating junc- tion. This junction is visible from the externat tien: O€ the state. If this junction is ever osed to cotlnect another transition segment. it n5ll become a contiming junction. since it Nil1 no longer terminace a transition. e3 teminates on an es~ernai ccrminating junc- cion. This junction is nsiblc From the emcrnal view of the state.

S h o w as... a drclc rc-ith the lctter 'H' (for &tory) connected to the con- tainer states boiinclary n-ith n dotted iine. a circlc a i th the letter 'H' (€or history) connected to the con- tainer states boiindary wicli ii

solid line.

a circle with the letter 'H' (for history) connected to the con- tainer states bauntlary with ;L

SOM line with a r r w irt the circle.

TabIe D.2: Tenninating .Iiinctians (co history).

1. Exit action For S 1

3. Action code For c l

4. Entry action for S L

5. E n t l action For SI1

SI1 rernains the crirrent activc state after the transition is cornplete,

Figure D.5: Transition SeIection Rdes

Appendix E

Capsule Statemachine Diagrams

This appendis contnins stnte diagarns for the seven capsules testetl in this thesis.

E . l Experiment 1: Ce11 Phone Component

1 disconnected

Figure E.1: Ceii f hone TOP state.

active

I

Figure E.2: CeII Phone ConnectetI.

E.2 Experiment 2: Cell from Game of Life

I

Figure €2: Ceii From Game of Life TOP state.

E.3 Experiment 3: TkafEic Light Controller

I I I - ..- , tirneout

1

Figure E.4: Trafic Ligiits TOP Stace.

1

Figure E.6: Trd!ic Lights Flxhing;\mbcr.

timeout

transitioning

I

Figure E.C Trafic Lights Working.

f - amber

*

greenTimeout

Figure E.3: Traffic Lights Transitioning.

# + tirneout2

True greenTheout

I ' #

Figure E.9: Traffic Lights Green.

E.4 Experiment 4: Cruise Control

Figure E.10: Criiise Controt TOP itate.

watchDog

off f

error

t l

resume

-

Figure EU: Cruise Control On.

Figure E.12: Cruise Control Active.

E.5 Experiment 5: Radar Control Unit

I 1

\ init f initiaiiic ' scan ' rwning

l 1 w '. .J

init scan

stopScan

9

Figure E.13: Radar Control Cuit TOP State.

E.6 Experiment 6: Pop Machine

Figure E.14: Pop 4Iachine TOP.

Figure EX: Pop Machine initialize.

Figure E.16: Pop Machine Operating.

E. 7 Experiment 7: Elect ronic Warfare Jammer

Initial {ZU ~2, reset

Figure E.lT: EWS Jammer TOP State.

L5S

Figure E.18: EWS .ranimer \Varking.