multimedia c# support of qos-aware and adaptive

Oliver Lampl, DI (FH)

MULTIMEDIA C#

Support of QoS-Aware and Adaptive Programming by

Extending a General Purpose Programming Language

DISSERTATION

zur Erlangung des akademischen Grades

Doktor der technischen Wissenschaften

Alpen-Adria Universitat Klagenfurt

Fakultat fur Technische Wissenschaften

1. Begutachter: O.Univ.-Prof. Dipl.-Ing. Dr. Laszlo Boszormenyi

Institut: Institut fur Informationstechnologie

Alpen-Adria Universitat Klagenfurt

2. Begutachter: O.Univ.-Prof. Dr. Dr. h.c. Hanspeter Mossenbock

Institut: Institut fur Systemsoftware

Johannes Kepler Universitat Linz

Februar 2010

Ehrenwortliche Erklarung

Ich erklare ehrenwortlich, dass ich die vorliegende wissenschaftliche

Arbeit selbstandig angefertigt und die mit ihr unmittelbar verbundenen

Tatigkeiten selbst erbracht habe. Ich erklare weiters, dass ich keine an-

deren als die angegebenen Hilfsmittel benutzt habe. Alle aus gedruckten,

ungedruckten oder dem Internet im Wortlaut oder im wesentlichen Inhalt

ubernommenen Formulierungen und Konzepte sind gemaß den Regeln fur

wissenschaftliche Arbeiten zitiert und durch Fußnoten bzw. durch andere

genaue Quellenangaben gekennzeichnet.

Die wahrend des Arbeitsvorganges gewahrte Unterstutzung ein-

schließlich signifikanter Betreuungshinweise ist vollstandig angegeben.

Die wissenschaftliche Arbeit ist noch keiner anderen Prufungsbe-

horde vorgelegt worden. Diese Arbeit wurde in gedruckter und elektronis-

cher Form abgegeben. Ich bestatige, dass der Inhalt der digitalen Version

vollstandig mit dem der gedruckten Version ubereinstimmt.

Ich bin mir bewusst, dass eine falsche Erklarung rechtliche Folgen

haben wird.

Unterschrift:

Klagenfurt, 4. Februar 2010

To Iris and my family

Contents

List of Tables vi

List of Figures viii

List of Listings xi

Acknowledgements xv

Abstract xvii

1 Introduction 1

1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.2 Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.3 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

1.4 Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2 Quality of Service 11

2.1 QoS - Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2.1.1 Real-Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

2.2 Constraint Declaration . . . . . . . . . . . . . . . . . . . . . . . . . . 16

2.2.1 Specifying QoS . . . . . . . . . . . . . . . . . . . . . . . . . . 18

2.2.2 QoS Specification Languages . . . . . . . . . . . . . . . . . . . 28

2.2.3 Semantic Correctness of Constraints . . . . . . . . . . . . . . 36

2.3 Constraint Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . 37

i

2.3.1 Q-Compiler - 2KQ+ . . . . . . . . . . . . . . . . . . . . . . . 38

2.3.2 QoSME . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

2.3.3 QoO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

2.3.4 Esterel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

2.4 Adaptation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

2.5 Multimedia Frameworks . . . . . . . . . . . . . . . . . . . . . . . . . 46

2.5.1 Java Media Framework . . . . . . . . . . . . . . . . . . . . . . 48

2.5.2 Direct Show . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

2.5.3 GStreamer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

2.5.4 FFmpeg/FOBS . . . . . . . . . . . . . . . . . . . . . . . . . . 50

2.5.5 Infopipes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

2.5.6 QUINNA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

2.5.7 DJINN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

2.5.8 NMM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

3 Requirements 53

3.1 Quality of Service Specification . . . . . . . . . . . . . . . . . . . . . 53

3.2 Quality of Service Monitoring . . . . . . . . . . . . . . . . . . . . . . 54

3.3 Programming Language Extension . . . . . . . . . . . . . . . . . . . 55

4 Quality Aware Programming 57

4.1 MMC# Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

4.2 MMC# Specification . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

4.2.1 Keywords . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

4.2.2 Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

4.2.3 Constraint declaration . . . . . . . . . . . . . . . . . . . . . . 64

4.2.4 Assignment operators . . . . . . . . . . . . . . . . . . . . . . . 67

4.2.5 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

4.2.6 Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

4.3 MMC# by Example . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

ii

4.3.1 Constraint Declaration . . . . . . . . . . . . . . . . . . . . . . 76

4.3.2 Transformation Rules for QL . . . . . . . . . . . . . . . . . . 77

4.3.3 Exception based QoS Violations . . . . . . . . . . . . . . . . . 80

4.4 Constraint Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

4.4.1 Constraint Splitting . . . . . . . . . . . . . . . . . . . . . . . . 85

4.4.2 Constraint Condition Normalization . . . . . . . . . . . . . . . 86

4.4.3 Constraint Condition Comparison . . . . . . . . . . . . . . . . 87

4.4.4 Constraint Reconstruction . . . . . . . . . . . . . . . . . . . . 88

4.4.5 Exploiting Optimization Potential . . . . . . . . . . . . . . . . 91

4.5 Quality Aware Programming Model . . . . . . . . . . . . . . . . . . . 93

5 Evaluation 97

5.1 Quality Aware Programming Reviewed . . . . . . . . . . . . . . . . . 97

5.1.1 QoS Specification Requirements . . . . . . . . . . . . . . . . . 98

5.1.2 QoS Monitoring Requirements . . . . . . . . . . . . . . . . . . 100

5.1.3 Requirements for Language Extensions . . . . . . . . . . . . . 101

5.2 Alternative Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . 102

5.2.1 Library Implementation . . . . . . . . . . . . . . . . . . . . . 102

5.2.2 Aspect Oriented Approach . . . . . . . . . . . . . . . . . . . . 103

5.2.3 Template Metaprogramming with C++ . . . . . . . . . . . . . 104

5.3 Historization Challenges . . . . . . . . . . . . . . . . . . . . . . . . . 105

5.3.1 History Size . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105

5.3.2 Event Recording . . . . . . . . . . . . . . . . . . . . . . . . . 105

5.3.3 History Initialization . . . . . . . . . . . . . . . . . . . . . . . 106

5.4 Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108

5.4.1 Run-Time Performance . . . . . . . . . . . . . . . . . . . . . . 108

5.4.2 Compile-Time Performance . . . . . . . . . . . . . . . . . . . 110

5.4.3 Impact of Garbage Collection . . . . . . . . . . . . . . . . . . 111

5.4.4 Operating System Impact . . . . . . . . . . . . . . . . . . . . 115

iii

6 Quality Aware Programming Patterns 117

6.1 QoS Monitoring Patterns . . . . . . . . . . . . . . . . . . . . . . . . . 117

6.1.1 Application Scenarios . . . . . . . . . . . . . . . . . . . . . . . 117

6.1.2 Single Variable Monitoring . . . . . . . . . . . . . . . . . . . . 121

6.1.3 Multiple Variable Monitoring . . . . . . . . . . . . . . . . . . 122

6.1.4 Required QoS Monitoring . . . . . . . . . . . . . . . . . . . . 123

6.1.5 Provided QoS Monitoring . . . . . . . . . . . . . . . . . . . . 123

6.2 QoS Requirements of a Video Player . . . . . . . . . . . . . . . . . . 124

6.2.1 Simplistic Model . . . . . . . . . . . . . . . . . . . . . . . . . 124

6.2.2 Model Using Playback Buffer . . . . . . . . . . . . . . . . . . 129

6.2.3 Client-Proxy-Server Model . . . . . . . . . . . . . . . . . . . . 131

6.3 Rated Buffer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133

7 Quality Aware Multimedia Framework 137

7.1 Framework Architecture . . . . . . . . . . . . . . . . . . . . . . . . . 138

7.2 Adaptive Quality Aware Components . . . . . . . . . . . . . . . . . . 139

7.3 QoS Implementation Models . . . . . . . . . . . . . . . . . . . . . . . 141

7.3.1 BET Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141

7.3.2 Link Based Model . . . . . . . . . . . . . . . . . . . . . . . . . 144

7.3.3 Adaptivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145

7.4 Existing Implementation . . . . . . . . . . . . . . . . . . . . . . . . . 147

7.4.1 Component Managers . . . . . . . . . . . . . . . . . . . . . . 147

7.4.2 Receivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150

7.4.3 Transmitters . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150

7.4.4 Transceivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151

7.5 Implementations of Video Players with QoS Costraints . . . . . . . . 152

7.6 Simple Video Player with QoS Constraints . . . . . . . . . . . . . . . 154

7.7 An Experimental Video Player . . . . . . . . . . . . . . . . . . . . . . 155

iv

8 Implementation 157

8.1 Mono Compiler - Customizing C# . . . . . . . . . . . . . . . . . . . 157

8.1.1 Mono Project . . . . . . . . . . . . . . . . . . . . . . . . . . . 157

8.1.2 Structure of the Mono C# Compiler . . . . . . . . . . . . . . 159

8.1.3 Embedding Language Extensions . . . . . . . . . . . . . . . . 162

8.2 QoS Management Framework . . . . . . . . . . . . . . . . . . . . . . 164

8.2.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164

8.2.2 Components and Interfaces . . . . . . . . . . . . . . . . . . . 165

8.2.3 Manually Implemented QoS Monitoring . . . . . . . . . . . . 169

9 Conclusion and Further Work 173

A QoS Management Framework - Extended 177

A.1 History Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177

A.1.1 Exchangeability . . . . . . . . . . . . . . . . . . . . . . . . . . 177

A.1.2 Time Measurement . . . . . . . . . . . . . . . . . . . . . . . . 179

A.1.3 Event History Size . . . . . . . . . . . . . . . . . . . . . . . . 181

A.1.4 Event History Reset . . . . . . . . . . . . . . . . . . . . . . . 182

B Simple Automatic Parallelization 185

B.1 Parallelization - Related Work . . . . . . . . . . . . . . . . . . . . . . 185

B.2 Language Extensions - Parallelizations . . . . . . . . . . . . . . . . . 186

B.2.1 Extended Foreach Statement . . . . . . . . . . . . . . . . . . . 186

B.2.2 Forall Statement . . . . . . . . . . . . . . . . . . . . . . . . . 188

B.2.3 Statement Level Attributes . . . . . . . . . . . . . . . . . . . 190

B.3 Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192

B.4 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192

Bibliography 196

v

List of Tables

2.1 Overview of Used Event Types and Constants. . . . . . . . . . . . . . 21

4.1 Common constraints in multimedia systems. . . . . . . . . . . . . . . 78

4.2 Transformation Rules for QL. . . . . . . . . . . . . . . . . . . . . . . 78

4.3 List of Extracted Conditions. . . . . . . . . . . . . . . . . . . . . . . 86

4.4 List of Normalized Conditions. . . . . . . . . . . . . . . . . . . . . . . 87

4.5 Relation Matrix. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

4.6 Relation-based Actions. . . . . . . . . . . . . . . . . . . . . . . . . . . 90

7.1 Overview of Existing QaMF Component Managers. . . . . . . . . . . 150

7.2 Overview of Existing QaMF Receiver Components. . . . . . . . . . . 151

7.3 Overview of Existing QaMF Transmitter Components. . . . . . . . . 152

7.4 Overview of Existing QaMF Transceiver Components. . . . . . . . . . 153

vii

List of Figures

1.1 Comparison between adaptive quality-aware programming and a com-

mon approach. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.1 Jitter, allowed tolerance for a given rate. . . . . . . . . . . . . . . . . 22

2.2 Throughput, (I) everything at the beginning, (II) constant rate, (III)

everything at the end. . . . . . . . . . . . . . . . . . . . . . . . . . . 24

2.3 2KQ+ Overall Architecture [qco09]. . . . . . . . . . . . . . . . . . . . 40

2.4 Architecture of QoSME[Flo96]. . . . . . . . . . . . . . . . . . . . . . 41

2.5 Example of remote method call in a QuO application [LBS+]. . . . . 43

2.6 Pipes and Filters Patter [GS93]. . . . . . . . . . . . . . . . . . . . . . 46

2.7 Structure of a Multimedia Component [GT95]. . . . . . . . . . . . . . 47

4.1 Comparison between adaptive quality-aware programming and a com-

mon approach. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93

5.1 Performance of Time Assignment Statement. . . . . . . . . . . . . . . 110

5.2 Performance of Time Parameter Passing. . . . . . . . . . . . . . . . . 111

6.1 Video Player Model. . . . . . . . . . . . . . . . . . . . . . . . . . . . 125

6.2 Video Player Model With Buffer. . . . . . . . . . . . . . . . . . . . . 130

6.3 Simple leaky bucket [SN04]. . . . . . . . . . . . . . . . . . . . . . . . 130

7.1 Quality Aware Multimedia Framework Architecture. . . . . . . . . . . 139

7.2 Bounded Execution Time Model. . . . . . . . . . . . . . . . . . . . . 143

ix

7.3 QoS Models. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144

7.4 Constraints of a Video Player. . . . . . . . . . . . . . . . . . . . . . . 154

7.5 Model of a Real-life Video Player Application. . . . . . . . . . . . . . 155

7.6 Video player application GUI. . . . . . . . . . . . . . . . . . . . . . . 156

8.1 Simplified Mono Architecture [Web09i]. . . . . . . . . . . . . . . . . . 158

8.2 Mono C# Compiler Overview [dI02]. . . . . . . . . . . . . . . . . . . 160

8.3 QoS Management Framework Overview. . . . . . . . . . . . . . . . . 165

8.4 QoS Management Class Model. . . . . . . . . . . . . . . . . . . . . . 166

B.1 Parallelization Framework Overview. . . . . . . . . . . . . . . . . . . 190

B.2 Parallelized DCT. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193

B.3 Parallelized DCT Efficiency. . . . . . . . . . . . . . . . . . . . . . . . 194

B.4 Parallelized AVC. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195

x

Listings

1.1 Example of monitoring the rate of 25 elements per second (common

approach). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

1.2 Example of monitoring the rate of 25 elements per second (Quality

Aware Programming model). . . . . . . . . . . . . . . . . . . . . . . . 8

2.1 RateServiceI interface. . . . . . . . . . . . . . . . . . . . . . . . . . . 29

2.2 Example illustrating contract types and contracts. . . . . . . . . . . . 29

2.3 Example illustrating a profile. . . . . . . . . . . . . . . . . . . . . . . 30

2.4 A sample CDL contract - replication. . . . . . . . . . . . . . . . . . . 31

2.5 Sample HQML specification. . . . . . . . . . . . . . . . . . . . . . . . 33

2.6 Application level QoS monitoring. . . . . . . . . . . . . . . . . . . . . 35

2.7 Resource level QoS specification. . . . . . . . . . . . . . . . . . . . . . 35

2.8 A simple QoS monitor in Esterel [BS98]. . . . . . . . . . . . . . . . . 44

4.1 Example of monitoring the rate of 25 elements per second (Quality

Aware Programming model). . . . . . . . . . . . . . . . . . . . . . . . 60

4.2 Type compatibilty. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

4.3 Dynamic and static timed variable declaration. . . . . . . . . . . . . . 64

4.4 Definition of throughput (25 frames per second) combined with jitter. 67

4.5 Mixing Timed and Standard Assignment. . . . . . . . . . . . . . . . . 68

4.6 Timed Write Example. . . . . . . . . . . . . . . . . . . . . . . . . . . 69

4.7 Timed Read Example. . . . . . . . . . . . . . . . . . . . . . . . . . . 69

4.8 Timed Parameter Passing with Named Constraints. . . . . . . . . . . 71

4.9 Timed Parameter Passing with Declarative Constraints. . . . . . . . . 71

xi

4.10 Parameterizable Declarative QoS Constraint specifying frame rate and

jitter. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

4.11 Parameterizable QoS Constraint Instantiation specifying 25 frames/sec

with 10 msec jitter. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

4.12 Extending a Declarative QoS Constraint. . . . . . . . . . . . . . . . . 74

4.13 Using time and non-time based constraints. . . . . . . . . . . . . . . 75

4.14 Example demonstrating constraint declaration variants. . . . . . . . . 76

4.15 Definition of throughput (25 frames per second). . . . . . . . . . . . . 78

4.16 Definition of throughput (25 frames per second) combined with jitter. 79

4.17 Definition of bounded execution time. . . . . . . . . . . . . . . . . . . 80

4.18 Exception based QoS violation. . . . . . . . . . . . . . . . . . . . . . 80

4.19 Exception with QoS parameters. . . . . . . . . . . . . . . . . . . . . . 81

4.20 Declarative QoS Constraint Specifying Delay. . . . . . . . . . . . . . 83

4.21 Declarative QoS Constraint Specifying QoS Requirements. . . . . . . 85

4.22 Reconstructed QoS Constraint. . . . . . . . . . . . . . . . . . . . . . 89

4.23 Weaker Constraint Example. . . . . . . . . . . . . . . . . . . . . . . . 89

4.24 Inconsistent Constraint Example. . . . . . . . . . . . . . . . . . . . . 89

4.25 Monitoring the rate of 25 elements per second (common approach). . 94

4.26 Monitoring the rate of 25 elements per second (Quality Aware Pro-

gramming model). . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94

5.1 Quality Aware Programming Model - basic code structure. . . . . . . 98

5.2 Refinement of Declarative QoS Constraints. . . . . . . . . . . . . . . 99

5.3 Delay of event events. . . . . . . . . . . . . . . . . . . . . . . . . . . . 106

5.4 Minimum and Maximum Delay constraint. . . . . . . . . . . . . . . . 107

6.1 Applying a frame for display. . . . . . . . . . . . . . . . . . . . . . . . 118

6.2 Decoding a frame and preparing it for display. . . . . . . . . . . . . . 118

6.3 A streaming server monitoring network communication. . . . . . . . . 119

6.4 Web-Service providing temperature conversion from Fahrenheit to Cel-

sius. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120

xii

6.5 Web-Service client invoking the temperature conversion service. . . . 121

6.6 Example of single variable monitoring. . . . . . . . . . . . . . . . . . 122

6.7 Example of multiple variable monitoring. . . . . . . . . . . . . . . . . 122

6.8 Example of QoS precondition monitoring. . . . . . . . . . . . . . . . 123

6.9 Example of QoS postcondition monitoring. . . . . . . . . . . . . . . . 124

6.10 Minimal Rate Constraint for Producers. . . . . . . . . . . . . . . . . 133

6.11 Minimal/Maximal Rate Constraint for Consumers. . . . . . . . . . . 134

6.12 Put Method Used by Producers. . . . . . . . . . . . . . . . . . . . . . 134

6.13 Get Method Used by Consumers. . . . . . . . . . . . . . . . . . . . . 135

7.1 BET Constraint Example. . . . . . . . . . . . . . . . . . . . . . . . . 142

7.2 QoS implementation of a component. . . . . . . . . . . . . . . . . . . 142

7.3 Adaptation Code Example. . . . . . . . . . . . . . . . . . . . . . . . 145

7.4 Fault-tolerant Adaptation Code Example. . . . . . . . . . . . . . . . 146

7.5 Using a Component Manager. . . . . . . . . . . . . . . . . . . . . . . 148

7.6 Component Adaptor Interface. . . . . . . . . . . . . . . . . . . . . . . 149

8.1 Manually Implemented Delay Constraint. . . . . . . . . . . . . . . . . 170

8.2 Manually Implemented Delay Monitoring. . . . . . . . . . . . . . . . 171

A.1 QoS History Manager Factory. . . . . . . . . . . . . . . . . . . . . . . 178

A.2 Time Measurement with DateTime.Now. . . . . . . . . . . . . . . . . 179

A.3 Example of Constraint Violation with DateTime. . . . . . . . . . . . 180

A.4 Time Measurement with Stopwatch. . . . . . . . . . . . . . . . . . . . 180

A.5 History Initialization and Event Recording. . . . . . . . . . . . . . . . 181

A.6 Cascading History Reset. . . . . . . . . . . . . . . . . . . . . . . . . . 182

B.1 Foreach Statement. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187

B.2 Extended Foreach Statement. . . . . . . . . . . . . . . . . . . . . . . 187

B.3 DCT Implementation in C#. . . . . . . . . . . . . . . . . . . . . . . 187

B.4 DCT Implementation in C# with the Extended Foreach Statement. . 188

B.5 DCT Implementation in Modula-3 with the Extended Foreach Statement.188

B.6 Forall Statement in C#. . . . . . . . . . . . . . . . . . . . . . . . . . 189

xiii

B.7 Parallel Statements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191

B.8 Parallel Foreach Statement - Extended Foreach Statement. . . . . . . 191

B.9 Parallel For Statement. . . . . . . . . . . . . . . . . . . . . . . . . . . 191

xiv

Acknowledgements

I am deeply grateful to my advisor Prof. Laszlo Boszormenyi for making this thesis

possible. I thank him for his ongoing encouragement, patience and tireless support

during my work on this thesis. During numerous discussions which led to many

ideas his openness and constructive criticism helped me to avoid misleading research

directions. I would also like to thank Prof. Hanspeter Mossenbock of the Johannes

Kepler Universitat Linz for providing valuable input and his willingness to peer-review

this thesis.

Furhtermore, I would like to acknowledge Ewald Petutschnig for implementing

many parts of the multimedia framework.

I am very thankful to my family who motivated and supported me during this

work.

Last but not least, and most importantly, I would like to thank my wife Iris for

her long-lasting patience, support, love and understanding.

xv

Abstract

This thesis investigates how adaptive QoS-aware multimedia applications are devel-

oped and the required support from the programming language and the compiler as

well as the run-time system.

Multimedia applications have become an increasingly important part of our lives.

Many frameworks such as JMF, DirectShow, FFmpeg, GStreamer can help in the

development of such software. All of these frameworks, however, are subject to one

major limitation: they are unable to handle Quality of Service (QoS) issues. Such

frameworks do typically provide timing mechanisms to correctly display multimedia

content, but QoS considerations are shifted away to the resource layer or other mid-

dleware. As a result most multimedia applications work greedily on a best effort basis

and provide no adaptivity.

Providing QoS-awareness and adaptive behavior in multimedia applications is

cumbersome and error-prone. Many different QoS specification languages which try

to support QoS-aware application development have been developed. In many cases

this is done by shifting away QoS processing facilities from the programmer and hiding

them inside predefined components. Instead, QoS can be directly integrated into a

common general purpose programming language like C#.

This thesis introduces the specification of MMC# as an extension to the C#

programming language focusing on adaptive, QoS-aware programming. The new lan-

guage features provide a solution to the three common problems in the field of QoS

processing: (1) constraint declaration, (2) constraint monitoring and (3) providing

adaptive behavior in multimedia applications. Furthermore, declarative definition of

xvii

QoS requirements directly within the programming language allows to apply semantic

constraint analysis - partly at compile-time, partly at run-time - to check the cor-

rectness of specified requirements and further provide optimizations by automatically

removing not required constraints. Furthermore, a new multimedia framework that

includes integrated support for QoS and adaptivity is introduced. This framework

uses predefined QoS metrics and additionally provides mechanisms to freely define

QoS requirements to fit an application’s needs.

xviii

CHAPTER

1 Introduction

This chapter provides a brief overview about the topic of the thesis and clearly outlines

the position in context of related research. Furthermore, contributions of this thesis

are briefly summarized and the structure of this work is described.

1.1 Motivation

Multimedia streaming and play-back is the periodical process of dealing with long

sequences of data (e.g. video frames) under so-called “soft real-time” constraints.

The implementation of this substantial pattern of common multimedia applications

is cumbersome and error-prone. Much research work has been done in the field

of the development of models to understand streaming [APBP02]. Furthermore,

the formal representation of “soft real-time” has led to the development of a huge

number of different QoS specification languages, each focusing on different aspects of

multimedia applications. Complex frameworks and architectures have been proposed

and implemented to map these specifications and observe their compliance with the

system behavior [WGN02]. The implementation of quality aware software is shifted

away by using additional languages and hiding QoS within middle-ware systems. I

suggest, instead of defining QoS and adaptivity as an external aspect of an application,

it should be natively supported by programming languages.

Up to the present, the software engineering community has largely neglected the

CHAPTER 1. INTRODUCTION Page 2

problems surrounding the development of multimedia applications that dynamically

adapt their behavior to both the required Quality of Service (QoS) and to available

resources. QoS and adaptation, of course, have been recognized as important issues

and have been subsequently discussed in the relevant literature, but these discussions

have usually restricted themselves to a limited context and rarely concentrated on

the design issues of QoS aware, adaptive applications [GBCD98, DHT95, WGN02].

Most adaptations are done on the system level - whatever system level in an

actual case exactly means. However, the “system” often does not know what kind of

adaptation is acceptable for the user. It may perform unnecessary actions on the one

side and miss important ones on the other. It is only on the application level that

knows exactly what is tolerable and what is not. Applications should be designed to

be a priori QoS aware (i.e. depending on a set of QoS parameters) and adaptive. The

QoS parameters should not be statically predefined. Rather, the programmer should

be able to define them easily and dynamically.

Current multimedia frameworks are far away from supporting well such kind of

applications. Instead, state of the art multimedia applications are usually greedy;

they simply try to take advantage of as many resources as possible. Even so-called

adaptive applications restrict their adaptive behavior to share resources cooperatively

with different components of the same single application. Applications with unselfish,

cooperative behavior (i.e. they share resources among each other) are almost unheard

of because it is extremely cumbersome to develop such software with current tech-

nology. Neither language support nor QoS-aware frameworks are widely available to

ease this task.

As multimedia technology becomes mature, the design and development issues of

QoS aware, adaptive applications must become a major issue - maybe even one of

the grand challenges. If we want to avoid a crisis, similar to the well-known software

crisis of the late sixties and early seventies, we must be prepared to devote much more

of our attention to these issues than we do today. The first step towards unselfish

behavior is making it easy. A next step might be to enforce it - a task that also needs


appropriate tools.

1.2 Context

This thesis discusses the issue of Quality Aware Programming in the context of mul-

timedia application development which I define as the process of implementing QoS-

enabled software with adaptive behavior. More specifically, the implementation pro-

cess can be divided into three main problems:

1. Constraint definition - specification of quality requirements.

2. Constraint monitoring - evaluation of constraints during program execution.

3. Adaptation - modifying behavior of the application if constraints are violated.

Although the term quality has different meanings in different research areas, in

this context it represents the quality of the provided service, media or application

and is used as an expression for QoS, e.g. frame rate, resolution, delay or jitter. In

the multimedia literature quality does not refer to software quality, but is used to

describe media or transmission characteristics.

Quality Aware Programming is not new and has already been discussed in the

literature [WGN02]. Many QoS specification languages exist [JN04] and even more

frameworks and middle-ware systems [BS98] for applying these. The main focus of the

research community is in applying QoS in distributed multimedia systems primarily

on the communication processes. In contrast, this work concentrates on applying

QoS within the whole application and not hiding it within resource layers. This is

achieved through natively embedding constraint definition, monitoring and adaptive

behavior in a common programming language. QoS specifications languages try to

formally express QoS requirements and adaptation rules in a declarative way without

concentrating on how this behavior can be achieved. In contrast, this work introduces

a programming model which can be used within a common programming language to


implement QoS-enabled applications. It allows declaration of QoS constraints which

represent temporal and non-temporal requirements and their evaluation within the

same programming environment.

Constraint definition is a well-known research topic. Jin and Nahrstedt [JN04]

evaluated QoS specification languages in their work and provide a comprehensive

overview. They also define a taxonomy to classify them. Instead of providing a

completely new specification syntax, I have integrated constraint definition support

within a common programming language, by reusing existing, well-known language

constructs.

Constraint monitoring is the process of evaluating QoS constraints. This recur-

ring process is often hidden in QoS-based middle-ware systems, specialized multi-

media architectures and frameworks, e.g. the Q-Compiler [NWX00, WGN02], the

Sumo project [CB95, BS98] or QoSME [Flo96, Flo94, WYF+00]. Instead of using

such existing approaches, I introduce support for processing QoS requirements on

the programming language level. I show how a programming language can provide

simple facilities to make constraint monitoring as easy as any kind of usual exception

handling.

Adaptation techniques for multimedia data processing is a widely discussed re-

search area. Algorithms have been developed to reduce bandwidth of multimedia

content by dropping frames, reducing color depth, just to name a few. They just

need to be applied within applications. Diot et al [DHT95] have already mentioned

the importance of adaptive behavior in multimedia applications. Their work and the

work of many other authors focuses on limited resources primarily in the network.

All resources, however, should be considered and applications should not use process-

ing power, memory and other resources greedily. Instead, each application should be

implemented to be robust against limitations of any kind of resources. This requires

all applications to be natively adaptive.

Currently QoS implementation is cumbersome and error-prone. Listing 1.1 shows

an example of monitoring the throughput, constrained to 25 elements per second,


implemented in C# [ECM05]. Measurements need to be taken “manually” and data

evaluation is complex and hard to read. QoS constraints are mixed within conditional

statements and cannot be easily exchanged without reimplementing the whole block.

Instead of implementing such QoS monitoring manually, I suggest a small set of

novel language extensions that can be used to provide a clear code structure. Figure

1.1 illustrates how adaptive programming is currently done and how I suggest it to

be done using the Quality Aware Programming model. I separate the code into three

blocks: constraint declaration, application logic code and adaptation code.

1 int n = 0;

2 long oneSecond = 1*1000*1000*10; // in 100- nanosec

3 DateTime [] timeStamps = new DateTime [26];

4

5 while (DataAvailable()) {

6 timeStamps [n % 25] = DateTime .Now ;

7

8 Frame inData = FetchData ();

9

10 if (n >= 25

11 && timeStamps [(n % 25) ]. ToFileTime ()

12 - timeStamps [(n + 1) % 25]. ToFileTime () > oneSecond ) {

13 // constraint violated , do some adaptation

14 } else {

15 // process received data

16 }

17 n++;

18 }

Listing 1.1: Example of monitoring the rate of 25 elements per second (common

approach).

Instead of defining a brand new programming language I have extended the well-

known programming language C#. Although it might be possible to add QoS han-

dling features via a library or aspect oriented programming [Kic], the programming

language integration has got more advantages. It allows adding new features to in-

crease safety and readability of programs. Furthermore, code analysis can be done

to check the semantics of applied operations. Code parts can be optimized by the

compiler to increase performance.


Figure 1.1: Comparison between adaptive quality-aware programming and a common

approach.

Deploying a compiler, however, is more complicated than distributing a simple

library, which might have an impact on the acceptance of the new language features.

Nevertheless, the advantages prevail.

I define three important requirements for extending a programming language

which need to be satisfied:

1. Expressiveness - The new feature enhances the expressive power of the lan-

guage considerably.

2. Safety - The safety of programs using the new feature is enhanced.

3. Optimizations - The new feature enables some automatic optimizations.

To fulfill all three requirements, a set of concepts and tools is required to support

the developer both in terms of the programming language and on the framework level.

1.3 Contributions

In this work I introduce concepts to support the development process of quality

aware software in the multimedia application domain. The main focus is on making


the development of QoS aware and adaptive applications as easy as that of any other

software being able to handle exceptions.

I present the Quality Aware Programming model which defines a new structure for

programming adaptive quality aware applications. Instead of mixing up QoS imple-

mentation - which, in most cases, is a mixture of time measurements and conditional

statements for checking time-based constraints - with business logic it separates the

concerns of constraint definition, application logic and adaptation code. It will be

demonstrated how the application logic can implicitly cover constraint monitoring

without rewriting it for every different constraint.

The programming model is encouraged by a few enhancements on the program-

ming language level. I present an extension of the general purpose programming

language C# [ECM05], called MMC# (MultiMedia C#), which enables:

• the declarative definition of QoS constraints,

• implicit constraint monitoring by applying a new assignment operation called

timed (streaming) assignment,

• compiler-supported, automatic exception generation in the case of QoS viola-

tions.

Importantly, instead of proposing a new language for QoS specification (as many

authors do), I integrate it into a general purpose language, thus making it available to

virtually anybody. The programmer is free to separate essential basic business logic

from issues of adaptation, allowing adaptive behavior to be implemented as part of

the exception handling. In other words, handling of QoS violations is as simple as

that of exceptions.

The declarative definition of QoS specifications enables support for automatic

constraint analysis. This work shows how constraints can be analyzed and com-

pared to each other to improve safety of programs. Correctness checks of constraint

specifications enable identification of unnecessary parts and thus allow optimization


of constraint monitoring by minimizing the number of conditions. This analysis is

implemented within the compiler and executed at compilation time. In cases the

compiler has not got the complete knowledge of constraint parameter values, code is

generated which verifies the constraint correctness at run-time.

Listing 1.2 is a reimplementation of the previous code example (Listing 1.1) ap-

plying the Adaptive Quality Aware Programming model. It uses the concept of timed

data types to implement constraint monitoring. Timed data types add the time as a

n + 1st dimension and allow storing events which are triggered by the timed (stream-

ing) assignment statement. In the presented example (Listing 1.2) the timed variable

inData is attached to the declarative definition of the QoS constraint throughput.

Each time the streaming assignment is executed an event is triggered and stored

within the timing history.

The history of events is the basis for evaluating the constraint. If the constraint

is violated, an exception is thrown and the corresponding exception handler executes

the specified adaptation code.

1 // constraint declaration , timed data type with constraint definition

2 // @ represents the all quantor , n declares an index variable

3 // the array access is used to retrieve timing history data

4 // the applied constraint is frame rate: 25 frames per second

5 Frame [~@n{inData [n] - inData [n -25] < Units.Sec (1) }] inData ;

6


8 try {

9 inData ~: FetchData (); // constraint monitoring


11 } catch (QoSException) { // exception caused in case of constraint violation


13 }

14 }

Listing 1.2: Example of monitoring the rate of 25 elements per second (Quality Aware

Programming model).

To illustrate the advantages of the new model, I introduce patterns of good be-

havior for quality aware programming. These patterns show both the declaration of


different constraints and their handling. I demonstrate how to apply execution time

constraints, throughput constraints or even a free combinations of them. Further-

more, this work shows which constraints are useful within multimedia applications

and which are not required or can be covered by others.

All these findings have influenced the development of a new multimedia framework.

I introduce QaMF (Quality Aware Multimedia Framework), a new component-based

multimedia framework natively supporting QoS that relies on MMC#. Components

are defined quality aware and can be freely connected to each other to build complex

applications. Furthermore, I define patterns that allow QoS aware components to

be connected in a consistent way. In simple experimental applications consistent

connections can be applied easily and more or less automatically. However, in the

general case this task is too complex to be automated, which raises the importance

of patterns of good behavior.

1.4 Structure

This thesis is divided into nine chapters. Chapter 2 describes existing approaches

to specify and monitor QoS. In Chapter 3 requirements for QoS specification and

monitoring are discussed and requirements for extending a programming language are

defined. Chapter 4 introduces the Quality Aware Programming model including the

specification of MMC# and the semantic analysis which is applied to QoS constraint

specifications. Chapter 5 evaluates the novel programming model and points out

introduced overhead and performance of the implementation. Usage patterns for the

new features are presented in Chapter 6. Chapter 7 introduces the Quality-aware

Multimedia Framework (QaMF) which utilizes the presented language features and

the usage patterns. The applied compiler modifications and the run-time system

support are discussed in Chapter 8. Last but not least, Chapter 9 concludes this

thesis and outlines further work.

CHAPTER

2 Quality of Service

This chapter discusses existing research in the field of QoS. Starting with a brief defini-

tion of the term QoS, it deals with related research work in the field of QoS constraint

declaration, monitoring and adaptation. In addition, this chapter presents an over-

view about common multimedia frameworks, including JMF [Web09c], GStreamer

[WT09] and Direct Show [Web09h] with respect to their QoS capabilities.

2.1 QoS - Definition

QoS stands for Quality of Service. This is a general term used in many different areas

to specify quantitative and qualitative aspects of a requested service.

In literature QoS is often associated with temporal and non-temporal aspects of

communication systems. This can lead from reliability and availability requirements

of certain network services to concrete timing aspects of data transfer. In this thesis

QoS represents quality parameters in the field of multimedia systems. These param-

eters cover temporal and non-temporal behavior of multimedia applications as well

as properties of multimedia content.

In case of communication systems, QoS often refers to data transfer providing

some kind of guarantees. Such guarantees can reach from availability contracts to

concrete bandwidth definitions. In many cases, QoS processing of communication

systems is implemented directly within network hardware, like routers which make

CHAPTER 2. QUALITY OF SERVICE Page 12

use of QoS algorithms to provide different levels of quality to different network types.

One way to provide QoS is to use some sort of queuing algorithm. Simple networks

use FIFO (First In - First Out) to process their traffic. In this case, packets which are

received first are also processed first. If different packets need different processing,

e.g. critical packets need to be served urgently, whereas uncritical packets may be

delayed for some time, the queuing algorithm needs to be changed. Lots of different

algorithms exist in this area like Weighted Fair Queuing (WFQ), Random Early

Detection (RED), Traffic Shaping, just to name a few [Tan03].

There are many different definitions of QoS. Frequently the definition is directly

related to communication systems.

“Quality of service is a concept based on the statement that not all

applications need the same performance from the network over which

they run. Thus, applications may indicate their specific requirements to

the network, before they actually start transmitting information data.”

[Flu95]

“Quality of Service indicates the defined and controlling behavior of a

service expressed through quantitative measurable parameter(s).” [Sch01]

Literature provides many different definitions of QoS mainly in the context of

networking. Burgstahler et al present a comparison of the following two definitions

of QoS [BDH+03].

“The collective effect of service performance which determines the de-

gree of satisfaction of a user of the service.” (ITU-T) [IT94]

“A set of service requirements to be met by the network while trans-

porting a flow.” (IETF) [CNRS98]

In contrast to the network view on QoS, I want to concentrate on QoS in multime-

dia applications in general, in particular the integration of QoS processing facilities


into a common programming language. Although I am concentrating on QoS with

respect to temporal behavior, I use the following definitions for the terms QoS and

QoS constraint:

QoS QoS represents the temporal and non-temporal monitorable charachteristics of

a component, application or complete software system and is not limited to a

specific context like networking.

QoS Constraint A QoS constraint represents relations of time variant or time in-

variant properties with respect to certain boundaries.

2.1.1 Real-Time

Multimedia applications require processing of data in real-time. Standardization

institutes define real-time processing in a computer system as follows [SN04]:

“A real-time process is a process which delivers the results of the pro-

cessing in a given time-span.”

Steinmetz and Nahrstedt give a detailed overview of real-time processing in [SN04].

A real-time system operates under certain timing conditions often referred to as

deadline-driven processing. Processing needs to be done until a certain deadline is

reached. Such a deadline represents a point in time which is the latest acceptable time

limit for a certain processing result. In contrast to a “normal” computer system, the

result of a real-time system does not only depend on the computational correctness

but also needs to be available within a certain time-span. If the result is not available

until the deadline is reached, it is useless for the system.

Two different types of deadlines exist:

• Soft deadlines - In case a soft deadline is missed, the system does not produce

unacceptable results. This is the case for multimedia system. If the display of a

single frame cannot be achieved within a certain time frame, it can be dropped.


It is no problem if one single deadline is missed, but too many deadlines should

not be missed. A real-time system dealing with soft deadlines is called a soft

real-time system.

• Hard deadlines - In contrast to soft deadlines, all results need to be available

within a certain time span. If this cannot be reached, severe failures which

might lead to fatal damage can occur. To achieve completely deterministic

timing behavior of hard real-time systems, execution times of single commands

need to be known a priori. Worst case execution times are calculated to get an

understanding of the timing requirements of the applications. In many cases

a complete schedule is generated during the compilation process to control the

timing behavior.

Hard real-time systems are required to provide predictable fast response and a very

high degree on fault-tolerance. Kirsch [Kir02] describes three different programming

models to support the development of real-time applications:

• Synchronous Model - is based on the idea of zero time computation. Re-

actions on events are immediately available. Furthermore, a program is de-

terministic if it provides at least one reaction for each event. The compiler

is responsible for mapping the application to an executional context. A syn-

chronous application always runs at the speed of its context. In many cases,

the compiler supports additional checks to prove the features of reactivity and

synchrony which requires to calculate the worst case execution time. Esterel

[BG92] is a common representative of synchronous languages.

• Scheduled Model - is based on soft-time. In contrast to the previous model,

this is not an abstract time. Soft-time is the required amount of time to finish a

scheduled task. It depends on performance, utilization and applied scheduling

scheme. In order to not miss hard deadlines, the soft-time is required to be lower

or equal real-time. Operations and mechanisms which might lead to unbound

soft-time, e.g. semaphores, are often banned from languages used.


• Timed Model - is based on fixed execution times. The output of programs

or tasks is available after a certain amount of time. In case the computation is

finished earlier, the output is delayed. In case the computation takes too long, an

exception is triggered. This model fits best for embedded control systems which

required low jitter and predictable timing behavior. Checking the property of

time safety which means proving that all tasks finish in time is very difficult as

it requires testing schedulability and calculating execution times.

Kopetz [Kop00, Kop97] examines the design process and development of hard

real-time systems in detail. The software engineering process needs to take into

account the temporal behavior of the hardware which the system will be deployed on.

Kopetz states that a two level design is required to specify component functionality

and the interactions between components. Furthermore, communication needs to

be predictable to successfully implemented distributed real-time systems and fault-

tolerance needs to be provided in general instead of being implemented within each

application separately. After completing the development cycle detailed validation

needs to be applied to validate the behavior of real-time systems. It is required to

know the maximum amount in time which a certain execution takes which is referred

to as worst case execution time. In case of large real-time systems, simulations help to

check correct executional behavior. For very critical parts of real-time systems even

formal verification should be applied to verify correctness properties before deploying

such a real-time system.

Multimedia applications do not have the same real-time requirements as command

and control systems in application areas like factory automation or aircraft piloting

[SN04]. In case of simple video-playback applications, fault-tolerance is less strict

because it is not a severe failure if a video frame is missed. Missing of deadlines should

be avoided and should not occur too often. Time critical operations in multimedia

applications are periodic and much easier to schedule than sporadic ones.

This thesis focuses on soft real-time systems which try to achieve real-time pro-

cessing but do not guarantee any timing behavior. It is the simplicity of the presented


approach which enables a broader field of application compared to hard real-time sys-

tems and allows embedding temporal behavior monitoring within a general purpose

programming language. However, it is not limited to them.

2.2 Constraint Declaration

The specification of QoS requirements is an important task when building multimedia

applications. All requirements need to be formulated to provide correct service be-

havior for the end-user. The end-user is not necessarily interested in technical details

and often wants to specify quality in a non-technical, qualitative form of e.g. high,

middle and low quality [SN04].

Jin and Nahrstedt [JN04, JN] evaluate QoS specification languages in their work

and identified the following main requirements for QoS languages:

• A QoS language should describe quantitative (performance specific) and quali-

tative (behavior specific) requirements as well as adaptation rules.

• The definition of QoS must be declarative in nature.

• The language should be accompanied by automated mapping/compilation to

the underlying system.

Furthermore, they distinguish different layers of QoS.

• User level QoS - high, medium, low.

• Application level QoS - hardware-, platform independent QoS.

• Resource/System level QoS - hardware-, platform dependent QoS.

The user layer is used to define constraints at a very high level, such as low,

medium and high quality. User level QoS covers perceptual QoS parameters divided

into two main areas: perceptive quality and pricing choices. The application layer


is used to define machine readable constraints. These constraints may result out of

a translation of the user quality requirements. These constraints do not require a

concrete knowledge of the underlying resource layer and express media quality, media

relations and adaptation rules. Last but not least, application constraints can be

mapped to resource or system constraints on the resource layer. These constraints

can include e.g. bit rate, task processing time, or scheduling requirements.

In many cases input constraints need to be mapped to constraints which a system

can interpret. Often user constraints are specified in the way: “Give me good quality

for a low price.” An end-user is usually not interested in defining the best resolution

for the output device or bit rate for the connection capabilities. It is also possible

that the end user does not even know anything about these QoS parameters. The

easiest way to deal with user quality constraints is to define service classes like good

quality, medium quality and low quality and map these service classes to concrete

application level constraint definitions [SN04, JN04].

Current research is focusing more and more on QoE (Quality of Experience)

[wik09b, BHHA08]. QoE represents the perceptive user quality and concentrates

on providing the correct mapping from user level QoS to application level QoS. It is

important that the end-user is satisfied after consuming multimedia content. End-

user satisfaction cannot be directly linked to technical QoS parameters. Field studies,

labor experiments, expert panels and literature studies are required to build a knowl-

edge base. This knowledge base can then be used to map between QoE requirements

and technical QoS parameters.

This thesis does not cover the field of QoE although it might be possible to use

the proposed features for QoE-related implementations. I focus on technical QoS

parameters on the application layer and their declaration and monitoring. However,

this work is not limited to the application layer. It is possible to apply the proposed

features to the resource layer as well.

Jin and Nahrstedt [JN04, JN] review a huge set of QoS specification languages

in their work and define five criteria for the evaluation of QoS languages at the


application layer which I reuse as requirements for the specification of QoS constraints:

• Expressiveness - The specification allows to define a wide range of services

and the corresponding adaptation rules.

• Declarativity - The specification formulates requirements without specifying

how they are carried out.

• Independency - The specification should be independent from the functional

code.

• Extensibility - The specification allows to specify new QoS dimensions.

• Reusability - The specification allows reuse and refinement of existing QoS

specifications.

The wide range of QoS specification languages for the application layer ranges

from script-based, parameter-based, process-oriented, logic-based, markup-based and

aspect-oriented languages to object-oriented specification languages. This thesis fol-

lows a completely different approach. Instead of seeing QoS as something external, I

integrate QoS into a common programming language to make adaptive quality aware

programming a fundamental part of modern software development.

2.2.1 Specifying QoS

This section covers temporal and non-temporal constraints and their specification.

Temporal Constraint A temporal constraint represents a relation between time of

occurrence of different events with respect to certain boundaries, e.g. frame rate

or delay.

Non-Temporal Constraint A non-temporal constraint represents relations of time

invariant properties with respect to certain boundaries, e.g. video resolution or

limitation of memory utilization.


Quality Language - QL

The Lancaster University Sumo Project [BS98] uses QoS specifications defined as

annotations on exported interfaces. These annotations specify the offered QoS and the

requirements to the environment. These annotations are formally described with the

Quality Language (QL). QL represents a formal model for reasoning about temporal

QoS constraints. It uses events and their histories of occurrences to check whether a

given constraint can be met or not.

Temporal constraints describe certain conditions with respect to a given time, e.g.

the number of frames within a certain period of time or the amount of bits transmitted

over a network within a single second.

Blair and Stefani define a simple mathematical notion called Quality Language

(QL) [BS98] to describe temporal constraints. This first-order real-time logic uses

event types, events and histories to capture timing behavior.

• Event types - An event type ǫ represents particular state transitions in a

system, e.g. the arrival of a frame.

• Events - An event is the occurrence of a particular event type.

• Histories - A history stores a sequence of events of one particular event type.

It is accessed by using the history function τ .

∀n, |τ(ǫr, n) − τ(ǫr, n − k)| 6 δ (2.1)

The mathematical formula 2.1 expresses the QoS constraint throughput which is

defined as k events per δ time units. Event types are used with subscript letters to

identify a certain event type. In this case, ǫr represents a receiving event. This is

denoted with the subscript letter r. Further examples use different event types like

ǫs (sending event), ǫi (input event) or ǫo (output event).

The variable n represents an index variable which is applied to select the nth event

occurrence. It is used in combination with the all quantor to specify that the defined


condition needs to be satisfied for all events. In the given example, δ defines the

upper limit in time and substitutes a concrete value. If an upper and a lower bound

is required, subscript identifiers are added to δ to distinguish between different values.

The history function τ returns concrete points in time which are represented by

positive number values. These values are taken from a time domain and used to

calculate a time span which is compared to durations defined via the constant δ.

However, QL has also some limitations. It does not support stochastic or proba-

bilistic properties. This has implications to the expressiveness of QL. It is not possible

to express Mean Time Between Failure (MTBF) and Mean Time To Repair (MTTR)

as they express average values used for probabilistic reliability calculations.

This thesis uses QL as a formal representation of QoS constraints. Table 2.1 gives

an overview of event types and constants which are used within the following sections.

In general, QoS requirements can be divided into two groups: temporal and non

temporal constraints. The first group covers all constraints which are time dependent.

The second group covers all other constraints.

Temporal Constraints - Quantitative QoS

Timing requirements can be represented as time based constraints. These constraints

altogether represent a class of constraints which define timing behavior of a system or

a component. The most important constraints are throughput, jitter and latency, but

also error rate or end-to-end latency [SN04, BS98, Tan03, PH05] can be expressed.

This section gives a brief overview of these constraints and their formal represen-

tation.

Latency (Delay) Delay or latency (Figure 2.2) describes the amount of time be-

tween two consecutive events, e.g. the time required for transferring a single frame

over the network or the amount of time required for decoding a single frame out of a

video stream.


Event Type Description

ǫ General event type which represents e.g. incoming frames

or the sending of frames.

ǫr Event type representing the reception of data e.g. incoming

frames.

ǫi Event type representing incoming data.

ǫo Event type representing outgoing data.

ǫerror Event type representing the occurrence of an error, e.g.

constraint violation.

ǫA Event type representing network sending at communication

node A.

ǫB Event type representing network receiving at communica-

tion node B.

Constant Description

δ Constant value describing amount of time, e.g. delay value.

δr Constant value describing amount of time for a certain rate,

e.g. frame rate of one second.

δlow Constant value describing lower time boundary, e.g. mini-

mum jitter value.

δup Constant value describing upper time boundary, e.g. max-

imum jitter value.

Table 2.1: Overview of Used Event Types and Constants.


∀n, |τ(ǫ, n) − τ(ǫ, n − 1)| 6 δ (2.2)

This constraint type can also be used to specify a constant rate by defining the

delay between two events exactly (2.3). In practice it is hard to achieve a really exact

rate without any timing variance.

∀n, |τ(ǫ, n) − τ(ǫ, n − 1)| = δ (2.3)

For multimedia applications it is enough that latency is within an upper and a

lower bound. For video playback, frame presentation delay bounds need to be smaller

than the sensitivity of human sense organs, e.g. eyes and ears.

∀n, |τ(ǫ, n) − τ(ǫ, n − 1)| > δ (2.4)

If the definition of minimum (2.4) and maximum (2.2) delay is combined, a dif-

ferent constraint called jitter can be formulated.

Jitter Jitter (Figure 2.1) represents the allowed time ranges for varying the pro-

cessing delay. A video frame e.g. does not require to be displayed with an exact rate,

but the deviation from the display slot should not be violated too much.

0sec

-�

1sec

Figure 2.1: Jitter, allowed tolerance for a given rate.

There are different definitions of jitter [PH05, Tan03], e.g. the maximum variation

of the delay or the standard deviation of the delay. Mansour and Patt-Shamir [MPS01]

give a more detailed definition of jitter. They define two different jitter types:


• Delay jitter expresses how far off the difference of receiving events from the

ideal point in time when the event should have been received is. This means

comparing received events with an exact periodic sequence.

• Rate jitter defines the allowed interval in time between two consecutive events.

This thesis uses the definition of rate jitter and defines jitter as the maximum

variation of the delay between two consecutive events (2.5). Every further occurrence

of jitter is related to rate jitter.

∀n, δlow 6 |τ(ǫ, n) − τ(ǫ, n − 1)| 6 δup (2.5)

Jitter can be controlled and held within given limits by using buffers, but with

the impact of increasing the overall delay [Tan03].

Throughput Throughput (2.6) defines the number of processed data units per time

unit, e.g. the number of frames displayed per second.

∀n, |τ(ǫ, n) − τ(ǫ, n − k)| 6 δ (2.6)

Figure 2.2 illustrates a common misunderstanding when specifying throughput.

In case of video representation, often certain frame rates are mentioned e.g. 25 frames

per second. In this case, the user expects that the 25 frames are equally distributed

over the time slot of one second, but this cannot be specified with the help of a

throughput constraint. Throughput does not specify the exact position of frames

within a given time slot. To define a constant rate, the delay constraint should be

used.

It depends on the application if a constant rate is expected or not. In case of a

video decoder, output frames are not necessarily produced with a constant rate. When

looking at the example of MPEG [MPE], a distinction between different frame types -

I-, P- and B-frames is made. These different frame types require different processing

capabilities, e.g. bandwidth, CPU-time, or memory. The decoder is generally not


0sec

(I)

1sec

0sec

(II)

1sec

0sec

(III)

1sec

Figure 2.2: Throughput, (I) everything at the beginning, (II) constant rate, (III)

everything at the end.

able to produce output at a constant rate, but the display component of a video

player requires a nearly constant rate. For the human eye it is important to provide a

more or less constant delay between two frames. If the decoder is not able to process

the data with a constant rate, a buffer can help to minimize jitter. Such buffers like

the leaky token bucket buffer [Tan03] convert the varying output of a producer to a

constant rate.

∀n, |τ(ǫ, n) − τ(ǫ, n − k)| 6 δ

∀n, |τ(ǫ, n) − τ(ǫ, n − 1)| 6δ

k(2.7)

Figure 2.7 shows the difference between the definition of throughput and delay.

As delay covers timing behavior between single frames it is considered to be stronger

than throughput. Throughput is the maximum allowed sum of all delays within a

given time unit (2.8).

∀n, |τ(ǫ, n) − τ(ǫ, n − k)| 6 δ,

(

n∑

i=n−k+1

|τ(ǫ, i) − τ(ǫ, i − 1)|

)

6 δ (2.8)


If throughput is defined together with jitter, the timing definition is more strict.

The combined constraint is important for multimedia applications as it expresses soft-

real time behavior and allows small variations of processing rates. For example, a

video uses the constraint of 25 frames per second and an allowed jitter of 5 milliseconds

(2.9). In general, this can be expressed with formula (2.10).

∀n, |τ(ǫ, n) − τ(ǫ, n − 25)| 6 1000msec ∧ 35msec 6 |τ(ǫ, n) − τ(ǫ, n − 1)| 6 45msec

(2.9)

∀n, |τ(ǫ, n) − τ(ǫ, n − k)| 6 δr ∧ δlow 6 |τ(ǫ, n) − τ(ǫ, n − 1)| 6 δup (2.10)

The throughput constraint can be omitted as the jitter constraint limits the delay

of two consecutive events (2.11). If the jitter definition is tight enough, a more or less

constant rate is achieved.

δlow 6δr

k6 δup (2.11)

Bounded Execution Time Another important constraint is the bounded execu-

tion time constraint (Figure 2.12). It is the amount of time required for executing a

processing step, e.g. the time required for decoding a single frame.

∀n, |τ(ǫo, n) − τ(ǫi, n)| 6 δ (2.12)

Error Rate Error rate (2.13) is similar to throughput. In contrast, the error rate

is required to be greater than a given value. Errors are counted and if too many

errors occur within a given interval of time, a defined adaption can be applied, e.g. if

more than 10 errors occur within 20 seconds, adaptation is required. This constraint

type covers a certain amount of errors due to the fact that sporadic errors may occur

which can often be ignored.


∀n, |τ(ǫerror, n) − τ(ǫerror, n − k)| > δ (2.13)

End-To-End Delay End-to-end delay 2.14 is similar to the definition of bounded

execution time. It defines the amount of time (can also be expressed as lower bound)

which is acceptable to transfer data from A to B over a network. In this case, transfer

can also mean processing.

∀n, |τ(ǫA, n) − τ(ǫB, n)| 6 δ (2.14)

Non-Temporal Constraints - Qualitative QoS

In contrast to temporal constraints, this section covers all other constraints which do

not relate to time. In case of multimedia, there are many other constraints which

are required for successful video delivery or presentation, e.g. memory consumption,

CPU utilization, security, media specific properties or pricing.

All resources are limited and thus constraining memory, CPU usage or display size

are important. In case of mobile environments, resource limitations are even worse.

To avoid applications wasting resources greedily, all resources should be constrained.

In this thesis the same mathematical notation to describe non time-based con-

straints is used.

CPU Utilization To constrain the processing power (2.15), a simple variable called

cpuUtilization is used, which contains the current amount of CPU utilization. For

some application it might be interesting to adapt its behavior to reduce CPU utiliza-

tion.

cpuUtilization 6 λ, cpuUtilization = 0..1 (2.15)

Memory Usage Memory is also a limited resource. Especially mobile devices only

have limited memory resources. In this case, the application can adapt its behavior


if it recognizes that it uses too much memory itself or that another application uses

too much memory (2.16, 2.17).

applicationMemory 6 λappMemory, appMemory = amount in bytes (2.16)

freeMemory > λfreeMemory, freeMemory = amount in bytes (2.17)

Video Resolution (Quality) Depending on the presentation device, the display

resolution is also a possible QoS parameter (2.18). The resolution can be limited

for small devices, but also a minimum resolution for very large display devices like

HDTV screens can be required.

(width > λminWidth ∧ height > λminWidth)

∧ (width 6 λmaxWidth ∧ height 6 λmaxWidth) (2.18)

There are many other constraint types addressing different areas like video stream

properties, security, or pricing which can be defined but are not described in detail

in this thesis.

Common QoS constraints in Multimedia

Kim and Nahrstedt define a set of QoS parameters required for MPEG video stream-

ing [KN97]. They distinguish between different QoS types and requirement classes,

e.g. processing requirements, communication requirements, or CPU requirements.

When displaying a video, e.g. an MPEG video stream several QoS parameters are

required to be satisfied. First, a specific frame rate is required. Second, data needs

to be received without too much jitter and, third, a long start up latency should

be avoided. The third parameter is only required at the startup phase of the video.

During playback the first and the second constraint can be combined to monitor

timing behavior (2.19).


∀n, |τ(ǫ, n) − τ(ǫ, n − k)| 6 δ ∧ δlow 6 |τ(ǫ, n) − τ(ǫ, n − 1)| 6 δup (2.19)

2.2.2 QoS Specification Languages

Many different languages exist to describe QoS characteristics of a system or applica-

tion specialized on user layer, application layer or resource layer specifics. In general,

QoS characteristics can be differentiated into required QoS and provided QoS. This

is widely used within QoS-enabled communication systems. A caller defines a set of

QoS requirements which are matched against provided QoS. The process of match-

ing QoS capabilities is called QoS negotiation. After negotiation has been done, a

QoS contract will be created and communication starts. There are many different

languages which allow the definition of QoS and allowed QoS contracts.

Jin and Nahrstedt [JN04, JN] give a detailed overview of many different QoS spec-

ification languages. The following sections briefly describe a subset of the application

level QoS specification languages.

QML

QML (Quality of service Modeling Language) [FK98] was developed at HP labo-

ratories. It is a language to describe QoS properties of software components for

CORBA-based [OMG99] distributed object systems in an object oriented context.

QML offers three main abstraction mechanisms for QoS specification:

• Contract types - represent specific QoS aspects. It defines dimensions which

are required to characterize a QoS aspect. Each dimension which can be of type

set, enumerated or numeric has a domain of values. Performance or reliability

represent common contract types.

• Contracts - represent a particular QoS specification and are an instance of a

contract type.


• Profiles - associate contracts with interfaces, operations, operation arguments,

and operation results.

1 interface RateServiceI {

2 Rates latest (in Currency c1, in Currency c2) raises (InvalidC );

3 Forecast analysis (in Currency c) raises (Failed );

4 }

Listing 2.1: RateServiceI interface.

1 type Reliability = contract {

2 numberOfFailures : decreasing numeric no/year; // number of failures per year

3 TTR : decreasing numeric set; // time to repair

4 availability : increasing numeric ; // availability of the service

5 };

6

7 type Performance = contract {

8 delay : decreasing numeric msec; // introduced delay

9 throughput : increasing numeric mb/sec; // throughput in megabyte per second

10 };

11

12 systemReliability = Reliability contract {

13 numberOfFailures < 10 no/year; // less than 10 failures per year

14 TTR { // upper bound , mean time and

15 percentile 100 < 2000; // variance of time to repair

16 mean < 500;

17 variance < 0.3;

18 };

19 availability > 0.8; // availability of more than 80%

20 };

Listing 2.2: Example illustrating contract types and contracts.

An example of QML specification is illustrated by applying QoS properties to

RateServiceI (Listing 2.1) Interface Definition Language (IDL) service interface.

The QML specification (Listing 2.2) defines the contract types reliability and perfor-

mance. A contract for system availability is expressed based on the contract type

availability. The contract is applied to the interface together with performance re-

quirements restricting the delay parameter, e.g. the analysis call should be performed

in less than 4000 milliseconds using the rateServerProfile (Listing 2.3).


QML describes non-functional properties of software components such as relia-

bility, performance, security and timing. It cannot be executed to implement the

specified QoS requirements. QML is designed to support QoS in general and it is not

designed for a specialized application domain. One limitation, however, exists. It is

able to specify QoS properties at design time, but it does not address the problem

of adaptation. It is not possible to specify what actions need to be taken in case the

current runtime environment is not possible to satisfy the requirements [JN04].

1 rateServerProfile for RateServiceI = profile {

2 // associating reliability requirements

3 require systemReliability;

4 // associating performance requirements

5 from latest require Performance contract {

6 // specifying the allowed delay when calling "latest "

7 delay {

8 // 50% are required to be processed in less than 10 msec

9 percentile 50 < 10 msec;





14 // average processing should be done in less than 15 msec

15 mean < 15 msec

16 };

17 };

18

19 from analysis require Performance contract {

20 // analysis should be done in less than 4 seconds

21 delay < 4000 msec

22 };

23 };

Listing 2.3: Example illustrating a profile.

QDL

QDL (Quality Description Language) is an extension of CORBA [OMG99] functional

IDL. It allows specification of possible QoS states, system resources and mechanisms

for measuring and providing QoS and behavior for adaptation. It is part of the QuO


(Quality Object) framework developed at BBN [LSZB98, ZBS97, LBS+98].

1 contract repl_contract(

2 ...) is

3

4 negotiated regions are

5 // Low_Cost region definition including reality regions

6 // and the corresponding transitions

7 region Low_Cost : when ClientExpectedReplicas == 1 =>

8 reality regions are

9 region Low : when MeasuredNumberReplicas < 1 =>

10 region Normal : when MeasuredNumberReplicas == 1 =>

11 region High : when MeasuredNumberReplicas > 1 =>

12 transitions are

13 transition any ->Low : ClientCallback. availability_degraded ();

14 transition any ->Normal : ClientCallback. availability_back_to_normal ();

15 transition any ->High : ClientCallback. resources_being_wasted ();

16 end transitions ;

17 end reality regions ;

18 // Available region definition

19 region Available : when ClientExpectedReplicas >= 2 =>

20 reality regions are

21 ...

22 transitions are

23 ...


25 end reality regions ;constraint correctness

26 // transitions between Low_Cost and Available regions

27 transitions are

28 transition Low_Cost -> Available :

29 ReplMgr .adjust_degree_of_replication (ClientExpectedReplicas );

30 transition Available -> Low_Cost :

31 ReplMgr .adjust_degree_of_replication (ClientExpectedReplicas );


33 end negotiated regions ;

34 end repl_contract;

Listing 2.4: A sample CDL contract - replication.

QDL consists of a set of three languages:

1. Contract Description Language (CDL) - is used to specify a QoS contract

between a client and an object.


2. Structure Description Language (SDL) - is used to specify structural as-

pects. These include adaptation and strategies based on QoS measured in the

system.

3. Resource Description Language (RDL) - is used to abstract the physical

resources used by an object.

QoS is specified as an external aspect of the application and can be defined sepa-

rately to the functional code. It represents an aspect-oriented approach.

Listing 2.4 illustrates an example of CDL contract [LSZB98]. The contract con-

trols replication for a QuO application. The client has two possibilities: either request

one replica (Low Cost region) or more than one (Available region). QDL uses regions

to define QoS states and transitions which allow switching between these regions.

ClientExpectedReplicas and MeasuredNumberReplicas are system condition

objects. They indicate the client’s expected number of replicas and the actual num-

ber of replicas respectively. Reality transitions notify the client about changes in

the number of replicas. When a transition is executed, a client callback can be trig-

gered, e.g. if the value of MeasuredNumberReplicas changes to 1, the client method

availability back to normal is called.

QDL is expressive, declarative, independent and extensible, but there is no facility

for reuse. Although QML has several refinement possibilities, QDL is more expressive

than QML [JN].

HQML

HQML (XML-based Hierarchical QoS Markup Language) [GNY+02] is a QoS lan-

guage based on the XML standard [Web09l] for web based multimedia applications.

It is designed to specify requirements for three different levels:

• User level - is used to define user quality levels (e.g. high, low, average). These

QoS specifications are used to find a match between user requirements and the

QoS properties which the system provides.

• Application level - specifies all kinds of application level QoS properties (e.g.

frame rate, size). Such QoS specification is used by QoS proxies to provide QoS

support for the communication channels.

• System resource level - is used to specify resource requirements (e.g. memory,

CPU)

HQML is used to correctly provision all components within a communication net-

work to reserve the required resources to provide requested QoS properties. Further-

more, HQML allows to specify adaptation rules to reconfigure components on QoS

violations. Although it is expressive and very easy to use, it lacks of reuse facilities.

1 <AppConfig id = "100 ">

2 

3 <CriticalQoS type = "frame rate">

4 <Range unit = "fps ">

5 <UpperBound > 40 </UpperBound >

6 <LowerBound > 30 </LowerBound >

7 </Range >

8 </CriticalQoS >

9 ...

10 

11 <ReconfigRuleList >

12 

13 <ReconfigRule >

14 <Condition type = "Bandwidth ">

15 very low

16 </Condition >

17 <ReconfigAction type = "switch to" >

18 101

19 </ReconfigAction >

20 </ReconfigRule >

21 </ReconfigRuleList >

22 </AppConfig >

Listing 2.5: Sample HQML specification.

Listing 2.5 illustrates a short example of specifying an application using a frame

rate from 30 to 40 frames per second. Additionally, a reconfiguration block is defined.

In case the bandwidth is very low, an alternative configuration with id 101 is applied.


XQoS

XQoS [EGP+02] is another QoS specification language based on XML. This language

tries to provide global and evolving description of the QoS needs of applications. It

aims to be expressive, portable, scalable and modular. It describes QoS parameters

by using the following elements:

• Synchronization and timing elements - These elements are intended to

model parallel and synchronous data flows.

• Flow description elements - These elements define the type of media flow

and required QoS properties.

These elements can be combined to specify all flows within the communication

system and their required QoS properties. This specification is then applied within

the transport and network services to provide the specified QoS properties. There is

no way to specify adaptation rules for QoS violations.

QuAL

Florissi developed a process-based QoS specification language called QuAL (Quality

of service Assurance Language) [Flo96, Flo94]. It is based upon Concert/C [AGG+94],

an extension of the programming language C [KR88].

QuAL provides support for handling application specific and resource specific

QoS metrics. It is part of the QoSME (Quality of Service Management Environment)

which is a QoS framework providing QoS-aware operating system calls and QoS-aware

sockets.

Listing 2.6 illustrates an example which monitors the inter-arrival delay of the

incoming video frame at a certain point. It uses the function qual monitor to mon-

itor the QoS metric frame rate. QoS violations are automatically detected by the

runtime system and the application is informed. Application level QoS specifica-

tion is implemented manually by evaluating QoS measurements taken from the QoS

run-time.


1 /* Definition of a threshold for the difference between the arriving time of

2 consecutive messages . At 30 frames /s, the difference must not exceed 1/30 s. */

3 #define THOLD 1/30

4 /* Definition of a QoS Metric function */

5 double inter_arrival_delay(double1 start , double end , qos_ppp *profile )

6 {

7 double j = 0;

8 for (int i = 0; i < profile ->size; ++i) {

9 /* Check if difference between arriving times exceeded threshold . */

10 /* Assume there is no permutation or loss. */

11 if((( profile -> signatures [i+1].ta - profile -> signatures [i].ta) * 1000 ) > THOLD )

12 ++j;

13 }

14 return j;

15 }

16

17 main()

18 {

19 ...

20 /* Trigger monitoring of inter_arrival_delay for inport video */

21 qual_monitor( inter_arrival_delay , 5, 1, video );

22 ...

23 }

Listing 2.6: Application level QoS monitoring.

1 realtm {loss NULL; // no losses are allowed

2 rate sec 10 - sec 25; // mean rate is 10 to 25 elements per second

3 jiter ms 33, nocoercion ; // a maximum jitter of 33 milliseconds is allowed

4 recovery sec 3;} // any recover must take less than 3 seconds

5 receiveport // port declaration keyword

6 {image_t } // message type which is exchanged

7 image_input ; // port identifier

Listing 2.7: Resource level QoS specification.

QuAL extends Concert/C port concept to specify real-time requirements for net-

work communication. Listing 2.7 illustrates such a specification by defining the re-

quirements for an image input port.

One major drawback is that the QoS specification is spread over functional code

and the language is more imperative than declarative, but it is expressive and easy


to use.

2.2.3 Semantic Correctness of Constraints

The declarative definition of QoS requirements demands syntactic and semantic cor-

rectness. Therefore, formal analysis of QoS constraints should be an essential part

of each QoS management system. The system should be able to check QoS defini-

tions for their correctness and consistency. In many cases consistency checks are used

within the negotiation process of QoS. It is important that QoS constraints can be

translated to allow relations between QoS requirements and provided QoS. Constraint

parameters need to be semantically consistent over compared constraints [SN04].

One approach to check consistency of QoS annotations is presented by Blair and

Stefani in [BS98]. They use QL (Quality Language) to apply formal analysis. QL is a

first-order real-time logic which allows definition of weaker and stronger relations be-

tween constraints and consistency checking of provided and required QoS constraints.

It is possible to check if a constraint formula is logically satisfiable by checking the

logical validity of the negation of the formula. If the negation is valid, the original

constraint specification is inconsistent. This check proves that a given formula is

logically satisfiable but does not mean that the QoS requirements can be met. How-

ever, there are several limitations. QL does not support specification of stochastic

or probabilistic properties, e.g. QL cannot express Mean Time Between Failure and

Main Time To Repair.

Gu and Nahrstedt provide consistency checks within QoS environment configu-

ration [GN00]. This is achieved by defining application dependency graphs with the

help of a visual editing environment using HQML. They define a context free gram-

mar which allows correct specification of applications. Their work focuses on the

consistency of the whole system specification and not on particular QoS constraints.

The Q-Compiler meta-data compilation [WGN02, WNGX01] also supports con-

sistency checking. It compares QoS parameters of connected components to provide


consistent setup configurations within a distributed environment. However, this con-

sistency checking focuses more on consistent parameter translation when applying

user-level to application-level constraint parameter conversion.

Leue shows how formal verification of QoS requirements can be applied to Speci-

fication and Description Language (SDL)/ Message Sequence Chart (MSC) specified

systems by adding support for temporal logic [Leu94]. He illustrates the possibility

to apply a proof system for the particular timed logic calculus or model checking to

verify that a given system satisfies a set of QoS requirements.

As this thesis integrates QoS programming facilities within a common general

purpose programming language, it is important to analyze constraint definitions. The

syntactical correctness is automatically provided by the compiler. The challenge lies

in evaluating the semantical correctness of single constraints and detect optimization

potential.

2.3 Constraint Monitoring

Constraint monitoring is the repeatedly executed process of acquisition of informa-

tion about the state of the application and the evaluation of specified requirements

which are expressed as constraints. This includes performance information as well as

resource utilization.

This section gives a brief overview of multimedia middleware systems and the

implementation of constraint monitoring within these systems. It only covers a small

subset of current architectures and implementations but emphasizes different imple-

mentation strategies for QoS monitoring.

All these middleware systems focus on constraint monitoring of network commu-

nication and do not provide general QoS processing capabilities. Instead, this thesis

presents a novel approach to handle QoS directly within a common programming lan-

guage and without applying it to a specific context, e.g. the communication process.


2.3.1 Q-Compiler - 2KQ+

Q-Compiler [WGN02, WNGX01, qco09, WN01] is a programming framework for

quality-aware ubiquitous multimedia applications. The programming framework con-

sists of the following four parts:

• High level application specification

• Meta-data compilation

• Binding

• Run-time meta-data execution.

First, the developer needs to create an abstract application specification. The

creation of a complete high level application specification requires the following three

steps:

1. The developer creates a detailed application specification containing a func-

tional dependency graph of involved components, possible setup configurations,

component descriptions and connection descriptions.

2. A user to application specific quality translation template is created, which maps

between user level QoS requirements and application-specific QoS categories and

dimensions.

3. Adaptation rules are specified by defining actions which are executed on specific

events.

After the completion of the high level specification it can be compiled. The meta-

data compilation is executed in two different steps. First, an environment independent

translation is done. It compiles the high level application specification into QoS-

aware application descriptors. It checks the correctness of each setup configuration

and associates the appropriate user to application QoS mappings. Then it checks


the consistency between two connected components within each setup configuration.

It preconfigures existing generic QoS-enabling services with the help of a rule base.

The output is a portable meta code in XML format. This application descriptor is

independent of the environment.

Second, an environment dependent translation is executed. It is responsible for

customization and deployment. The translation is done in a dynamic and distributed

way, based on run-time meta-data execution:

1. All generic services are substituted by specific quality-aware configurations.

2. These configurations are automatically translated into QoS-enabling services.

3. Setup cost and running cost for each configuration are estimated.

4. Adaptation rules are compiled into an adaptation control script.

The output is the environment dependent, XML-based application descriptor

QoSCASpecs (QoS-aware component-based application specification).

The binding phase helps the developer to bind components into executable code.

Code is instrumented partially automatically or fully automatically depending on

the type of component. The code is parsed, and predefined tags or system calls like

socket calls are replaced by API calls to QoS-enabling services. After instrumentation

the code is rebuilt. This is done by selecting a target machine, downloading the

instrumented code and running the specific compilers (e.g., C/C++, Java). After

compilation the modified component is uploaded into a component repository. Having

the specific quality-aware configuration updated, the component is ready to use within

the deployment environment. The binding phase is responsible for adding API calls to

the QoS management framework to update QoS information and performance data.

It is the basis for QoS monitoring which is executed during run-time.

Run-time meta-data execution is applied on a component based reconfigurable

middleware 2KQ+ [WNGX01]. The middleware is required to provide functionality

of instantiation, management, control of quality-aware applications. The system is


Figure 2.3: 2KQ+ Overall Architecture [qco09].

running over Gaia services [RHR+01]. Gaia is a distributed operating system for ubi-

quitous smart room environment. Several services are required within the distributed

environment to successfully execute applications:

• Configuration selection service - selects the best configuration among ex-

isting setup configurations.

• Location discovery service - helps to find specific target machines.

• Instantiation service - instantiates components/applications on target ma-

chines.

• Registration service - registers instantiated components within a directory

service.

• Distributed environment monitoring service - provides information about

resource availability, mobility of users and devices in specific environment. This

part provides data for QoS monitoring.

• Adaptation management service - applies the adaptation control script and

modifies the behavior of the components/applications in the environment.


Figure 2.4: Architecture of QoSME[Flo96].

The run-time system monitors QoS with the help of profilers and executes adap-

tation control scripts in case of QoS violations. Figure 2.3 illustrates the overall

architecture of the 2KQ+ framework.

2.3.2 QoSME

QoSME (Quality of Service Management Environment) [Flo96, Flo94, WYF+00] in-

troduces additional programming language constructs within Concert/C [AGG+94]

to support monitoring, analysis and adaptation to QoS delivery. It uses QuAL (Sec-

tion 2.2.2) as QoS specification language and provides a single set of abstractions to

QoS Management (Figure 2.4):

• QoS in OS (QoSOS) - is a unified abstraction of the underlying operating

system.

• QoS in Sockets (QoSockets) - is an abstraction of communication sockets.

Automatic QoS monitoring is implemented with the help of QoS Management

Information Bases (QoS MIBs). These QoS MIBs store statics about QoS behavior


of the system. Specified constraints are checked with the help of QoS MIBs. In case

of violations, application handlers are invoked upon QoS degradation. This allows

implementation of adaptive behavior.

QoSME frees the application developer from QoS monitoring instrumentation

by automatically generating instrumentation when QuAL applications are compiled.

QoS MIBs keep applications informed about the QoS delivery performance of the

system. Furthermore, QoSME integrates QoS management facilities within stan-

dard network management frameworks. QoS MIB data is available through SNMP

[CFSD90, MR90, RM90].

2.3.3 QoO

QoO (Quality Objects Framework) [LSZB98, ZBS97, LBS+98] is a framework for de-

veloping distributed applications with QoS. It is a CORBA [OMG99] based approach

and provides the ability to specify, monitor and control aspects of QoS in applications.

QoO adds additional components to the CORBA, IDL concepts:

• Contracts - QDL (Quality Description Language) is used to specify QoS re-

quirements and adaptation rules.

• Remote Object Wrappers - QoS behavior and adaptation is handled via

remote object wrappers which have identical interfaces as the original objects.

• System Condition Objects - Additional system objects are spread over all

components to measure and control QoS parameters.

Figure 2.5 illustrates a remote method invocation. QoO hangs in between and is

completely transparent to the client. The QoO kernel [VZL+98] is invoked by each

remote method invocation and determines current QoS region specified with QDL. If

a region transition is done, a callback is executed.

Furthermore, the QoO framework defines three different development roles:

• Application developer - is responsible for functional code.


Figure 2.5: Example of remote method call in a QuO application [LBS+].

• QoS developer - is responsible for QoS requirements and their correct spec-

ification with QDL. This includes the definition of possible regions and their

transitions including adaptation rules.

• Mechanism developer - is responsible for system components, wrappers and

ORB functionality.

2.3.4 Esterel

Another interesting way for implementing QoS monitoring is using the programming

language Esterel [BG92, BS98]. Esterel is a synchronous and imperative program-

ming language providing high level constructs defining relationships between events.

Temporal execution semantics are clearly defined and programs are deterministic.

An Esterel program consists of parallel processes which are executed synchronously

and communicate to each other through signals. Blair and Stefani use Esterel to

implement reactive objects. These objects monitor QoS and can be automatically

generated out of QoS requirements formulated in QL (Quality Language) [BS98].


Listing 2.8 illustrates the implementation of a simple QoS monitor. When the

code is executed, it waits for events. If data is available, which is notified via the

data available event, the data needs to be consumed within a given time range. If

the event data consumed does not occur before a predefined timeout, a QoS violation

signal is emitted. The loop execution ends when a stop signal arrives.

1 module DeviceMonitor:

2 type QoS_violation_type;

3 constant DELAY , TOLERANCE : integer ,

4 QOS_MISSED_DEADLINE : QoS_violation_type;

5 input data_available , data_consumed , tick , stop;

6 output QoS_violation( QoS_violation_type);

7

8 trap terminate in

9 loop

10 await data_available;

11 do

12 await data_consumed

13 watching DELAY + TOLERANCE tick

14 timeout

15 emit QoS_violation(QOS_MISSED_DEADLINE )

16 end %do

17 end %loop

18 ||

19 await stop; exit terminate ;

20 end %trap;

Listing 2.8: A simple QoS monitor in Esterel [BS98].

The Sumo project [CB95, BS98] uses Esterel to implement reactive objects used

for QoS monitoring. It is a micro kernel based system to apply communication and

processing support for real-time and multimedia applications using the Chorus micro

kernel [RAA+91, BGG+91]. It mainly consists of two parts:

• Sumo-CORE - is a micro kernel based environment specialized on real-time

support for multimedia applications emphasizing communications, scheduling

and memory management. It includes QoS management functions to provide

QoS guarantees in heterogeneous environments.


• Sumo-ORB - is an open distributed processing environment for telecommuni-

cation services and applications. It extends CORBA with real-time functionality

to support distributed multimedia applications.

2.4 Adaptation

Adaptation is the process of changing the behavior of an application in reaction to

certain circumstances. In multimedia applications these circumstances are resource

limitations. To implement adaptation, information about the current state of the ap-

plication (e.g. memory consumption, CPU utilization, network bandwidth, external

events) needs to be monitored. With the help of the gathered information, a decision

can be made if and what type of adaptation is required.

Diot et al [DHT95] already mentions the importance of adaptive behavior of mul-

timedia applications and presents two types of adaptations which are possible for

almost any multimedia application.

• Delay adaptation - If the deviation of communication delays is known, a

playout buffer can be correctly configured and continuous playback is possible.

• Throughput adaptation - In case of network congestion or transmission er-

rors, the required throughput can be decreased by using specialized coding

techniques like hierarchical encoding of video streams or variable compression

techniques which modify the compression rates according to the available band-

width.

Although there are some applications which definitely require resource guarantees,

the mass of applications should be implemented with adaptive behavior, which is not

the case today. Current software systems only cover QoS aspects on the network

layer and allow adaptation only for communication by reducing bandwidth or layered

encoding. In many cases the amount of data is reduced to ensure that the required

quality can be provided at the destination site.


Figure 2.6: Pipes and Filters Patter [GS93].

Many adaptation techniques exist to manipulate video and audio quality. These

algorithms are not dealt within this thesis. This thesis illustrates how adaptations are

applied by defining exception handlers. The programmer implements separate code

to react on special circumstances. This thesis demonstrates adaptive programming

in general and does not discuss specific adaptation techniques.

2.5 Multimedia Frameworks

Multimedia applications have become an increasingly important part of our lives.

Many frameworks such as JMF [Web09c], DirectShow [Web09h], FFmpeg [Web09a],

GStreamer [WT09] can help in the development of such software. All of these frame-

works, however, are subject to one major limitation: they are unable to handle Qual-

ity of Service (QoS) issues. Such frameworks typically provide timing mechanisms to

correctly display multimedia content, but QoS is shifted away to the resource layer

or other middleware. As a result, most multimedia applications work greedily on a

best effort basis and hardly provide adaptivity features.

Many current multimedia frameworks are built around the pipe-and-filter pattern

[GS93] as it is typically the most suitable pattern for processing streams of data

(Figure 2.6). It consists of filters which are plugged together with pipes. Each filter

is independent and simply transforms incoming data from one or more input streams

to one or more output streams. This pattern allows to distribute different processing

steps into independent components.


Figure 2.7: Structure of a Multimedia Component [GT95].

Gibbs and Tsichritzis identify the fundamental requirements for multimedia frame-

works [GT95]. Component based multimedia frameworks require at least three main

types of components:

• Producers - create data.

• Consumers - receive data.

• Transformers - processes data.

Components are connected via ports (Figure 2.7). A producer has only outgoing

ports, a consumer only incoming and a transformer has incoming and outgoing ports.

These ports are used to transport media data through a component chain.

Most of the current multimedia frameworks lack QoS support or if it is available,

it is hidden inside the system components and programmers have no or little explicit

control over it. Such frameworks shift QoS to the resource layer, which is responsible

for providing the data in time. There is no other possibility for the application to

react on QoS violations than not displaying a frame. This results in the common stuck

frame and display judder phenomena during playback. Some frameworks require the

display component to conduct QoS monitoring to ensure correct rendering behavior.


These frameworks do not allow different components to be monitored, either. Other

frameworks support QoS via resource reservations.

Most of the frameworks developed within research projects claim to support QoS,

but implementations are often complex and adaptive behavior is left behind. More-

over, widely used multimedia frameworks like JMF, FFmpeg or GStreamer do not

provide QoS programming facilities at all.

2.5.1 Java Media Framework

Sun’s Java Media Framework (JMF) [Web09c] is an extension to the Java SDK and

provides an object oriented, platform independent framework for real-time storage,

transformation and display of multimedia content [ED04]. It allows processing of time

dependent data like video and audio, it is very easy to use but lacks QoS support. To

provide real-time processing, it uses its own time model, which allows the definition

of clocks for each media object. Each component has to provide its data in time

and, if a time frame is violated, the current data is dismissed. Components - also

called transformers - are subject to a particular execution time frame. For example,

the maximum delay of a processing chain for 25 frames per second is limited to 40

milliseconds.

The great disadvantage is that this functionality is hidden inside the framework

and the programmer cannot easily define specific timing requirements.

2.5.2 Direct Show

Microsoft’s Direct Show [Web09h] is an alternative to JMF for Windows platforms.

It is the successor of Video for Windows (VfW) and is based on the pipe-and-filter

pattern using Microsoft’s COM (Component Object Model). Three different compo-

nent types (source filters, transform filters and rendering filters) are used to build a

processing chain.

QoS is handled by notifications from the rendering filters. If too much or too

little data is received within a given time period, a notification message is sent to the


previous component in the chain. This message is passed on until a component reacts

to it. Each QoS notification message contains four different values:

• Type - Defined by the QualityMessageType enumeration; either Famine, indi-

cating that the filter is receiving too little data, or Flood, indicating that the filter

is receiving too much data.

• Proportion - The requested adjustment in the data rate, from a baseline of

1000. For example, 750 indicates 75% and 1500 indicates 150%.

• Late - Reference time indicating how late the most recent sample arrived. The

value is negative if the sample arrived early.

• TimeStamp - The timestamp of the most recent sample.

A default implementation is provided by the framework which can be customized

by implementing additional quality control interfaces.

However, it is very time consuming to get familiar with the complex architecture

and the heavy mass of documentation available for interfaces, methods and their

flags. Direct Show is based upon COM and no direct APIs are available for the

.NET framework. Wrappers need to be implemented manually, which is very time

consuming and may introduce several bugs.

2.5.3 GStreamer

GStreamer [WT09] is an open source multimedia framework whose architecture is

a mixture of a plugin-system and the pipe-and-filter pattern. It ships with lots of

different language bindings.

Its QoS processing is similar to that of Direct Show in that a data sink sends QoS

messages to previous components in the chain. Such messages contain information

about how a component should change its behavior to improve processing.


2.5.4 FFmpeg/FOBS

FFmpeg [Web09a] is an open source framework for processing videos implemented in

the C programming language. It contains implementations of many different codecs

and supports a large number of video formats. The framework provides a low level

API and lacks documentation and example code. In fact, it provides no real QoS

support but offers a multitude of transcoding possibilities and thus supports many

video adaptation techniques.

FOBS (Ffmpeg OBjectS ) [MtcvgaUD09, SP08] enhances the original API, wrap-

ping it with C++ objects and interfaces. The QoS limitations still exist.

2.5.5 Infopipes

Infopipes [BHK+02, KBH+01, SPK+04, SPM04] define an abstraction of streaming

with an architecture similar to the pipe-and-filter pattern, using the analogy of fluids.

Infopipes are built out of components like pumps, tubes, buffers, filters. A complete

network of components can be defined analogously to a network of water pipes, mov-

ing data streams through the components like fluid. Koster et al claim that remote

procedure calls are not suitable for information streams [KBH+01] as streams have

different communication requirements in comparison to simple procedure calls. Al-

though according to findings in literature Infopipes is supposed to support QoS, no

example demonstrates its implementation.

2.5.6 QUINNA

QUINNA [TBO05b, TBO05a, TOB04] is a component based QoS-architecture for

open systems that allows new components to be added during run-time and is based

on the following hypothesis [TBO05b]:

1. The system has global knowledge about available resources, meaning that a re-

quest cannot be delegated to another unknown system.


2. The system must be component-based itself in order to provide a homogeneous

components view at each layer (application, operating system and resources).

QUINNA implements QoS contracts. A centralized management component is

responsible for the initialization, adaptation and administration of these contracts.

The resources for the required QoS are reserved within the system and an observer

monitors if the requirements are met. If QoS requirements are not met, they are

adapted to a lower level and the system is reprovisioned again using definitions stored

within tables that are mapped to system level QoS properties.

2.5.7 DJINN

DJINN [Mit97, SM97, MNCK99, NKMC98] was developed to create distributed mul-

timedia applications which support dynamic reconfiguration at runtime. It is a mul-

timedia component framework which additionally provides QoS control, real-time op-

erations, group-oriented communication and high level application composition and

configuration.

Applications are built out of a series of connected components. Connectors are

also implemented as components. QoS is embedded by splitting each component

into two parts: an active part responsible for executing the implemented logic and

a second part containing a model of the configuration and QoS requirements. The

segregation of execution logic and configuration parameters allows dynamic configu-

ration at runtime. Reconfiguration is only allowed if no inconsistencies occur and all

QoS requirements are still met (resource requirements, frame rate) [SM97, MNCK98].

The required QoS is matched against the QoS which can be provided by a resource

management component.

2.5.8 NMM

Network-Integrated Multimedia Middleware (NMM) [LWRS08, Mar05, Mar01] is an-

other multimedia middleware which concentrates on the integration of the network.


It has been developed to support distributed multimedia applications and provides a

completely transparent view on distributed systems. All resource in the network can

be accessed by any node which allows transparent control of distributed devices and

components.

Processing elements are called nodes which use communication ports named jacks.

These jacks are associated with allowed formats which represent a detailed description

of the streaming data. Two jacks can only be put together if they match the same

format.

The automatic application setup (AAS) uses a high level description of nodes to

build an distributed flow graph which meets certain QoS constraints. In contrast

to JMF and DirectShow which are restricted to operate on one single system, NMM

supports distributed flow graphs based on a high-level description of the whole system.

The output of the AAS is a preconfigured distributed system which provides the

requested behavior in an optimal way. QoS processing is focused on the underlying

network. It tries to provide the best quality of experience available and media data is

automatically adapted to optimally use the bandwidth currently available. Different

monitoring and adaptation strategies are applied to adjust bit rate, frame rate, video

coding based on measured packet loss and signal strength.

CHAPTER

3 Requirements

This chapter describes the requirements for quality aware programming. These re-

quirements are divided into three parts. The first part covers general requirements for

QoS specifications. The second part deals with requirements for QoS monitoring. The

third part focuses on requirements for extending a programming language. All these

criteria have been defined for quality aware programming to ensure its applicability

to multimedia applications.

3.1 Quality of Service Specification

According to Jin and Nahrstedt [JN04] QoS specification languages should describe

quantitative (performance specific) and qualitative (behavior specific) requirements

as well as adaptation rules. In general, a QoS definition should be declarative in

nature and an automated mapping/compilation to the underlying system should be

available. The basis of requirements are five criteria which need to be satisfied by a

novel QoS specification syntax [JN04, FK98].

These requirements are taken from the general requirements definition for QoS

specification languages which are discussed in section 2.2 in detail.

• Expressiveness - QoS specifications should be easy to read and give a clear

understanding of the requirements. It should be possible to specify a wide

variety of QoS requirements.

CHAPTER 3. REQUIREMENTS Page 54

• Declarativity - The QoS specification should clearly point out what is required

rather than how requirement checks should be applied.

• Independency - The QoS specification should be independent from the applied

context. It should be semantically separate from other components and their

specifications. Different QoS specifications can be applied to the same interface.

• Extensibility - QoS specifications should be reusable and support for refine-

ment should be possible.

• Reusability - Different QoS specifications may share common parts of their

specification. This means QoS specifications should be reusable within the

declaration part, but the same QoS specification may also be attached to many

different components or services.

All these requirements are necessary to build a comprehensive novel programming

framework for QoS. As this work extends an existing general purpose programming

language, further requirements are needed.

3.2 Quality of Service Monitoring

The process of monitoring the behavior of an application should be provided au-

tomatically without requiring the developer to apply manual code instrumentation.

Constraint evaluation should be done automatically and violations should cause ex-

ceptions which can then be handled accordingly. Evaluation of QoS specifications

and constraint monitoring should not be mixed with application logic code.

Application developers are usually interested in building quality aware applica-

tions quickly and easily. Nevertheless, it should be possible for advanced developers

to modify the monitoring process and change the way constraints are evaluated if

needed.


Adaptation code should not be mixed with application logic code and should be

defined within a separate block. Exchangeability of adaptive behavior should be

possible.

3.3 Programming Language Extension

Current general purpose programming languages like C# and Java are widely known.

Instead of defining a completely new programming language, an existing one can be

extended. This reduces implementation effort and may increase the acceptance by

potential users.

One drawback is that the deployment of a compiler is more complicated than the

distribution of a simple library which may hinder the propagation.

I define that extending a programming language is feasible if and only if the

following three criteria are fulfilled. If these requirements are not met, the feature

should be implemented rather within a library.

1. Expressiveness - The new feature enhances the expressive power of the lan-

guage considerably.

2. Safety - The safety of programs using the new feature is enhanced.

3. Optimizations - The new feature enables some space for automatic optimiza-

tions.

Although these requirements can be applied to the development of software li-

braries as well, there are differences if they are applied to a programming language.

A library itself can only be as expressive as the programming language in which the

library is implemented. Well defined interfaces provide clear code structures and

make programs easily readable and understandable. The expressiveness relies on the

software model applied to achieve the problem solution and is limited to the expres-

siveness of the programming language. In case of libraries, expressiveness depends on

the clarity of the interface definition.


Safety can be achieved within libraries with the help of extensive testing and

wide-spread usage. The code hidden within libraries is assumed to be safe, but

the implementation of libraries can only facilitate the same safety properties which

the programming language provides. Additional safety is not introduced with the

help of libraries, only well tested code is applied which can reduce the amount of

programming errors during the development process. If a library requires a certain

number of methods to be invoked, the programmer can still forget to implement each

method call which leads to erroneous behavior during program execution. If the

programming language provides certain safety properties, the compiler can produce

compile time errors and force the programmer to write correct code.

All these requirements are evaluated after the presentation of the novel Quality-

Aware Programming model.

CHAPTER

4 Quality Aware Program-

ming

An investigation of different multimedia applications showed that substantial parts

of programs processing multimedia data follow some very common patterns:

1. In the compression/decompression/transformation part of such programs, large

multidimensional numerical arrays are partitioned into small independent blocks

and processed by algorithms like the Discrete Cosine Transformation (DCT).

2. In the video streaming and play-back part, long sequences of data (e.g. video

frames) are processed and/or transmitted periodically, under so-called “soft

real-time” constraints.

The manually created code for these recurring patterns is typically cumbersome,

error-prone and inefficient. These observations suggest that support for multimedia

programming on the level of a programming language is required.

A multimedia programming language should natively support simple parallel pro-

gramming as well as control of QoS.

During my research I have defined some programming language extensions to

handle the first pattern (Section B). I have extended the foreach statement to gain

access to loop variables within the iterations and allow array splitting for block based

access [LSB06]. The extended foreach statement is used to provide more simple and

expressive implementations of DCT-block based compression algorithms. Further-

more, I have defined a statement level attribute to support parallel statements and

CHAPTER 4. QUALITY AWARE PROGRAMMING Page 58

parallel loops within the C# programming language to enable simple automatic paral-

lelism. It was intended to automatically speed up performance of MPEG-1/2 codecs

on multi-processor or multi-core machines. This work was not continued after the

new AVC/H.264 [h26] coding standard was published. Modern video codecs emerge

complex coding structures and the scheme of independent block based processing is

not applicable any more [SFLB07a]. The encoding and decoding processes of modern

video codecs introduce many different dependencies not only between frames, but

also between simple data blocks. To automatically parallelize such coding algorithms

a complete dependency analysis is required to synchronize data access and retain cor-

rect processing. The needs of complete data dependency analysis, which is currently

not covered, reduced the advantages of the new feature.

To handle the second pattern, a notion of time and a way to express Quality of

Service (QoS) constraints are needed. A vast number of multimedia query languages

resp. language extensions exist [JN04, JN], nevertheless, to my knowledge, no com-

plete language support for quality aware programming embedded in current common

systems programming languages exists.

Instead of defining a brand new language, I investigated the possibilities of ex-

tending some existing programming languages with the minimal necessary features,

considering the basic principles defined in section 3.3.

Everything else should rather be put into a library than applied as a language

extension. Under these premises I suggest the following extensions for general purpose

programming languages:

1. Constraint Declaration - A very simple first-order logic based language ex-

tension, enabling to express QoS constraints.

2. Constraint Monitoring - Timed Data Types - A time dimension, which

can be added to any existing scalar or array type as an n+first dimension.

3. Adaptation - Reuse the existing exception mechanism to react on QoS viola-

tions.


The concept of timed data types and its underlying framework allow to embed

monitoring and control of quality of service definitions directly within a system pro-

gramming language like C# [ECM05, NEG+05, M06] or Modula-3 [LSB06, Nel91,

BW96]. First prototypes were implemented in both programming languages to see

its general availability. Due to too much effort in maintaining both languages and only

minor availability of multimedia libraries in Modula-3, the Modula-3 implementation

was stopped. C# has been chosen instead of Java [Web09b] due to its capabilities to

support low level API calls within unsafe blocks and support for embedding C++

easily which is advantageous for multimedia applications.

Considering the previously defined requirements, I have extended the program-

ming language C# by adding native support for quality of service. The current im-

plementation is based on Mono [Web09f], an open source implementation of the C#

programming language. The new QoS-aware language is called MMC# (MultiMedia

C#).

4.1 MMC# Concepts

The C# programming language is enriched by a few concepts to support QoS pro-

cessing.

• Timed data types.

• Declarative QoS constraint definition.

• A new value assignment operation with implicit QoS monitoring.

Time can be seen as the n + 1st dimension of a data type. MMC# introduces

timed types similar to array types. This means that time can be added to any other

data type, but in contrast to array types it is only recognized for special processing

instructions. In case of type compatibility, the additional dimension is omitted.

The example of throughput monitoring (Listing 4.1) emphasizes the main concepts

of MMC#. It uses the variable inData which is declared with a timed type. This


means that the variable is associated with an additional time dimension which records

all events triggered by the new value assignment operation. Furthermore, a QoS

constraint is attached to the variable which evaluates recorded events to check whether

a certain condition is held or not.

1 // constraint declaration , timed data type with constraint definition

2 // @ represents the all quantor , n declares an index variable

3 // the array access is used to retrieve timing history data

4 // the applied constraint is frame rate: 25 frames per second


6


8 try {





13 }

14 }

Listing 4.1: Example of monitoring the rate of 25 elements per second (Quality Aware

Programming model).

The QoS constraint is defined with the help of a declarative syntax which is similar

to the Quality Language (QL) (Section 2.2.1). The history of the timed variable is

extracted by using simple array access.

The new value assignment is called timed or streaming assignment and provides

automatic and implicit constraint monitoring and event recording. Each time the

variable inData is assigned it triggers an event and the attached constraint is evalu-

ated. In case the constraint is violated, an exception is thrown.

The following section describes the MMC# language specification in detail.

4.2 MMC# Specification

MMC# provides language extensions for implementing quality aware multimedia ap-

plications by reducing implementation cost and, of course, providing more safety to


multimedia programming. The following definitions are used within this specification:

Non-timed type T represents any value or reference type of the C# programming

language.

Timed type T[∼], where T is a non-timed type, T is also called the original non-

timed type.

Timed variable T[∼] var, a variable named var of timed type.

Timed parameter stream T[∼] var, a method parameter of timed type. In this

case, the additional keyword stream is used to identify timed method parame-

ters.

Timed assignment ( : / :) is a new kind of an assignment, where either on the

left or the right side a timed variable must stay.

Timed parameter passing is the use of timed method parameters with the pa-

rameter passing mode stream.

Constraint association is the association of a constraint to a timed variable. Con-

straint association is required to provide automatic QoS monitoring.

The language specification is done by extending the context free grammar defined

in the ECMA-C# Standard [ECM05]. Only modified terminals and non-terminals

are described. To distinguish newly created parts, these elements are marked bold.

Extending the syntactic grammar of C#:

4.2.1 Keywords

Two new keywords1 are defined:

constraint

stream

1This block refers to section 9.4.3 page 71 and following of the C# Standard [ECM05].


The keyword constraint is used for constraint declaration. The keyword stream

defines a new parameter passing mode.

4.2.2 Types

The type definition2 is extended to support timed types. Timed-types are value-types

and similar to array-types.

type:timed-typevalue-type

reference-type

type-parameter

timed-type:

non-timed-type [ ∼ constraint-referenceopt]

non-timed-type:

value-type

reference-type

constraint-reference:

primary-expression

constraint-declaration

A timed data type in this context is defined similarly to an array type. Time is

the n+1st dimension, which can be added to any data type. The idea behind this new

data type is to store all assigned values over the time within a history. Although this

value history is not kept in the memory, the assignment operation can be extended

to monitor the timing behavior. This monitoring operation is implemented with the

help of the timed (streaming) assignment statement (see section 4.2.4) and the timed

2This block refers to section 11 page 107 and following of the C# Standard [ECM05].


(streaming) parameter passing mode (see section 4.2.5). Each time a new value is

assigned an event is triggered. These events are recorded within a separate history

which allows to evaluate the timing behavior of a timed variable.

Timed types are type compatible to their original non-timed types. Thus, the time

dimension is ignored at type checks. The additional type information is only evaluated

for the timed assignment operation and the timed method parameter passing mode.

It is possible to use timed types for casting, however, only the original non-timed

type is effectively used for this operation (Listing 4.2).

The definition of timed constants is not allowed.

1 int [~] a;

2 int b;

3 int [~] c;

4

5 // can be used like a normal integer

6 a = 7;

7 // timed types are type compatible to their origal non -timed type , in this case int

8 b = a;

9 // cast can be used , but is not necessary due to type compatibility

10 c = (int [~])b;

Listing 4.2: Type compatibilty.

To allow QoS checking, constraints are associated to timed variables. Such con-

straints are of type System.IQoSObject and are either added during declaration or

assigned dynamically via the new statement (Listing 4.3). The interface definition is

explained in section 8.2.2 in detail. If the constraint is associated statically within

the declaration of the timed variable, it cannot be changed over the life-time of the

variable. In contrast, the constraint association in the new statement is dynamic. In

this case, a constraint needs to be specified within the timed type declaration of the

new statement. If the constraint is not specified, an error is generated. Associated

constraints are stored separately within the QoS management framework (Section

8.2).

A constraint provides the definition of relations of time variant or time invariant

properties with respect to certain boundaries. An associated history buffer stores


events which are triggered during timed operations on timed variables. The cast

operation and the standard assignment operation do not have any influence on the

associated constraint or the history buffer of a timed variable.

1 // Timed variable with dynamic QoS . The constraint is specified using the new

operator .

2 int [~] dynamicQoSVariable = new int [~( IQoSObject ) constraint ];

3

4 // The constraint is declared together with the variable and

5 // cannot be changed during the lifetime of the variable .

6 int [~( IQoSObject ) constraint ] staticQoSVariable;

Listing 4.3: Dynamic and static timed variable declaration.

4.2.3 Constraint declaration

constraint-declaration:

@ identifier { constraint-condition-with-exception }

constraint-condition-with-exception:

or-constraint-condition

or-constraint-condition : throw-statement

Constraints are declared with the help of the all quantor (∀). The @ character is

used instead of the usual mathematical notation as the (∀) symbol is not available

on the keyboard. The identifier symbolizes the name of the control variable which is

incremented each time the constraint is evaluated. The name of the control variable

must be unique within the applied context. The initial value is 0, which means that

no timed assignment was executed. The actual value of the control variable represents

the index of the actual event. The control variable can be used to access the timing

history of a timed type. Only past events can be accessed. The number of events

which are required to correctly evaluate a constraint is variable and depends on the

constraint declaration itself. In case the history is accessed with the value n − k and

there are not enough (≥ k) values available, the constraint is not evaluated at all.


The constraint condition evaluates to a boolean value. If the resulting value is

false, the constraint is violated and an exception is thrown.

A throw statement3 can be optionally used to throw user defined exceptions in case

of QoS violations. If no exception is specified, a System.QoSException is thrown.

Constraint conditions can be combined by logical operators && and || which allows

building complex logical expressions. Such complex conditions are evaluated as one

single block. This can be compared to building complex conditions for if statements.

or-constraint-condition:

and-constraint-condition

or-constraint-condition || and-constraint-condition

and-constraint-condition:

constraint-condition

and-constraint-condition && constraint-condition

A constraint condition is required to be of a type that can be implicitly converted

to a boolean value. It allows a set of comparison operators to be used.

constraint-condition:

qos-expression

qos-expression qos-conditional-operator qos-expression

qos-conditional-operator:

<

<=

==

>

>=

3This block refers to Section 15.9.5 page 243 of the C# Standard.


A qos-expression is used to apply mathematical operations within parts of con-

straints. It is clearly defined which operations are allowed to reduce complexity of

constraint declaration.

qos-expression:

qos-primary-expression

qos-expression qos-math-operator qos-expression

qos-math-operator:

+

-

*

/

%

qos-primary-expression:

primary-expression

Primary expressions4 are the simplest forms of expressions. They allow e.g. con-

stant values, variable access, parenthesis and array element access. This array element

access is applied to access the timing history of a timed variable within a constraint

declaration.

A constraint is represented by the implementation of the System.IQoSObject

interface (Section 8.2.2). This interface defines the method CheckQoS in which the

history of timed data types is checked against specified quality conditions. If the

condition evaluates to false, an exception is thrown (System.QoSException). A

constraint class can be implemented manually or generated by the compiler using the

explained declarative specification syntax.

In case of declarative QoS specification, the compiler automatically generates a

subclass of the current class which implements the System.IQoSObject interface. The

4This block refers to Section 14.5 page 165 of the C# Standard.


CheckQoS method is implemented by rewriting the provided constraint declaration

into if statements. The history access of timed variables is mapped to method

calls to GetSignalHistory of the QoS management framework (Section 8.2.2). This

method retrieves the time of the required event occurrence identified with the help

of the control variable. The complete interface is explained in section 8.2 in detail.

Additionally, check code is added by the compiler which tests if enough history values

are available to correctly evaluate the constraint. In case the constraint cannot be

held, a System.QoSException is thrown.

Furthermore, a constraint is only valid if there is enough information in the history.

If the history does not contain enough events, the declared constraint cannot be

checked and is always evaluated to true.

1 // combination of throughput and latency jitter

2 @n {var[n] - var[n -25] <= Units.MSec (1000) &&

3 Units .MSec (10) <= var[n] - var[n -1]

4 <= Units.MSec (50)}

Listing 4.4: Definition of throughput (25 frames per second) combined with jitter.

4.2.4 Assignment operators

assignment:

unary-expression assignment-operator expression

unary-expression ∼: expression

unary-expression :∼ unary-expression

Two new assignment operators5 are added. The so called timed- or streaming-

assignment assigns a value from or to a timed-variable. This operation is done under

QoS control. It advises the compiler to add QoS monitoring code. Two versions of

the assignment (~: for write - :~ for read) exist.

Each time the assignment is executed an event is recorded and the corresponding

constraint is evaluated. If the evaluation fails, an exception is thrown. This behavior

5This block refers to Section 14.14 page 218 and following of the C# Standard [ECM05].


is explained in section 4.2.3 in detail.

A write-timed-assignment requires the left part of the assignment to be a timed

type. Accordingly, a read-timed-assignment requires a timed type on the right side.

In multimedia streaming applications two different kinds of value assignments

are needed. The first is the usual assignment operation that does not need any

constraint-checking or quality requirements. This operation is expressed by the “nor-

mal” assignment operation (=) which is executed best effort and can be compared

with a multimedia management operation, e.g. copying of a video. In this case,

timed variables are used like normal variables. The additional type information as

well as timing history values are completely ignored and no QoS monitoring code is

executed.

1 int [~] a;

2 int b;

3 int [~] c;

4 // standard assignment

5 a = 1;b = 2;c = 3;

6 // timed write assignment

7 a ~: b;

8 a = 4;// no QoS monitoring executed

9 a ~: 5;

10 c ~: a;// value of a is used for assignment , the timing history of a is ignored

11

12 b ~: a; // compile time error

Listing 4.5: Mixing Timed and Standard Assignment.

In contrast, a streaming operation is applied for features like video playback which

is the process of displaying time dependent data under QoS control. Such streaming

behavior is achieved by the timed (streaming) assignment statements (resp. param-

eter passing). These timed assignment statements are implemented as usual assign-

ments with additional QoS monitoring code. In this case the additional type infor-

mation and the associated constraints are evaluated. If the timed assignment is used

with non-timed variables, the compiler generates an error message. However, it is

possible to mix up timed assignments and standard assignments with timed variables

(Listing 4.5).


The timed assignment is responsible for automatic constraint checking and the

triggering of events. These events are recorded in the event history of each timed

variable. In case of the write timed assignment, the constraint is defined by the left

hand, in case of the read timed assignment by the right hand value. Both timed

assignments trigger events and evaluate associated constraints. If the constraint can-

not be held, an exception is thrown (System.QoSException). Both directions - read

and write - are shown in the following examples (Listing 4.6, 4.7) and demonstrate

streaming a series of integer values to the screen.

1 int [~@n {value [n] - value[n-1] <= Units.MSec (10)}] value;

2 // var is assigned to some constraint

3 int [] array = {1,2,3,4,5,6,7,9};

4

5 for (int i = 0; i < array.Length ; i++) {

6 try {

7 // Timed writing ; Left -hand must be a timed variable .

8 value ~: array[i];

9 Console .WriteLine ("Value : " + value);

10 } catch ( QoSException e) {

11 Console .WriteLine ("QoS Violation !");

12 }

13 }

Listing 4.6: Timed Write Example.

1 int value ;

2 // array is assigned to some constraint

3 int [][~@n {value[n] - value [n-1] <= Units .MSec (10) }] array = {1,2,3,4,5,6,7,9};

4

5 for (int i = 0; i < array.Length ; i++) {

6 try {

7 // Timed reading ; Right -hand must be a timed variable .

8 value :~ array[i];

9 Console .WriteLine ("Value : " + value);


11 Console .WriteLine ("QoS Violation !");

12 }

13 }

Listing 4.7: Timed Read Example.


Both examples finish with the same result. All values of the array were assigned to

a single variable within a loop and displayed on the screen. This was executed under

the given constraint specifying a rate of 100 values per second. In the first example

(Listing 4.6), the quality requirements are specified on the destination variable and

in the second example (Listing 4.7) the definition relies on source variable.

4.2.5 Methods

parameter-modifier:

ref

outstreamargument:

expression

ref variable-reference

out variable-referencestream expressionA new parameter passing mode6 is added which is called timed (streaming) pa-

rameter passing mode. Stream parameters are passed by value, but with constraint

evaluation. Accordingly, a streamed argument is denoted with the keyword stream.

Streaming parameters require a timed data type within the parameter list of the

method declaration.

The new parameter passing mode is defined to distinguish between timed param-

eter passing and other parameter passing modes. In fact, a value can be streamed

to a method which means that it is passed to the method under QoS control. This

is expressed by the keyword stream. Stream parameters are passed as values under

given QoS constraints. They cause the triggering of events, which are recorded into

the history. The constraint is checked during method invocation. If the constraint

6This block refers to Section 17.5.1 page 287 and following of the C# Standard [ECM05].


cannot be held, an exception is thrown. This behavior is explained in section 4.2.3 in

detail.

1 IQoSObject constraint = ... // some constraint

2

3 // dynamic timed parameter

4 void foo1(stream object [~ constraint ] input) {

5 ... // do something

6 }

7

8 // timed parameter

9 void foo2(stream object [~@n{input [n]-input[n-1] < Units.MSec (100) }] input ) {


11 }

12

13 // stream data to the method

14 try {

15 foo1(stream new object ());

16 foo2(stream new object ());

17 } catch(QoSException e) {

18 // violation occurred

19 }

Listing 4.8: Timed Parameter Passing with Named Constraints.

1 // static timed parameter

2 void foo(

3 stream object [~@n {input[n] - input[n -1]

4 < Units.MSec (100) }] input) {


6 }

7

8 // stream data to method

9 try {

10 foo(stream new object ());

11 } catch(QoSException e) {

12 // violation occurred

13 }

Listing 4.9: Timed Parameter Passing with Declarative Constraints.

Two possibilities of timed parameter declaration exist. The first is dynamic and

uses an object reference to associate the constraint. This reference must be of type

System.IQoSObject and can be changed each time a new instance of the surrounding


object is created (Listing 4.8). The second possibility is to directly apply the declar-

ative constraint specification syntax. Such a constraint cannot be changed during

program execution.

The second possibility is to directly specify the constraint. To illustrate this

(Listing 4.9), a method is implemented which defines the maximum delay between

subsequent calls not to exceed 100 milliseconds.

4.2.6 Classes

class-member-declaration:

constant-declarationparameterizable- onstraint-de larationfield-declaration

method-declaration

property-declaration

event-declaration

indexer-declaration

operator-declaration

constructor-declaration

finalizer-declaration

static-constructor-declaration

type-declaration

parameterizable-constraint-declaration:

constraint-modifiersopt constraint identifier

(formal-parameter-listopt) => constraint-declarations ;

constraint-modifiers:

constraint-modifier

constraint-modifiers constraint-modifier

constraint-modifier:


private

protected

public

static

The class member declaration7 is extended to provide parameterizable constraints.

Parameterizable constraints allow the definition of constraints by specifying constraint

templates as class members via the keyword constraint. These constraints can be

instantiated on demand and thus provide reusability of constraint declarations.

Constraints can be combined by logical operators && and ||. This allows building

complex constraints out of simple ones, e.g combining throughput and jitter con-

straints. In case of a QoS constraint violation, an exception is thrown. The semantic

of logical concatenation of using the operator && is that every constraint needs to be

met. In contrast, when using the operator ||, one single constraint satisfaction is

enough to fulfill the whole constraint. This kind of concatenation allows the combi-

nation of different separatly defined constraints.

constraint-declarations:

or-constraint-declaration

or-constraint-declaration:

and-constraint-declaration

or-constraint-declaration || and-constraint-declaration

and-constraint-declaration:

constraint-declaration

and-constraint-declaration && constraint-declaration

Parameterizable constraints allow QoS constraints to be reused. Such constraints

are declared in the same way as other class members and make use of the same

7This block refers to Section 17.2 page 269 and following of the C# Standard [ECM05].


accessibility features. Furthermore, they allow the definition of parameters which can

then be used inside the constraint. During constraint instantiation actual parameters

are mapped to the formal parameters in the constraint definition.

The code example shown in Listing 4.10 defines a frame rate constraint and the

corresponding tolerated jitter. The first parameter identifies a timed variable which

contains a timing history of its streaming assignments. The second parameter defines

the frame rate and the third the allowed jitter. Listing 4.11 shows how to assign a

new instance of the previously defined constraint to a timed variable. A constraint

with a rate of 25 assignments per second and an allowed jitter of plus or minus 10

milliseconds is attached.

1 private constraint FrameRateAndJitter

2 (object [~] i, int rate , int jitter

3 , int timeRange ) =>

4 @n{i[n] - i[n-rate]

5 < QoS.Units.MSec(timeRange )}

6 && @n{i[n] - i[n-1]

7 > rate/QoS .Units .MSec( timeRange )

8 -QoS .Units .MSec(jitter )

9 && i[n] - i[n-1]

10 < rate/QoS .Units .MSec( timeRange )

11 +QoS .Units .MSec(jitter )};

Listing 4.10: Parameterizable Declarative QoS Constraint specifying frame rate and

jitter.

1 object [~] o; // timed variable

2 o = new object [~ FrameRateAndJitter(o, 25

3 , 10, 1000) ];

Listing 4.11: Parameterizable QoS Constraint Instantiation specifying 25 frames/sec

with 10 msec jitter.

1 private constraint FrameRateAndJitterWithLatency

2 (object [~] i, int rate , int jitter ,

3 , int timeRange , int maxLatency ) =>

4 @n{i[n] - i[n-1] < QoS.Units.MSec(maxLatency )}

5 && FrameRateAndJitter(i, rate , jitter , timeRange );

Listing 4.12: Extending a Declarative QoS Constraint.


Parameterizable constraints can be reused in any other code fragment. Addition-

ally, parameterizable constraints can be nested inside other constraints to provide

extensibility. The example (Listing 4.12) shows how to extend existing constraints

by adding the QoS requirement of maximum latency per frame.

It is also possible to express non-temporal constraints. Listing 4.13 illustrates an

example of monitoring delay, CPU usage and memory consumption. The specified

temporal constraint allows a delay of 10 milliseconds. Additionally, a non-time-based

constraint restricts execution if processing time exceeds 20% or the amount of avail-

able memory is less or equal 512Mb. Although the control variable is not used within

the non-temporal constraint declaration, it is required for syntactical correctness.

1 using System .Diagnostics ;

2

3 // ...

4 // PerformanceCounter is a predefined class to measure system performance values

5 // and is located within the System .Diagnostics package . It provides an interface

6 // method NextValue to retrieve the next measurement value .

7

8 private constraint qos_Delay_CPU_RAM(object [~] p, PerformanceCounter cpu ,

PerformanceCounter ram)

9 => @n{p[n] -p[n-1] <= QoS.Units .MSec (10)}

10 && @n{cpu.NextValue () < 20 && ram .NextValue () > 512};

11

12 // ...

13

14 // create CPU performance profiler

15 PerformanceCounter cpuCounter = new PerformanceCounter ();

16 cpuCounter .CategoryName = "Processor ";

17 cpuCounter .CounterName = "% Processor Time";

18 cpuCounter .InstanceName = "_Total ";

19

20 // create memory performance profiler

21 PerformanceCounter ramCounter = new PerformanceCounter("Memory ", "Available MBytes ");

22 // attach constraint

23 object [~ qos_Delay_CPU_RAM(val , cpuCounter , ramCounter )] val ;

24

25 // ...

Listing 4.13: Using time and non-time based constraints.


4.3 MMC# by Example

This section provides programming examples to see how MMC# can be applied.

4.3.1 Constraint Declaration

1 using System ; using QoS;

2

3 public class Program {

4 // Parameterizable constraint declaration

5 private constraint qos_delay (object [~] p, int delay )

6 => @n{p[n] -p[n-1] <= Units.MSec(delay)};

7

8 // method testing constraints

9 public void ExampleConstraintDeclarations () {

10 // static parameterized constraint assignment

11 int [~ qos_delay (valueParamCon1 , 20) ] valueParamCon1;

12 int [~] valueParamCon2; // dynamic parameterized constraint assignment

13 valueParamCon2 = new int [~ qos_delay (valueParamCon2 , 20) ];

14

15 // static anonymous constraint assignment

16 int [~@n{valueCon1 [n] - valueCon1 [n-1] <= Units .MSec (20) }] valueCon1 ;

17 int [~] valueCon2 ; // dynamic anonymous constraint assignment

18 valueCon2 = new int [~@n{valueCon2 [n] - valueCon2 [n -1] <= Units.MSec (20) }];

19

20 int [~] valueNamedCon;

21 // defining a named constraint instance

22 IQoSObject qosDelayCon = @n{ valueNamedCon[n] - valueNamedCon[n-1]

23 <= Units.MSec (20)};

24 valueNamedCon = new int [~ qosDelayCon ];

25

26 for (int i = 0; i < 100; i++) {

27 valueParamCon1 ~: i; valueParamCon2 ~: i;

28 valueCon1 ~: i; valueCon2 ~: i; valueNamedCon ~: i;

29 }

30 }

31

32 public static void Main(string [] args) { // main method

33 new Program (). ExampleConstraintDeclarations ();

34 }

35 }

Listing 4.14: Example demonstrating constraint declaration variants.


Listing 4.14 illustrates possibilities to declare constraints and use different con-

straint initialization methods.

The first block explains handling of parameterizable constraints. The timed

variables valueParamCon1 and valueParamCon2 use the parameterizable constraint

qos delay. This constraint requires two parameters: a timed variable (p) and the

delay value specified in milliseconds (delay). Parameterizable constraints define con-

straint templates which can be reused and associated to many different timed variables

by using different parameters.

The next block illustrates constraints which are specified when declaring or ini-

tializing a variable (valueCon1/valueCon2). Such constraints are called anonymous

constraints and can be declared on demand. It is not possible to reuse an anonymous

constraint.

Last but not least, a named constraint instance is used. This is achieved by assign-

ing the anonymous constraint declaration to a variable of type System.IQoSObject.

Named constraints can be used to pass constraint definitions as method parameters

or store them as member fields.

All blocks within the example express the same QoS requirement - delay of 20

milliseconds.

4.3.2 Transformation Rules for QL

The syntax used for QoS specification is similar to QL [BS98]. QL allows easy and

expressive declaration of time based constraints. This section describes how QL

formulas can be transformed into MMC# constraint specification syntax.

Table 4.1 shows common time based constraints used in multimedia streaming

applications. The function τ is called history function and allows access to the history

of the events via an index variable. Events are referenced as ǫ and an index denotes the

type of the event, e.g. ǫr and ǫs symbolize the receiving and the sending event. This

allows the representation of lots of different time based constraints, e.g. throughput,

latency or jitter.


QoS Constraint QL Syntax

Throughput ∀n, |τ(ǫr, n) − τ(ǫr, n − k)| 6 δ

Latency ∀n, |τ(ǫr, n) − τ(ǫr, n − 1)| 6 δ

Jitter ∀n, δ1 6 |τ(ǫr, n) − τ(ǫr, n − k)| 6 δ2

Bounded Execution Time ∀n, |τ(ǫi, n) − τ(ǫo, n)| 6 δ

Table 4.1: Common constraints in multimedia systems.

The MMC# constraint declaration syntax can be deduced from the existing math-

ematical notation of QL by applying the following transformation rules (Table 4.3.2).

1. Exchange the symbol ∀ with the symbol @.

2. Map the history function τ to array access of timed variables. Substitute the

event name by the name of a timed variable.

3. Put the expression itself into braces.

4. Replace variables in the constraint by concrete values or variables defined in

your code.

Table 4.2: Transformation Rules for QL.

As an example these rules are applied to the definition of throughput. The event

ǫ is replaced with the timed variable named var. δ is set to 1000 milliseconds. The

utility class Units provides conversions between different time values. It contains

static methods to convert expressions into the internally used time representation.

The resulting constraint is a definition for throughput of 25 events per second

(Listing 4.15).

1 // 25 assignments per second

2 @n {var[n] - var[n -25] <= Units.MSec (1000) }

Listing 4.15: Definition of throughput (25 frames per second).


n is declared as the counter variable inside the constraint. It starts with 0 and

represents the index of the timing history. A point in the future cannot be accessed

(n+1), only values in the past are allowed.

Constraint evaluation is omitted until enough values are available.

1 // single constraint


3 @n {var[n] - var[n -25] <= Units.MSec (1000) &&

4 var[n] - var[n-1] >= Units .MSec (10) &&

5 var[n] - var[n-1] <= Units .MSec (50)}

6

7 // complex constraint


9 @n {var[n] - var[n -25] <= Units.MSec (1000) } &&

10 @n {var[n] - var[n-1] >= Units .MSec (10) &&

11 var[n] - var[n-1] <= Units .MSec (50)}

Listing 4.16: Definition of throughput (25 frames per second) combined with jitter.

The next example (Listing 4.16) shows how to combine constraints to build more

complex QoS specifications. In this example, quality requirements for throughput and

jitter defined in one single constraint are expressed. This is achieved by concatenating

constraint conditions using logical OR (||) and logical AND (&&) operators (Listing

4.16). The syntax is similar to defining conditional statements in C#. The constraint

of 25 frames per second is combined with a jitter definition which expects events with

a delay between 10 and 50 milliseconds.

The simple constraint uses one constraint to combine throughput and jitter. It is

evaluated as one single block. The complex constraint concatenates two completely

separate constraint declarations, which are evaluated separately.

Constraints are not limited to single timed variables. A maximum delay between

input and output of streaming data can be defined. In this case, the run-time support

tracks two events whose histories are compared to each other. The constraint for

bounded execution time (see Table 1 ) uses two different events. ǫi expresses the

incoming event and ǫo the outgoing.

Given two timed variables input and output, the bounded execution time of


events between input and output can be limited to a certain value, e.g. 50 milliseconds

(Listing 4.17).

1 // bounded execution time

2 @n {output [n] - input [n] <= Units .MSec (50)}

Listing 4.17: Definition of bounded execution time.

The presented declarative QoS specifications can be assigned to any declared timed

variable. The compiler automatically generates classes out of the constraint definition

which are of base type System.IQoSObject. Objects of these classes are instantiated

and assigned to timed variables. The class itself contains an implementation of an

interface method which is called during event triggering to check if the constraint is

held or not. The history of the variables is accessed through the QoS management

framework from the system class library.

4.3.3 Exception based QoS Violations

1 // An exception for too slow execution .

2 public class TooSlowException : QoSException {

3 //...

4 }

5

6 // An exception for too fast execution .

7 public class TooFastException : QoSException {

8 //...

9 }

10

11 // A constraint which uses different exceptions

12 // to request different behavior of adaption code.

13 @n{output [n] - input[n] <= Units.MSec(maxTimeMSec )

14 : throw new TooSlowException () &&

15 output [n] - input[n] >= Units.MSec(minTimeMSec )

16 : throw new TooFastException ()

17 }

Listing 4.18: Exception based QoS violation.

Adaptation code is nested inside exception handlers. These code blocks are trig-

gered in case of QoS constraint violations. If no explicit exception is defined, a


System.QoSException is thrown. This general QoS violation exception does not

identify the part of a constraint which caused the violation. To allow more flexible

exception handling, different parts of constraints can throw different exceptions. A

bounded execution time constraint is defined having an upper and a lower boundary.

The distinction between violation of one of these two boundaries is done using two

different exceptions (Listing 4.18).

If the current execution time exceeds the specified maximum execution time, a

TooSlowException is thrown. If the execution monitored is too fast, a TooFastEx-

ception is thrown. Adaption code can make use of this information and correct the

behavior of the application, e.g. speed up or slow down the execution. Further-

more, it is possible to pass event history values as exception parameters to use timing

deviation for the applied adaptation code.

1 // An exception containing the time difference .

2 public class QoSExceptionWithInfo : QoSException {

3 long timeDeviation;

4

5 public QoSExceptionWithInfo(long timeDeviation)

6 : base("Deviation " + timeDeviation) {

7 this.timeDeviation = timeDeviation;

8 }

9

10 public long GetTimeDeviation () {

11 return timeDeviation;

12 }

13 }

14

15 // A constraint exception parameters

16 @n{p[n] - p[n -1] <= Units.MSec (20)

17 : throw new QoSExceptionWithInfo ((p[n] - p[n -1]) - Units .MSec (20))

18 }

Listing 4.19: Exception with QoS parameters.

Listing 4.19 illustrates an example of passing history values to the exception. In

this case, the time deviation of the required value of 20 milliseconds is passed to the

exception and stored within the member field diffTime.

In contrast to Java, C# does not guarantee that all exceptions are caught. This


is due to compatibility reasons to other .NET languages. Although the compiler does

not check possible exceptions, exception handling is safe in the sense that no exception

is lost. If an exception is not caught, the whole program is terminated [M06].

4.4 Constraint Analysis

In some cases, complex constraint definitions may be hard to understand and mistakes

can occur. Correctness checks executed by the compiler and by compiler generated

code at run-time help the application developer to define correct constraints. To

support the developer this semantic analysis must be performed by the compiler.

The declarative syntax allows automatic validation of syntax and semantic behav-

ior at compile time. Syntactic correctness is achieved by correct grammar definition.

Literature findings show that current research focuses on negotiation of required and

provided QoS and correct translation of QoS parameters. This thesis explains an

approach which analyzes specified constraints partially at compile time, partially at

run-time and provides correctness checks and optimization possibilities.

One approach to check if a constraint is semantically wrong is a formal proof based

on static code analysis. Two criteria are required to be met.

τ(ǫx, n) − τ(ǫx, n − 1) > 0, total order of events (4.1)

τ(ǫy, n) − τ(ǫx, n) > 0, x happens before y (4.2)

A previous history entry always precedes the current one. It is assumed that the

total order of events of the same source is guaranteed (4.1).

If a timed assignment of a timed variable x happens before that of y within the

same block, the corresponding history values appear in the same order (4.2). This

criterion requires sequential execution of the statements of the observed block.

If these two criteria are not met, the constraint is semantically wrong. If they

are held, it cannot be guaranteed that the constraint can be held, but at least it


is possible that it evaluates to true. This static code analysis only helps in finding

incorrect constraints and does not allow for further optimizations. The two criteria

are assumed to be held and are not part of the implemented semantic constraint

analysis.

Beyond finding incorrect constraints, I try to analyze constraints in deep and

optimize their conditions. The semantic analysis is divided into two parts:

1. Compile-time analysis

2. Run-time analysis

The first part is executed during compile-time and evaluates constant expressions

which are subsequently compared to each other. It converts each constraint into

a linear function and compares the gradient of each function with each other. This

helps to figure out which constraints are weaker and which are stronger (Listing 4.20).

Depending on the result of the comparison, the compiler optimizes the constraint or

generates a compile-time error message or warning. If the constraint contains dynamic

elements like parameters or method calls, additional code is generated to carry out

the check during run-time.

1 // constraints with constant values

2 @n{frame[n] - frame[n-1] < QoS.Units .MSec (50)} && // stronger constraint

3 @n{frame[n] - frame[n-1] < QoS.Units .MSec (100)} // weaker constraint

Listing 4.20: Declarative QoS Constraint Specifying Delay.

The second part of the compiler generated semantic analysis is performed at

run-time, during constraint instantiation. The compiler generates code to check

the correctness of a constraint during its instantiation. In case of parameterizable

constraints, all parameter values are processed similar as constant expressions at

compile-time. In case of inconsistent constraint definitions, an exception is thrown.

This can be compared to array bound checking [NI05], which is used in mainstream

programming languages like C# or Java [Web09b].


Although semantic analysis examines constraints at compile-time and during in-

stantiation, some uncertainty of correctness remains when using user defined methods.

Adding method calls allows constraints to be extensible for any required behavior,

but they can also harm a constraint definition. A method call may last too long

and the constraint check itself may take longer than the action being monitored.

Furthermore, such method calls may lead to erroneous behavior which is eventually

not captured by exception handling code. The current implementation of semantic

analysis allows user defined methods but does not analyze them in detail. Methods

defined in the system class QoS.Units can be used as the compiler implementation

has detailed knowledge about their behavior.

The constraint analysis process consists of the following steps:

1. Split constraint definition into conditional blocks.

2. Convert condition blocks into a normalized form.

3. Compare each normalized constraint condition to each other and fill evaluation

matrix.

4. Decide constraint reconstruction based on the evaluation matrix to provide

optimized code or warning and error messages.

At the end of the analysis, a matrix with relations between constraints and parts

of constraints is available. These relations help the compiler making decisions about

possible optimizations. A relation between two constraints in the matrix can be

expressed by one of the following types:

• Weaker (W ) - A weaker constraint is logically implied by a stronger one.

• Equality (E) - Two constraints express an equivalent behavior.

• Stronger (S) - It is the inverse relation to the Weaker relation.


• Inconsistent (I) - Inconsistent constraints represent contradictions. Such con-

straints will never evaluate to true.

• Different (−) - Two constraints are different and thus both are required to

express the specified requirement.

• Dynamic Constraint (D) - The constraint uses dynamic elements and cannot

be evaluated at compile-time. Dynamic elements are method calls or class

member accesses which result in varying values during run-time.

4.4.1 Constraint Splitting

Concatenated constraints are too complex to be analyzed as a whole. Therefore, the

whole constraint is split into its conditional parts. Time independent constraint parts

are omitted. These parts are not covered by the constraint analysis.

An example is used to illustrate the constraint analysis. It analyzes the following

QoS requirements which are expressed in Listing 4.21:

• A frame rate of 25 frames per second.

• The delay between two consecutive frames should not exceed 100 milliseconds.

• The deviation of delay between two consecutive frames should not exceed 20

milliseconds.

• A group of 5 frames needs to be processed within 500 milliseconds.

1 @n{frame[n] - frame[n -25] < QoS.Units.Sec (1) } &&

2 @n{frame[n] - frame[n-1] < QoS.Units .MSec (100)} &&

3 @n{ frame[n] - frame[n -1] > QoS.Units.MSec (20)

4 && frame[n] < frame[n -1] + QoS.Units.MSec (60)} &&

5 @n{frame[n] - frame[n-5] < QoS.Units .MSec (500)}

Listing 4.21: Declarative QoS Constraint Specifying QoS Requirements.


Part Condition

C1 frame[n] - frame[n-25] < QoS.Units.Sec(1)

C2 frame[n] - frame[n-1] < QoS.Units.MSec(100)

C3a frame[n] - frame[n-1] > QoS.Units.MSec(20)

C3b frame[n] < frame[n-1] + QoS.Units.MSec(60)


Table 4.3: List of Extracted Conditions.

After constraint splitting is finished, the following conditional blocks have been

identified (Table 4.4.1).

Although the example demonstrates && concatenated conditions, constraint anal-

ysis covers || concatenated conditions, as well. The only difference is made during

constraint reconstruction. In case of logical or, the weaker constraints are taken into

account while omitting the stronger ones.

4.4.2 Constraint Condition Normalization

To make constraints comparable, they need to be transformed into a normalized form

(Table 4.4). All constraints are converted into one of these forms (4.3). If a constraint

condition cannot be converted into one of these forms, it is considered to be different

to any other constraint.

∀n, τ(ǫ, n) − τ(ǫ, n − k) < δ

∀n, τ(ǫ, n) − τ(ǫ, n − k) = δ

∀n, τ(ǫ, n) − τ(ǫ, n − k) > δ (4.3)


Part Condition

C1 frame[n] - frame[n-25] < QoS.Units.Sec(1)


C3a frame[n] - frame[n-1] > QoS.Units.MSec(20)

C3b frame[n] - frame[n-1] < QoS.Units.MSec(60)


Table 4.4: List of Normalized Conditions.

4.4.3 Constraint Condition Comparison

Constraint conditions are compared based on the normalized form. The parameters

k and δ are the condition parameters. They can be a constant value or defined as

a parameter in case of parameterizable constraints. If they are constant, they can

be used for further investigations. In case they are passed as parameters they are

marked as a dynamic constraint (D).

If constraint parts use the same timed variables, they can be compared. If they

use different timed variables, it is not possible to compare them.

Condition relations are evaluated by comparing k, δ and the comparison operator

(4.4). Method calls to the class QoS.Units can be omitted, due to the fact that the

method body of the static methods only contain formulas with constant values.

Constraints can be represented as linear funtions and compared to each other

by comparing their gradient. Additionally, the applied conditional operator and the

number of recognized events (k) is distinguished. The evaluation is illustrated by

comparing conditions C2 and C3b (4.5), which results in a weaker relation between C2

and C3b.

The relation matrix is the result of the comparison of these conditions. Table 4.5

illustrates the complex relations between the single conditions.


RA→B =

E, if δA = δB ∧ kA = kB

W, if opA = opB =

<, if δA

kA− δB

kB> 0 ∧ kA > kB

=, if δA

kA

> δB

kB

>, if δA

kA

− δB

kB

< 0 ∧ kA < kB

S, if opA = opB =

<, if δA

kA

− δB

kB

6 0 ∧ kA 6 kB

=, if δA

kA

< δB

kB

>, if δA

kA− δB

kB> 0 ∧ kA > kB

D, if δA ∨ δB ∨ kA ∨ kB are dynamic

I, if opA 6= opB ∧

<>, if δA

kA− δB

kB< 0

><, if δA

kA− δB

kB> 0

−, else

(4.4)


opC2=′<′, kC2

= 1, δC2= 100

C3b frame[n] - frame[n-1] < QoS.Units.MSec(60)

opC3b=′<′, kC3b

= 1, δC3b= 60

RA→B, opA = opB =′<′ ∧δC2

kC2

−δC3b

kC3b

> 0 ∧ kA > kB

RC2→C3b= W, ′ <′=′<′=′<′ ∧ 100

1− 60

1> 0 ∧ 1 > 1 (4.5)

4.4.4 Constraint Reconstruction

The relation matrix allows reasoning about the importance of specific conditions (Ta-

ble 4.6). If the matrix (Table 4.5) contains equal conditions other than self equality

(main diagonal), these conditions can be reduced to one single representative. Weaker

conditions can be omitted and only stronger ones are taken into account. Different


C1 C2 C3a C3b C4

C1 E - - - -

C2 - E - W S

C3a - - E - -

C3b - S - E S

C4 - W - W E

Table 4.5: Relation Matrix.

conditions must be kept to preserve the semantics of the whole constraint. Incon-

sistent constraint declarations cause the compiler to produce error messages and the

compilations fails. If the constraint is modified by the compiler, warnings are pro-

duced which inform the developer about the inefficient definition of the constraints.

1 @n{frame[n] - frame[n -25] < QoS.Units.Sec (1) }

2 &&

3 @n{frame[n] - frame[n-1] > QoS.Units .MSec (20)

4 && frame[n] < frame[n -1] + QoS.Units.MSec (60)}

Listing 4.22: Reconstructed QoS Constraint.

1 @n{ // delay of 30 milliseconds

2 input[n] - input [n-1] < QoS.Units .MSec (30)

3 && // frame rate of 25 frames per sec

4 input[n] - input [n-25] < QoS.Units .MSec (1000) }

Listing 4.23: Weaker Constraint Example.

1 @n{ // maximum delay of 30 milliseconds

2 input[n] - input [n-1] < QoS.Units .MSec (30)

3 && // minimum delay of 50 milliseconds

4 input[n] - input [n-1] > QoS.Units .MSec (50)}

Listing 4.24: Inconsistent Constraint Example.

If dynamic conditions are encountered, the compiler generates checking code which

is executed at run-time during constraint initialization. This checking code compares

passed parameter values with other parameters or constant values defined within the


Relation Description

W If one constraint is semantically weaker than another, the weaker one

can automatically be removed. A weaker constraint is logically implied

by a stronger one. Listing 4.23 presents an example of this. If the delay

between two consecutive frames never exceeds 30 msec, this implies that

the frame rate is below 25 fps (it is actually 33,33 fps).

E If two constraints are equivalent, one of them can be removed.

S If one constraint is semantically stronger than another, the weaker one

can be removed automatically.

I If two constraints contradict each other, a compile-time error is gener-

ated. Listing 4.24 illustrates that either the first evaluates to true and

the second fails or the second evaluates to true and the first fails. Such

a constraint is obviously erroneous and should be removed.

− Two constraints are different and thus both are required to express the

specified requirement. No action is required.

D The constraint uses dynamic elements and cannot be evaluated at

compile-time. Dynamic elements are method calls or class member

accesses which result in varying values during run-time. Check code is

generated which evaluates constraint correctness during initialization.

Table 4.6: Relation-based Actions.


constraint. If the check fails, an exception is thrown to notify wrong arguments used

for constraint initialization (System.QoSInitializationException).

Based on the information within the relation matrix a new constraint can be built.

4.4.5 Exploiting Optimization Potential

This section discusses further optimization potential with the help of semantic con-

straint analysis. These optimizations are currently not implemented and might be

exploited in future to increase run-time performance. Two different approaches are

presented:

• Violation tolerance - can be introduced to reduce the number of exception

thrown. If QoS violations occur sporadic, it is possible to omit some of them.

• Monitoring accuracy - can be reduced to decrease the number of recorded

events and the number of QoS constraint checks executed per time frame.

Violation Tolerance

Multimedia systems are often implemented as soft real-time systems in which errors

should be avoided but do not lead to severe system failure. In case of a video playback

system, it does not matter if some frames are missed. The actual implementation of

MMC# focuses on complete detection and does not allow to specify a certain tolerance

value. It is possible to detect sporadic QoS violations due to limited resources. Such

violations can often be ignored and dismissed as they do not harm the process of

video playback as a whole.

Tolerance can be defined as a certain absolute number of violations which are

ignored, e.g. if a video playback does not detect more than five QoS violations,

nothing needs to be done. Tolerance can also be defined as the ignored number of

violations per time frame. This notion is even better than the absolute number. If

the frequency of violations increases, the application needs to react on them.


The actual implementation allows to add a certain tolerance within the exception

handler itself, but does not allow tolerance to be specified within the constraint dec-

laration. If this were the case, the overhead introduced by exception handling could

be reduced to a minimum. Only violations which are required for proper application

behavior need to be thrown by the monitoring code.

Monitoring Accuracy

Another possibility of optimizations to increase run-time performance is to reduce

accuracy of the constraint monitoring code. The question arises if really every con-

straint monitoring is required to provide correct application behavior. If the accuracy

of QoS checking is reduced, the fault-tolerance is automatically increased due to the

fact that some QoS violations are not detected at all.

The actual implementation monitors every event occurrence and records it with

the help of the QoS-management framework. This means that every event which is

recorded is also evaluated against specified QoS constraints. Small violations might

trigger adaptations which can lead to reduced display size or lowered quality.

The constraint specification formulates the requirements with the help of a math-

ematical definition. The example of throughput can be used to illustrate reducing

accuracy of constraint evaluation (4.6). It is slightly modified to only check every mth

occurrence.

∀(m · n), |τ(ǫr, m · n) − τ(ǫr, m · n − k)| 6 δ, m ∈ N (4.6)

If m is set to one, the constraint provides the full accuracy. If m is increased, the

accuracy is reduced. In this case, it is not required to monitor all events any more. It

is enough to monitor only every mth event. The number of events monitored within

a certain time frame as well as the number of constraint checks is reduced to 1

m. This

increases the performance significantly and reduces the overhead to almost 1

mof the

time. In this case not all QoS violations are detected.

The value of m depends on the probability of the occurrence of QoS violations


and can be determined with the help of a model for the surrounding environment.

Even better, an adaptive monitoring approach can be implemented which decides the

value of m depending on the detected QoS violations.

The reduction of accuracy can be applied to any other constraint, as well.

4.5 Quality Aware Programming Model

Currently, too many applications avoid being adaptive. Adaptive quality-aware pro-

gramming is rather hard and complex. Adaptation is handled on the middleware or on

the system level and its implementation is completely hidden inside QoS frameworks.

Figure 4.1: Comparison between adaptive quality-aware programming and a common

approach.

The Quality Aware Programming model defines a new structure for programming

adaptive quality-aware applications. It allows adaptive programming to be as easy as

exception handling and does not shift the responsibility for adaptive behavior to an

underlying framework or middleware system. A part of the control over adaptivity

can be shifted to the application level. In many cases it is only the application which

knows what adaptation is required.

The example of monitoring the throughput constraint of 25 frames per second

is implemented in Listing 4.25. Implementation is done without the help of special


frameworks or other utilities and uses plain C#. Time recording is done manually

and QoS checking is done directly within the application logic code. Modification of

the constraints would require a complete reimplementation of the whole block.

1 int n = 0;

2 long oneSecond = 1*1000*1000*10; // in 100- nanosec

3 DateTime [] timeStamps = new DateTime [26];

4


6 timeStamps [n % 25] = DateTime .Now ;

7

8 Frame inData = FetchData ();

9

10 if (n >= 25

11 && timeStamps [(n % 25) ]. ToFileTime ()

12 - timeStamps [(n + 1) % 25]. ToFileTime () > oneSecond ) {


14 } else {


16 }

17 n++;

18 }

Listing 4.25: Monitoring the rate of 25 elements per second (common approach).

1 // constraint declaration

2 // timed data type with constraint definition


4


6 try {





11 }

12 }

Listing 4.26: Monitoring the rate of 25 elements per second (Quality Aware

Programming model).

Existing adaptive programming is often a mixture of time measurements and

conditional statements for checking time-based constraints. Adaptive programming


should be encouraged by presenting this programming model which demerges the

code into a well defined structure (Figure 4.1). Together with the presented language

extensions the novel programming model makes adaptive programming easier and

safer both on the system and on the application level.

Adaptive quality-aware programming defines a clear structure by separation of

concerns. Instead of defining conditional statements, declarative definition of QoS

constraints are applied. Adaptive code is placed inside an exception handler and is

only executed if the application code cannot achieve specified requirements.

An implementation with the Quality Aware Programming model (Listing 4.26)

allows separation of concerns and increases readability and reduces maintenance effort

if modifications are required. Adaptive behavior is separated from the application

logic code and placed inside the exception handler.

The Quality Aware Programming model is implemented with the help of the pre-

sented language extensions and the QoS management framework (Section 8.2). The

QoS management framework provides the implementation of event history recording,

time measurements and administrates constraints. The compiler automatically em-

beds method calls to the QoS management framework and generates classes which

implement the declared constraint specifications. The main advantage of this ap-

proach is to provide exchangeability of the QoS framework components, e.g. event

history recording, time measurements without modifying the compiler.

CHAPTER

5 Evaluation

This chapter evaluates the Quality Aware Programming model and the corresponding

language extensions. This is done by comparing the defined requirements with the

implementation. Furthermore, implementation alternatives, historization challenges

and performance aspects are discussed.

5.1 Quality Aware Programming Reviewed

The Quality Aware Programming model defines a clear structure for the implemen-

tation of adaptive QoS-aware applications. Listing 5.1 illustrates the basic building

block used for implementing QoS-aware applications. It consists of three parts which

can be directly mapped to the three problems of constraint specification, constraint

monitoring and adaptation.

• Constraint declaration - The QoS constraint specification is placed outside

the application logic code and uses a declarative specification syntax which

allows to apply constraint analysis. This analysis checks the correctness of

specified constraints and additionally optimizes complex constraint definitions.

• Application logic with automatic QoS monitoring - The application logic

is implemented without interfering monitoring code. Time measurement code

and conditional statements which check QoS constraints are not required to be

CHAPTER 5. EVALUATION Page 98

implemented manually. The application developer only uses the timed assign-

ment or timed method parameters if QoS-awareness is required. Everything

else is automatically done by the compiler.

• Adaptation code - Implementing adaptive behavior is as easy as any other

kind of exception based programming. In case of a QoS violation, the moni-

toring code throws an exception automatically. Adaption code is placed inside

an exception handler which processes the caught QoS-related exception. This

behavior can be used to parameterize application logic to change its process-

ing strategies. The Quality Aware Programming model does not cover specific

adaptation techniques and their implementation challenges.

1 object [~@n {...}] obj ; // constraint declaration

2 try {

3 ... // application logic


5 ... // adaptation code

6 }

Listing 5.1: Quality Aware Programming Model - basic code structure.

5.1.1 QoS Specification Requirements

The requirements for specifying QoS constraints are formulated in section 3.1 by the

following definitions:

• Expressiveness

• Declarativity

• Independency

• Extensibility

• Reusability


Expressiveness is provided by using a declarative constraint syntax similar to the

mathematical constraint specification syntax of QL (Quality Language). Further-

more, time is represented as the n + 1st dimension of any data type which allows

access to time values like simple array element access. Although the main focus of

the programming language is to simplify time based QoS constraint handling, it is also

possible to express a wide variety of different QoS requirements including temporal

and non-temporal constraints.

The constraint syntax is declarative in nature. It allows specification of QoS

requirements similar to QL. The syntax focuses on formulating requirements and does

not define how QoS requirements are checked. Additionally, a declarative constraint

can contain specific throw statements applying user-defined exceptions to distinguish

between different QoS violations. Such exceptions can be evaluated to find the correct

adaptation technique.

Constraints are independent of the applied context. Constraints can be declared

locally in the context in which they are required as well as somewhere else in the

code. Parameterizable constraints can be defined, e.g. within a library or within the

classes containing QoS-aware behavior.

1 public constraint Throughput

2 (object [~] obj , int rate , int timeRange ) =>

3 @n{obj[n] - obj[n-rate] < QoS.Units .MSec(timeRange )};

4

5 public constraint ThroughputAndJitter =>

6 (object [~] obj , int rate , int timeRange , int allowedJitter) =>

7 Throughput (obj , rate , timeRange ) &&

8 @n{obj[n] - obj[n-1] > QoS.Units .MSec(timeRange /rate - allowedJitter)

9 && obj[n] - obj[n-1] < QoS.Units .MSec(timeRange /rate + allowedJitter)};

Listing 5.2: Refinement of Declarative QoS Constraints.

Extensibility and Reusability are achieved by declaring parameterizable constraints.

A general definition of QoS requirements is declared which then can be reused within

another QoS constraint specification. This enables refinement. General QoS con-

straints can be defined and then reused within other constraints to declare more spe-

cific constraints. Listing 5.2 illustrates the refinement of the throughput constraint


by appending an additional jitter constraint.

Furthermore, a parameterizable constraint can be attached to many different

timed variables and used within many different contexts.

5.1.2 QoS Monitoring Requirements

The requirements for monitoring QoS constraints are formulated in section 3.2 by the

following definitions:

• Automatic constraint monitoring

• Exception based QoS violations

• Independence of application logic and adaptation code

• Open QoS processing framework

Timed assignment and timed parameter passing mode include automatic code in-

strumentation to provide monitoring and constraint checking facilities. The developer

is only required to define which actions should be executed under QoS control.

In case of QoS violations, exceptions are thrown. These exceptions only occur

when applying a timed assignment or using the timed parameter passing mode. The

run-time part of the constraint analysis can also cause an exception in case the run-

time applicability of the provided constraint conditions is not ensured.

Due to using exceptions in case of QoS violations, the application logic code can

be free of adaptation code. Adaptation code is placed inside exception handlers.

However, application logic needs to be parameterizable at run-time to allow changing

its processing behavior.

QoS management is part of the system class library. Its components can be

exchanged to implement different management strategies. A factory class can be con-

figured by advanced developers to use different implementations of the historization

process.


5.1.3 Requirements for Language Extensions

The requirements for extending an existing programming language are formulated in

section 3.3 by the following definitions:

• Expressiveness

• Safety

• Optimizations

Extending a programming language is very controversial. Normally, extensions

which make the life of programmers easier should be placed inside a library, but in

the case of QoS-aware programming the disadvantages prevail.

A library cannot add new operators easily although some languages exist where the

behavior of existing operators can be overloaded. Furthermore, application logic still

requires calls for constraint monitoring. It is possible to define complex frameworks

which use additional specification languages which can be found in the literature.

These frameworks make development of QoS-aware software feasible but not easier. A

library implementation of a QoS-aware framework is not able to provide declarative,

language embedded constraint specifications which can be analyzed and optimized

automatically. This is only possible if constraints are defined within separate spec-

ification languages which are evaluated during execution of QoS-aware applications.

A library cannot provide compile time support for declarative QoS constraints or

correctness checks. Implementing QoS manually is hard to achieve and error-prone.

One forgets easily to add constraint monitoring or time recording method calls which

might be difficult to figure out during test phases.

Nevertheless, there are many different approaches which use library based QoS

handling. In many cases, this is achieved by providing a separate layer with QoS-

enabled services and API calls which need to be called. Such approaches focus mainly

on network based QoS like socket calls. The approach of this thesis is different.

It tries to make QoS-aware software development as simple as any other form of


software, using exception handling. This is achieved by providing few extensions on

the programming language level.

To make the presented language extension even more plausible, the language ex-

tensions have been defined with respect to the requirements: expressiveness, safety

and optimization potential.

The Quality Aware Programming model along with its language extensions in-

creases the expressiveness of QoS-aware application by defining a clear structure and

a declarative way to specify QoS requirements. The safety is significantly increased

by automating the process of constraint monitoring. Furthermore, correctness checks

increase the safety even more, due to the fact that the definition of inconsistent

temporal constraints is not possible. Last but not least, optimization potential is uti-

lized by automatically reconstructing complex constraint definitions. Specifications

of complex constraints can often lead to multiple conditions which overlap or are not

needed at all. Furthermore, it is possible to add a QoS violation tolerance or to re-

duce monitoring accuracy to achieve an increase in performance. These findings need

further study and can be exploited in future.

5.2 Alternative Approaches

Several alternative approaches have been evaluated. This section gives a brief over-

view about advantages and disadvantages of other strategies.

5.2.1 Library Implementation

On the one hand, a library has several advantages. It is very easy to distribute

and allows encapsulation of functionality which is accessible through well defined

APIs. A library may increase safety due to reuse of existing, well tested parts of

software. On the other hand, libraries are limited. They are containers of existing

reusable components, but do not increase the expressiveness of the programming

language. Well defined interfaces help to increase readability and can make libraries


easily understandable. In case of QoS constraint specification, no novel syntax can

be introduced which allows constraint declaration with the help of mathematical

formulas. Instead, only methods and calling conventions are used to express certain

behavior.

The main advantage of a library implementation is that there is no need to change

the compiler itself. A set of methods can be used to express predefined constraints,

trigger timing measurements and check the applied constraints. In contrast, the

modification of the compiler can increase the expressiveness by adding novel syntactic

elements. It further helps to make the code more safe, as the compiler can enforce

certain code structures. It has the possibility to analyze code at compile time and

thus can provide error messages before executing incorrect code. The compiler helps

the programmer to detect mistakes at compile time.

Findings in the literature have shown that many different QoS-aware libraries

which encapsulate network or operating system calls exist. Complex architectures

hidden in libraries do not necessarily increase the number of adaptive applications

developed.

5.2.2 Aspect Oriented Approach

AOP (Aspect Oriented Programming) [Kic] is a modern approach in the field of

software engineering. Aspects of software are implemented separately. Instead of

spreading such code over the whole application, an aspect (crosscutting concern) is

defined and join points which indicate where the aspect is used are specified. An

additional program weaves the aspect code into the object code after the application

has been compiled. Logging, persistence and also QoS are examples of aspects of an

application. AspectJ [asp09b] and Aspect# [asp09a] are common representatives of

aspect oriented languages.

There is one big limitation using AOP. Join points of existing AOP frameworks

can currently only be before or after a method invocation. There is no possibility to

apply an aspect to a certain piece of code within a method. This cannot be done due


to the fact that reflection on single statements is not available. However, it is possible

to apply aspects on method calls, field access or constructor/destructor calls.

Compared to common libraries, AOP has the advantage that code can be placed

outside the application logic. However, the application code must be well structured

- which is always an advantage - to find method invocations where QoS awareness

can be applied. Furthermore, constraint declaration within the host language is

not possible. Therefore, this approach does not bring significant increase to the

expressiveness of a given programming language.

5.2.3 Template Metaprogramming with C++

Another possibility for implementing QoS-aware applications is template metapro-

gramming with C++. Templates provide compile-time parameterization for classes or

functions and thus allow the definition of generic QoS data types similar to the timed

types presented. Operator overloading can be used to change the semantics of the

assignment operator and provide QoS monitoring code. Expression templates [Vel95]

might be usable for constraint declaration, but implementation of constraint analysis

is practically not possible due to the enormous complexity of template metaprogram-

ming.

Template metaprogramming [wik09c, CEC00] shifts parts of runtime evaluation

to the compilation process which may increase compile time significantly for large

projects with massive use of templates. It allows the definition of generic code.

However, readability of the code is reduced and the code is quite hard to understand

and maintain. Debugging is very difficult and it is not possible to produce specific

error messages during the compilation process. Portability and compiler limits harm

the power of template metaprogramming. Advanced language features might not

be supported by every compiler implementation. Complex types can quickly reach

compiler limits which vary depending on the compiler implementation.


5.3 Historization Challenges

The historization process covers the recording of time events which are triggered

during timed assignments or method invocations with parameters using the timed

parameter passing mode. Several questions emerge when looking at the historization

process in detail.

• How many events need to be stored to allow correct constraint evaluation?

• Is it necessary to capture all events?

• How is constraint evaluation handled if the history is empty?

5.3.1 History Size

The runtime environment is required to record events during execution. The size of

the history needs to be limited to avoid wasting memory. The amount of events which

are stored within the history should be chosen with knowledge about constraints, e.g.

a delay constraint requires two values for comparison.

The constraint analysis provides information about the number of events that

are required to be recorded. This value can already be provided at compile-time if

constant values are used for history access. In case dynamic values are used, the

run-time part of the constraint analysis manages the history size accordingly. If the

history size cannot be determined, a default size of 100 events is assumed.

5.3.2 Event Recording

In general, all events are recorded, irrespective of their requirement. This is done au-

tomatically by compiler generated code. It depends on the applied constraint if every

event is required. All examples provided within this thesis require a complete history

recording for correct constraint evaluation. This means that no event can be omit-

ted. It might be possible to define constraints which only require a certain subset of


events. Listing 5.3 illustrates an example where only even events need to be recorded

within the history to successfully evaluate the constraint. Such constraints have not

been recognized in multimedia applications and because of that this functionality is

currently not covered by the actual implementation.


2 @n {n % 2 == 1 || var [n] - var[n-2] <= Units .MSec (10)}

Listing 5.3: Delay of event events.

5.3.3 History Initialization

History initialization is another interesting aspect. The current implementation uses

an empty history as the starting point. If enough events have been recorded, the

constraint evaluation starts. In case a constraint is violated, all associated histories

are reinitialized. This is necessary in order to not carry out erroneous processing

which can lead to false violations.

The current behavior fits well in case of checking maximum timing behavior. It

allows correct evaluation if a certain process takes too long. It does not fit well to

monitor minimum timing behavior because QoS monitoring always starts with an

empty history. In this case, the application behavior depends on the implemented

timing characteristics. QoS monitoring and control only work after the history con-

tains enough values to evaluate the associated constraint. If adaptivity is used to

slow down execution speed, it is possible that constraint monitoring always detects a

constraint violation when the constraint is checked for the first time.

Listing 5.4 provides an example of monitoring the minimum and the maximum

timing behavior of a loop iteration. The example makes use of the Thread.Sleep

method to delay execution and defines a required delay which has to be greater or

equal 100 milliseconds and less or equal 200 milliseconds. In case a processing step is

too fast, the sleep interval (sleepTime) is increased. If the processing step takes too

long, the interval is decreased. The sleep interval starts with value 0 and expects the

processing to take more than the lower bound of the constraint.


1 class DelayViolation : QoSException {

2 private long diffTime ;

3

4 public DelayViolation(long diffTime ) : base ("Violated by " + diffTime ) {

5 this.diffTime = diffTime ;

6 }

7

8 public long GetDiffTime () { return this.diffTime ;}

9 }

10

11 private constraint minMaxDelay (object [~] o, long min , long max ) =>

12 @n{o[n]-o[n -1] >= QoS.Units.MSec(min ) : throw new DelayViolation(QoS.Units .MSec(min

) - (o[n]-o[n -1]))} &&

13 @n{o[n]-o[n -1] <= QoS.Units.MSec(max ) : throw new DelayViolation(QoS.Units .MSec(max

) - (o[n]-o[n -1]))};

14

15 public bool ProcessData () {/* ... some processing */}

16

17 // ...

18

19 bool[~ minMaxDelay (dataOk , 100, 200)] dataOk = true;

20 long sleepTime = 0; // delay to reduce processing speed

21

22 while (dataOk ) {

23 try {

24 System .Threading .Thread .Sleep ((int )sleepTime );

25 dataOk ~: ProcessData (); // process data

26 } catch (DelayViolation e) {

27 if (e.GetDiffTime () > 0) { // too fast , increase sleep time

28 sleepTime = e.GetDiffTime ();

29 } else { // too slow , reduce sleep time , e.GetDiffTime is already negative

30 sleepTime = sleepTime + e.GetDiffTime ();

31 if (sleepTime < 0) sleepTime = 0;

32 }

33 }

34 }

Listing 5.4: Minimum and Maximum Delay constraint.

It is possible to provide a precalculated history with an optimal timing behavior.

This may work fine in certain cases, but does not necessarily provide correct pro-

cessing behavior. Succeeding constraint violations can occur due to the fact that a


precalculated timing history was used.

5.4 Performance

5.4.1 Run-Time Performance

The presented QoS monitoring facilities introduce an additional overhead to program

execution. It is obvious that the timed assignment operation cannot compete with

the standard assignment statement which can be mapped to a single CPU instruction

in most cases. Instead of executing a single statement, additional functionality is

invoked with the help of the QoS management framework. The overhead produced

is approximately one order of magnitude compared to the original operation. Thus,

the question arises: Is it feasible to use the presented features at all?

In a typical application only a tiny portion of normal assignments need to be

replaced with streaming operations which are executed under QoS control. Thus,

the effective overhead caused by adaptive programming is much less than it might

be expected at first. The performance overhead is investigated by splitting up the

execution costs of a standard video decoding process. The amount of time used for

the decoding process of an H.264/AVC [h26] implementation is analyzed in detail

in [SFLB07a]. One of the most expensive parts of the overall process is the inverse

transformation which needs more than 40% of the time.

When applying the language features for quality aware programming, they pro-

duce only an overhead less than 1% of the whole process. The timed assignment

statement is added to assign fully reconstructed frames to a display variable. Exe-

cuting the application with a required frame rate of 25 frames per second causes an

overhead of much less than one millisecond.

Several measurements were executed to show the performance impact of the timed

assignment statement. The complexity of the constraint declaration as well as the

number of timed assignments need to be taken into account. A constraint declaration

can embed method calls which might harm constraint evaluation time.


All measurements were executed on a dual-core Intel Centrino (2GHz) machine

with 4GB RAM, running on Windows. Both execution time using Mono and the .NET

framework have been evaluated. The time needed for constraint checking increases

linearly with the number of constraints assigned to a timed variable. Furthermore,

the time required for constraint monitoring increases linearly with the number of

timed assignments (Figure 5.1).

An interesting aspect is that the .NET run-time executes the code faster than the

Mono run-time. Measurements showed that the difference in performance is related to

the execution time of a single method call. The measurements were done executing

one million method calls several times and pointed out that the performance of a

single method call in Mono 1.9.1 was slower compared to the performance of the .NET

framwork version 3.0. As the compiler adds additional method calls to implement

QoS monitoring, the performance difference in executing simple method calls explains

the difference between the two platforms. More detailed measurements are required

to give an extensive explanation about the performance difference which are not in

scope of this work.

The timed parameter passing mode performs similarly to the timed assignment

statement (Figure 5.2). Comparing the timed assignment or timed method parameter

passing mode with a manual implementation as shown in section 4.5, the implemented

language features are two times slower. The performance difference is explained easily

as the manual implementation is directly tied to the applied context and only contains

a minimum number of statements. In contrast, the timed assignment makes use of the

QoS management framework (Section 8.2) and includes several method calls (Section

8.2.3) instead of a single if-statements.

Although the absolute execution time of timed assignments presented seems to be

extremely slow compared to the standard assignment, the amount of quality checking

does not harm execution of a video player or other multimedia applications.

Two different implementations of time recording have been implemented within

the QoS management framework. In general, multimedia applications monitoring


Figure 5.1: Performance of Time Assignment Statement.

frame rates do not necessarily need high resolution timers. Such standard timer

accuracy depends on the currently used framework and allows monitoring in the

range of a few milliseconds. Quality checks on millisecond level may not work as the

order of magnitude of the accuracy of the monitoring reaches the constraint value

itself.

Currently, the QoS management framework implementation does support high

resolution timers which provide correct monitoring on ticks. One tick represents 100

nano seconds in the .NET framework. This accuracy allows exact monitoring of events

within the level of microseconds.

5.4.2 Compile-Time Performance

Another important performance aspect is compile-time performance. Due to the fact

that the compiler implementation has been changed the modifications could have an

impact on the duration of the compilation process.

In fact, additionally generated code introduces a compile-time overhead, but this

overhead is very small and depends on the use of the presented language features.


Figure 5.2: Performance of Time Parameter Passing.

Measurements show that the implementation of a quality aware video playback ap-

plication causes an overhead of less than 1% of the overall compilation time.

Constraint analysis at compile-time is very fast, too. The actual cost of this

additional step during compilation is hard to measure. Quality aware applications will

make use of a very small number of complex constraints and not applying constraint

analysis on thousands of constraints. The additional overhead provided by this step

is relatively small and depends on the number of complex constraints and on their

complexity.

All in all, the produced overhead at compile-time is very small and can be ne-

glected.

5.4.3 Impact of Garbage Collection

Garbage collection is a general problem for multimedia applications. If garbage col-

lection takes too long or executes too often, it may harm soft real-time execution.

Multimedia applications require enough CPU time for executing time critical code.

If the application instantiates and frees many objects within short periods of time,


it is possible that the garbage collector decreases the performance and prevents the

application from achieving its soft real-time behavior. Although the presented lan-

guage extension is not explicitly bound to a managed environment and could have

also been implemented directly within C++ or another non-managed environment,

the impact of garbage collection needs to be discussed.

Much research was done finding appropriate garbage collection algorithms and

implementations for multimedia applications in managed and also unmanaged envi-

ronments. This thesis does not concentrate on alternative implementation approaches

for different programming environments. Furthermore, it discusses the impact of

garbage collection on the presented language extensions as well as the general impact

on multimedia applications.

Zerzelidis and Wellings [ZW05] define requirements for a real-time .NET frame-

work based on the requirements for a real-time java platform defined within the NIST

report [nis99]. In their opinion, two possibilities for garbage collection exist. Either it

is completely disabled or if provided it must be real-time aware. Garbage collection

needs to be preemptive and requires a bounded pre-emption latency in case a higher

priority task needs to run.

Claypool et al [CCH+00] discuss garbage collection impact on video processing in

Java. They analyzed an MPEG video player and show the processing capabilities of

Java with respect to video display. They state that in general a Java virtual machine

often spends between 15% and 20% of its time on garbage collection. Furthermore,

they measured that the performance impact on video processing is negligibly small

by executing their video player with garbage collection enabled and disabled.

McGachey et al [MATH+08] use a different approach to provide soft real-time

garbage collection. They combine transactional memory and concurrent garbage col-

lection to reduce pause times of applications. They address that concurrent garbage

collection becomes more and more important due to the increasing number of multi-

core CPUs available.


Garbage collection in .NET [Ricb, Rica] is implemented using three different gen-

erations of objects to improve performance. The life-time of objects is very different

as there are objects which are available during the whole life-time of the application

and objects with a very short life-time which are instantiated, used and immediately

dismissed. The generational approach allows a distinction between the life-time of

objects and assumes that many objects can be immediately garbage collected after

their use. It uses different areas in memory for objects of different generations. Each

time the garbage collection runs for a specific generation it increments a counter and

as the life-time of an object increases it moves to the next generation. This allows

garbage collection of low generations to run more often and thus much faster. Ob-

jects in higher generations are more likely to have a very long life-time and do not

need to be garbage collected very often. Within the different generations conventional

garbage collector algorithms can be applied.

In the case of .NET the Mark-And-Compact strategy is used. During the marking

phase it recognizes application roots which are start objects from which reference

graphs are built to detect reachable objects. After marking the compact phase moves

used memory to the bottom of the heap and the rest is declared as unused memory.

This automatically removes fragmentation and decreases memory allocation times to

a minimum because free memory does not need to be searched within a fragmented

heap. However, the compact phase has the disadvantage that all pointers to the heap

need to be updated.

Furthermore, .NET provides two different main modes for garbage collection

[Hun09]. In the concurrent mode .NET tries to avoid stopping the program while

collection is in progress. The application will not pause but its performance de-

creases. This mode is used for interactive applications to increase the impression that

the application is still responsive.

In contrast, the synchronous mode suspends the application while garbage collec-

tion runs. This mode is actually more efficient as the application process stops and

garbage collection is run on its own.


In this thesis, the QoS monitoring process has been measured according to the im-

pact of garbage collection. In fact, the actual implementation of the QoS management

framework and the implemented historization process does not interfere with garbage

collection. It uses history array objects which are used to implement a ring buffer.

This ring buffer is allocated during constraint instantiation storing time stamps of

events. The current implementation of the QoS management framework requires only

0.12% of its time for garbage collection when running a simple QoS monitoring loop

evaluating the QoS requirement delay. The memory consumption of this example is

constant due to the memory aware implementation of the historization process within

the QoS management framework.

In contrast, a video player (section 7.6) implementation needs between 8% and

10% of its time for garbage collection. This huge amount can be easily explained and

appears due to the fact that each processing step of the video player allocates its own

block of data. This means that many objects are created which are ready for garbage

collection after displaying one single frame. Concurrent garbage collection in .NET

provides responsiveness of applications and allows smooth display of frames without

notifying annoying pauses during playback.

Furthermore, the ITEC Video Explorer [Sch09] represents a highly optimized ap-

plication which provides video exploration capabilities for H.264/AVC videos. During

video playback it only spends approximately 0.2% of its execution time with garbage

collection. This is achieved by massive object reuse. Almost any object required for

the decoding process is instantiated during application startup. This reduces garbage

collection times as there is nearly nothing to be collected. The .NET garbage collector

is highly optimized and it is advisable to leave its execution to the .NET framework

instead of calling it manually.

The application slow-down which is introduced by garbage collection can force QoS

violations. These QoS violations occur in case a specific garbage collection execution

takes too long. If the application needs 10% of its execution time for garbage collection

and the garbage collector is executed 10 times within a single second, an execution


delay of approximately 10 milliseconds is introduced. In this case, delay constraints

monitoring a delay not to be more than 10 milliseconds will lead to QoS violations.

In general, the impact of the garbage collector depends on the implementation of

the application. The amount and size of objects which are instantiated affect the tim-

ing behavior of the garbage collector. The detailed understanding of how the garbage

collector works can help to avoid unnecessary slow down or huge memory consump-

tion. For high performance programs the time required for object allocation might

be an important issue, too. This problem can often be easily solved by using object

pooling [obj09] which reuses already allocated memory and thus avoids additional

time for allocating and destroying objects on demand.

5.4.4 Operating System Impact

When applying too strict constraints monitoring values within the micro second level,

scheduling of the operating system can have an impact on correct processing. Too

small constraints can easily lead to QoS violations which occur due to the fact that

the operating system scheduler withdraws CPU time from the current application to

process another application or service. As current operating systems like Windows are

not real-time operating systems a specific task execution time cannot be guaranteed

neither a fixed amount of clock-cycles cannot be achieved.

In general, QoS requirements in multimedia applications do not necessarily need

such a high accuracy. In most cases, it is enough to process a task within a few

milliseconds and thus not requiring more precise QoS constraint monitoring.

CHAPTER

6 Quality Aware Program-

ming Patterns

This chapter describes programming patterns which are useful for quality aware pro-

gramming. These usage patterns for constraint monitoring can be easily applied to

newly created as well as already existing applications.

6.1 QoS Monitoring Patterns

QoS monitoring patterns are used as hints for application developers who add QoS ca-

pabilities to their applications. These patterns can be easily integrated in multimedia

applications, but are not limited to them. Although implementation is very sim-

ple, some things need to be considered to provide correct monitoring and processing

behavior.

Before the programming patterns are explained, a few application examples pro-

vide an overview about the fields of application.

6.1.1 Application Scenarios

This section sums up a few application scenarios to show the wide range of possible

applications which can be enhanced by quality aware programming. Adding QoS

handling requires only small changes to the programs.

CHAPTER 6. QUALITY AWARE PROGRAMMING PATTERNS Page 118

Simple Video Player

Writing a video player is not much effort in many general purpose programming

languages. Many handy libraries exist which can decode binary input streams frame

by frame. A simple way to implement a video display component is to define a form

where the display method is overwritten to show a bitmap. Exchanging these bitmaps

can be implemented using a small method applying the streaming parameter passing

mode (Listing 6.1).

1 private Bitmap frame;

2

3 // set new frame for display

4 void SetFrame (

5 stream Bitmap [~@n {frame[n] - frame[n -25]

6 < QoS.Units.MSec (1000) }] frame) {

7 this.frame = frame;

8 this.Refresh (); // causing display of frame

9 }

Listing 6.1: Applying a frame for display.

1 // decode a frame

2 void DecodeFrame () {

3 // decoding process

4 Bitmap frame = GetDecodedFrame();

5

6 try {

7 form.SetFrame (stream frame);

8 } catch (QoSException e) {

9 // lower resolution

10 // speed up decoding process

11 ... // apply code for adaptation

12 }

13 }

Listing 6.2: Decoding a frame and preparing it for display.

Every time the method SetFrame is called it causes the current display to be

refreshed using a new bitmap passed as a parameter using the streaming parameter

passing mode. As long as the defined constraint can be held, the video playback runs

normally. In case of a QoS violation, the method throws an exception and allows the


application to react within the decoding process to speed up the performance. Here,

the adaptational behavior can be implemented by throwing away frames or lowering

the resolution of the decoded frames. The adaptivity itself relies on the possibilities of

the used encoding/decoding implementation. The method SetFrame itself does not

provide adaptivity, rather it enables automatic QoS monitoring easily (Listing 6.2).

Depending on the used decoder, different strategies can be applied to achieve

an optimal display process. The compiler-generated code only ensures that QoS

requirements are monitored and violations are detected automatically.

Streaming Server

1 private Socket client ; // client reference

2 private bool inTransfer = true;

3 // timed variable

4 private Byte [][~@n {byteData [n] - byteData [n-1]

5 < QoS.Units.MSec (50) }] byteData ;

6

7 // prepare single block

8 private Byte [] PrepareData () {

9 // construct binary data for transfer

10 }

11

12 // transfer binary data over the network

13 private void Transfer () {

14 // as long as data exists

15 while (inTransfer ) {

16 try {

17 // stream prepared data under constraint over the network

18 byteData ~: PrepareData ();

19 client .Send(byteData , byteData .Length , 0 );


21 // current constraint was not held

22 AdaptData ();

23 }

24 }

25 }

Listing 6.3: A streaming server monitoring network communication.


The implementation of a simple streaming server can be extracted out of most

network communication examples. The provided example does not concentrate on

file organization, packetization, or multi-threading issues. In general, QoS-aware

programming can be applied everywhere in the source code. In case of streaming, the

network throughput of the server is very important. Applying the timed assignment

supports to ensure that a given throughput is held over an existing network connection

(Listing 6.3).

A method call is responsible for preparing data blocks of fixed length. These

data blocks are transferred over a network via a socket. A timed variable is used

to ensure that a specified bit-rate is held. If the time needed for transfer exceeds

the defined boundaries, an exception is thrown. Then the exception handler calls a

method to adapt the data to be transferred or changes the constraint to a value which

is achievable.

WebService Client

To show an example out of a different application domain, a Web-Service using the

timed parameter passing mode is presented. The following source code [msd07] illus-

trates a simple Web-Service implementation which converts temperature values from

Fahrenheit to Celsius (Listing 6.4).

1 public class TemperatureService

2 : System .Web.Services .WebService {

3

4 [System .Web.Services .WebMethod ()]

5 public double ConvertTemperature(

6 double dFahrenheit ) {

7 return (( dFahrenheit - 32) * 5) / 9;

8 }

9 }

Listing 6.4: Web-Service providing temperature conversion from Fahrenheit to

Celsius.


The client code accesses the Web-Service through the automatically generated

proxy class MyTemperatureService. The stub is created out of the WSDL [web07]

specification. In this example, the temperature service has to be called at least every

second. If this requirement is violated, an exception is thrown.

1 // constraint used for convert method

2 private IQoSObject wsConstraint =

3 @n { dFahrenheit [n] - dFahrenheit [n -1]

4 <= QoS .Units .MSec (1000) };

5

6 // does webservice call

7 public double convert (

8 /* dynamic stream parameter */

9 stream double [~ wsConstraint] dFahrenheit ) {

10 // initialize proxy object

11 MyTemperatureService ws

12 = new MyTemperatureService ();

13 // call the webservice

14 return ws. ConvertTemperature(dFahrenheit );

15 }

Listing 6.5: Web-Service client invoking the temperature conversion service.

The client (Listing 6.5), e.g. a sensor application, calls the method to convert the

temperature values. QoS is achieved by using the timed parameter passing mode.

Each successive call has to be completed within the specified range of time (1000

milliseconds). If this timing behavior cannot be held, an exception is thrown.

6.1.2 Single Variable Monitoring

The single variable monitoring pattern uses one single timed variable which is moni-

tored. Every QoS requirement which can be expressed via one single timed variable

can be attached, e.g. throughput, delay or jitter. This pattern fits best when con-

straining consecutive execution within a block of code, e.g within a loop or successive

method calls (Listing 6.6).

The same behavior can be achieved by using timed method parameter passing

mode. In this case, the timed assignment of the example needs to be replaced by a


method call.

1 object [~ someConstraint] monitoredObject;

2

3 while (/*... */) {

4 // ...

5 try

6 {

7 monitoredObject ~: // ... some value


9 // ... react on violation

10 }

11 // ...

12 }

Listing 6.6: Example of single variable monitoring.

6.1.3 Multiple Variable Monitoring

There are other constraints like the bounded execution time which require two timed

variables to be monitored. This constraint evaluates the execution time of blocks. To

implement bounded execution time, two timed variables are required. The first timed

variable captures the start of the execution of the block. The second timed variable

captures the end. Listing 6.7 illustrates the structure of the implementation.

1 object [~] monitoredStart;

2 // some BET constraint , e.g.

3 // @n{ monitoredEnd[n] - monitoredStart[n] < QoS .Units .MSec (10)}

4 object [~ someBETConstraint] monitoredEnd;

5

6 try {

7 monitoredStart ~: // ... some value

8 {

9 // ... processing block

10 }

11 monitoredEnd ~: // ... some value



14 }

Listing 6.7: Example of multiple variable monitoring.


Although there are two timed variables used, the first one does not require a

constraint as it is just used for event recording which are then evaluated by the

constraint attached to the second variable. This example can be extended to use

more timed variables and complex dependencies between measured execution times.

6.1.4 Required QoS Monitoring

In many cases, the correct processing behavior of a piece of code is dependent on QoS

prerequisites. These QoS requirements which need to be checked before execution

can be embedded with the help of a single timed variable.

Listing 6.8 illustrates the monitoring of preconditions. The timed variable as-

signment is placed before the execution of the actual processing code. In case the

precondition evaluated by inputMonitor is not held, an exception is thrown and the

processing code may not be executed.

1 // Precondition: cpuUtilization < 80%

2 // If used within a loop , delay , throughput , jitter can be used as well.

3 object [~ qosPreconditionConstraint ] inputMonitor;

4

5 try {

6 inputMonitor ~: // ... some value



9 }

10

11 {

12 // ... processing block which requires certain QoS

13 }

Listing 6.8: Example of QoS precondition monitoring.

6.1.5 Provided QoS Monitoring

The provided QoS parameters can be evaluated, too. It is similar to requirements

monitoring (section 6.1.4). In contrast, the output is monitored with the help of a

timed variable.


Listing 6.9 illustrates the monitoring implementation of provided QoS constraints.

The processing code is always executed. In case the QoS requirements are not held,

an exception is thrown which signals the QoS violation and allows the program to

react.

1 // Postcondition: freeMemory > 512 MB

2 // If used within a loop , delay , throughput , jitter can be used as well.

3 object [~ qosPostconditionConstraint ] outputMonitor;

4

5 {

6 // ... processing block which provides certain QoS

7 }

8

9 try {

10 outputMonitor ~: // ... some value



13 }

Listing 6.9: Example of QoS postcondition monitoring.

6.2 QoS Requirements of a Video Player

This section discusses QoS requirements and how they can be specified in the context

of a video player application. Video playback is a widespread and common multi-

media application. QoS capabilities can be applied in several different ways which is

described within the following sections.

6.2.1 Simplistic Model

In general, a video player consists of three independent parts. The first part is

responsible for gathering the data, e.g. from the local disk or the network. This data

is passed on to the next part. The second part processes the media data. In many

cases this is done by a video decoder which transforms the compressed input data

into images ready for display. The last part displays the images. It is also possible


that these images are transformed to fit display needs. Figure 6.1 illustrates these

three blocks.

Data Source Decoder Display- -

Figure 6.1: Video Player Model.

There are many different possibilities to apply QoS to this model. Existing mul-

timedia frameworks like JMF [Web09c] restrict the bounded execution within their

applications. When applying bounded execution time, the following constraints are

required.

∀n, |τ(ǫDataSourceOut, n) − τ(ǫDataSourceIn, n)| 6 δDataSource

∀n, |τ(ǫDecoderOut, n) − τ(ǫDecoderIn, n)| 6 δDecoder

∀n, |τ(ǫDisplayOut, n) − τ(ǫDisplayIn, n)| 6 δDisplay

∀n, |τ(ǫOut, n) − τ(ǫIn, n)| 6 δ

δDataSource + δDecoder + δDisplay = δ (6.1)

On the one hand, bounded execution time can be applied to the whole application.

If a frame rate of 25 frames per second is required, 40 milliseconds are used to limit

bounded execution. In case a QoS violation occurs, the application can react but

does not exactly know which part missed the allowed time range. On the other

hand, if each part constrains its execution time, the problem is to correctly distribute

the overall execution time to each component (6.1). One great disadvantage with this

approach is that bounded execution time applied to each part can only work correctly

if processed data is of the same size. The data source can read a whole image at once,

which then is displayed after decoding. If compressed data like MPEG [MPE] encoded


video is used, the presented QoS model does not work well, due to different frame

types.

Another possibility is to constrain the delay at the display part of the video player.

This mechanism is implemented, e.g. within DirectShow [Web09h]. The delay is

measured for each consecutive frame. If it is violated, the application detects it but

does not know which part of the video player is responsible for the violation (6.2).

∀n, |τ(ǫDisplay, n) − τ(ǫDisplay, n − 1)| 6 δDisplay (6.2)

In general, each part of the video player requires its own constraints. The data

source reads data in amount of bits per time unit. This requirement can be formulated

as throughput constraint. The decoder requires a certain data throughput for the

decoding process. Its output is again throughput, but in amount of frames per second

as data is transformed from a bit stream into a continuous flow of frames. The

display component is the most restrictive one and requires a more or less constant

rate which can be described using delay. If different constraints are applied, the

overall application constraint cannot be determined easily. One possibility is to use

the delay of the display part to define an overall application constraint of type bounded

execution time (6.3).

∀n, |τ(ǫDataSource, n) − τ(ǫDataSource, n − kDataRate)| 6 δDataSource

∀n, |τ(ǫDecoderIn, n) − τ(ǫDecoderIn, n − kDataRate)| 6 δDecoderIn

∀n, |τ(ǫDecoderOut, n) − τ(ǫDecoderOut, n − kFrameRate)| 6 δDecoderOut

∀n, |τ(ǫDecoderOut, n) − τ(ǫDecoderOut, n − 1)| 6 δDecoderJitterMax

∀n, |τ(ǫDecoderOut, n) − τ(ǫDecoderOut, n − 1)| > δDecoderJitterMin

∀n, |τ(ǫDisplay, n) − τ(ǫDisplay, n − 1)| 6 δDisplay

∀n, |τ(ǫOut, n) − τ(ǫIn, n)| 6 δApp (6.3)

Although the different constraints seem to be independent, their values can be


set in relation to each other (6.4). The decoder output constraint limits the upper

bandwidth requirement. The frame rate can be multiplied with the size required for

one single frame (frameSize). In general, video frames are transferred as compressed

data which means that the decompressed frame size specifies an upper boundary for

the bandwidth. Furthermore, the jitter value relate to the display delay and the

output frame rate of the decoder needs to be fast enough to provide frames at least

with an average rate less or equal to the specified delay.

δDataSource = δDecoderIn

kDataRate

δDataSource

6kFrameRate · frameSize

δDecoderOut

δDecoderJitterMin 6 δDisplay

δDisplay 6 δDecoderJitterMax

kFrameRate

δDecoderOut

6 δDisplay (6.4)

Although specifying QoS constraints for each part in detail is much more complex,

the application benefits from detailed knowledge where a constraint violation occurs.

Furthermore, this allows specific adaptation to be applied at the place where it is

required.

Throughput vs Jitter

A common QoS requirement for video players is frame rate. For the end user it

is enough to specify the number of pictures which should appear within one single

second. Frame rate can be represented via the QoS constraint throughput. This

constraint is defined as the amount of events per time unit and does specify the exact

position of events within the time unit. The following QoS requirements seem to

express the same behavior but when looking at them in detail, they are completely

different.

1. 1 frame per 40 milliseconds


2. 25 frames per 1 second

3. 1500 frames per 1 minute

4. 90000 frames per 1 hour

All of these constraints define a frame rate of 25 frames per second. When looking

at the mathematical definition, the four constraints can have four completely different

meanings. The first constraint is the most strict one. It can be compared to an exact

rate definition with 40 milliseconds delay. The second constraint requires 25 frames

per second, but does not specify when these 25 frames occur within one second (section

2.2.1). The last two constraints are even less exact. If a video player were fast enough,

it could display 90000 frames within the first second and then wait till the end of the

hour is reached.

Another problem when evaluating these constraint definitions is the amount of

memory which is required. The first definition requires only one event to be stored,

whereas the last constraint requires to store 90000 events which needs a huge amount

of memory.

Instead of defining a throughput constraint, the delay between two consecutive

events is enough to reach the required behavior. In more detail, the frame rate of 25

frames per second can be redefined to a frame-to-frame delay of 40 milliseconds. Exact

rates are hard to achieve and often it is enough to reach approximately the required

rate due to the inertia of human sense organs. Furthermore, current operating systems

are no real-time systems, so it might not be possible to achieve a constant rate at all.

Instead of defining a rate constraint, jitter can be used.

As jitter seems to be the ideal constraint for video players, it needs to be applied

at the correct place.

It is obvious that jitter cannot be applied to the data source. The constraint is

defined as minimum and maximum delay between two events. Network traffic cannot

be mapped to this constraint, because it does not say anything about bandwidth or

packet size. If jitter is applied, the data source has to deliver a data packet within a


given rate, e.g. 40 milliseconds ± 10 milliseconds. This would mean that the required

bandwidth needs to be mapped to the transferred packet size. When changing the

resolution of a video, it would require to change the packet size for transfer instead

of varying between packet size and packet rate.

Another possibility would be to apply delay jitter to the decoding component.

At first, this seems to be possible. A decoder delivers frames with a certain frame

rate, but the time slices between the output frames differ. It depends on the video

codec used. If an MPEG video [MPE] is decoded, the processing requirements for

frames are not equal. Due to different frame types, it is not possible to guarantee

that a decoding component can deliver output images with a fixed delay. In general,

a decoder is required to deliver a certain throughput, but the delay between frames

can vary more than the jitter constraint for display would allow.

The display component is the only component where jitter can be applied correctly.

It defines the timing requirements to successfully display a video for the human eye

and allows small variations which cannot be detected by a human being.

Actual video player implementations do not use detailed QoS monitoring. In

fact, they only apply QoS measurements to the display component to display frames

correctly rather than monitoring the whole process to figure out which parts of the

application require the appropriate adaptation.

6.2.2 Model Using Playback Buffer

Different types of codecs provide variable timing behavior. In many cases the decoder

introduces jitter due to the fact that multiple frame types are used. Each frame type

results in variable amount of processing power, bandwidth or memory required which

directly affects the timing behavior. To reduce jitter introduced via the network and

the decoder, an additional buffer is required. The buffer is responsible for temporarily

storing decoded frames to provide smooth output for the display part.

The previously defined simplified model is extended by a buffer between the de-

coder and the display part (figure 6.2). This buffer converts the output variations of


Source Decoder Buffer Display- - -

Figure 6.2: Video Player Model With Buffer.

the decoder to an output with constant delay.

The buffer can be implemented via the leaky bucket concept [SN04]. This buffer

converts timing variations of input elements to output elements with constant rate.

Figure 6.3 illustrates this behavior.

Figure 6.3: Simple leaky bucket [SN04].

To provide correct display, the buffer fill state needs to be taken into account as

well. In case the buffer gets full, the output rate can be slightly increased to reduce

the buffer fill state. In contrast, if the buffer gets empty, the output rate can be

decreased to allow the buffer to be filled. This behavior needs to be covered by QoS

constraints, as well, in order to not produce QoS violations when reacting on the


playout buffer fill state. It can be achieved by constraining requirements to certain

ranges instead of concrete values.

6.2.3 Client-Proxy-Server Model

The previous models show QoS requirements of a video player based on a local imple-

mentation of all parts. Instead of processing the whole video locally on one physical

machine, the model can be split into three different parts running on physically in-

dependent hardware communicating over the network.

• Streaming server - is a multimedia application providing data at a constant

rate.

• Streaming proxy cache - is a multimedia application which caches multi-

media content to store it physically near the clients and which provides video

adaptations to transform content to fulfill user needs.

• Streaming client - is a multimedia application which is responsible for dis-

playing multimedia content.

These three applications have different QoS requirements which are discussed in

this section in detail.

A streaming server [KSB09] has the task to stream out media data. It consists of a

data source part which reads data from the local disk and a second component which

is responsible for transferring the data over the network. In many cases, adaptation

possibilities exist within streaming servers to transform content before delivery. This

thesis does not cover disk allocation strategies or caching methods for multimedia

content.

∀n, |τ(ǫDataSource, n) − τ(ǫDataSource, n − kDataRate)| 6 δDataSource

∀n, |τ(ǫNetworkOut, n) − τ(ǫNetworkOut, n − kDataRate)| 6 δNetworkOut (6.5)


To generate a constant rate, a buffer is required between data allocation and

transfer over the network. In this case, the throughput constraint (6.5) fits best to

define the amount of data which needs to be transferred over the network.

∀n, |τ(ǫNetworkIn, n) − τ(ǫNetworkIn, n − kDataRate)| 6 δNetworkIn

∀n, |τ(ǫNetworkOut, n) − τ(ǫNetworkOut, n − kDataRate)| 6 δNetworkOut (6.6)

The streaming proxy cache [RK01, SBH+03] is an additional application to reduce

network bandwidth and cost. It allows media data caching to provide it for further

clients and often implements adaptation techniques to manipulate multimedia con-

tent. This can be done to reduce local storing amount or to adjust the content to

achieve certain client requirements, e.g. display size, bandwidth. Again, the input

and output requirements of a proxy cache are throughput constraints. It gathers data

with a defined throughput either from a streaming server or the local cache and pro-

vides it with a defined output throughput requirement specified by the client (6.6). If

video adaptation is also implemented by the proxy, an adaptation engine transforms

the multimedia content. It depends on the implementation of the proxy if the adap-

tation process runs in background as a management operation which does not require

any QoS metrics or if it is applied when a client requests content. If QoS metrics are

needed, it depends on the adaptation technique if data throughput in amount bytes

or a frame rate fits better. It may be possible to use completely different constraints,

as well. Moreover, it has to be considered that additional processing always increases

the end-to-end delay.

The streaming client is similar to the previously defined local video player model.

The only difference is that data is read from the network and not from the local disk.

Again, each component within each application requires separate QoS monitor-

ing to provide fully adaptive behavior and to detect QoS violations at their source.

However, the same constraints can be applied as defined for the previous models. Fur-

ther investigations are required to define a distributed model dealing with buffering,

communication issues and QoS negotiation.


6.3 Rated Buffer

QoS processing can also be implemented directly within buffers. In this section, a

bounded buffer is utilized to store data units generated by different producers and

provide a more or less constant rate to consumers.

The bounded-buffer problem is a classical multi-process synchronization problem

[Web09j]. Two or more separate processes share a common ring buffer with fixed size.

Two different process types are used. Processes of the first type are called producers

and store data into the buffer. The second type contains consumers which retrieve

data out of the buffer. Synchronization is required to assure correct processing.

Producers are prevented from overfilling the buffer and consumers are forced to wait

if the buffer is empty.

1 // min rate for data production

2 private constraint ProducerRateConstrol (object [~] element , int timeUnit , int avgRate )

=>

3 @n {element [n] - element [n-avgRate ] <= QoS .Units .MSec(timeUnit )

4 : throw new TooSlowException(element [n], element [n-1], QoS.Units.MSec(timeUnit )

/ avgRate )

5 };

Listing 6.10: Minimal Rate Constraint for Producers.

In case of the rated buffer, the QoS requirement of a certain output rate is added.

This requires the bounded buffer to be modified to add QoS processing capabili-

ties. The output rate is defined to provide a smooth more or less constant rate to

consumers.

The rated buffer automatically slows down participants in case they are too fast

and notifies them via an exception if they are too slow. The size of the buffer is

defined as bucket size and controlled with the help of timed variables. It requires

an average rate from the producer and forces the consumer to wait if it is too fast.

Producers are automatically forced to wait if the buffer is full. This is achieved by

object synchronization which is a default behavior of the bounded buffer. In case

the producers are too slow and do not reach a required average rate, an exception


is thrown to notify a QoS violation. This allows the producers to react and modify

their processing strategies to speed up performance (Listing 6.10).

1 // max /min rate for data retrieval

2 private constraint ConsumerRateConstrol (object [~] element , int timeUnit , int avgRate ,

int maxRate ) =>

3 @n {element [n] - element [n-1] > QoS.Units.MSec(timeUnit )/ maxRate

4 : throw new TooFastException(element [n], element [n-1], QoS.Units.MSec(timeUnit )

/ maxRate )}

5 && @n {element [n] - element [n-1] < QoS.Units.MSec(timeUnit )/ avgRate

6 : throw new TooSlowException(element [n], element [n-1], QoS.Units .MSec(

timeUnit ) / avgRate )

7 };

Listing 6.11: Minimal/Maximal Rate Constraint for Consumers.

Consumers are forced to wait to achieve a certain rate. This is implemented with

the help of a constraint specifying a required average and a maximum rate (Listing

6.11).

1 object [~] inObj ; // timed variable for input

2 // initialized with ProducerRateConstrol constraint

3

4 /**

5 * Put an object into the buffer

6 * Wait if the buffer is full

7 */

8 public void Put (Object obj) {

9 lock (this) {

10 if (n == size) Monitor .Wait(this); // wait if the buffer is full

11 buf[tail] = obj;

12 tail = (tail + 1) \% size; n++;

13 Monitor .Pulse(this); // notify next waiting thread

14 inObj ~: obj;

15 } // lock

16 } // Put

Listing 6.12: Put Method Used by Producers.

The Put method is called by producers to store data into the buffer (Listing 6.12).

If the buffer gets full, the producer is automatically delayed by the synchronization

block. In order to keep an accurate buffer fill state, a timed variable (inObj) is at-

tached to the ProducerRateConstrol constraint. The constraint requires an average


rate to be achieved by the producer. If the producer is not able to put data into the

buffer with the average rate, an exception is thrown to notify the producer to speed

up its processing.

1 int sleepTime = 0; // initially sleep time is zero

2 object [~] outObj ; // timed variable for output

3 // initialized with ConsumerRateConstrol constraint

4

5 /**

6 * Get an object out of the buffer

7 * If there is no object in the buffer ,

8 * wait till one has been put in.

9 */

10 public object Get () {

11 lock (outputLock ) {

12 if (sleepTime > 0)

13 System .Threading .Thread .Sleep (sleepTime ); // delay execution

14 lock (this) {

15 if (n == 0) Monitor .Wait(this); // Check if buffer is empty

16 try {

17 outObj ~: buf [head ];

18 } catch ( TooFastException e) {

19 sleepTime += e.delay; // increase delay if too fast

20 } catch ( TooSlowException e) {

21 sleepTime -= e.delay; // decrease delay if too slow

22 if (sleepTime < 0)

23 sleepTime = 0; // avoid negative value

24 }

25 head = (head + 1) \% size; n--;

26 Monitor .Pulse(this); // notify next waiting thread

27 return outObj ;

28 } // lock (this)

29 } // lock (outputLock )

30 } // Put

Listing 6.13: Get Method Used by Consumers.

Handling a more or less constant output rate is much more complicated than

requiring a minimum input rate (Listing 6.13). A sleepTime variable is used to

suspend the current thread execution. An additional lock is required to synchronize

all consumers to achieve the required rate. A control loop is provided by extracting

wrong execution times out of the QoS exceptions and modifying the sleep interval


accordingly. While the current consumer thread is sleeping, producers are not blocked

in storing their data. Furthermore, notifications triggered when putting data into the

buffer are ignored, as no consumer thread is waiting for a notification.

If the consumer requests data which is detected by the constraint monitoring as

too fast, a TooFastException is thrown. This exception indicates the amount of time

required to achieve a specified rate and influences the current sleep interval. In case

the execution is too slow - this, in fact, means that the sleep time is too high - it is

decreased accordingly.

CHAPTER

7 Quality Aware Multimedia

Framework

This chapter presents QaMF (Quality Aware Multimedia Framework), a novel multi-

media framework exploiting the Quality Aware Programming model and the according

language extensions provided by MMC#. The motivation for creating it was the miss-

ing or poor support of QoS processing facilities in existing widely used multimedia

frameworks [Web09c, Web09h, WT09, Web09a].

Similarly to many other frameworks, QaMF uses the pipe-and-filter pattern [GS93]

but concentrates on adaptive, quality aware programming. Its main focus is not to

provide implementations of video codecs like FFmpeg or GStreamer. In contrast, it

natively adds QoS to any multimedia application and makes adaptive programming

substantially easier. Instead of implementing video codecs manually, the framework

embeds FFmpeg [Web09a] within its encoding and decoding components. This allows

QaMF to enrich current frameworks and tools with QoS by integrating such tools into

adaptive quality aware components.

QoS is provided by applying two different models: the BET (bounded execution

time) and the LB (link based) model. This approach is not completely new. Parts

of these models can be found within existing frameworks. For example, JMF uses

the BET model for constraining its transformer chain. GStreamer and Direct Show

constrain execution times at rendering components which can also be implemented

with QaMF. DJINN, QUINNA and Infopipes have more QoS capabilities, but QoS

definition is cumbersome and no programming language support is available.

CHAPTER 7. QUALITY AWARE MULTIMEDIA FRAMEWORK Page 138

QoS monitoring is done without preemption, this means that constraints are eval-

uated through timed assignments and do not interrupt the currently executed code.

In other words, the framework can only react after QoS violations, which is in fact no

problem for multimedia applications. QaMF was designed for soft and not for hard

real-time systems.

7.1 Framework Architecture

QaMF is a component based multimedia framework for building quality aware multi-

media applications. Its main concern is to natively support QoS and to provide tools

for building multimedia applications in an easy and flexible way. Application is used

in a very general sense here. A streaming file system is also considered as a multime-

dia application as well as a video player. Both can be implemented easily with the

help of QaMF. Applications are constructed by combining a series of predefined or

self-written components. The result is a processing pipe where data is put through.

A component manager is responsible for either pushing or pulling data through the

pipe.

The framework distinguishes between three different types of components.

• Transmitters (data sources) - Transmitters produce data or forward data

which is received from some input/output device such as the network interface

or a camera.

• Transceivers (data manipulation components) - Transceivers are used to

manipulate data, combining the roles of receiver and transmitter. A transceiver,

for example, can implement a decoding component which receives encoded data

as input and provides decoded data as its output.

• Receivers (data sink) - Receivers are data consumers such as display compo-

nents or elements that send data to an input/output device, e.g. the network

interface or the disk.


Figure 7.1: Quality Aware Multimedia Framework Architecture.

This framework is structured into three different areas, not to be confused with

the three types of components. Centralized communication handling is carried out

within the ComponentManager which is responsible for controlling the execution of

the component chain and also handles QoS violation events. The second block is

the component chain itself, which usually consists of several components. These

components are designed to be quality aware and raise events in case QoS violations

occur. The component chain is responsible for data processing. The third area

is the set of adaptors which are required to define specific actions to handle QoS

violations. Adaptors can enhance or reduce the quality of a given stream using

arbitrary adaptation techniques, e.g. stream switching, frame dropping, or, in the

ideal case, dynamically select the necessary layers of an scalable video coding (e.g.

SVC) stream [Web09k].

7.2 Adaptive Quality Aware Components

The framework utilizes components as fundamental building blocks of its applications,

essentially designed using the structure defined by Gibbs [Gib95]. Gibbs states that

components have at least three different types of interfaces: synchronous (method


calls), asynchronous (events) and isochronous (streaming). All three different types

of interfaces are useful for inter-component communication. Synchronous interfaces

are used for initialization, asynchronous interfaces are used for signaling, e.g. QoS

violations, and the isochronous interfaces are used for media data transfer. In mul-

timedia applications, media data transfer (streaming) is subject to soft real-time

requirements. Each component which provides streaming should be quality aware

and support adaptation.

As the interfaces of multimedia components are defined, media transport interfaces

need to have QoS monitoring capabilities. Two different scenarios are identified:

• Internal QoS monitoring - The component monitors its own behavior with-

out respect to any other component.

• External QoS monitoring - The component monitors the processing of itself

and all predecessors within the component chain.

Internal QoS monitoring can be easily modeled by bounded execution time. This

constraint allows monitoring of the processing time of a single block of code and does

not bother surrounding components. If every component uses bounded execution time

monitoring, the constraints of all components within a chain can be aggregated to

calculate an overall bounded execution time of the whole application. However, this

simple approach has one big disadvantage. It requires that all components process

the same amount of data, e.g. one complete frame.

External QoS monitoring can be applied directly to the isochronous interfaces

of a component. This allows the definition of different constraint types for differ-

ent communication ports. The input data can be monitored best with a throughput

constraint. The output can be monitored best using jitter and processing delay. Fur-

thermore, the input constraint represents QoS capabilities required by previously ex-

ecuted processing steps, while the output constraint allows the definition of provided

QoS requirements.


Additionally to QoS capabilities, components should be adaptive in nature. This

means that each component needs to provide parameterization possibilities which

affect the processing behavior of the component and can be utilized if adaptation is

required. Which type of adaptation technique can be applied to a single component

depends on the type of the component and its functionality.

7.3 QoS Implementation Models

For the implementation of QoS, the Quality Aware Programming model (section 4.5)

is used and natively embedded into each component. More complex components can

be realized by grouping components together to build one complex component.

The development focus of QaMF lies in its native support for QoS. QoS process-

ing and adaptive programming facilities are achieved by applying two different QoS

handling strategies:

• BET model - Bounded Execution Time model

• LB model - Link Based model

The BET model uses only one single constraint type: bounded execution time. In

contrast, the LB model allows each component link to be monitored by user defined

constraints. This model allows more flexibility but also introduces more complexity.

The LB Model can be applied to every application built with the Quality Aware

Programming model, whereas the range of application for the BET model is limited.

7.3.1 BET Model

Bounded execution time restricts the amount of time between two different events such

as the sending and receiving of frames over a communication channel by applying a

lower and an upper border. Listing 7.1 illustrates the definition of the BET constraint

within the MMC# programming language.


The BET model defines bounded execution time constraints for every component

by using two timed variables. The first (inputConstraint) is assigned at the begin-

ning of the components logic code, the second one (outputConstraint) is assigned

at the end of the components code. Listing 7.2 shows an example of the skeleton of

a QoS-aware implementation of a component.

1 private constraint BETConstaint

2 (object [~] input , object [~] output , int minTime , int maxTime ) =>

3 @n{ // maximum time of maxTime milliseconds

4 output [n] - input[n] < QoS .Units .MSec(maxTime )

5 && // minimum time of minTime milliseconds

6 output [n] - input[n]> QoS.Units.MSec(minTime )};

Listing 7.1: BET Constraint Example.

In Listing 7.2 data processing is represented by the method call ProcessData.

If the time required for processing the data exceeds the defined bounded execution

time, a System.QoSException is thrown. To increase framework flexibility, the adap-

tion code is not directly implemented within the exception handler of each compo-

nent. Instead, the adaptation code is defined separately and invoked via the method

RaiseQoSViolationEvent, which allows adaptation behavior to be invoked without

modifying the code of the components. This method sends an event to the component

manager which selects the appropriate adaptation technique.

1 IData newData = null;

2 try {

3 // timed assignment

4 inputConstraint ~: data;

5

6 newData = ProcessData (data);

7

8 // timed assignment

9 outputConstraint ~: newData ;

10 } catch (QoSException ex) {

11 RaiseQoSViolationEvent (ex);

12 }

Listing 7.2: QoS implementation of a component.


Figure 7.2: Bounded Execution Time Model.

Two timed timed assignments are used by default to implement BET constraint

monitoring. Time measurements are handled with the help of the timed assignment

statements. The outputConstraint variable is associated with the BET constraint

and checks the bounded execution time of the ProcessData method. This code

structure also allows the monitoring of different constraints which is utilized by the

Linke Based model.

The BET model represents the inner timing behavior of a component. Timing

requirements are monitored inside the component rather than capturing the sur-

rounding code. It is a simplified QoS model using only one single constraint type.

This property makes all constraints directly comparable, thus permitting automatic

constraint matching. In addition, this helps the system to verify if it is possible to con-

nect two components. Furthermore, the sum of all constraints allows the framework

to monitor the overall behavior of the whole application (Figure 7.2).

Bounded execution time gives the application developer the opportunity to con-

strain the execution time of a single block of code. Although this model is often used

by existing frameworks such as JMF [Web09c], it is insufficient for general use. For

example, to apply BET to a video player with a frame rate of 25 frames per second,


Figure 7.3: QoS Models.

a BET of 40 milliseconds needs to be specified for all components together assuming

zero time for the transmission between the components. The latter is acceptable,

as slow links can be modeled as components. The problem is that this uniformed

constraint is too strict. Depending on the complexity of each component, the time

frame needs to be split.

BET-based QoS monitoring might be satisfactory for a simple labor-experiment,

but is obviously not feasible in a realistic environment, e.g. in an MPEG [MPE] video

player where the processing of I-, P-and B-frames takes different amounts of time.

7.3.2 Link Based Model

The LB model monitors the input and output communication ports of a component by

applying freely defined constraints. This means that the input constraint verifies that

the data which is provided meets certain QoS requirements. Accordingly, the output

constraint verifies that the resulting output also fulfills certain QoS requirements. In

contrast to the BET model, the timing behavior of previous components in the chain

is monitored, too (Figure 7.3).

In comparison to the BET model, applying different constraints at different steps

is more complex. However, the implementation of the QoS monitoring inside the

components remains the same (Listing 7.2). The timed variables are just attached

with different constraints.

Combining different kinds of constraints has the disadvantage that constraints


cannot be matched easily and thus automatic compatibility checks cannot be done

when plugging components together. Accordingly, a global application constraint

cannot be automatically generated by summing up all the constraints of the compo-

nents. As a result, a separate constraint needs to be defined to specify global QoS

requirements, a task which can be difficult in many cases.

7.3.3 Adaptivity

The previous sections presented models for monitoring QoS requirements. If a system

detects QoS violations, it must react accordingly. In the multimedia domain, chang-

ing the behavior of the application is achieved by executing adaptation code on the

stream under delivery. In the Quality Aware Programming model adaptivity is im-

plemented with the help of the exception handler. A component catches all possible

QoS exceptions. Instead of reacting directly within the component, events are raised

on QoS violations.

1 override protected void DoAdaption (IComponent source , QoSException qosExceptionParam)

2 {

3 // adaptation code for the NetworkTransmitter component

4 NetworkTransmitter dataProvider

5 = source as NetworkTransmitter;

6

7 // check type of exception parameter

8 // this behavior could also be implemented with the help of throw and try /catch

9 if ( qosExceptionParam is TooSlowException)

10 dataProvider. SwitchToLowerQuality ();

11 else if (qosExceptionParam is TooFastException)

12 dataProvider. SwitchToHigherQuality ();

13 }

Listing 7.3: Adaptation Code Example.

The component manager receives these events, stops executing the component

chain and calls dynamically bound adaptation code. Depending on the event type,

several different adaptation scenarios can be selected. This allows the adaptation code

to be exchanged without modifying the component itself. Adaptation code is defined


by implementing the IComponentAdaptor interface. After the adaptation code has

been executed, the component chain continues normal processing.

Listing 7.3 illustrates the standard adaptation pattern, presenting an example

of simple source switching adaptation technique. If the program receives an excep-

tion of type TooSlowException, it switches to a lower quality source. On the other

hand, if it processes incoming information “too fast”, it receives an exception of type

TooFastException and switches to a better quality level.

In general, this pattern can be used to implement any sort of adaptation code

which, for example, may include frame dropping or source-switching. Video stream-

ing may apply more advanced techniques like rate shaping [BDS96, JE97], layered en-

coding [RHE99], frame dropping [WKC+97] or combinations of these [RNK+97]. The

adaptor can make use of detailed knowledge about the environment, media format or

encoder/decoder implementation. The framework does not provide new adaptation

techniques. It rather defines the correct place for implementing them.

1 // error rate constraint , only violated if too

2 // many errors occur within a certain time range

3 private constraint ErrorRate

4 (object [~] o, int rate , int timeRange ) =>

5 @n{o[n] - o[n-rate]

6 > QoS .Units .MSec( timeRange )};

7 // ...

8 private object [~] adaptationConstraint

9 = new object [~ ErrorRate (

10 adaptationConstraint , 5, 1000) ];

11 // ...

12 override protected void DoAdaption (IComponent source , QoSException qosExceptionParam)

13 {

14 try {

15 // timed assignment , which checks the error rate

16 // if too many errors occur , an exception is thrown and the adaptation executed

17 adaptationConstraint ~: source ;

18 } catch (QoSException) {

19 ReallyDoAdaption (...);

20 }

21 }

Listing 7.4: Fault-tolerant Adaptation Code Example.


If QoS violations occur in multimedia applications, it is often the case that the first

n occurrences of the constraint violations are to be ignored to be resistant against

sporadic errors. Fault tolerance can be easily added by defining complex QoS re-

quirements or embedding quality aware programming patterns within the adaptation

code. Listing 7.4 illustrates adaptive behavior that executes adaptation code only

if the number of QoS violations exceeds a given rate. If the QoS violations occur

infrequently, no adaptation is applied.

7.4 Existing Implementation

The current implementation of the framework already contains many different compo-

nents providing different functionalities. All these components can be freely combined

to build a specific adaptive quality-aware multimedia application by combining them

within a component manager.

This section gives a brief overview about the existing implementation of QaMF.

7.4.1 Component Managers

The component manager itself is implemented as a quality-aware component, thus

providing QoS constraints and their appropriate monitoring which enables to check

the overall QoS requirements of the whole application.

Listing 7.5 illustrates how components are combined by adding them to the ex-

ecution chain of the component manager. The binding (cm.Bind()) initializes QoS

constraints and prepares all components to be ready to start processing. After bind-

ing has finished, the execution is started (cm.Start()) which advises the component

manager to start component chain processing.

The component manager is responsible for controlling assigned components. It

handles QoS violations and controls pushing or pulling of data through the compo-

nents. Several implementation strategies exist. The current implementation of QaMF

uses a single-threaded implementation model. In this case, the component manager


invokes the first component and passes on the data with the help of synchronous pro-

cessing calls. Another possibility is a multi-threaded implementation which slightly

increases complexity because of additional buffering which then is required.

1 // create component manager

2 IComponentManager cm = new SimpleComponentManager ();

3

4 // create components

5 IComponent source = new FileTransmitter();

6 IComponent decoder = new FFMpegDecoderTransceiver ();

7 IComponent display = new PictureBoxReceiver ();

8

9 // component initialization code

10 // ... // initialize constraints

11 // ... // pass required initialization parameters

12 FileTransmitter.PropertyClass ftProperties = (FileTransmitter. PropertyClass) source .

Properties ;

13 ftProperties.SourceFile = "movie .mpg "; // read file named movie .mpg

14 ftProperties.Rate = 1024*1024; // initialize predefined constraint with the rate 1

MByte /sec

15 // ...

16

17 // add components

18 cm.Add (source );

19 cm. AddSuccessor(source , decoder );

20 cm. AddSuccessor(decoder , display );

21

22 // bind components

23 cm.Bind();

24

25 // start execution

26 cm.Start ();

Listing 7.5: Using a Component Manager.

The component manager can be compared to a group component which allows

grouping other components together to provide a further level of abstraction. The

current implementation contains grouping possibilities to build the following complex

components.

• GroupTransmitter - is a grouping component which represents a transmitter.

It produces data and can be used as a starting component.


• GroupTransceiver - is a grouping component to build a complex processing

step out of simple components. It receives data and also passes on data to the

next component.

• GroupReceiver - is a grouping component which represents a receiver. It is a

data consumer and can be used as an end point of the component chain.

Although there are similarities to group components, the component manager

has much more responsibilities. It controls the overall chain execution and provides

starting, stopping and pause functionality. Furthermore, it handles all QoS violations.

It receives all QoS violation events and selects an appropriate adaptation strategy out

of existing implementations.

There is one major limitation to QaMF. It currently does not support splitting

or joining of the component chain. This might be an interesting feature when ap-

plying parallelism. To support this feature, additional split and join components are

required. In addition, a multi-threaded component manager is required to control all

chains.

QaMF currently supports the following implementations of component managers

(Table 7.1):

To provide adaptive behavior implementations of the IComponentAdaptor inter-

face can be added to each component manager (Listing 7.6). The ComponentAdaptors

property stores all applied adaptor implementations within a list. In case of a QoS

violation, the component manager loops over the list of adaptors and calls their in-

terface method to execute the adaptation code.

1 public interface IComponentAdaptor {

2 // adaptation method , called in case of a QoS violation

3 void Adapt (IComponent source , QoSException qosMessage );

4

5 // initialization methode , used to initialize the adaptor

6 void Initialize ( IComponentManager cm , IComponent [] allComponents);

7 }

Listing 7.6: Component Adaptor Interface.


Component Manager Description

SimpleComponentManager The SimpleComponentManager is a single-

threaded component manager implementation.

It allows controlling of chains and handles QoS

violations by selecting appropriate adaptation

strategies.

VisualComponentManager The VisualComponentManager is an extension to

the SimpleComponentManager. It provides a com-

prehensive GUI to build applications by placing

components together. Each component is config-

ured via a property grid control. A generated ap-

plication is called scenario and can be stored. This

special component manager is intended to provide

a simple test framework for newly created compo-

nents.

Table 7.1: Overview of Existing QaMF Component Managers.

7.4.2 Receivers

Receivers are components which can be used as data sinks. They represent the end

of a component chain and can write received data e.g. to the screen, the local disk

or a network socket.

Table 7.2 contains an overview of the most important receiver components.

7.4.3 Transmitters

Many transmitters are available as counterparts of existing receiver components.

They are data sources which allow gathering data e.g. from the local disk or a

network socket. Furthermore, transmitters have been implemented which generate

data on their own.


Component Description

FileWriter The FileWriter is responsible for storing received

data within a specified file.

SocketWriter The SocketWriter is responsible for writing data

on a network socket. This allows building dis-

tributed multimedia applications.

PictureBoxReceiver The PictureBoxReceiver draws received data on

the screen. It is a standard end-point for a video

player, for example.

BitmapReceiver The BitmapReceiver is similar to the Picture-

BoxReceiver. It converts received frames to

Bitmap objects and notifies registered objects via

the event NewBitmap.

Table 7.2: Overview of Existing QaMF Receiver Components.

Table 7.3 contains an overview of the most important transmitter components.

7.4.4 Transceivers

Media processing is done by transceiver components. These components read data,

process this data and provide it for the next component within the component chain.

Encoders and decoders are common representatives of such components, but also

different processing can be applied which can lead from simple buffering till complex

image manipulation.

Table 7.4 contains an overview of the most important transceiver implementations

including components for encoding/decoding of video data.

Furthermore, transceivers can be used to encapsulate existing library or frame-

work functionality. This has been applied e.g. by the FFMpegDecoderTransceiver or



BouncingBallBitmapGenerator The BouncingBallBitmapGenerator is an easy to

use data source which generates animated pictures

of filled circles moving around.

FileTransmitter The FileTransmitter is the counterpart of the

FileWriter. It reads data from a file and passes

it on to the next component within the chain.

SocketTransmitter The SocketTransmitter is responsible for reading

data from a network socket. This allows building

distributed multimedia applications.

YuvGenerator The YuvGenerator is similar to the Bouncing-

BallBitmapGenerator and provides data in the

YUV420p format. The original implementation is

part of the ffmpeg-project [Web09a].

Table 7.3: Overview of Existing QaMF Transmitter Components.

FFMpegEncoderTransceiver. QaMF utilizes video coding capabilities of the FFMpeg-

library and additionally surrounds the integration with QoS processing support, which

is currently not available in FFMpeg.

7.5 Implementations of Video Players with QoS

Costraints

This section provides different possibilities for implementing a video player. All the

presented players have been implemented with QaMF with the help of quality aware

component composition.



BufferedTransceiver The BufferedTransceiver temporarily stores data within a

buffer. It releases data with a predefined rate to the next

component in the chain. This component uses two separate

threads. The first thread is responsible for receiving data

and storing it within the buffer, while the second thread

passes on the data depending on a predefined time interval.

DropDataTransceiver The DropDataTransceiver is used for testing purpose and

simulates dropped packets which are randomly selected.

FFMpegDecoderTransceiver To decode received data, the FFMpegDecoderTransceiver

can be used. As QaMF does not reinvent current imple-

mentations, existing codec implementations are reused. In

this case the FFMpeg-library is integrated to decode re-

ceived compressed data.

FFMpegEncoderTransceiver This component represents the counterpart to the previ-

ously explained FFMpegDecoderTransceiver. It is used for

encoding frames.

ItecAvcDecoderTransceiver The ItecAvcDecoderTransceiver is another decoding com-

ponent used for processing AVC/H.264 compressed videos

using a C# based implementation [SFLB07b, Sch09].

IncreasingDelayTransceiver The IncreasingDelayTransceiver is used for testing and

provides an increasing delay within the component chain.

It slows down data processing.

SleepingTransceiver The SleepingTransceiver is similar to the IncreasingDelay-

Transceiver. It uses a fixed delay to slow down data pro-

cessing.

Table 7.4: Overview of Existing QaMF Transceiver Components.


Figure 7.4: Constraints of a Video Player.

7.6 Simple Video Player with QoS Constraints

In this section a simple video player which consists of three different components

is presented. The first component is responsible for providing data for the second

component. The data can be read from disk or is received over a network. The second

component decodes the data and generates images which then can be displayed by

the third component. The third component is thus the end of the component chain.

If the data transferred between the components stands in a one-to-one relation which

means that every input data packet results in one single output frame for display, the

BET model can be applied. Each data block produced by the first component results

in an output block of the decoding component. This is achievable for a simple YUV

player where the transmitter reads whole YUV-images and the decoder transforms

them into RGB-images prior to display.

In general, video players use many different decoders and modern video formats do

not provide a one-to-one mapping of input data to output data. It is quite complex to

achieve a one-to-one mapping of data due to different frame bounds and processing re-

quirements. Therefore, each component in the chain must apply different constraints

(Figure 7.4). The first component which receives the data is normally subject to a

throughput constraint. A certain number of bytes are required to successfully decode

and display a video. The decoder component can also use a throughput constraint,

but here throughput is expressed in frames per second. The display component is the

most sensitive in the chain. If the decoder varies its throughput too much, a playback

buffer is required as output should be displayed using a more or less constant delay.


7.7 An Experimental Video Player

An experimental video player demonstrates QoS-programming facilities of QaMF.

This video player is able to play videos which can be decoded with FFMpeg and the

ITEC AVC/H.264 decoder [Sch09]. The player uses the following processing model

(Figure 7.5).

Figure 7.5: Model of a Real-life Video Player Application.

In contrast to the previously illustrated constraint model, this video player requires

a buffer to smooth the output of the decoding component. Figure 7.5 presents the

component chain of the video player application. It consists of a FileTransmitter

which is responsible for reading data blocks from the local disk. The data is passed

to the FFMpegDecoderTransceiver, which decodes the data and provides frames to be

displayed. A DelayTransceiver allows manipulation of the processing time to simulate

performance problems during the decoding process. This is used to simulate adaptive

behavior when reading data from the local disk. The RatedBitmapReceiver displays

the received frames with a constant rate. A BufferedTransceiver is applied between

display and decoding components. It is responsible for compensating jitter produced

by the decoding component.


Furthermore, the player does not only monitor QoS requirements specified. In

addition, it provides an implementation of the source switching adaptation technique.

In case a QoS violation is detected, the source is switched either to a lower quality

to reduce required processing power or to a better quality if enough resources are

available (Figure 7.6). This thesis is not focusing on providing novel adaptation

techniques. It rather shows how they should be applied with the Quality Aware

Programming model.

Figure 7.6: Video player application GUI.

CHAPTER

8 Implementation

This chapter describes the implementation of the presented language extensions and

their run-time support. It explains how compiler modifications have been applied.

Furthermore, an overview about the run-time environment is given to understand the

behavior of the QoS management framework.

8.1 Mono Compiler - Customizing C#

C# is the main programming language of the .NET framework. This thesis utilizes

the Mono C# compiler as the basis for implementation.

8.1.1 Mono Project

The Mono project [Web09f, Web09g] is an open source implementation of the .NET

platform mainly developed for UNIX, but also supports Windows and Mac OS X.

Its aim is to provide a cross-platform for .NET applications to enable windows appli-

cations to run on UNIX based systems and Mono developed applications to run on

Windows based systems. Figure 8.1 illustrates the simplified architecture.

Mono includes the following components:

• Common Language Infrastructure (CLI) Virtual Machine - The CLI

virtual machine contains a class-loader, JIT (just-in-time) compiler and GC

(garbage collector) infrastructure.

CHAPTER 8. IMPLEMENTATION Page 158

• Class Library - The provided class library is .NET compatible and also in-

cludes Mono-provided classes.

• C# Compiler - The C# compiler is the main compiler of the Mono framework.

There is already support for other languages like Visual Basic, Oberon or Object

Pascal.

Figure 8.1: Simplified Mono Architecture [Web09i].

The .NET framework uses a virtual machine like environment called the Common

Language Runtime (CLR) and is intended for use with several different programming

languages. The .NET platform has compilers [cli09] that target the virtual machine

from a number of languages: Managed C++, Java Script, Eiffel, Component Pas-

cal, just to name a few. The CLR and the Common Type System (CTS) provide

a common run-time to all of these languages. Independent of the language syntax

and structures, the target output is the Common Intermediate Language (CIL) which

provides interoperability between all CLI compatible languages. When a CIL appli-

cation is executed, the code is compiled to native machine code for the appropriate

architecture.


8.1.2 Structure of the Mono C# Compiler

The Mono C# compiler is an essential part of the Mono framework. It is an ECMA-

Standard [ECM05] compliant compiler implementation for the C# programming lan-

guage targeting the standard compliant CLI [ECM06]. Each new version of the

programming language is provided via a separate compiler executable.

• mcs - A C# compiler targeting .NET 1.x framework.

• gmcs - A C# compiler targeting .NET 2.0 and 3.5 framework.

• smcs - A C# compiler targeting .NET 2.1 framework including Silverlight APIs

[Web09e].

• dmcs - A C# 4.0 compatible compiler (currently under development).

The first prototype of the novel language features presented within this thesis have

been implemented within mcs. Later on the development was switched to gmcs.

Mono C# compiler [dI09] is completely written in the C# programming lan-

guages and makes heavy use of .NET API calls including System.Reflection and

System.Reflection.Emit.

The compilation process is divided into the following blocks:

1. Parsing - The compiler parses a set of source files. This is done with the

JAY parser which is a port of Berkeley Yacc to Java that was later ported to

C# by Miguel de Icaza. Required assemblies are loaded after parsing has been

completed.

2. Type Hierarchy Processing - After parsing has finished the type hierarchy is

resolved and populated to the type system. After this step the program skeleton

is complete.

3. Code Generation - This phase is divided into two parts. The first part is

responsible for providing semantic analysis and check if the code is correct.


The second one really emits the code. After executing code generation, the

output is saved to the disk.

[dI09] provides a detailed overview about the file organization of the compiler and

explains the responsibilities of each code block. Figure 8.2 illustrates the different

phases during compiler execution.

Figure 8.2: Mono C# Compiler Overview [dI02].

Lexical Analysis

During the first phase of the compiler execution all input is processed by the tokenizer.

The tokenizer splits the input text into small parts which then are analyzed by the

JAY parser.

If the tokenizer returns a token, its location is recorded within a property which

is accessible by the parser. This information is used to provide correct error messages

pointing to the location of the problem including line number and column. If the


location cannot be determined by the compiler, error messages pointing to line 0 are

produced which represent anonymous error messages.

C# has only limited pre-processing capabilities compared to the programming

language C [KR88]. Pre-processor directives are handled via a separate method which

is invoked when detecting pre-processing directives starting with the symbol #.

Syntax Analysis

The JAY parser takes the previously generated tokens and evaluates the syntax of

the source program. The grammar of the parser does not exactly match the grammar

provided within the standard, because the standard has been written to be human

readable.

Each statement or expression identified by the parser is represented by a class.

If the parser finds an appropriate token sequence, it instantiates an object of the

corresponding class representing the parsed language construct.

The output of the syntax analysis is an abstract syntax tree (AST) which contains

expression trees and all statements required for code generation. It is the basis for

the semantic analysis to check the correctness of the code and the evaluation of all

used types.

Semantic Analysis

The abstract syntax tree is the internal representation of the provided program. It

is the input for the semantic analysis. This analysis resolves all types starting with

the interface hierarchy. Next, classes, structures, constants and enumerations follow.

Afterwards, methods, indexers, properties, delegates and events are resolved. At

the end of this process all elements containing code have been analyzed, all type

information has been evaluated and if no error has been encountered, the AST is

considered to be semantically correct.


Code Generation

The resolved abstract syntax tree is the basis for code generation. Context objects

(EmitContext) are used to keep track of processed namespaces, type containers and

the current state of code generation. With the help of the ILGenerator, the program

is translated into the Common Intermediate Language (CIL).

8.1.3 Embedding Language Extensions

Several different possibilities to modify the Mono C# compiler exist. The implemen-

tation of the language features presented within this thesis have been applied to gmcs,

which is the compiler version targeting the .NET framework 2.0.

The tokenizer is based on a manual implementation and needs to be modified

accordingly in case new tokens are required. For the current extensions, this has been

done for the new assignment statements and keywords introduced within MMC#.

The parser is implemented as a JAY-parser specification which is a mixture of

grammar definitions and related C# code. This code is executed when a given token

sequence is recognized. The modifications to the grammar are directly applied within

the “cs-parser.jay” file. The parsing process generates an AST which consists of

abstract representations of expressions and statements.

There are many different possibilities of how a certain modification can be im-

plemented. Instead of emitting code manually, the presented language features are

based on modifications directly applied to the abstract syntax tree. This can be ex-

plained considering the following example. If an assignment statement is detected,

an instance of the class Assign is created. This object contains a left-hand and a

right-hand expression which are the target and the source of the assignment operation

respectively. During semantic analysis, this object is resolved by identifying the types

of the expressions. The code generation phase emits the appropriate code.

Instead of defining a new TimedAssign class, the JAY processing code was adopted

by adding extra method invocations to the run-time available QoS management frame-

work to provide QoS monitoring behavior. These surrounding elements are added


to the representation of the assignment and reuse existing elements of the current

compiler implementation to provide the new functionality. The existing assignment

implementation was parameterized to distinguish between different assignment state-

ments.

This implementation technique has the advantage that correct code generation is

left to the existing implementation of the compiler. This approach allows easy modi-

fications to the front-end of the compiler without worrying about the code generation

in detail.

The use of timed data types as overlay types - this means that the type informa-

tion related to time is ignored when applying type compatibility checks - allows the

generated CIL code to be standard conform. It further enables compiled applications

to run on different .NET implementations and, in addition, allows QoS-aware libraries

to be reused within standard .NET or Mono programs.

If timed data types had been integrated into the Common Type System (CTS),

other programming languages could use the new types. This has the disadvantage

that a QoS-aware .NET runtime needs to be shipped with each application instead

of reusing existing installations. This approach, however, was not chosen to provide

compatibility to existing implementations which might also increase acceptance.

Applied Modifications

The following files were modified to implement the presented language enhancements

for Quality Aware Programming.

• assign.cs - This file contains all classes which are required to implement an

assignment operation and the necessary object initializer. The MMC# modifi-

cation is required for the constraint assignment when initializing a time variable

with a new constraint.

• cs-parser.jay - This is the parser definition file which reads the input source

code and populates the abstract syntax tree. It contains most of the modifica-

tions which have been applied to support QoS capabilities.


• cs-tokenizer.cs - The tokenizer is extended to accept new keywords and the

timed assignment operation.

• delegate.cs - This file contains all classes which deal with delegates and method

invocations. The implementation of timed parameters requires modification to

this file.

• ecore.cs - This file contains base classes for expressions and types. It includes

an implementation of the timed data type which can be applied to any other

data type.

• expression.cs - All expressions are implemented within this file. Time param-

eter arguments are implemented as well as the cast operation which is required

to ignore the timed type to provide correct type compatibility with non-timed

types.

• parameter.cs - This file is responsible for method parameters. It contains the

extensions for timed method parameters.

8.2 QoS Management Framework

This section describes the QoS management framework which provides support for

the Quality Aware Programming model at run-time. First, an overview is presented

followed by interface definitions and some examples.

8.2.1 Overview

The architecture of the QoS management framework is quite simple. The focus lies

on providing historization functionality and a common interface for evaluating con-

straints. It is responsible for storing the occurrences of events and executes constraint

evaluation (Figure 8.3).


Figure 8.3: QoS Management Framework Overview.

The QoS management framework allows registration of event sources called signals

which can be variables, member fields or method parameters and stores attached con-

straints of type IQoSObject. The compiler automatically generates sub types of type

IQoSObject which contain the constraint implementations. These generated classes

are added as nested classes and become part of the actual compiled code. Instances

of such classes are invoked via the common interface to evaluate their constraints.

All calls to the history are done via the IQoSHistoryManager interface. The history

manager provides storage space for event occurrences. The actual implementation

provides ring buffers for optimized performance which are implemented as arrays.

The structure of the compiler generated code is shown in section 8.2.3 by manually

implemented QoS monitoring.

8.2.2 Components and Interfaces

Figure 8.4 illustrates the class hierarchy and briefly describes all involved interfaces

and classes.

Two main interfaces exist which provide access to all required functionality. The


Figure 8.4: QoS Management Class Model.

first interface IQoSObject defines a common way to trigger constraint evaluation.

The second interface IQoSHistoryManager is responsible for providing access to the

event histories.

IQoSObject

The IQoSObject interface is the base interface for all constraint implementations. It

defines methods required for QoS processing:

• CheckQoS - This is the method called when constraint evaluation is required. Its

implementation contains check code for specified QoS requirements belonging

to a timed variable or parameter and the corresponding calls to the IQoSHisto-

ryManager implementation to retrieve recorded events.

• Reset - In case a QoS violation is detected, the histories of all related timed

variables are cleared. This is a cascading process which might lead to removing

all histories of currently used timed variables if their constraints are somehow

related.


• GetContext - The history management is organized storing histories of timed

variables according to their context. This allows unique identification of histo-

ries according to their associated variables.

In case the declarative constraint definition is used, the compiler generates a class

which implements this interface. It automatically provides the implementation for

the three required methods. Furthermore, it automatically instantiates the classes

and assigns the constraint objects to their timed variables.

If a QoS violation is detected, the Reset method of the constraint which caused

the violation is executed. This method clears the histories of all timed variables which

are related to the current constraint. It further retrieves all other constraint objects

of related timed variables and calls their Reset methods. This is done as long as a

timed variable with a non-empty history is found.

IQoSHistoryManager

The IQoSHistoryManager is the main interface within the QoS management frame-

work. The implementation of the history manager is responsible for recording his-

tory events, providing access to histories and further associate timed variables with

IQoSObject implementations. As timed data types are only available during the

compilation of MMC# and not directly integrated within the Common Type Sys-

tem (CTS) of the Common Language Infrastructure (CLI) and Common Language

Runtime (CLR), these types are handled by the QoS history manager.

The interface definition contains the following methods:

• GetInstance - creates a new instance of the QoS history manager.

• RegisterSignal - registers a new signal. In this context, a signal represents

a source of events. This source can be a timed variable or a timed method

parameter.

• InitHistorySize - sets the initially required size for the history of events for

a given signal.


• GetMaxStoredHistory - retrieves the maximum size of the history, which re-

turns the value provided via the InitHistorySize method call.

• AssignQoSToSignal - attaches a constraint to a signal (timed variable or timed

method parameter).

• GetQoSOfSignal - retrieves a previously attached constraint. This method is

applied when constraint checking is executed to get a certain constraint.

• IsQoSAssignedToSignal - determines whether a constraint is attached to the

signal or not.

• AddSignalToHistory - adds the current event to the history and stores its

timestamp.

• GetCurrentTimeValue - returns the current time in ticks.

• ClearSignalHistory - clears all values out of the history to provide a clear

history for startup. This method is called in case a QoS constraint is violated.

This behavior is required to not carry off processing errors.

• GetSignalHistory - retrieves a history value for a signal.

• GetSignalHistoryCount - gets the number of actually stored history values.

This method is used to check if there are enough values to execute constraint

evaluation.

Units

QoS.Units is a utility class providing conversions of time values to their internal

representation within the QoS management framework. The actual implementation

expresses time in ticks (100 nanoseconds). The following conversion methods are

supported:

• Min - Converts the input from minutes into the internal representation.


• Sec - Converts the input from seconds into the internal representation.

• MSec - Converts the input from milliseconds into the internal representation.

• MicroSec - Converts the input from microseconds into the internal representa-

tion.

QoSHistoryManagerFactory

The QoSHistoryManagerFactory provides exchangeability of IQoSHistoryManager

implementations. The following methods are provided:

• SetQoSHistoryManager - is used to define the implementation of the history

manager. This method can be invoked at the beginning of an application to

apply a user defined history manager implementation.

• GetQoSHistoryManager - retrieves the history manager. If the history manager

has not been instantiated, this method creates a new instance of the history

manager by invoking its GetInstance method. If required, a context object

can be passed to this method resulting in a newly created history manager

instance for each context.

8.2.3 Manually Implemented QoS Monitoring

It is possible to implement QoS constraints and the according monitoring manually.

To achieve this, a subtype of type IQoSObject needs to be defined. Listing 8.1

illustrates the manual implementation of a delay constraint. It restricts the delay of

the signal CustomSignal to the specified amount of milliseconds.

If a manual constraint implementation is used, also the event recording can be

implemented manually. The code provided in Listing 8.2 demonstrates how the pre-

viously defined constraint can be applied.

Currently, mixing manual constraint definitions with automatic QoS monitoring

is sophisticated. The compiler defines context objects which refer to the scope of


variables. The actual implementation does not provide access to these context objects.

A mapping function would be required to determine the context of a signal from the

history manager which is currently not implemented.

1 // manually implemented delay constraint

2 // @n{var [n] - var [n-1] < Units .MSec(delay)}

3 // code would be generated by the compiler if specification syntax was used

4 class MyDelay : IQoSObject {

5 int n = 0; // counter

6 long delay = 0;

7

8 public MyDelay (long delayInMSec ) { this.delay = delayInMSec ; }

9

10 public int CheckQoS () {

11 IQoSHistoryManager mgr = QoSHistoryManagerFactory .GetQoSHistoryManager ();

12 // wait till there are enough values in the history

13 n++;

14 if (n < 2) return 0;

15 // get history values

16 long time1 = mgr. GetSignalHistory("CustomSignal", n-1);

17 long time2 = mgr. GetSignalHistory("CustomSignal", n-2);

18 // compare history values

19 if (time1 - time2 >= Units .MSec(delay ))

20 throw new QoSException("MyDelay violated ");

21 // no QoS violation

22 return 0;

23 }

24

25 public void Reset () {

26 n = 0; // reset counter variables

27 }

28

29 public object GetContext () {

30 return null;

31 }

32 }

Listing 8.1: Manually Implemented Delay Constraint.


1 IQoSHistoryManager mgr = QoSHistoryManagerFactory . GetQoSHistoryManager ();

2 mgr .RegisterSignal("CustomSignal");

3 mgr .AssignQoSToSignal("CustomSignal", new MyDelay (100));

4

5 // ...

6 while (/* someCondition */) {

7 // ... action to be monitored , code generated by ~: / :~

8 mgr. AddSignalToHistory("CustomSignal");

9 try {

10 mgr.GetQoSOfSignal("CustomSignal").CheckQoS ();


12 // react on QoS violation

13 mgr.ClearSignalHistory(" CustomSignal");

14 }

15 }

Listing 8.2: Manually Implemented Delay Monitoring.

CHAPTER

9 Conclusion and Further

Work

Quality Aware Programming has been introduced and discussed within this thesis.

The problems of QoS specification, monitoring and adaptation have been explained

in detail and a solution has been provided. Although a lot of research has been

done in the field of QoS specification languages and many different frameworks have

been developed to support QoS monitoring, to the best of my knowledge, no explicit

support for temporal QoS handling and adaptive programming in a common general

purpose programming language is available. Support for exception based adaptive

behavior does not exists either.

This thesis presents a solution for the problems of QoS specification, monitoring

and adaptivity by extending the common general purpose programming language

C#. The type system has been modified to support the notion of time which is a

necessity when specifying temporal constraints. A declarative syntax has been defined

to allow constraint definition. These constraints can then be evaluated during run-

time by applying a new value assignment operation. Adaptivity is made as easy as

any other kind of exception based programming. In case of QoS violations, exceptions

are thrown to notify that the currently specified QoS behavior cannot be achieved.

Furthermore, the declarative syntax of QoS specification enables semantic analysis

which checks correctness of constraints and facilitates optimizations.

All the new language features are the basis of the Quality Aware Programming

CHAPTER 9. CONCLUSION AND FURTHER WORK Page 174

model defining a clear and simplified structure for QoS-aware application develop-

ment. It allows separation of concerns by extracting the constraint definition and

time measuring code.

Furthermore, several programming patterns applying this novel programming

model have been explained. They demonstrate basic building blocks for implementing

QoS-aware applications.

The focus of multimedia applications led to the development of a new multimedia

framework (QaMF) utilizing the novel programming model and the defined program-

ming patterns. It focuses on the development of adaptive quality-aware multimedia

applications instead of providing new codec implementations. It supports user de-

fined QoS constraints and applies automatic semantic analysis at compile-time and

at run-time to check the correctness of constraints. The suggested code structure,

consisting of constraint declaration, application logic and adaptation code reduces er-

rors during development and maintenance of multimedia applications. Furthermore,

this thesis shows how QoS is handled in detail by presenting two models for QoS im-

plementation. The BET model uses one single constraint type and therefore enables

automatic constraint compatibility checks. The Link Based model can be applied to

support complex custom constraints.

The work for this thesis has left open some further questions which remain subject

for further study:

• Detailed analysis of system influence on QoS monitoring including operating

system scheduling as well as garbage collection needs to be done.

• The current semantic analysis only compares conditions of single constraints.

Some kind of relation between constraints needs to be defined to make complete

constraints comparable and allow compatibility checks between constraints.

• Evaluate possibilities to define a normalized notion which allows constraints to

be compared in general including temporal and non-temporal constraints.


• Provide compile-time checks when applying the Link Based model which can be

compared to type compatibility checks when connecting different components.

• Introduce a notion to specify violation tolerance to reduce the number of thrown

exceptions in case of sporadic QoS violations.

• Check the possibilities of reducing the the run-time cost of the monitoring pro-

cess by decreasing the number of recorded events and the number of QoS con-

straint checks executed. The price for this improvement were reduced accuracy.

• Investigate propabilistic models of system reliability to find an optimal way of

reducing monitoring costs to gain maximum performance.

• Find an algorithm to automatically balance a certain tolerance value and the

required monitoring accuracy to fit the application needs.

• Implement proactive QoS behavior. The current work focuses on reactive QoS

processing which means that QoS violations are detected and adaptation code is

executed. Tighter QoS constraints can be used instead to detect QoS violations

in advance and provide certain actions to proactively change the application

behavior.

The aim of this thesis is to provide mechanisms and tools to transform quality

aware multimedia programming into a first-class citizen. The focus lies on build-

ing multimedia applications which cooperate with each other and are not greedy.

Each multimedia application should be adaptive and not just stop playback if data

or processing power is missing. It should be possible to create federative multime-

dia applications that work collaboratively and react to bottlenecks (such as network

throughput or even CPU or memory shortages).

The main motivation behind this research is that the best place for doing adapta-

tion is in the application itself because it is the application which knows what quality

is acceptable to the user.

APPENDIX

A QoS Management Frame-

work - Extended

This chapter continues the description of implementation details from section 8.2. The

main focus lies on describing the different implementations of the history manager

and its exchangeability at run-time.

A.1 History Manager

The history manager interface has already been explain in section 8.2.2. This section

focuses on the exchangeability of the history manager implementation and different

possibilities to implement time measurements as well as the implementation of the

history itself and its cascading reset.

A.1.1 Exchangeability

The history manager factory enables the framework to use different history manager

implementations. The factory implementation is illustrated in Listing A.1. It uses

one single instance to create new history managers. Each history manager which

is attached to a special context is stored within the hash table to provide it for

consecutive access.

If a different history manager implementation is required, the factory needs to

be initialized accordingly before the first use of timed assignments or timed method

parameters.

APPENDIX A. QOS MANAGEMENT FRAMEWORK - EXTENDED Page 178

1 // factory creates one single instance of the qos history manager

2 public class QoSHistoryManagerFactory {

3 // default implementation which is used

4 // currently DefaultQoSHistoryManager and PreciseQoSHistoryManager available

5 static IQoSHistoryManager manager = new PreciseQoSHistoryManager ();

6 static Hashtable managers = new Hashtable ();

7

8 // define new qos history manager

9 public static void SetQoSHistoryManager (IQoSHistoryManager manager ) {

10 if (manager != null) {

11 lock (managers ) {

12 managers .Clear ();

13 QoSHistoryManagerFactory .manager = manager ;

14 }

15 }

16

17 // factory method returns qos history manager

18 public static IQoSHistoryManager GetQoSHistoryManager () {

19 return GetQoSHistoryManager (null);

20 }

21

22 // factory method returns qos history manager

23 public static IQoSHistoryManager GetQoSHistoryManager(object context ) {

24 if (context == null)

25 context = "default ";

26 IQoSHistoryManager m = null;

27 lock (managers ) {

28 m = (IQoSHistoryManager) managers [context ];

29 if (m == null) {

30 m = manager .GetInstance ();

31 managers [context ] = m;

32 }

33 }

34 return m;

35 }

36 }

Listing A.1: QoS History Manager Factory.


A.1.2 Time Measurement

Time measurement is implemented with the interface method GetCurrentTimeValue.

Currently, two different implementations of time measurements exist in the QoS man-

agement framework. The first implementation uses the DateTime.Now property which

is able to measure the time with an accuracy of 10 milliseconds. The second im-

plementation uses the Stopwatch class which has a measurement accuracy of 100

nanoseconds.

DateTime.Now

System.DateTime is part of the system class library and represents an instant in time

[msd09a]. Time is measured in 100-nanosecond units called ticks.

The public, static, read-only property Now retrieves a DateTime object represent-

ing the current time. The accuracy which can be reached using this time measure-

ments technique is limited to the system timer and lies in the range of 10 to 16

milliseconds [msd09d]. If a higher resolution timer is required, multimedia timers or

high-resolution timers should be used.

The method ToFileTime converts the DateTime object into a long value repre-

senting the time in ticks which is then stored within the history (Listing A.2).

1 public class DefaultQoSHistoryManager : IQoSHistoryManager {

2

3 ...

4

5 // get the current time in ticks with the help of DateTime .Now

6 public virtual long GetCurrentTimeValue () {

7 return DateTime .Now. ToFileTime (); // returns time in ticks

8 }

9

10 ...

11 }

Listing A.2: Time Measurement with DateTime.Now.

This type of time measurement can be applied to many different applications and

provides correct monitoring behavior for many constraints. In case delay or jitter


values reach the range of 10 milliseconds, different time measurement implementations

should be considered.

Listing A.3 demonstrates a simple delay constraint monitoring which will always

lead to a constraint violation when the DateTime.Now value is updated.

1 // use low resolution time measurement

2 QoSHistoryManagerFactory .SetQoSHistoryManager (new DefaultQoSHistoryManager ());

3

4 // delay constraint using a value lower than the measurement accuracy

5 int [~@n{val[n] - val[n-1] < Units .MSec (5) }] val ;

6 for (int i = 0; i < 10; i++)

7 {

8 val ~: i; // will cause constraint violation when DateTime .Now is updated

9 System .Threading .Thread .Sleep (3);

10 }

Listing A.3: Example of Constraint Violation with DateTime.

Stopwatch

1 public class PreciseQoSHistoryManager : DefaultQoSHistoryManager {

2

3 ...

4

5 // get the current time in ticks with the help of Stopwatch

6 public override long GetCurrentTimeValue () {

7 return Stopwatch . GetTimestamp(); // returns time in ticks

8 }

9

10 ...

11 }

Listing A.4: Time Measurement with Stopwatch.

The System.Diagnostics.Stopwatch class provides high-resolution time mea-

surement functionality [msd09c]. An instance of the Stopwatch class can be used to

measure a time span using the Start and Stop methods. Additionally, the Stopwatch

class provides the static method GetTimestamp to call the unmanaged Win32 API


QueryPerformanceCounter which retrieves the current time. Depending on the un-

derlying hardware and software support for high-resolution timers, this value is up-

dated up to every 100-nanoseconds which can be exploited to provide precise mea-

surement results.

A.1.3 Event History Size


2 const int DEFAULT_HISTORY_SIZE = 100;

3 public class HistoryElement { // nested container , used to manage element history

4 public long[] timestamps ; public int currentIndex; public IQoSObject constraint ;

5 }

6 Dictionary <string , HistoryElement > histories

7 = new Dictionary <string , HistoryElement >(); // constraint /history storage

8

9 // add new time stamp to history

10 public virtual void InitHistorySize(string signal , int size) {

11 // create new history element

12 HistoryElement elem = new HistoryElement();

13 // if size is not provided use default size

14 if (size <= 0) size = DEFAULT_HISTORY_SIZE;

15 // initialize history element , constraint is initialized by another method

16 elem.timestamps = new long(size); elem.currentIndex = 0; elem.constraint = null;

17 // add history object

18 histories [signal ] = elem;

19 }

20

21 // add new time stamp to history

22 public virtual void AddSignalToHistory(string signal ) {

23 HistoryElement elem = histories [signal ]; // get the history object for the signal

24 elem.timestamps [elem.currentIndex % elem.timestamps . GetLength (0)]

25 = GetCurrentTimeValue (); // add new time stamp in ring buffer

26 elem. currentIndex++; // increase the index

27 }

28 ...

29 }

Listing A.5: History Initialization and Event Recording.

The actual implementation of the timing history stores time stamp values of data

type long within an array. The array size is determined during constraint analysis


and passed to the history manager via the method InitHistorySize. If the required

size cannot be determined a default size of 100 events is assumed. When time a timed

assignment is executed, the current time value is added to the event history with the

help of the AddSignalToHistory method.

The history is implemented with the help of the HistoryElement class which con-

tains an array of time stamp values and the current index (Listing A.5). Furthermore,

it stores the associated constraint object.

Custom implementations of the history manager can vary the history size by

overriding the InitHistorySize method.

A.1.4 Event History Reset

The reset of event histories is a cascading process which needs to reset all histories

of all signals which are somehow related to the current constraint, e.g. a constraint

uses the timed variables A and B and no other constraint exists then the histories of

both A and B need to be cleared.


2

3 // clear the signal history of a signal

4 public virtual void ClearSignalHistory(string signal ) {

5 // get the history object for the signal

6 HistoryElement elem = histories [signal ];

7 if (elem.currentIndex > 0) {

8 // set the current index to 0 which indicates an empty history

9 elem.currentIndex = 0;

10

11 if (elem. constraint != null)

12 elem.constraint .Reset (); // cascading Reset

13 }

14 }

15 ...

16 }

Listing A.6: Cascading History Reset.

This is implemented with the help of the interface method Reset of the compiler

generated QoS constraint classes. The method contains calls to history manager


method ClearSignalHistory (Listing A.6) which clears the history of the passed

signal and retrieves the constraint objects associated to the signal and calls the Reset

method. This is executed as long as a non empty history is found.

APPENDIX

B Simple Automatic Paral-

lelization

This chapter gives a brief overview about work in the field of parallel programming.

Multimedia applications can easily benefit of automatic parallelization capabilities

within the programming language. Many codecs or compression algorithms are block

based and in many cases each block can be processed independently (e.g DCT). This

structure obviously calls for simple, automatic parallelization.

B.1 Parallelization - Related Work

Generally, two different approaches to introduce parallelism exist. In the synchronous

approach, one instruction is used to work on multiple data elements. The asyn-

chronous approach on the other hand allows the execution of different instruction

streams simultaneously.

Philipsen and Tichy [PT92] implement a machine independent forall loop both in

a synchronous and asynchronous version targeting multiple architectures with shared

and distributed memory. Furthermore, they showed in [HHP+93] that with the use of

an adequate working environment debugging of parallel programs written with forall

loops is feasible and does not pose a big problem.

Knudsen [Knu96] introduces a queue to distribute the workload of an asynchronous

forall loop on multiple execution units of a shared memory system as provided by multi

processor and multi core computers. Implementing dynamic workload generation via

APPENDIX B. SIMPLE AUTOMATIC PARALLELIZATION Page 186

distributing nested procedures by a queue causes very little overhead in the range of a

few percents. This approach reaches a high utilization even in the last loop iterations

without the need of static analysis.

Zima and Chapman [ZC91] present how dependence analysis can be used by com-

pilers to automatize vectorization and parallelization. Source to source transforma-

tion using forall loops as a target construct is suggested. Explicit synchronization

barriers are used rather than implicitly inserted barriers at the end of each parallel

loop. The described techniques can also be used to verify the mutual independence

of different loop iterations, a condition which is demanded but not yet checked by

Knudsens approach, nor by ours. Loop carried dependencies would require explicit

assumptions about loop behavior.

In this work Knudsen’s approach has been adopted both in our Modula-3 and C#

implementation. In the latter, however, objects are passed instead of using nested

procedures because nested procedures are not supported in C#. Supplying an asyn-

chronous forall seems to be sufficient in most cases of multimedia programming. Asyn-

chronous foralls are also open to vectorization, thus allowing a further possibility for

speedups.

B.2 Language Extensions - Parallelizations

B.2.1 Extended Foreach Statement

Pixel manipulations can be implemented using multidimensional arrays. A lot of en-

coder or decoder implementations make use of such data structures. In programming

languages like Modula or C/C++, such arrays are iterated using simple for state-

ments. In Java and C# new loop statements have been developed in order to iterate

over collections or arrays. In C# the foreach statement [ECM05] is used to iterate

over expressions that can be evaluated to a type that implements the IEnumerable

interface, or a type that declares a GetEnumerator method which then returns an

object of type IEnumerator [Web09d]. Listing B.1 illustrates the syntax.


1 foreach ( type identifier in expression )

2 embedded -statement

Listing B.1: Foreach Statement.

These enumerators iterate over the stored elements. For each element the embed-

ded statement is executed. In multimedia applications it is often required to refer

to the index of the elements accessed. Therefore the foreach statement is extended

to define the expression for retrieving the elements of the array (Listing B.2). This

enables the programmer to define index variables which can be accessed during the

loop. To avoid unpredictable side effects, index access is read only.

1 foreach ( type identifier = expression in expression )


Listing B.2: Extended Foreach Statement.

The following example (Listing B.3) represents a simple implementation of the

discrete cosine transformation (DCT) as used for JPEG implementations [Ste00] im-

plemented in C#.

1 double value = 0;

2

3 for (int u = 0; u < 8; u++) {

4 for (int v = 0; v < 8; v++) {

5 for (int i = 0; i < 8; i++) {

6 for (int j = 0; j < 8; j++) {

7 value += (source [i,j] - 128) * Math.Cos (((2*i + 1) * u * Math.PI) / 16)

8 * Math.Cos (((2*j + 1) * v * Math.PI) / 16) ;

9 }

10 }

11 coefficients[u,v] = value / 4;

12 }

13 }

Listing B.3: DCT Implementation in C#.

Instead of using four conventional nested for statements, the code fragment can

be reimplemented by applying two extended foreach loops without considering the

size of the array being iterated over (Listing B.4).


1 foreach (double c = coefficients[u,v] in source ) {

2 foreach (int p = source [i,j] in source ) {

3 value += (p - 128) * Math.Cos (((2* i + 1) * u * Math.PI) / 16)

4 * Math.Cos (((2* j + 1) * v * Math.PI) / 16);

5 }

6 coefficients[u,v] = value / 4;

7 }

Listing B.4: DCT Implementation in C# with the Extended Foreach Statement.

The extended foreach has been implemented in Modula-3 too (Listing B.5).

1 FOREACH c = coefficients[u,v] IN coefficients VIA u,v DO

2 FOREACH p = source [i,j] IN source VIA i,j DO

3 value := value + ( FLOAT(p-128, LONGREAL ) * cos (FLOAT ((2*i+1)*u,LONGREAL )*Pi /16.0

D0)

4 * cos (FLOAT ((2*j+1)*v,LONGREAL )*Pi /16.0

D0));

5 END

6 coefficients[u,v] := value / 4.0 D0;

7 END

Listing B.5: DCT Implementation in Modula-3 with the Extended Foreach Statement.

The extended foreach loop expresses more clearly how elements are assigned within

the loop. Furthermore, no array ranges have to be considered. The compiler internally

generates the correct code for iteration and therefore provides more safety. No infinite

loops can be created and the statement enables the programmer to express index based

calculations with arrays. Like the original foreach in C#, the current element of the

loop can be directly accessed.

B.2.2 Forall Statement

To optimize performance, parallelism can be added to the loop to distribute its exe-

cution into several threads. This executional optimization can be reached due to the

fact that a foreach statement as defined in [ECM05] does not necessarily guarantee a

special order of execution. The semantics only defines that each element of the given

collection or array is accessed. This fact can be used to implement a foreach loop


in which each of the iterations can be executed simultaneously. The syntax is very

similar to the shown foreach or the extended foreach statement, but the semantics

differ (Listing B.6).

1 forall ( type identifier in expression )


3

4 forall ( type identifier = expression in expression )


Listing B.6: Forall Statement in C#.

The block executed each time the loop iterates is embedded into a job class. The

instance of this class represents a job which is executed for each loop iteration. In-

stead of executing the jobs in sequential order, they are put into a queue to feed

workers which can operate in parallel. These workers are controlled by a manage-

ment framework which observes the execution of all workers. Furthermore, it ensures

that sequential execution follows after all parallel work is done. The compiler itself

generates code to fork the workers, distribute the work to each worker and synchro-

nize all workers when all work has been completed. After all parallel work is done,

the program continues normally.

The default worker pool implementation is integrated into the system class library

of Mono [Web09f]. To give programmers the ability to implement their own worker

pools, the default behavior of jobs and the worker pool is defined by the interfaces

IJob and IWorkerPool (Figure B.1).

The worker pool can be exchanged during runtime by replacing the default worker

pool implementation with the WorkerPoolFactory. Each time a forall statements loop

body is executed, an IJob object is created and added to the worker pool. At the end

of the loop the waitTillFinished method is called to ensure that work has been done

before continuing. The implementation uses the default multi-threading libraries.

The same behavior can be achieved using asynchronous method invocation. This

feature of the Common Language Runtime [BS02] can be used to inherently introduce

concurrency into a program. The forall statement has also been implemented in


Figure B.1: Parallelization Framework Overview.

Modula-3 using nested procedures [5, 6, 16] instead of passing objects.

B.2.3 Statement Level Attributes

A further language extension was added to support statement level parallelism with-

out introducing special loop statements. The concept of attributes was extended

to support statement level annotations. This allows to specify parallel execution of

blocks or loops by defining an attribute directly before the statement.

The attribute ([Parallel(degree)]) is used to define parallelism. It uses the

parameter degree to define the degree of parallelism. The current run-time frameworks

maps the degree to the amount of threads used.

The parallelization attribute can be added to the following language structures:

• Blocks - A code block can be executed in parallel (Listing B.7). Each statement


is executed in parallel to all others. The closing bracket represents a synchro-

nization point at which execution waits until all parallel statements have been

executed.

• Foreach Statements - The loop body is executed in parallel (Listing B.8).

• Extended Foreach Statement - The loop body is executed in parallel.

• For Statement - The loop body is executed in parallel (Listing B.9).

In order to apply parallel execution, the developer needs to consider that there is

no guaranteed order of execution and no dependency checking.

1 [Parallel (2) ] // the two statements are executed in parallel

2 {

3 statement1 ; // some statement

4 statement2 ; // some statement

5 }

Listing B.7: Parallel Statements.

1 [Parallel (4) ]

2 foreach (int i in array) {

3 ... // loop body

4 }

Listing B.8: Parallel Foreach Statement - Extended Foreach Statement.

1 [Parallel (4) ]

2 for (int i = 0; i < 10; i++) {

3 ... // loop body

4 }

Listing B.9: Parallel For Statement.

However, applying parallelism is not without issues. The programmer has to

consider the overhead for thread creation, control and the cost of object initialization.

Normal loops can be terminated using return or break statements. This cannot be

easily achieved when using parallelized execution. So the use of these statements

has been forbidden, because of the misleading semantics. If break or return is used,


the programmer wants the execution of the loop to stop, but the parallel execution

disables immediate loop termination. Another problem is exception handling within

the loop. This is achieved by catching exceptions within the worker threads, and

throw them at the end of the execution to allow the programmer to use the default

exception handling mechanisms of the language.

B.3 Performance

The performance enhancement is demonstrated using the simple block based DCT

(Listing B.4). The forall statement is applied to parallelize the processing of blocks

where the number of threads and the number of blocks are varied. The measure-

ment was done on a quad processor machine showing linear scalability (Figure B.2)

and little overhead (Figure B.3). The execution time of the forall statement using

one single thread is only a few percents higher than the execution time of an imple-

mentation using for statements. This overhead is caused by object initialization and

thread control. When increasing the number of threads, the execution time decreases

significantly, but when the number of threads reaches the number of processors, the

management overhead grows considerably. Similar results are presented in [Knu96]

which proofs the efficiency of our implementation.

It is also possible to speed up the decoding process of an AVC/H.264 video (Figure

B.4). In contrast to the DCT block based example, the AVC/H.264 coding structure

is more complex. Many dependencies even between single blocks are introduced.

Simple automatic parallelization is not possible. Dependencies need to be considered

and synchronization is required. Nevertheless, a respectable speedup was achieved.

B.4 Evaluation

As the parallelization enhancement is implemented as a language extension, the same

requirements need to be evaluated as defined in section 3.3.


Figure B.2: Parallelized DCT.

• Expressiveness - The expressive power of the extended foreach statement has

been demonstrated by rewriting a DCT. The new statement tries to express the

array loops with more simplicity in syntax but, of course, higher complexity in

semantics.

• Safety - The safety of the programs is enhanced due to the fact that the loop

termination condition is hidden in the compiler generated code and guarantees

that the loop comes to an end after it has accessed all elements of the data

structure. The parallel statements automatically synchronize at the end of the

parallelization block and wait till all executed tasks are finished.

• Optimizations - The forall loop and the other parallel statements can be used

to optimize the execution time of the program (demonstrated by examples and

graphs), where the loop body is separated into work packages and put into

an efficient parallel processing framework. The extended foreach statements

has further optimization potential. It is possible to automatically split huge

multidimensional arrays into smaller ones.


Figure B.3: Parallelized DCT Efficiency.

As multi-core or multi-processor machines become widely available, novel libraries

have been introduced to make parallel programming easier. Such libraries like the

Task Parallel Library (TPL) [msd09b, wik09a] make use of anonymous delegates to

implement parallel for and foreach statements.

The work according to simple automatic parallelization was not continued after

the new AVC/H.264 [h26] coding standard was published (Section 4). Modern codecs

introduce very complex coding structures to facilitate compression strategies based

on dependencies between simple data blocks. A complete dependency analysis would

be required to automatically parallelize such codec implementation.


Figure B.4: Parallelized AVC.

Bibliography

[AGG+94] Joshua S. Auerbach, Arthur P. Goldberg, German S. Goldszmidt,

Ajei S. Gopal, Mark T. Kennedy, Josyula R. Rao, and James R.

Russell. Concert/c: a language for distributed programming. In

WTEC’94: Proceedings of the USENIX Winter 1994 Technical Con-

ference on USENIX Winter 1994 Technical Conference, pages 8–8,

Berkeley, CA, USA, 1994. USENIX Association.

[APBP02] R. Koster J. Walpole A. P. Black, J. Huang and C. Pu. Infopipes: an

abstraction for multimedia streaming. In Multimedia Systems (spe-

cial issue on Multimedia Middleware) 8(5), pages 406–419. Springer

Verlag, 2002.

[asp09a] Castle project, aspect#, aop framework for the cli, 2009.

http://www.castleproject.org/AspectSharp/.

[asp09b] Eclipse, the aspectj project, 2009. http://www.eclipse.org/aspectj/.

[BDH+03] L. Burgstahler, K. Dolzer, C. Hauser, J. Jahnert, S. Junghans,

C. Macian, and W. Payer. Beyond technology: the missing pieces

for qos success. In RIPQoS ’03: Proceedings of the ACM SIGCOMM

workshop on Revisiting IP QoS, pages 121–130, New York, NY, USA,

2003. ACM.

197

BIBLIOGRAPHY Page 198

[BDS96] I. Busse, B. Deffner, and H. Schulzrinne. Dynamic qos control of

multimedia applications based on rtp. In Computer Communications,

1996.

[BG92] Gerard Berry and Georges Gonthier. The esterel synchronous pro-

gramming language: design, semantics, implementation. Sci. Com-

put. Program., 19(2):87–152, 1992.

[BGG+91] Allan Bricker, Michel Gien, Marc Guillemont, Jim Lipkis, Douglas

Orr, and Marc Rozier. A new look at microkernel-based operating

systems: Lessons in performance and compatibility, 1991.

[BHHA08] Peter Brooks, Bjørn Hestnes, Svein Heiestad, and Carl Aaby. Commu-

nicating quality of experience data for the development of multimedia

services. 2008.

[BHK+02] Andrew P. Black, Jie Huang, Rainer Koster, Jonathan Walpole, and

Calton Pu. Infopipes: an abstraction for multimedia streaming. vol-

ume 8, pages 406–419, Secaucus, NJ, USA, 2002. Springer-Verlag New

York, Inc.

[BS98] Gorden S. Blair and Jean-Bernard Stefani. Open Distributed Process-

ing and Multimedia. Addison-Wesley Longman Publishing Co., Inc.,

Boston, MA, USA, 1998.

[BS02] Don Box and Chris Sells. Essential .Net Volume 1: The Common

Language Runtime. Addison Wesley, 2002.

[BW96] Laszlo Boszormenyi and Carsten Weich. Programming in Modula-3 -

An Introduction in Programming with Style. Springer, 1996.

[CB95] Geoff Coulson and Gordon Blair. Architectural principles and tech-

niques for distributed multimedia application support in operating

systems. SIGOPS Oper. Syst. Rev., 29(4):17–24, 1995.


[CCH+00] Mark Claypool, Tom Coates, Shawn Hooley, Eric Shea, and Chris

Spellacy. Video performance in java. In Proceedings of the 2000 in-

formation resources management association international conference

on Challenges of information technology management in the 21st cen-

tury, pages 222–225, Hershey, PA, USA, 2000. IGI Publishing.

[CEC00] Krzysztof Czarnecki, Ulrich Eisenecker, and Krzysztof Czar-

necki. Generative Programming: Methods, Tools, and Applications.

Addison-Wesley Professional, June 2000.

[CFSD90] J.D. Case, M. Fedor, M.L. Schoffstall, and J. Davin. Simple Network

Management Protocol (SNMP). RFC 1157 (Historic), May 1990.

[cli09] List of cli languages, 2009. From Wikipedia, the free encyclopedia,

http://en.wikipedia.org/wiki/Microsoft .NET Languages.

[CNRS98] E. Crawley, R. Nair, B. Rajagopalan, and H. Sandick. A Framework

for QoS-based Routing in the Internet. RFC Editor, United States,

1998.

[DHT95] C. Diot, C. Huitema, and T. Turletti. Multimedia applications should

be adaptive. In Proceedings HPCS’95, Mystic (CN), August 1995.

[dI02] Miguel de Icaza. The mono c# compiler, 2002. From

Mono-Project, http://primates.ximian.com/ miguel/slides-europe-

nov-2002/Mono C Sharp Overview 1007.sxi.

[dI09] Miguel de Icaza. The internals of the mono c# compiler, 2009. From

Mono-Project Subversion.

[ECM05] C# language specification, June 2005. Standard ECMA-334.

[ECM06] Common language infrastructure (cli), June 2006. Standard ECMA-

335.


[ED04] Horst M. Eidenberger and Roman Divotkey. Media Processing in

Java. Heidelberg, dpunkt-Verlag, 2004.

[EGP+02] Ernesto Exposito, Mathieu Gineste, Romain Peyrichou, Patrick

Senac, Michel Diaz, and Serge Fdida. Xml qos specification language

for enhancing communication services. In ICCC ’02: Proceedings of

the 15th international conference on Computer communication, pages

76–90, Washington, DC, USA, 2002. International Council for Com-

puter Communication.

[FK98] Svend Frølund and Jari Koistinen. Qml: A language for quality of

service specification. Technical Report HPL-98-10, Hewlett-Packard

Software Technology Laboratory, 1998.

[Flo94] Patrıcia Gomes Soares Florissi. Quality of service management au-

tomation in integrated distributed systems. In CASCON ’94: Pro-

ceedings of the 1994 conference of the Centre for Advanced Studies on

Collaborative research, page 18. IBM Press, 1994.

[Flo96] Patricia Gomes Soares Florissi. QOSME: QOS management environ-

ment. PhD thesis, New York, NY, USA, 1996.

[Flu95] Francois Fluckiger. Understanding networked multimedia: applica-

tions and technology. Prentice Hall International (UK) Ltd., Hert-

fordshire, UK, UK, 1995.

[GBCD98] Tom Fitzpatrick Gordon, Gordon S. Blair, Geoff Coulson, and Nigel

Davies. A software architecture for adaptive distributed multimedia

systems, 1998.

[Gib95] Simon Gibbs. Multimedia component frameworks. pages 305–319,

Hertfordshire, UK, 1995. Prentice Hall International (UK) Ltd.


[GN00] Xiaohui Gu and Klara Nahrstedt. Visual quality of service specifica-

tion for distributed heterogenous systems. Technical report, Cham-

paign, IL, USA, 2000.

[GNY+02] Xiaohui Gu, Klara Nahrstedt, Wanghong Yuan, Duangdao

Wichadakul, and Dongyan Xu. An xml-based quality of service en-

abling language for the web. Journal of Visual Language and Com-

puting, Special Issue on Multimedia Language for the Web, 13:61–95,

2002.

[GS93] David Garlan and Mary Shaw. An introduction to software archi-

tecture. In V. Ambriola and G. Tortora, editors, Advances in Soft-

ware Engineering and Knowledge Engineering, pages 1–39, Singapore,

1993. World Scientific Publishing Company.

[GT95] Simon J. Gibbs and Dionysios Tsichritzis. Multimedia programming:

objects, environments and frameworks. ACM Press/Addison-Wesley

Publishing Co., New York, NY, USA, 1995.

[h26] ISO/IEC JTC 1/SC 29/WG 11: ISO/IEC FDIS 14496-10Information

Technology - Coding of audio-visual objects - Part 10: Advanced

Video Coding. (2003).

[HHP+93] Stefan U. Hanßgen, Ernst A. Heinz, Michael Philippsen, Paul Lukow-

icz, and Walter F. Tichy. The modula-2* environment for parallel

programming. In In Proceedings of the Working Conference on Mas-

sively Parallel Programming Models, pages 43–52. IEEE Society Press,

1993.

[Hun09] Andrew Hunter. Understanding garbage collection in .net, 2009.

http://www.simple-talk.com/dotnet/.net-framework/understanding-

garbage-collection-in-.net/.


[IT94] ITU-T. Recommendation e.800: Terms and definitions related to

quality of service and network performance including dependability’.

August 1994.

[JE97] S. Jacobs and A. Eleftheriadis. Adaptive video applications for non-

qos networks. In 5 th International Workshop on Quality of Service

(IWQOS’97), pages 161–165, Columbia University, New York, USA,

1997.

[JN] Jingwen Jin and Klara Nahrstedt. Classification and comparison of

qos specification languages for distributed multimedia applications.

[JN04] Jingwen Jin and Klara Nahrstedt. Qos specification languages for

distributed multimedia applications: A survey and taxonomy. IEEE

MultiMedia, 11(3):74–87, 2004.

[KBH+01] Rainer Koster, Andrew P. Black, Jie Huang, Jonathan Walpole, and

Calton Pu. Infopipes for composing distributed information flows. In

In the Proceedings of the ACM Multimedia Workshop on Multimedia

Middleware, pages 44–47. ACM, 2001.

[KBP07] Anne-Marie Kermarrec, Luc Bouge, and Thierry Priol, editors. Euro-

Par 2007, Parallel Processing, 13th International Euro-Par Confer-

ence, Rennes, France, August 28-31, 2007, Proceedings, volume 4641

of Lecture Notes in Computer Science. Springer, 2007.

[Kic] G. Kiczales. Aspect-oriented programming. ACM Comput. Surv.,

page 154.

[Kir02] Christoph M. Kirsch. Principles of real-time programming. In Al-

berto L. Sangiovanni-Vincentelli and Joseph Sifakis, editors, EM-

SOFT, volume 2491 of Lecture Notes in Computer Science, pages

61–75. Springer, 2002.


[KN97] Kiwook Kim and Klara Nahrstedt. Qos translation and admission

control for mpeg video. In 5th International Workshop on Quality of

Service (IWQoS’97, pages 359–362, 1997.

[Knu96] Svend Erik Knudsen. Statement-sets. In Laszlo Boszormenyi, editor,

ACPC, volume 1127 of Lecture Notes in Computer Science, pages

161–173. Springer, 1996.

[Kop97] Hermann Kopetz. Real-Time Systems: Design Principles for Dis-

tributed Embedded Applications. Kluwer Academic Publishers, Nor-

well, MA, USA, 1997.

[Kop00] Hermann Kopetz. Software engineering for real-time: a roadmap. In

ICSE ’00: Proceedings of the Conference on The Future of Software

Engineering, pages 201–211, New York, NY, USA, 2000. ACM.

[KR88] Brian W. Kernighan and Dennis Ritchie. The C Programming Lan-

guage, Second Edition. Prentice-Hall, 1988.

[KSB09] Peter Karpati, Tibor Szkaliczki, and Laszlo Boszormenyi. Designing

and scaling distributed vod servers. Multimedia Tools Appl., 41(1):55–

91, 2009.

[LBS+] Joseph P. Loyall, David E. Bakken, Richard E. Schantz, John A.

Zinky, David A. Karr, Rodrigo Vanegas, and Kenneth R. Anderson.

Qos aspect languages and their runtime integration (html version).

http://www.dist-systems.bbn.com/papers/1998/LCR/lcr.html.

[LBS+98] Joseph P. Loyall, David E. Bakken, Richard E. Schantz, John A.

Zinky, David A. Karr, Rodrigo Vanegas, and Kenneth R. Ander-

son. Qos aspect languages and their runtime integration. In LCR

’98: Selected Papers from the 4th International Workshop on Lan-

guages, Compilers, and Run-Time Systems for Scalable Computers,

pages 303–318, London, UK, 1998. Springer-Verlag.


[Leu94] Stefan Leue. Qos specification based on sdl/msc and temporal logic.

In Proceedings of Workshop on Distributed Multimedia Applications

and Quality of Service Verification, 1994.

[LSB06] Oliver Lampl, Elmar Stellnberger, and Laszlo Boeszoermenyi. Pro-

gramming language concepts for multimedia application development.

In Modular Programming Languages, pages 23–36. Springer, Septem-

ber 2006.

[LSZB98] J. P. Loyall, R. E. Schantz, J. A. Zinky, and D. E. Bakken. Specify-

ing and measuring quality of service in distributed object systems. In

ISORC ’98: Proceedings of the The 1st IEEE International Sympo-

sium on Object-Oriented Real-Time Distributed Computing, page 43,

Washington, DC, USA, 1998. IEEE Computer Society.

[LWRS08] Marco Lohse, Florian Winter, Michael Repplinger, and Philipp

Slusallek. Network-integrated multimedia middleware (nmm). In MM

’08: Proceeding of the 16th ACM international conference on Multi-

media, pages 1081–1084, New York, NY, USA, 2008. ACM.

[M06] Hanspeter Mossenbock. Softwareentwicklung mit C# 2.0.

dpunkt.verlag, Heidelberg, Deutschland, 2006.

[Mar01] Marco Lohse and Philipp Slusallek and Patrick Wambach. Extended

Format Definition and Quality-driven Format Negotiation in Multi-

media Systems. In Multimedia 2001 – Proceedings of the Eurographics

Workshop, pages 65–74. Springer, 2001.

[Mar05] Marco Lohse and Philipp Slusallek. Towards Automatic Setup of Dis-

tributed Multimedia Applications. In Proceedings of The 9th IASTED

International Conference on Internet and Multimedia Systems and

Applications (IMSA), pages 359–364. ACTA Press, 2005.


[MATH+08] Phil McGachey, Ali-Reza Adl-Tabatabai, Richard L. Hudson, Vijay

Menon, Bratin Saha, and Tatiana Shpeisman. Concurrent gc lever-

aging transactional memory. In PPoPP ’08: Proceedings of the 13th

ACM SIGPLAN Symposium on Principles and practice of parallel

programming, pages 217–226, New York, NY, USA, 2008. ACM.

[Mit97] Scott Mitchell. Djinn: A programming framework for distributed

multimedia groupware applications. Technical Report no. 727, De-

partment of Computer Science, Queen Mary & Westfield College,

University of London, 1997.

[MNCK98] Scott Mitchell, Hani Naguib, George Coulouris, and Tim Kindberg.

Dynamically reconfiguring multimedia components: A model-based

approach. In Proceedings of the 8th ACM SIGOPS European Work-

shop, Sintra, Portugual, pages 40–47. ACM Press, 1998.

[MNCK99] Scott Mitchell, Hani Naguib, George Coulouris, and Tim Kindberg.

A qos support framework for dynamically reconfigurable multimedia

applications. In Proceedings of the IFIP WG 6.1 International Work-

ing Conference on Distributed Applications and Interoperable Systems

II, pages 17–30, Deventer, The Netherlands, 1999. Kluwer, B.V.

[MPE] International organisation for standardisation iso/iec

jtc1/sc29/wg11 coding of moving pictures and audio, mpeg-2,

http://www.chiariglione.org/mpeg/standards/mpeg-2/mpeg-2.htm.

[MPS01] Yishay Mansour and Boaz Patt-Shamir. Jitter control in qos net-

works. IEEE/ACM Trans. Netw., 9(4):492–502, 2001.

[MR90] K. McCloghrie and M.T. Rose. Management Information Base for

network management of TCP/IP-based internets. RFC 1156 (His-

toric), May 1990.


[msd07] Creating an xml web service method, 2007.

http://msdn.microsoft.com.

[msd09a] Datetime structure (system), 2009. From MSDN,

http://msdn.microsoft.com/en-us/library/system.datetime.aspx.

[msd09b] Parallel programming in the .net framework, 2009. From MSDN,

http://msdn.microsoft.com/en-us/library/dd460693(VS.100).aspx.

[msd09c] Stopwatch class (system.diagnostics), 2009.

From MSDN, http://msdn.microsoft.com/en-

us/library/system.diagnostics.stopwatch.aspx.

[msd09d] Windows time (windows), 2009. From MSDN,

http://msdn.microsoft.com/en-us/library/ms725496(VS.85).aspx.

[MtcvgaUD09] Omnividea Multimedia and the computer vision group at UPM-

DISAM. Fobs - ffmpeg objects, 2009. http://fobs.sourceforge.net.

[NEG+05] Christian Nagel, Bill Evjen, Jay Glynn, Morgan Skinner, Karli Wat-

son, and Allen Jones. Professional C# 2005. Wrox Press Ltd., Birm-

ingham, UK, UK, 2005.

[Nel91] Greg Nelson, editor. Systems programming with Modula-3. Prentice-

Hall, Inc., Upper Saddle River, NJ, USA, 1991.

[NI05] Thi Viet Nga Nguyen and Francois Irigoin. Efficient and effective

array bound checking. ACM Trans. Program. Lang. Syst., 27(3):527–

570, 2005.

[nis99] National institute of standards and technology, requirements for real-

time extensions for the java plattform, 1999. http://www.nist.gov/rt-

java.


[NKMC98] H. Naguib, T. Kindberg, S. Mitchell, and G. Coulouris. Modelling qos

characteristics of multimedia applications. Proc. 13 th IEEE Real-

Time Systems Symposium (RTSS ’98), Madrid, Spain, 1998.

[NWX00] Klara Nahrstedt, Duangdao Wichadakul, and Dongyan Xu. Dis-

tributed qos compilation and runtime instantiation. 2000.

[obj09] Object pool, 2009. From Wikipedia, the free encyclopedia,

http://en.wikipedia.org/wiki/Object pool.

[OMG99] OMG. CORBA/IIOP 2.3, OMG Technical Document 98-12-01. Ob-

ject Management Group, Framingham. MA, December 1999.

[PH05] Jarmo Prokkola and Mikko Hanski. Qos measurement methods and

tools. Technical Report E4SU00169, VTT Technical Research Centre

of Finland, 2005.

[PT92] Michael Philippsen and Walter F. Tichy. Modula-2 and its compila-

tion. In Proceedings of the First International ACPC Conference on

Parallel Computation, pages 169–183, London, UK, 1992. Springer-

Verlag.

[qco09] Qcompiler: A meta-level compilation framework for enabling ubi-

quitous, distributed component-based qos-aware programming, 2009.

http://cairo.cs.uiuc.edu/projects/middleware/QCompiler/index.html.

[RAA+91] M. Rozier, V. Abrossimov, F. Armand, I. Boule, M. Gien, M. Guille-

mont, F. Herrmann, C. Kaiser, S. Langlois, P. Leonard, and

W. Neuhauser. Overview of the chorus distributed operating systems.

Computing Systems, 1:39–69, 1991.

[RHE99] Reza Rejaie, Mark Handley, and Deborah Estrin. Quality adapta-

tion for congestion controlled video playback over the internet. In

SIGCOMM, pages 189–200, 1999.


[RHR+01] Manuel Roman, Christopher K. Hess, Anand Ranganathan, Pradeep

Madhavarapu, Bhaskar Borthakur, Prashant Viswanathan, Renato

Cerqueira, Roy H. Campbell, and M. D Mickunas. Gaiaos: An in-

frastructure for active spaces. Technical report, Champaign, IL, USA,

2001.

[Rica] Jeffrey Richter. Garbage collection - part 2: Automatic memory

management in the microsoft .net framework. From the Decem-

ber 2000 issue of MSDN Magazine, http://msdn.microsoft.com/en-

us/magazine/bb985011.aspx.

[Ricb] Jeffrey Richter. Garbage collection: Automatic memory man-

agement in the microsoft .net framework. From the Novem-

ber 2000 issue of MSDN Magazine, http://msdn.microsoft.com/de-

de/magazine/bb985010(en-us).aspx.

[RK01] Reza Rejaie and Jussi Kangasharju. Mocha: a quality adaptive mul-

timedia proxy cache for internet streaming. In NOSSDAV ’01: Pro-

ceedings of the 11th international workshop on Network and operating

systems support for digital audio and video, pages 3–10, New York,

NY, USA, 2001. ACM.

[RM90] M.T. Rose and K. McCloghrie. Structure and identification of man-

agement information for tcp/ip-based internets. RFC 1155 (Stan-

dard), May 1990.

[RNK+97] Ranga S. Ramanujan, James A. Newhouse, Maher Kaddoura, Atiq

Ahamad, Eric R. Chartier, and Kenneth J. Thurber. Adaptive

streaming of mpeg video over ip networks. In LCN ’97: Proceedings

of the 22nd Annual IEEE Conference on Local Computer Networks,

pages 398–409, Washington, DC, USA, 1997. IEEE Computer Society.


[SBH+03] Peter Schojer, Laszlo Boszormenyi, Hermann Hellwagner, Bernhard

Penz, and Stefan Podlipnig. Architecture of a quality based intelligent

proxy (qbix) for mpeg-4 videos. In WWW ’03: Proceedings of the

12th international conference on World Wide Web, pages 394–402,

New York, NY, USA, 2003. ACM.

[Sch01] Jens Burkhard Schmitt. Heterogeneous Network Quality of Service

Systems. Kluwer Academic Publishers, Norwell, MA, USA, 2001.

[Sch09] Klaus Schoeffmann. Immediate Video Exploration – Enabling Explo-

rative Search in Videos for Instantaneous Use by Fast Content Anal-

ysis and Integration of Users’ Expertise. PhD thesis, University of

Klagenfurt, May 2009.

[SFLB07a] Klaus Schoffmann, Markus Fauster, Oliver Lampl, and Laszlo

Boszormenyi. An evaluation of parallelization concepts for baseline-

profile compliant h.264/avc decoders. In Kermarrec et al. [KBP07],

pages 782–791.

[SFLB07b] Klaus Schoffmann, Markus Fauster, Oliver Lampl, and Laszlo

Boszormenyi. An evaluation of parallelization concepts for baseline-

profile compliant h.264/avc decoders. In Kermarrec et al. [KBP07],

pages 782–791.

[SM97] George Coulouris-Tim Kindberg Scott Mitchell, Hani Naguib. A

framework for distributed multimedia applications. Submitted to

NOSSDAV ’97, the 7th International Workshop on Network and Op-

erating Systems Support for Digital Audio and Video, St. Louis, USA,

1997.

[SN04] Ralf Steinmetz and Klara Nahrstedt. Multimedia Systems. Springer,

Apr 2004.


[SP08] Jose San Pedro. Fobs: an open source object-oriented library for

accessing multimedia content. In MM ’08: Proceeding of the 16th

ACM international conference on Multimedia, pages 1097–1100, New

York, NY, USA, 2008. ACM.

[SPK+04] Galen Swint, Calton Pu, Younggyun Koh, Ling Liu, Wenchang Yan,

Charles Consel, Koichi Moriyama, and Jonathan Walpole. Infopipes:

The isl/isg implementation evaluation. In NCA ’04: Proceedings of

the Network Computing and Applications, Third IEEE International

Symposium, pages 293–296, Washington, DC, USA, 2004. IEEE Com-

puter Society.

[SPM04] Galen S. Swint, Calton Pu, and Koichi Moriyama. Infopipes: Con-

cepts and isg implementation. In WSTFEUS, pages 19–23. IEEE

Computer Society, 2004.

[Ste00] Ralf Steinmetz. Multimedia-Technologie: Grundlagen, Komponenten

und Systeme. Springer Verlag, Oct 2000. 3. Auflage (erstmalig mit

CD).

[Tan03] Andrew S. Tanenbaum. Computer Networks. Prentice Hall, fourth

edition, 2003.

[TBO05a] Jean-Charles Tournier, Jean-Philippe Babau, and Vincent Olive. An

evaluation of qinna, a component-based qos architecture for embedded

systems. In Hisham Haddad, Lorie M. Liebrock, Andrea Omicini, and

Roger L. Wainwright, editors, SAC, pages 998–1002. ACM, 2005.

[TBO05b] Jean-Charles Tournier, Jean-Philippe Babau, and Vincent Olive.

Qinna, a component-based qos architecture. In George T. Heineman,

Ivica Crnkovic, Heinz W. Schmidt, Judith A. Stafford, Clemens A.

Szyperski, and Kurt C. Wallnau, editors, CBSE, volume 3489 of Lec-

ture Notes in Computer Science, pages 107–122. Springer, 2005.


[TOB04] J-C. Tournier, V. Olive, and J-P. Babau. The qinna experiment, a

component-based qos architecture for real-time systems. In Work-

shop on Architectures for Cooperative Embedded Real-Time Systems

in conjunction with the 25th IEEE RTSS, Lisbonne, Portugal, De-

cember 2004.

[Vel95] Todd Veldhuizen. Expression templates. C++ Report, 7(5):26–31,

June 1995.

[VZL+98] Rodrigo Vanegas, John A. Zinky, Joseph P. Loyall, David Karr,

Richard E. Schantz, and David E. Bakken. QuO’s Runtime Support

for Quality of Service in Distributed Objects. Proceedings of Middle-

ware 98, the IFIP International Conference on Distributed Systems

Platform and Open Distributed Processing (Middleware 98), Septem-

ber 1998.

[web07] Web services description language (wsdl) 1.1, 2007.

http://www.w3.org/TR/wsdl.

[Web09a] Ffmpeg, 2009. http://www.ffmpeg.org/.

[Web09b] Java, 2009. http://java.sun.com/.

[Web09c] Java media framework, 2009. http://java.sun.com/javase /technolo-

gies/desktop/media/jmf/.

[Web09d] Microsoft developer network: C# programmer’s reference, 2009.

http://msdn.microsoft.com.

[Web09e] Microsoft silverlight, 2009. http://silverlight.net/.

[Web09f] Mono: Open source .net development framework, 2009.

http://www.mono-project.com.


[Web09g] Mono (software), 2009. From Wikipedia, the free encyclopedia,

http://en.wikipedia.org/wiki/Mono (software).

[Web09h] Msdn - direct show, 2009. http://msdn.microsoft.com/en-

us/library/ms783323.aspx/.

[Web09i] .net framework architecture, 2009. From Mono-Project,

http://www.mono-project.com/.NET Framework Architecture.

[Web09j] Producer-consumer problem, 2009. From Wikipedia, the free ency-

clopedia, http://en.wikipedia.org/wiki/Producer-consumer problem.

[Web09k] Scalable video coding, 2009. From Wikipedia, the free encyclopedia,

http://en.wikipedia.org/wiki/Scalable Video Coding.

[Web09l] Xml - extensible markup language, 2009. http://www.w3.org/XML/.

[WGN02] Duangdao Wichadakul, Xiaohui Gu, and Klara Nahrstedt. A pro-

gramming framework for quality-aware ubiquitous multimedia appli-

cations. In MULTIMEDIA ’02: Proceedings of the tenth ACM inter-

national conference on Multimedia, pages 631–640, New York, NY,

USA, 2002. ACM.

[wik09a] Parallel extensions, 2009. From Wikipedia, the free encyclopedia,

http://en.wikipedia.org/wiki/Parallel Extensions.

[wik09b] Qualiy of experience, 2009. From Wikipedia, the free encyclopedia,

http://en.wikipedia.org/wiki/Quality of experience.

[wik09c] Template metaprogramming, 2009. From Wikipedia, the free ency-

clopedia, http://en.wikipedia.org/wiki/Template metaprogramming.


[WKC+97] J. Walpole, R. Koster, S. Cen, C. Cowan, D. Maier, D. McNamee,

C. Pu, D. Steere, and L. Yu. A player for adaptive mpeg video stream-

ing over the internet. In 26th Applied Imagery Pattern Recognition

Workshop AIPR-97, Washington, DC, USA, 1997.

[WN01] Duangdao Wichadakul and Klara Nahrstedt. Distributed qos com-

piler. Technical report, 2001.

[WNGX01] Duangdao Wichadakul, Klara Nahrstedt, Xiaohui Gu, and Dongyan

Xu. 2k: An integrated approach of qos compilation and reconfig-

urable, component-based run-time middleware for the unified qos

management framework. In Middleware ’01: Proceedings of the

IFIP/ACM International Conference on Distributed Systems Plat-

forms Heidelberg, pages 373–394, London, UK, 2001. Springer-Verlag.

[WT09] Erik Walthinsen and Wim Taymans. Gstreamer - open source multi-

media framework, 2009. http://gstreamer.freedesktop.org.

[WYF+00] Phil Wang, Yechiam Yemini, Danilo Florissi, John A. Zinky, and Pa-

tricia Florissi. Experimental qos performances of multimedia appli-

cations. In INFOCOM, pages 970–979, 2000.

[ZBS97] John A. Zinky, David E. Bakken, and Richard D. Schantz. Architec-

tural support for quality of service for corba objects, 1997.

[ZC91] Hans Zima and Barbara Chapman. Supercompilers for parallel and

vector computers. ACM, New York, NY, USA, 1991.

[ZW05] A. Zerzelidis and A. J. Wellings. Requirements for a real-time .net

framework. SIGPLAN Not., 40(2):41–50, 2005.

multimedia c# support of qos-aware and adaptive

Documents