ASX: An Object-Oriented Framework forDeveloping Distributed Applications

Douglas C. Schmidtschmidt@ics.uci.edu

Department of Information and Computer ScienceUniversity of California, Irvine, CA 92717, (714) 856-4105

An earlier version of this paper appeared in the Proceed-ings of the6th USENIX C++ Conference, Cambridge, MA,April 11-14, 1994.

Abstract

The ADAPTIVE Service eXecutive (ASX) is a highly modularand extensible object-oriented framework that simplifies thedevelopment and configuration of distributed applicationson shared memory multi-processor platforms. This paperdescribes the structure and functionality of theASX frame-work’s object-oriented architecture. In addition, the paperpresents the results of performance experiments conductedusing ASX-based implementations of connection-orientedand connectionless protocols from the TCP/IP protocol fam-ily. These experiments measure the performance impact ofalternative methods for parallelizing communication proto-col stacks. Throughout the paper, examples are presented toindicate how the use of object-oriented techniques facilitateapplication extensibility, component reuse, and performanceenhancement.

1 Introduction

Distributed computing is a promising technology for im-proving collaboration through connectivity and interwork-ing; performance through parallel processing; reliability andavailability through replication; scalability and portabilitythrough modularity; extensibility through dynamic configu-ration and reconfiguration; and cost effectiveness through re-source sharing and open systems. Despite these benefits, dis-tributed applications (such as on-line transaction processingsystems, global mobile communication systems, distributedobject managers, video-on-demandservers, and communica-tion subsystem protocol stacks) are often significantly morecomplex to develop and configure than non-distributed ap-plications.

A significant portion of this complexity arises from limita-tions with conventional tools and techniques used to developdistributed application software. Conventional applicationdevelopment environments (such as UNIX, Windows NT,and OS/2) lack type-safe, portable, re-entrant, and extensible

system call interfaces and component libraries. For instance,endpoints of communication in the widely used socket net-work programming interface are identified via weakly-typedI/O descriptors that increase the potential for subtle run-timeerrors [1]. Another major source of complexity arises fromthe widespread use of development techniques based uponalgorithmic decomposition [2], which limit the extensibility,reusability, and portability of distributed applications.

Object-oriented techniques offer a variety of principles,methods, and tools that help to alleviate much of the com-plexity associated with developing distributed applications.To illustrate how these techniques are being successfully ap-plied in several research and commercial settings, this pa-per describes the structure and functionality of the ADAP-TIVE Service eXecutive (ASX). ASX is an object-orientedframework containing automated tools and reusable compo-nents that collaborate to simplify the development, config-uration, and reconfiguration of distributed applications onshared memory multi-processor platforms.

Components in theASXframework are designed to decou-ple (1) application-independent components provided by theframework that handle interprocess communication, eventdemultiplexing, explicit dynamic linking, concurrency, andservice configuration from (2) application-specific compo-nents inherited or instantiated from the framework that per-form the services in a particular distributed application. Theprimary unit of configuration in theASX framework is theservice. A service is a portion of a distributed applica-tion that offers a single processing capability to communi-cating entities. Services may be simple (such as returningthe current time-of-day) or highly complex (such as a real-time distributed PBX event traffic monitor [3]). By employ-ing object-oriented techniques to decouple the application-specific service functionality from the reusable application-independent framework mechanisms,ASXfacilitates the de-velopment of applications that are significantly more exten-sible and portable than those based on conventional algorith-mic decomposition techniques. For example, it is possible todynamic reconfigure one or more services in anASX-basedapplication without requiring the modification, recompila-tion, relinking, or restarting of a running system [4].

In addition to describing the object-oriented architectureof theASXframework, this paper examines results obtained

1

by using the framework to conduct experiments on proto-col stack performance in multi-processor-based communica-tion subsystems. In the experiments, theASXcomponentshelp control for several relevant confounding factors (suchas protocol functionality, concurrency control schemes, andapplication traffic characteristics) in order to precisely mea-sure the performance impact of different methods for par-allelizing communication protocol stacks. For example, inthe experiments described in Section 3, connectionless andconnection-oriented protocol stacks were developed by spe-cializing existing components in theASX framework viatechniques involving inheritance and parameterized types.These techniques hold the protocol functionality constantwhile allowing the parallel processing structure of the pro-tocol stacks to be altered systematically in a controlled man-ner.

This paper is organized as follows: Section 2 outlinesthe primary features of theASX framework and describesits object-oriented architecture, Section 3 examines empiri-cal results from experiments conducted using the frameworkto parallelize communication protocol stacks; and Section 4presents concluding remarks.

2 The ADAPTIVE Service eXecutiveFramework

2.1 Overview

The ADAPTIVE Server eXecutive (ASX) is an object-oriented framework that is specifically targeted for the do-main of distributed applications. The framework simpli-fies the construction of distributed applications by improv-ing the modularity, extensibility, reusability, and portabil-ity of both the application-specific network services andthe application-independent OS interprocess communication(IPC), demultiplexing, explicit dynamic linking, and concur-rency mechanisms that these services utilize.

A framework is an integrated collection of componentsthat collaborate to produce a reusable architecture for a fam-ily of related applications [5]. Object-oriented frameworksare becoming increasingly popular as a means to simplifyand automate the development and configuration process as-sociated with complex application domains such as graphi-cal user interfaces [6], databases [7], operating system ker-nels [8], and communication subsystems [9]. The compo-nents in a framework typically includeclasses(such as mes-sage managers, timer-based event managers, demultiplexers[10], and assorted protocol functions and mechanisms [11]),class hierarchies(such as an inheritance lattice of mecha-nisms for local and remote interprocess communication [1]),class categories(such as event demultiplexers [12]), andobjects(such as a service dispatch table). By emphasiz-ing the integration and collaboration of application-specificand application-independent components, frameworks en-able larger-scale reuse of software compared with simplyreusing individual classes or stand-alone functions.

StreamFramework

ServiceServiceConfiguratorConfigurator

APPLICATIONSAPPLICATIONSAPPLICATIONAPPLICATION--SPECIFICSPECIFIC

ConcurrencyConcurrencyglobalglobal

InterprocessInterprocessCommunicationCommunication

APPLICATION-INDEPENDENT

ServiceServiceInitializationInitialization

ReactorReactor

NetworkNetworkServicesServices

APPLICATIONSAPPLICATIONSAPPLICATIONSAPPLICATIONS

Figure 1: Class Categories in theASXFramework

The ASX framework incorporates concepts from severalother modular communication frameworks including SystemV STREAMS [13], thex-kernel [14], and the Conduit [9](a survey of these and other communication frameworks ap-pears in [15]). These frameworks all contain features thatsupport the flexible configuration of communication subsys-tems by inter-connecting building-block protocol and servicecomponents. In general, these frameworks encourage the de-velopment of standard reusable communication-related com-ponents by decoupling application-specific processing func-tionality from the surrounding framework infrastructure. Asdescribed below, theASXframework also contains additionalfeatures that help to further decouple application-specificservice functionality from (1) the type of locking mecha-nisms used to synchronize access to shared objects, (2) theuse of message-based vs. task-based parallel processingtechniques, and (3) the use of kernel-level vs. user-level ex-ecution agents.

2.2 The Object-Oriented Architecture of ASX

The architecture of theASX framework was developed in-crementally by generalizing from extensive design and im-plementation experience with a range of distributed applica-tions including on-line transaction processing systems [16],telecommunication switch performance monitoring systems[4], and multi-processor-based communication subsystems[17]. After building several prototypes and iterating througha number of alternative designs, the class categories illus-trated in Figure 1 were identified and implemented. A classcategory is a collection of components that collaborate toprovide a set of related services [2] such as communicationsubsystem services used to implement protocol stacks. A

2

complete distributed application may be formed by combin-ing components in each of the following class categories viaC++ language features such as inheritance, aggregation, andtemplate instantiation:

� The Stream class category – These components areresponsible for coordinating theconfigurationand run-time executionof a Stream, which is an object contain-ing a set of hierarchically-related services (such as thelayers in a communication protocol stack) defined by anapplication

� TheReactor class category – These components areresponsible fordemultiplexingtemporal events gen-erated by a timer-driven callout queue, I/O eventsreceived on communication ports, and signal-basedevents anddispatchingthe appropriate pre-registeredhandler(s) to process these events

� The Service Configurator class category –These components are responsible fordynamically link-ing or dynamically unlinkingservices into or out of theaddress space of an application at run-time

� The Concurrency class category – These compo-nents are responsible forspawning, executing, synchro-nizing, andgracefully terminatingservices at run-timevia one or more threads of control within one or moreprocesses

� The IPC SAP class category – These components en-capsulate standard OS local and remote IPC mecha-nisms (such as sockets and TLI) within a type-safe andportable object-oriented interface

Lines connecting the class categories in Figure 1 indicatedependency relationships. For example, components thatimplement the application-specific services in a particulardistributed application depend on theStream components,which in turn depend on theService Configuratorcomponents. Since components in theConcurrencyclass category are used throughout the application-specificand application-independent portions of theASXframeworkthey are marked with theglobal adornment. Note that the“namespaces” feature accepted recently by the ANSI C++committee provides explicit C++ language support for thesetypes of class category relationships.

This section examines the main components in each classcategory. Relationships between components in theASXframework are illustrated throughout the paper via Boochnotation [2]. Solid rectangles indicate class categories,which combine a number of related classes into a commonname space. Solid clouds indicate objects; nesting indicatescomposition relationships between objects; and undirectededges indicate some type of link exists between two objects.Dashed clouds indicate classes; directed edges indicate in-heritance relationships between classes; and an undirectededge with a small circle at one end indicates either a compo-sition or uses relation between two classes.

NETWORK INTERFACE

OR PSEUDO-DEVICES

STREAMSTREAMTailTail

MultiplexorMultiplexor

APPLICATIONAPPLICATION

StreamStream

STREAMSTREAMHeadHead

APPLICATIONAPPLICATION

StreamStream

UP

ST

RE

AMD

OW

NS

TR

EA

M

MESSAGEMESSAGE

OBJECTOBJECT

WRITEWRITE

TASKTASK

OBJECTOBJECT

READREAD

TASKTASK

OBJECTOBJECT

MODULEMODULE

OBJECTOBJECT

open()=0close()=0put()=0svc()=0

Figure 2: Components in theStream Class Category

2.3 The Stream Class Category

Components in theStream class category are responsiblefor coordinating one or more Streams. A Stream is an ob-ject used to configure and execute application-specific ser-vices into theASX framework. As illustrated in Figure 2, aStream uses both inheritance and object composition to linktogether a series of service-specificModules . Modulesare objects that developers use to decompose the architectureof a distributed application into a series of inter-connected,functionally distinct layers. Each layer implements a clusterof related service-specific functions (such as an end-to-endtransport service, a presentation layer formatting service, ora real-time PBX signal routing service).

A layer that performs multiplexing and demultiplexingof message objects between one or more related Streamsmay be developed using aMultiplexor object. AMultiplexor is a C++ template container class that pro-vides mechanisms to route messages between one or moreModules in a collection of related Streams.Module andMultiplexor objects may be configured into a Stream bydevelopers at installation-time or by applications at run-time.

EveryModule contains a pair ofTask objects that par-tition a layer into its constituent read-side and write-sideservice-specific processing functionality. ATask providesan abstract domain class that may be specialized to target aparticular application-specific domain (such as the domainof communication protocol stacks [18] or the domain of net-work management applications [19]). Likewise, aModuleprovides a flexible composition mechanism that allows in-

3

stances of the application-specificTask domain classes tobe configured dynamically into a Stream. These mechanismsincrease the extensibility and applicability of theASXframe-work by not limiting it to a particular domain.

A complete Stream is represented as an inter-connectedseries of independentModule and/orMultiplexor ob-jects that communicate by exchanging messages with ad-jacent objects. Modules and Multiplexors may bejoined together in essentially arbitrary configurations in or-der to satisfy application requirements and enhance com-ponent reuse. Service-specific functions in adjacent inter-connectedModules collaborate by exchanging typed mes-sages via a well-defined message passing interface.

TheASXframework employs a number of object-orienteddesign techniques (such as design patterns [20] and hierar-chical decomposition) and C++ language features (such asinheritance, dynamic binding, and parameterized types [21]).These techniques and language features enable developers toincorporate service-specific functionality into a Stream with-out modifying the application-independent framework com-ponents. For example, incorporating a new layer of servicefunctionality into a Stream involves the following steps:

1. Inheriting from theTask interface and selectively over-riding several methods (described below) in theTasksubclass to implement service-specific functionality

2. Allocating a newModule that contains two instances(one for the read-side and one for the write-side) of theservice-specificTask subclass

3. Inserting theModule into a Stream object

To avoid reinventing familiar terminology, many C++class names in theStream class category correspond tosimilar componentry available in the System V STREAMSframework. However, the techniques used to support ex-tensibility and concurrency in the two frameworks are sig-nificantly different. For example, adding service-specificfunctionality to theASX Stream classes is performed byinheriting from several interfaces and implementations de-fined by existingASXframework components. Using inher-itance to add service-specific functionality provides greatertype-safety than the pointer-to-function idiom used in Sys-tem V STREAMS. As described in Section 2.6.1 below, theASX Stream classes also completely redesign and reim-plement the co-routine-based, “weightless”1 service process-ing mechanisms used in System V STREAMS. TheseASXchanges enable more effective use of multiple PEs on sharedmemory multi-processing platforms by reducing the like-lyhood of deadlock and simplifying flow control betweenTasks in a Stream.

The remainder of this section discusses the primarycomponents of theASX Stream class category (i.e.,Stream class, theModule class, theTask class, and theMultiplexor class) in detail.

1A weightless process executes on a run-time stack that is also used byother processes. This greatly complicates programming and increases thepotential for deadlock. For example, a weightless process may not suspendexecution to wait for resources to become available or events to occur [22].

2.3.1 The Stream Class

The Stream class defines the application interface toa Stream. AStream object provides a bi-directionalget /put -style interface that allows applications to ac-cess a stack of one or more hierarchically-related serviceModules . Applications send and receive data and controlmessages through the inter-connectedModules that com-prise a particular Stream object. TheStream class alsoimplements apush /pop -style interface that allows appli-cations to configure a Stream at run-time by inserting andremoving objects of theModule class described below.

2.3.2 The Module Class

TheModule class defines a distinct layer of service-specificfunctionality. A Stream is formed by inter-connecting a se-ries of Module objects. Module objects in a Stream areloosely coupled, and collaborate with adjacentModule ob-jects by passing typed messages. EachModule object con-tains a pair of pointers to objects that are service-specificsubclasses of theTask class described shortly below.

As shown in Figure 2, two defaultModule objects(Stream Head and Stream Tail ) are installed auto-matically when a Stream is opened. These twoModule sinterpret pre-definedASX framework control messages anddata messages that circulate through a Stream at run-time.TheStream Head class provides a message buffering in-terface between an application and a Stream. TheStreamTail class typically transforms incoming messages froma network or from a pseudo-device into a canonical inter-nal message format that may be processed by higher-levelcomponents in a Stream. Likewise, for outgoing messages ittransforms messages from their internal format into networkmessages.

2.3.3 The Task Abstract Class

TheTask abstract class2 defines an interface that is inher-ited and implemented by derived classes to provide service-specific functionality for read-side and write-side process-ing. OneTask subclass handles read-side processing formessages sent upstream to itsModule layer and the otherhandles write-side processing messages send downstream toits Module layer.

TheTask class is an abstract class since its interface de-fines four pure virtual methods (open , close , put , andsvc ) that are described below. DefiningTask as an ab-stract class enhances reuse by decoupling the application-independent components provided by theStream class cat-egory from the service-specific subclasses that inherit fromand use these components. Likewise, the use of pure virtualmethods allows the C++ compiler to ensure that a subclass of

2An abstract class in C++ provides an interface that contains at least onepure virtual method[23]. A pure virtual method provides only an interfacedeclaration, usually without any accompanying definition. Subclasses of anabstract class must provide definitions for all its pure virtual methods beforeany objects of the class may be instantiated.

4

active

2: put()

1: put()

activeactive

(1)

ModuleA

ModuleB

ModuleC PROCESS

OR THREAD

WRITE

TASK

OBJECT

READ

TASK

OBJECT

MODULE

OBJECT

ModuleA

ModuleB

ModuleC

2: svc()

1: put()

4: svc()

3: put()

active

active

active

(2)

Figure 3: Alternative Methods for Invokingput andsvc Methods

Task honors its obligation to provide the following service-specific functionality:

� Initialization and Termination Methods: Subclassesderived from Task must implementopen and closemethods that perform service-specificTask initializationand termination activities. These activities typically allocateand free resources such as connection control blocks, I/O de-scriptors, and synchronization locks. Theopen andclosemethods of aModule ’s write-side and read-sideTask sub-classes are invoked automatically by theASX frameworkwhen theModule is inserted or removed from a Stream,respectively.

� Service-Specific Processing Methods:Subclasses ofTask must also define theput and svc methods, whichperform service-specific processing functionality on mes-sages that arrive at aModule layer in a Stream. When mes-sages arrive at the head or the tail of a Stream, they are es-corted through a series of inter-connectedTasks as a resultof invoking theput and/orsvc method of eachTask in theStream.

A put method is invoked when aTask at one layer ina Stream passes a message to an adjacentTask in anotherlayer. Theput method runssynchronouslywith respect toits caller,i.e., it borrows the thread of control from theTaskthat originally invoked itsput method. This thread of con-trol typically originates either “upstream” from an applica-tion process, “downstream” from a pool of threads that han-dle I/O device interrupts [14], or internal to the Stream froman event dispatching mechanism (such as a timer-driven call-out queue used to trigger retransmissions in a connection-oriented transport protocolModule ).

Thesvc method is used to perform service-specific pro-cessingasynchronouslywith respect to otherTask s in itsStream. Unlikeput , thesvc method is not directly invokedfrom an adjacentTask . Instead, it is invoked by a sepa-rate thread associated with itsTask . This thread providesan execution context and thread of control for theTask ’ssvc method. This method runs an event loop that continu-ously waits for messages to arrive on theTask ’s Message

Queue. A Message Queue is a standard component ina Task that is used to buffer a sequence of data messagesand control messages for subsequent processing in thesvcmethod. When messages arrive, thesvc method dequeuesthe messages and performs theTask subclass’s service-specific processing tasks.

Within the implementation of aput or svc method,a message may be forwarded to an adjacentTask in theStream via theput next Task utility method. Theput next method calls theput method of the nextTaskresiding in an adjacent layer. This invocation ofput mayborrow the thread of control from the caller and handle themessage immediately (i.e., the synchronous processing ap-proach illustrated in Figure 3 (1)). Conversely, theputmethod may enqueue the message and defer handling to itssvc method that is executing in a separate thread of con-trol (i.e., the asynchronous processing approach illustratedin Figure 3 (2)). As discussed in Section 3, the particularprocessing approach that is selected often has a significantimpact on performance and ease of programming.

In addition to theopen , close , put , and svc purevirtual method interfaces, eachTask also contains a num-ber of reusable utility methods (such asput next , getq ,andputq ) that may be used by service-specific subclassesto query and/or modify the internal state of aTask object.This internal state includes a pointer to the adjacentTaskon a Stream, a back-pointer to aTask ’s enclosingModule(which enables it to locate its sibling), aMessage Queue ,and a pair of high and low water mark variables that are usedto implement layer-to-layer flow control between adjacentModules in a Stream. The high water mark indicates theamount of bytes of messages theMessage Queue is will-ing to buffer before it becomes flow controlled. The lowwater mark indicates the level at which a previously flowcontrolledTask is no longer considered to be full.

Two types of messages may appear on aMessageQueue: simple and composite. A simple message containsa singleMessage Block and a composite message con-tains multipleMessage Block s linked together. Compos-ite messages generally consist of acontrolblock followed byone or moredatablocks. A control block contains bookkeep-

5

: Reactor

: EventHandler

REGISTEREDOBJECTS

FR

AM

EW

OR

KL

EV

EL

KE

RN

EL

LE

VE

L

AP

PL

ICA

TIO

NL

EV

EL

SELECT() OR POLL () INTERFACE

: EventHandler

:TimerQueue

open() register_handler() remove_handler() schedule_timer() cancel_timer() handle_events()

Reactor

EventHandler

Ahandle_input()handle_output()handle_exception()handle_signal()handle_timeout()handle_close()get_handle()

DerivedClass

handle_input()handle_output()get_handle()

EVENT HANDLERHIERARCHY

: EventHandler : Event

Handler

: SignalHandlers

: DescriptorTable

Figure 4: Components in theReactor Class Category

ing information (such as destination addresses and lengthfields), whereas data blocks contain the actual contents of amessage. The overhead of passingMessage Block s be-tweenTask s is minimized by passing pointers to messagesrather than copying data.

2.3.4 The Multiplexor Class

A Multiplexor defines mechanisms that demultiplexmessages destined for differentModules that comprisea set of inter-related Streams.Multiplexor s are usedto routeMessage Block s between inter-related streams(such as those used to implement complex protocol fami-lies in the Internet and the ISO OSI reference models). AMultiplexor is implemented via a C++ template classcalled Map Manager . Map Manager is parameterizedby an external identifier(such as a network address, portnumber, or type-of-service field) and aninternal identifier(such as a pointer to aModule ). These template parame-ters are instantiated in protocol-specificStream classes toproduce specializedMap Manager objects that perform ef-ficient intra-Stream message routing. EachMap Managerobject contains a set ofModule objects that may be linkedabove and below aMultiplexor in essentially arbitraryconfigurations.

2.4 The Reactor Class Category

Components in theReactor class category are responsiblefor demultiplexing (1) temporal events generated by a timer-driven callout queue, (2) I/O events received on communica-tion ports, and (3) signal events and dispatching the appro-priate pre-registered handler(s) to process these events. The

Reactor encapsulates the functionality of theselect andpoll I/O demultiplexing mechanisms within a portable andextensible C++ wrapper [12].Select andpoll are UNIXsystem calls that detect the occurrence of different types ofinput and output events on one or more I/O descriptors si-multaneously. To improve portability, theReactor pro-vides the same interface regardless of whetherselect orpoll is used as the underlying I/O demultiplexor. In addi-tion, theReactor contains mutual exclusion mechanismsdesigned to perform callback-style programming correctlyand efficiently in a multi-threaded event processing environ-ment.

TheReactor contains a set of methods illustrated in Fig-ure 4. These methods provide a uniform interface to manageobjects that implement various types of service-specific han-dlers. Certain methods register, dispatch, and remove I/Odescriptor-based and signal-based handler objects from theReactor . Other methods schedule, cancel, and dispatchtimer-based handler objects. As shown in Figure 4, thesehandler objects all derive from theEvent Handler ab-stract base class. This class specifies an interface for eventregistration and service handler dispatching.

The Reactor uses the virtual methods defined inthe Event Handler interface to integrate the demulti-plexing of I/O descriptor-based, timer-based, and signal-based events. I/O descriptor-based events are dis-patched via thehandle input , handle output , andhandle exceptions methods; timer-based events aredispatched via thehandle timeout method; and Signal-based events are dispatched via thehandle signalmethod. Subclasses ofEvent Handler may augmentthe base class interface by defining additional methods anddata members. In addition, virtual methods in theEventHandler interface may be selectively overridden to imple-ment application-specific functionality. Once the pure virtualmethods in theEvent Handler base class have been sup-plied by a subclass, an application may define an instance ofthe resulting composite service handler object.

When an application instantiates and registers a compositeI/O descriptor-based service handler object, theReactorextracts the underlying I/O descriptor from the object. Thisdescriptor is stored in a table along with I/O descriptors fromother registered objects. Subsequently, when the applicationinvokes its main event loop, these descriptors are passed asarguments to the underlying OS event demultiplexing sys-tem call (e.g.,select or poll ). As events associated witha registered handler object occur at run-time, theReactorautomatically detects these events and dispatches the appro-priate method(s) of the service handler object associated withthe event. This handler object then becomes responsible forperforming its service-specific functionality before returningcontrol to the mainReactor event-loop.

2.5 The Service Configurator Class Category

Components in theService Configurator class cat-egory are responsible for explicitly linking or unlinking ser-

6

1

1

1

1

ServiceConfig

REACTOR

n

ServiceRepository

1

(2) The Service_RepositoryClass

(3) The Service_ConfigClass

ServiceRepository

ServiceRepository

Iterator

F

1

n

1

1

1

ServiceObject

A

(1) Service_ObjectInheritance Hierarchy

EventHandler

A

SharedObject

A

ServiceObject

Task

A

n

ServiceObject

EventHandler

APPLICATION-SPECIFIC

FUNCTIONALITY

Task

A

Task

AAPPLICATION-

INDEPENDENT

FUNCTIONALITY

Figure 5: Components in theService Configurator Class Category

vices dynamically into or out of the address space of an ap-plication at run-time. Explicit dynamic linking enables theconfiguration and reconfiguration of application-specific ser-vices without requiring the modification, recompilation, re-linking, or restarting of an executing application [4]. TheService Configurator components discussed belowinclude the theService Object inheritance hierarchy(Figure 5 (1)), theService Repository class (Fig-ure 5 (2)), and theService Config class (Figure 5 (3)).

2.5.1 The Service Object Inheritance Hierarchy

TheService Object class is the focal point of a multi-level hierarchy of types related by inheritance. The interfacesprovided by the abstract classes in this type hierarchy may beselectively implemented by service-specific subclasses in or-der to accessService Configurator features. Thesefeatures provide transparent dynamic linking, service han-dler registration, event demultiplexing, service dispatching,and run-time control of services (such as suspending and re-suming a service temporarily). By decoupling the service-specific portions of a handler object from the underlyingService Configurator mechanisms, the effort neces-sary to insert and remove services from an application at run-time is significantly reduced.

The Service Object inheritance hierarchy consistsof the Event Handler and Shared Object abstractbase classes, as well as theService Object abstractderived class. TheEvent Handler class was describedabove in theReactor Section 2.4. The behavior of theother classes in theService Configurator class cate-gory is outlined below:

� The Shared Object Abstract Base Class: This abstractbase class specifies an interface for dynamically linking and

unlinking objects into and out of the address space of an ap-plication. This abstract base class exports three pure virtualmethods:init , fini , andinfo . These functions imposea contract between the reusable components provided by theService Configurator and service-specific objectsthat utilize these components. By using pure virtual meth-ods, theService Configurator ensures that a servicehandler implementation honors its obligation to provide cer-tain configuration-related information. This information issubsequently used by theService Configurator toautomatically link, initialize, identify, and unlink a serviceat run-time.

The init method serves as the entry-point to an objectduring run-time initialization. This method is responsible forperforming application-specific initialization when an objectderived fromShared Object is dynamically linked. Theinfo method returns a humanly-readable string that con-cisely reports service addressing information and documentsservice functionality. Clients may query an application to re-trieve this information and use it to contact a particular ser-vice running in the application. Thefini method is calledautomatically by theService Configurator class cat-egory when an object is unlinked and removed from an appli-cation at run-time. This method typically performs termina-tion operations that release dynamically allocated resources(such as memory or synchronization locks).

The Shared Object base class is defined indepen-dently from theEvent Handler class to clearly separatetheir two orthogonal sets of concerns. For example, certainapplications (such as a compiler or text editor) might benefitfrom dynamic linking, though they might not require timer-based, signal-based, or I/O descriptor-based event demul-tiplexing. Conversely, other applications (such as anftpserver) require event demultiplexing, but might not require

7

dynamic linking.

� The Service Object Abstract Derived Class: Supportfor dynamic linking, event demultiplexing, and service dis-patching is typically necessary to automate the dynamicconfiguration and reconfiguration of application-specific ser-vices in a distributed system. Therefore, theServiceConfigurator class category defines theServiceObject class, which is a composite class that combinesthe interfaces inherited from both theEvent Handlerand theShared Object abstract base classes. Duringdevelopment, application-specific subclasses ofServiceObject may implement thesuspend and resume vir-tual methods in this class. Thesuspend and resumemethods are invoked automatically by theServiceConfigurator class category in response to certain ex-ternal events (such as those triggered by receipt of theUNIX SIGHUP signal). An application developer may de-fine these methods to perform actions necessary to sus-pend a service object without unlinking it completely, aswell as to resume a previously suspended service object.In addition, application-specific subclasses must implementthe four pure virtual methods (init , fini , info , andget handle ) that are inherited (but not defined) by theService Object subclass.

To provide a consistent environment for defining, config-uring, and using Streams, theTask class in theStreamclass category is derived from theService Object in-heritance hierarchy (illustrated in Figure 5 (1)). This en-ables hierarchically-related, application-specific services tobe linked and unlinked into and out of a Stream at run-time.

2.5.2 The Service Repository Class

The ASX framework supports the configuration of appli-cations that contain one or more Streams, each of whichmay have one or more inter-connected service-specificModules . Therefore, to simplify run-time administration,it may be necessary to individually and/or collectively con-trol and coordinate theService Object s that comprisean application’s currently active services. TheServiceRepository is an object manager that coordinates localand remote queries and updates involving the services of-fered by an application. A search structure within the objectmanager binds service names (represented as ASCII strings)with instances of compositeService Object s (repre-sented as C++ object code). A service name uniquely identi-fies an instance of aService Object stored in the repos-itory.

Each entry in theService Repository contains apointer to theService Object portion of an service-specific C++ derived class (shown in Figure 5 (2)). This en-ables theService Configurator classes to automat-ically load, enable, suspend, resume, or unloadServiceObject s from a Stream dynamically. The repository alsomaintains a handle to the underlying shared object file foreach dynamically linkedService Object . This han-

dle is used to unlink and unload aService Objectfrom a running application when its service is no longerrequired. An iterator class is also supplied along with theService Repository . This class may be used to visitevery Service Object in the repository without com-promising data encapsulation.

2.5.3 The Service Config Class

As illustrated in Figure 5 (3), theService Config classintegrates several otherASXframework components (such astheService Repository , theService Object in-heritance hierarchy, and theReactor ). The resulting com-positeService Config component is used to automatethe static and/or dynamic configuration of concurrent appli-cations that contain one or more Streams. TheServiceConfig class uses a configuration file to guide its config-uration and reconfiguration activities. Each application maybe associated with a distinct configuration file. This file char-acterizes the essential attributes of the service(s) offered byan application. These attributes include the location of theshared object file for each dynamically linked service, aswell as the parameters required to initialize a service at run-time. By consolidating service attributes and installation pa-rameters into a single configuration file, the administration ofStreams within an application is simplified. Application de-velopment is also simplified by decoupling the configurationand reconfiguration mechanisms provided by the frameworkfrom the application-specific attributes and parameters spec-ified in a configuration file. Further information on the con-figuration format utilized by theService Config classis presented in [4].

2.6 The Concurrency Class Category

Components in theConcurrency class category areresponsible for spawning, executing, synchronizing, andgracefully terminating services at run-time via one or morethreads of control within one or more processes. The follow-ing section discusses the two main groups of classes (SynchandThread Manager ) in theConcurrency class cate-gory.

2.6.1 The Synch Classes

Components in theStream , Reactor , and ServiceConfigurator class categories described above con-tain a minimal amount of internal locking mechanisms toavoid over-constraining the granularity of the synchroniza-tion strategies used by an application [22]. In particular,only components in theASXframework that would not func-tion correctly in a multi-threaded environment (such as en-queueingMessage Block s onto aMessage Queue ,demultiplexingMessage Blocks onto internalModuleaddresses stored in aMultiplexor object, or register-ing an Event Handler object with theReactor ) areprotected by synchronization mechanisms provided by the

8

Synch classes. TheSynch classes provide type-safe C++interfaces for two basic types of synchronization mecha-nisms: Mutex and Condition objects [24]. AMutexobject is used to ensure the integrity of a shared resourcethat may be accessed concurrently by multiple threads ofcontrol. A Condition object allows one or more cooper-ating threads to suspend their execution until a condition ex-pression involving shared data attains a particular state. TheASXframework also provides a collection of more sophisti-cated concurrency control mechanisms (such asMonitors ,Readers Writer locks, and recursiveMutex objects)that build upon the two basic synchronization mechanismsdescribed below.

A Mutex object may be used to serialize the execution ofmultiple threads by defining a critical section where only onethread executes its code at a time. To enter a critical section,a thread invokes theMutex::acquire method. To leavea critical section, a thread invokes theMutex::releasemethod. These two methods are implemented via adaptivespin-locks that ensure mutual exclusion by using an atomichardware instruction. An adaptive spin-lock operates bypolling a designated memory location using the hardware in-struction until (1) the value at this location is changed bythe thread that currently owns the lock (signifying that thelock has been released and may now be acquired) or (2) thethread that is holding the lock goes to sleep (at which pointthe thread that is spinning also goes to sleep to avoid needlesspolling) [25]. On a shared memory multi-processor, the over-head incurred by a spin-lock is relatively minor since pollingaffects only the local instruction and data cache of the CPUwhere the thread is spinning. A spin-lock is a simple and ef-ficient synchronization mechanism for certain types of short-lived resource contention. For example, in theASX frame-work, eachMessage Queue in a Task object contains aMutex object that prevents race conditions from occurringwhenMessage Block s are enqueued and dequeued con-currently by multiple threads of control running in adjacentTasks .

A Condition object is a somewhat different synchro-nization mechanism that enables a thread to suspend itselfindefinitely (via theCondition::wait method) until acondition expression involving shared data attains a par-ticular state. When another cooperating thread indicatesthat the state of the shared data has changed (by invok-ing the Condition::signal method), the associatedCondition object wakes up the suspended thread. Thenewly awakened thread then re-evaluates the condition ex-pression and potentially resumes processing if the shareddata is now in an appropriate state. For example, eachMessage Queue in theASXframework contains a pair ofCondition objects (namednotfull andnotempty ),in addition to aMutex object. TheseCondition objectsimplement flow control between adjacentTask s. Whenone Task attempts to insert aMessage Block into aneighboringTask that has reached its high water mark, theMessage Queue::enTask method performs awaitoperation on thenotfull condition object. This operation

IPC_SAP

A

SOCK_SAP TLI_SAP FIFO_SAPSPIPE_SAP

SOCKET

API

TRANSPORT

LAYER

INTERFACE API

STREAM PIPE

API

NAMED PIPE

API

Figure 6: Components in theIPC SAP Class Category

atomically relinquishes the PE and puts the calling threadto sleep awaiting notification when flow control conditionsabate. Subsequently, when the number of bytes in the flowcontrolledTask ’s Message Queue fall below its low wa-ter mark, the thread running the blockedTask is automat-ically awakened to finish inserting the message and resumeits processing tasks.

Unlike Mutex objects,Condition object synchroniza-tion is not implemented with a spin-lock since there is gen-erally no indication of how long a thread must wait for aparticular condition to be signaled. Therefore,Conditionobjects are implemented via sleep-locks that trigger a contextswitch to allow other threads to execute. Section 3 discussesthe consequences of spin-locks vs. sleep-locks on applica-tion performance.

2.6.2 The Thread Manager Class

TheThread Manager class contains a set of mechanismsthat manage groups of threads that collaborate to imple-ment collective actions (such as a pool of threads that ren-der different portions of a large image in parallel). TheTheManager class shields applications from many incompati-bilities between different flavors of multi-threading mecha-nisms (such as POSIX threads, MACH cthreads, and Solaristhreads).

In addition, the Thread Manager class providesa number of mechanisms (such assuspend all andresume all ) that suspend and resume a set of collabo-rating threads atomically. This feature is useful for dis-tributed applications that execute one or more services con-currently. For example, when initializing a Stream com-posed ofModule s that execute in separate threads of con-trol and collaborate by passing messages between threads,it is important to ensure that allTasks in the Streamare completely inter-connected before allowing messages toflow through the Stream. The mechanisms in theThreadManager class allow these initialization activities to occuratomically.

9

2.7 The IPC SAP Class Category

Components in theIPC SAP class category encapsulatestandard OS local and remote IPC mechanisms (such assockets and TLI) within a type-safe and portable object-oriented interface.IPC SAP stands for “InterProcess Com-munication Service Access Point.” As shown in Figure 6,a forest of class categories are rooted at theIPC SAP baseclass. These class categories includesSOCK SAP(whichencapsulates the socket API),TLI SAP (which encapsu-lates the TLI API),SPIPE SAP (which encapsulates theUNIX SVR4 STREAM pipe API), andFIFO SAP (whichencapsulates the UNIX named pipe API).

Each class category inIPC SAP is itself organized as aninheritance hierarchy where every subclass provides a well-defined subset of local or remote communication mecha-nisms. Together, the subclasses within a hierarchy comprisethe overall functionality of a particular communication ab-straction (such as the Internet-domain or UNIX-domain pro-tocol families). Inheritance-based hierarchical decomposi-tion facilitates the reuse of code that is common among thevariousIPC SAP class categories. For example, the C++interface to the lower-level UNIX OS device control systemcalls likefcntl andioctl are inherited and shared by allthe other components in theIPC SAP class category.

3 Performance Experiments on theCommunication Subsystem

To illustrate how the components of theASX frameworkare used in practice, this section describes results from per-formance experiments that measure the impact of alterna-tive methods for parallelizing communication subsystems.A communication subsystem is a distributed system thatconsists ofprotocol functions(such as routing, segmenta-tion/reassembly, connection management, end-to-end flowcontrol, remote context management, demultiplexing, mes-sage buffering, error protection, session control, and pre-sentation conversions) andoperating system mechanisms(such as process management, asynchronous event invoca-tion, message buffering, and layer-to-layer flow control) thatsupport the implementation and execution of protocol stacksthat contain hierarchically-related protocol functions [15].

Advances in VLSI and fiber optic technology are shiftingperformance bottlenecks from the underlying networks to thecommunication subsystem [26]. Designing and implement-ing multi-processor-based communication subsystems thatexecute protocol functions and OS mechanisms in parallelis a promising technique for increasing protocol processingrates and reducing latency. To significantly increase com-munication subsystem performance, however, the speed-upobtained from parallel processing must outweight the con-text switching and synchronization overhead associated withparallel processing.

A context switch is generally triggered when an execut-ing process either voluntarily or involuntarily relinquishes

the processing element (PE) it is executing upon. Dependingon the underlying OS and hardware platform, performing acontext switch may involve dozens to hundreds of instruc-tions due to the flushing of register windows, instruction anddata caches, instruction pipelines, and translation look-asidebuffers [27]. Synchronization mechanisms are necessary toserialize access to shared objects (such as messages, mes-sage queues, protocol context records, and demultiplexingtables) related to protocol processing. Certain methods ofparallelizing protocol stacks incur significant synchroniza-tion overhead from managing locks associated with process-ing these shared objects [28].

A number ofprocess architectureshave been proposedas the basis for parallelizing communication subsystems[26, 29, 28]. A process architecture binds one or more pro-cessing elements (PEs) together with the protocol tasks andmessages that implement protocol stacks in a communica-tion subsystem. Figure 7 (1) illustrates the three basic ele-ments that form the foundation of a process architecture:

1. Control messages and data messages– which are sentand received from one or more applications and net-work devices

2. Protocol processing tasks– which are the units of proto-col functionality that process the control messages anddata messages

3. Processing elements(PEs) – which execute protocoltasks. There are two fundamental types of process ar-chitectures (task-basedandmessage-based) that struc-ture these three basic elements differently.

Two fundamental types of process architectures (task-basedand message-based) may be created by structuringthe three basic process architecture elements shown in Fig-ure 7 (1) in different ways. Task-based process architecturesare formed by binding one or more PEs to different units ofprotocol functionality (shown in Figure 7 (2)). In this ar-chitecture, tasks are the active objects, whereas messagesprocessed by the tasks are the passive objects. Parallelismis achieved by executing protocol tasks in separate PEs andpassing data messages and control messages between thetasks/PEs. In contrast, message-based process architecturesare formed by binding the PEs to the protocol control mes-sages and data messages received from applications and net-work interfaces (as shown in Figure 7 (3)). In this architec-ture, messages are the active objects, whereas tasks are thepassive objects. Parallelism is achieved by escorting multipledata messages and control messages on separate PEs simul-taneously through a stack of protocol tasks. Section 3 ex-amines how the choice of process architecture significantlyaffects context switch and synchronization overhead. A sur-vey of alternative process architectures appears in [15].

Selecting an effective process architecture is an importantdesign decision in application domains other than communi-cation subsystems. For example, real-time PBX monitoringsystems [3] and video-on-demand servers also perform non-communication-related tasks (such as database query pro-cessing) that benefit from a carefully structured approach

10

PEPEPE PEPEPEPEPEPE PEPEPE

SHARED MEMORYSHARED MEMORY

(1)(1) COMPONENTS IN A COMPONENTS IN A

THREADING ARCHITECTURETHREADING ARCHITECTURE

PEPEPE

PEPEPE

PEPEPE

PEPEPE

(2)(2) TASK TASK--BASEDBASED

THREADING ARCHITECTURETHREADING ARCHITECTURE

activeactive

(3)(3) MESSAGE MESSAGE--BASEDBASED

THREADING ARCHITECTURETHREADING ARCHITECTURE

PEPEPE PEPEPEPEPEPE PEPEPE

activeactive

activeactive

activeactive

activeactive

activeactive

activeactive

activeactive

MESSAGEMESSAGE

PE

PROCESSINGPROCESSING

ELEMENTELEMENT

PROTOCOLPROTOCOL

TASKTASK

Figure 7: Process Architecture Components and Interrelationships

to parallelism. This section focuses primarily upon the im-pact of process architectures on communication subsystemperformance since network protocol behavior and function-ality is well-understood and the terminology is relativelywell-defined. Moreover, a large body of literature existswith which to compare performance results presented in Sec-tion 3. The remainder of this section describes relevantaspects of performance experiments that measure the im-pact of different process architectures on connectionless andconnection-oriented protocol stacks.

3.1 Multi-processor Platform

All experiments were conducted on an otherwise idle Sun690MP SPARCserver, which contains 4 SPARC 40 MHzprocessing elements (PEs), each capable of performing at 28MIPs. The operating system used for the experiments is re-lease 5.3 of SunOS, which provides a multi-threaded kernelthat allows multiple system calls and device interrupts to exe-cute in parallel [25]. All the process architectures in these ex-periments execute protocol tasks in separateunboundthreadsmultiplexed over 1, 2, 3, or 4 SunOSlightweight processes(LWPs) within a process. SunOS 5.3 maps each LWP di-rectly onto a separate kernel thread. Since kernel threads arethe units of PE scheduling and execution in SunOS, this map-ping enables multiple LWPs (each executing protocol pro-cessing tasks in an unbound thread) to run in parallel on theSPARCserver’s PEs.

Rescheduling and synchronizing a SunOS LWP involvesa kernel-level context switch. The time required to performa context switch between two LWPs was measured to be ap-proximately 30usecs. During this time, the OS performssystem-related overhead (such as flushing register windows,instruction and data caches, instruction pipelines, and trans-lation lookaside buffers) on the PE and therefore does notprocess protocol tasks. Measurements also revealed that itrequires approximately 3 micro-seconds to acquire or re-lease aMutex object implemented with a SunOS adap-tive spin-lock. Likewise, measurements indicated that ap-

proximately 90 micro-seconds are required to synchronizetwo LWPs usingCondition objects implemented usingSunOS sleep-locks. The larger amount of overhead for theCondition operations compared with theMutex opera-tions occurs from the more complex locking algorithms in-volved, as well as the additional context switching incurredby the SunOS sleep-locks that implement theConditionobjects.

3.2 Communication Protocols

Two types of protocol stacks are used in the experiments,one based on the connectionless UDP transport protocol andthe other based on the connection-oriented TCP transportprotocol. The protocol stacks contain the data-link, trans-port, and presentation layers.3 The presentation layer is in-cluded in the experiments since it represents a major bottle-neck in high-performance communication systems due pri-marily to the large amount of data movement overhead it in-curs [30, 29].

Both the connectionless and connection-oriented proto-col stacks were developed by specializing existing compo-nents in theASXframework via techniques involving inheri-tance and parameterized types. These techniques are used tohold the protocol stack functionality constant while system-atically varying the process architecture. For example, eachprotocol layer is implemented as aModule whose read-sideand write-side inherit standard interfaces and implementa-tions from theTask class. Likewise, synchronization anddemultiplexing mechanisms required by a protocol layer orprotocol stack are parameterized using template arguments

3Preliminary tests indicated that the PE, bus, and memory performanceof the SunOS multi-processor platform was capable of processing messagesthrough the protocol stack at a much faster rate than the platform’s 10 MbpsEthernet network interface was capable of handling. Therefore, for the pro-cess architecture experiments, the network interface was simulated with asingle-copy pseudo-device driver operating in loop-back mode. For thisreason, the routing and segmentation/reassembly functions of the networklayer processing were omitted from these experiments since both the senderand receiver portions of the test programs reside on the same host machine.

11

that are instantiated based on the type of process architecturebeing tested.

Data-link layer processing in each protocol stack is per-formed by theDLP Module . ThisModule transforms net-work packets received from a network interface or loop-backdevice into a canonical message format used internally bythe Stream components. The transport layer component ofthe protocol stacks are based on the UDP and the TCP im-plementation in the BSD 4.3 Reno release. The 4.3 RenoTCP implementation contains the TCP header prediction en-hancements, as well as the slow start algorithm and conges-tion avoidance features. The UDP and TCP transport proto-cols are configured into theASXframework via theUDPandTCP Modules , respectively.

Presentation layer functionality is implemented in theXDR Module using marshalling routines produced by theONC eXternal Data Representation (XDR) stub genera-tor (rpcgen ). The ONC XDR stub generator automati-cally translates a set of type specifications into marshallingroutines that encode/decode implicitly-typed messages be-fore/after they are exchanged among hosts that may possessheterogeneous processor byte-orders. The ONC presentationlayer conversion mechanisms consist of a type specificationlanguage (XDR) and a set of library routines that implementthe appropriate encoding and decoding rules for built-in inte-gral types (e.g.,char, short, int, and long) and real types (e.g.,float and double). In addition, these library routines may becombined to produce marshalling routines for arbitrary user-defined composite types (such as record/structures, unions,arrays, and pointers). Messages exchanged via XDR areimplicitly-typed, which improves marshalling performanceat the expense of flexibility. TheXDR functions selectedfor both the connectionless and connection-oriented proto-col stacks convert incoming and outgoing messages into andfrom variable-sized arrays of structures containing both in-tegral and real values. This conversion processing involvesbyte-order conversions, as well as dynamic memory alloca-tion and deallocation.

3.3 Process Architectures

3.3.1 Design of the Task-based Process Architecture

Figure 8 illustrates theASXframework components that im-plement a task-based process architecture for the TCP-basedconnection-oriented and UDP-based connectionless protocolstacks. Protocol-specific processing for the data-link andtransport layer are performed in twoModules clustered to-gether into one thread. Likewise, presentation layer and ap-plication interface processing is performed in twoModulesclustered into a separate thread. These threads cooperate ina producer/consumer manner, operating in parallel on theheader and data fields of multiple incoming and outgoingmessages.

The LP DLP::svc and LP XDR::svc methods per-form service-specific processing in parallel within a Streamof Modules . When messages are inserted into aTask ’s

STREAMHeads

STREAMTail

APPLICATION

Modules

C1 C2

NETWORK DEVICES

OR PSEUDO-DEVICES

APPLICATION

Modules

PROCESS

OR THREAD

WRITE

TASK

OBJECT

READ

TASK

OBJECT

MODULE

OBJECT

MESSAGE

OBJECT

Figure 8: A Task-based Process Architecture

Message Queue , thesvc method dequeues the messagesand performs theTask subclass’s service-specific process-ing tasks (such as data-link layer processing or presentationlayer processing). Depending on the “direction” of a mes-sage (i.e., incoming or outgoing), each cluster ofModulesperforms its associated protocol functions before passing themessage to an adjacentModule running asynchronously ina separate thread. Messages are not copied when passed be-tween adjacentTasks since threads all share a common ad-dress space. However, moving messages between threadstypically invalidates per-PE data caches.

The connectionless and connection-oriented task-basedprocess architecture protocol stacks are designed in a sim-ilar manner. The primary difference is that the objects in theconnectionless transport layerModule implement the sim-pler UDP functionality that does not generate acknowledge-ments, keep track of round-trip time estimates, or managecongestion windows. The design of the task-based processarchitecture test driver always uses PEs in multiples of two:one for the cluster of data-link and transport layer processingModules and the other for the cluster of presentation layerand application interface processingModules .

3.3.2 Design of the Message-based Process Architecture

Figure 9 illustrates a message-based process architecture forthe connection-oriented protocol stack. When an incom-ing message arrives, it is handled by theMP DLP::svcmethod, which manages a pool of pre-spawned threads.Each message is associated with a separate thread that es-corts the message synchronously through a series of inter-connectedTask s in a Stream. Each layer of the protocolstack performs its protocol functions and then makes an up-call [31] to the next adjacent layer in the protocol stack by

12

STREAMHead

STREAMTail

APPLICATION

Modules

C1

Modules

NETWORK DEVICES

OR PSEUDO-DEVICES

PROCESS

OR THREAD

WRITE

TASK

READ

TASKMODULE

Figure 9: A Message-based Process Architecture

invoking theTask::put method in that layer. Theputmethod executes the protocol tasks associated with its layer.For instance, theMP TCP::put method utilizesMutexobjects that serialize access to per-connection control blocksas separate messages from the same connection ascend theprotocol stack in parallel.

The connectionless message-based protocol stack is struc-tured in a similar manner. However, the connectionlessprotocol stack performs the simpler set of UDP functional-ity. Unlike theMP TCP::put method, theMP UDP::putmethod handles each message concurrently and indepen-dently, without explicitly preserving inter-message ordering.This reduces the amount of synchronization operations re-quired to locate and update shared resources.

3.4 C++ Features Used to Simplify ProcessArchitecture Implementation

Many of the protocol functions, process architecture syn-chronization mechanisms, andASX framework supportcomponents (such as demultiplexing and message buffer-ing classes) are reused throughout the process architec-ture test programs described above. For example, processarchitecture-specific synchronization strategies may be in-stantiated by selectively instrumenting protocol functionswith different types of mutual exclusion mechanisms. Whencombined with C++ language features such as inheritanceand parameterized types, these objects help to decouple pro-tocol processing functionality from the concurrency controlscheme used by a particular process architecture.

For example, objects of classMultiplexor use aMapManager component to demultiplex incoming messages toModules . Map Manager is a search structure containerclass that is parameterized by an external ID, internal ID,and a mutual exclusion mechanism, as follows:

template <class EX_ID, class IN_ID, class MUTEX>class Map_Manager {public:

bool bind (EX_ID, IN_ID *);bool unbind (EX_ID);bool find (EX_ID ex_id, IN_ID &in_id);

private:MUTEX lock;// ...

The type ofMUTEXthat this template class is instantiatedwith depends upon the particular choice of process architec-ture. For instance, theMap Manager used in the message-based implementation of the TCP protocol stack describedin Section 3.3.2 is instantiated with the following class pa-rameters:typedef Map_Manager <TCP_Addr, TCB, Mutex>

MP_Map_Manager;

This particular instantiation ofMap Manager locates thetransport control block (TCB) associated with theTCP ad-dress of an incoming message. TheMap Manager classuses theMutex class described in Section 2.6.1 to ensurethat itsfind method executes as a critical section. This pre-vents race conditions with other threads that are inspectingor inserting entries into the connection map in parallel.

In contrast, the task-based process architecture implemen-tation of the TCP protocol stack described in Section 3.3.1does not require the same type of concurrency control withina connection. In this case, demultiplexing is performedwithin the svc method in theLP DLP readTask of thedata-link layerModule , which runs in its own separatethread of control. Therefore, theMap Manager used forthe connection-oriented task-based process architecture is in-stantiated with a differentMUTEXclass, as follows:typedef Map_Manager <TCP_Addr, TCB, Null_Mutex>

LP_Map_Manager;

The implementation of theacquire andrelease meth-ods in theNull Mutex class are essentially “no-op” inlinefunctions that may be removed completely by the compileroptimizer.

The ASX framework employs a C++ idiom that in-volves using a class constructor and destructor to acquireand release locks on synchronization objects, respectively[32]. TheMutex Block class illustrated below defines a“block” of code over which aMutex object is acquired andthen automatically released when the block of code is exitedand the object goes out of scope:template <class MUTEX>class Mutex_Block{public:

Mutex_Block (MUTEX &m): mutex (m) {this->mutex.acquire ();

}˜Mutex_Block (void) {

this->mutex.release ();}

private:MUTEX &mutex;

}

This C++ idiom is used in the implementation of theMapManager::find method, as follows:

13

template <class EX_ID, class IN_ID, class MUTEX> intMap_Manager<EX_ID, IN_ID, MUTEX>::find

(EX_ID ex_id, IN_ID &in_id){

Mutex_Block<MUTEX> monitor (this->lock);

if (/* ex_id is successfully located */)return 0;

elsereturn -1;

}

When the find method returns, the destructor for theMutex Block object automatically releases theMutexlock. Note that theMutex lock is released regardless ofwhich arm in theif/else statement returns from thefindmethod. In addition, this C++ idiom also properly releasesthe lock if an exception is raised during processing in thebody of thefind method.

3.5 Process Architecture Experiment Results

This section presents measurement results obtained fromthe data reception portion of the connection-oriented andconnectionless protocol stacks implemented using the task-based and message-based process architectures describedabove. Three types of measurements were obtained for eachcombination of process architecture and protocol stack:totalthroughput, context switching overhead, andsynchronizationoverhead.

Total throughput was measured by holding the protocolfunctionality, application traffic patterns, and network in-terfaces constant and systematically varying the process ar-chitecture to determine the resulting performance impact.Each benchmarking session consisted of transmitting 10,0004 Kbyte messages through an extended version of the widelyavailablettcp protocol benchmarking tool. The originalttcp tool measures the processing resources and overalluser and system time required to transfer data between atransmitter process and a receiver process communicatingvia TCP or UDP. The flow of data is uni-directional, with thetransmitter flooding the receiver with a user-specified num-ber of data buffers. Various sender and receiver parameters(such as the number of data buffers transmitted and the sizeof application messages and protocol windows) may be se-lected at run-time.

The version ofttcp used in our experiments was en-hanced to allow a user-specified number of communicatingapplications to be measured simultaneously. This featuremeasured the impact of multiple connections on process ar-chitecture performance (two connections were used to testthe connection-oriented protocols). Thettcp tool was alsomodified to use theASX-based protocol stacks configured viathe process architectures described in Section 3.5. To mea-sure the impact of parallelism on throughput, each test wasrun using 1, 2, 3, and 4 PEs successively, using 1, 2, 3, or4 LWPs, respectively. Furthermore, each test was performedmultiple times to detect the amount of spurious interferenceincurred from other internal OS tasks (the variance betweentest runs proved to be insignificant).

1 2 3 40

5

10

15

20

25

30

35

Ave

rage

Thr

ough

put (

Mbi

ts/s

ec)

Number of Processing Elements

CL Message

CL Layer

CO Message

CO Layer

Figure 10: Process Architecture Throughput

Context switching and synchronization measurementswere obtained to help explain differences in the through-put results. These metrics were obtained from the SunOS5.3 /proc file system, which records the number of vol-untary and involuntary context switches incurred by threadsin a process, as well as the amount of time spent waiting toobtain and release locks on mutex and condition objects.

Figure 10 illustrates throughput (measured in Mbits/sec)as a function of the number of PEs for the task-based andmessage-based process architectures used to implement theconnection-oriented (CO) and connectionless (CL) protocolstacks. The results in this figure indicate that paralleliza-tion definitely improves performance. Each 4 Kbyte mes-sage effectively required an average of between 3.2 and 3.9milliseconds to process when 1 PE was used, but only .9 to1.9 milliseconds to process when 4 PEs were used. How-ever, the message-based process architectures significantlyoutperformed their task-based counterparts as the numberof PEs increased from 1 to 4. For example, the perfor-mance of the connection-oriented task-based process archi-tecture was only slightly better using 4 PEs (approximately16 Mbits/sec, or 1.92 milliseconds per-message processingtime) than the message-based process architecture was us-ing 2 PEs (14 Mbits/sec, or 2.3 milliseconds per-messageprocessing time). Moreover, if a larger number of PEs hadbeen available, it appears likely that the performance im-provement gained from parallel processing in the task-basedprocess architectures would have leveled off sooner than themessage-based tests due to the higher rate of growth for con-text switching and synchronization shown in Figure 11 andFigure 12.

Figure 11 illustrates the number ofinvoluntaryandvolun-

14

1 2 3 40

200

400

600

800

1000

1200

1400

1600

1800

2000

Vol

unta

ry C

onte

xt S

witc

hes

Number of Processing Elements1 2 3 4

0

500

1000

1500

2000

2500

3000

3500

4000

4500In

volu

ntar

y C

onte

xt S

witc

hes

Number of Processing Elements

CL Message

CL Layer

CO Message

CO Layer

Figure 11: Process Architecture Context Switching Overhead

tary context switches incurred by the process architecturesmeasured in this study. An involuntary context switch occurswhen the OS kernel preempts a running thread. For exam-ple, the OS preempts running threads periodically when theirLWP time-slice expires in order to schedule other threadsto execute. A voluntary context switch is triggered when athread puts itself to sleep until certain resources (such as I/Odevices or synchronization locks) become available. For ex-ample, when a protocol task attempts to acquire a resourcethat may not become available immediately (such as obtain-ing a message from an empty list of messages in aTask ),the protocol task puts itself to sleep by invoking thewaitmethod of a condition object. This action causes the OSkernel to preempt the current thread and perform a contextswitch to another thread that is capable of executing proto-col tasks immediately.

As shown in Figure 11, The task-based process archi-tectures exhibited slightly higher levels of involuntary con-text switching than the message-based process architectures.This is due mostly to the fact that the task-based tests re-quired more time to process the 10,000 messages and weretherefore pre-empted a greater number of times. Further-more, the task-based process architectures also incurred sig-nificantly more voluntary context switches, which accountsfor the substantial improvement in overall throughput ex-hibited by the message-based process architectures. Theprimary reason for the increased context switching is thatthe locking mechanisms used by the message-based processarchitectures utilize adaptive spin-locks (which rarely trig-ger a context switch), rather than the sleep-locks used bytask-based process architectures (whichdo trigger a contextswitch).

Figure 12 indicates the amount of execution time/procreported as being devoted to waiting to acquire and releaselocks in the connectionless and connection-oriented bench-mark programs. As with context switching benchmarks,the message-oriented process architectures incurred consid-erably less synchronization overhead, particularly when 4PEs were used. As before, the spin-locks used by message-based process architecture reduce the amount of time spentsynchronizing, in comparison with the sleep-locks used by

1 2 3 40

1

2

3

4

5

6

7

8

Lock

Wai

t Tim

e (s

ecs)

Number of Processing Elements

CL Message

CL Layer

CO Message

CO Layer

Figure 12: Process Architecture Locking Overhead

the task-based process architectures.

4 Concluding Remarks

Despite an increase in the availability of operating systemand hardware platforms that support networking and paral-lel processing [25, 33, 22, 34], developing distributed appli-cations that effectively utilize parallel processing remains acomplex and challenging task. The ADAPTIVE Service eX-ecutive (ASX) provides an extensible object-oriented frame-work that simplifies the development of distributed appli-cations on shared memory multi-processor platforms. TheASX framework employs a variety of advanced OS mech-anisms (such as multi-threading and explicit dynamic link-ing), object-oriented design techniques (such as encapsula-

15

tion, hierarchical classification, and deferred composition)and C++ language features (such as parameterized types, in-heritance, and dynamic binding) to enhance software qualityfactors (such as robustness, ease of use, portability, reusabil-ity, and extensibility) without degrading application perfor-mance. In general, the object-oriented techniques and C++features enhance the software quality factors, whereas theadvanced OS mechanisms improve application functionalityand performance.

A key aspect of concurrent distributed application perfor-mance involves the type of process architecture selected tostructure parallel processing of tasks in an application. Em-pirical benchmark results reported in this paper indicate thatthe task-based process architectures incur relatively high-levels of context switching and synchronization overhead,which significantly reduces their performance. Conversely,the message-based process architectures incur much lesscontext switching and synchronization, and therefore exhibithigher performance. TheASX framework helped to con-tributed to these performance experiments by providing a setof object-oriented components that decouple the protocol-specific functionality from the underlying of process archi-tecture, thereby simplifying experimentation.

The ASX framework components described in this paperare freely available via anonymous ftp fromics.uci.eduin the filegnu/C++ wrappers.tar.Z . This distributioncontains complete source code, documentation, and exam-ple test drivers for the C++ components developed as partof the ADAPTIVE project [35] at the University of Califor-nia, Irvine. Components in theASX framework have beenported to both UNIX and Windows NT and are currently be-ing used in a number of commercial products including theAT&T Q.port ATM signaling software product, the EricssonEOS family of telecommunication switch monitoring appli-cations, and the network management portion of the Mo-torola Iridium mobile communications system.

Acknowledgements

Thanks to Paul Stephenson of Ericsson for countless hoursof discussion on devising object-oriented techniques for de-veloping distributed application frameworks.

References[1] D. C. Schmidt, “IPCSAP: An Object-Oriented Interface to

Interprocess Communication Services,”C++ Report, vol. 4,November/December 1992.

[2] G. Booch, Object Oriented Analysis and Design with Ap-plications (2nd Edition). Redwood City, California: Ben-jamin/Cummings, 1993.

[3] D. C. Schmidt and P. Stephenson, “An Object-OrientedFramework for Developing Network Server Daemons,” inProceedings of the2nd C++ World Conference, (Dallas,Texas), SIGS, Oct. 1993.

[4] D. C. Schmidt and T. Suda, “An Object-Oriented Frameworkfor Dynamically Configuring Extensible Distributed Commu-nication Systems,”IEE/BCS Distributed Systems Engineering

Journal (Special Issue on Configurable Distributed Systems),vol. 2, pp. 280–293, December 1994.

[5] R. Johnson and B. Foote, “Designing Reusable Classes,”Journal of Object-Oriented Programming, vol. 1, pp. 22–35,June/July 1988.

[6] M. A. Linton and P. R. Calder, “The Design and Implemen-tation of InterViews,” inProceedings of the USENIX C++Workshop, November 1987.

[7] D. Batory and S. W. O’Malley, “The Design and Implementa-tion of Hierarchical Software Systems Using Reusable Com-ponents,”ACM Transactions on Software Engineering andMethodology, vol. 1, pp. 355–398, Oct. 1992.

[8] R. Campbell, V. Russo, and G. Johnson, “The Design ofa Multiprocessor Operating System,” inProceedings of theUSENIX C++ Workshop, pp. 109–126, USENIX Association,November 1987.

[9] J. M. Zweig, “The Conduit: a Communication Abstraction inC++,” in Proceedings of the2nd USENIX C++ Conference,pp. 191–203, USENIX Association, April 1990.

[10] N. C. Hutchinson, S. Mishra, L. L. Peterson, and V. T.Thomas, “Tools for Implementing Network Protocols,”Soft-ware Practice and Experience, vol. 19, pp. 895–916, Septem-ber 1989.

[11] D. C. Schmidt, B. Stiller, T. Suda, A. Tantawy, and M. Zit-terbart, “Language Support for Flexible, Application-TailoredProtocol Configuration,” inProceedings of the18th Con-ference on Local Computer Networks, (Minneapolis, Min-nesota), pp. 369–378, IEEE, Sept. 1993.

[12] D. C. Schmidt, “Reactor: An Object Behavioral Pattern forConcurrent Event Demultiplexing and Event Handler Dis-patching,” in Pattern Languages of Program Design(J. O.Coplien and D. C. Schmidt, eds.), pp. 529–545, Reading, MA:Addison-Wesley, 1995.

[13] D. Ritchie, “A Stream Input–Output System,”AT&T BellLabs Technical Journal, vol. 63, pp. 311–324, Oct. 1984.

[14] N. C. Hutchinson and L. L. Peterson, “Thex-kernel: An Ar-chitecture for Implementing Network Protocols,”IEEE Trans-actions on Software Engineering, vol. 17, pp. 64–76, January1991.

[15] D. C. Schmidt and T. Suda, “Transport System ArchitectureServices for High-Performance Communications Systems,”IEEE Journal on Selected Areas in Communication, vol. 11,pp. 489–506, May 1993.

[16] D. C. Schmidt, “Acceptor and Connector: Design Patternsfor Actively and Passively Initializing Network Services,”in Workshop on Pattern Languages of Object-Oriented Pro-grams at ECOOP ’95, (Aarhus, Denmark), August 1995.

[17] D. C. Schmidt and T. Suda, “Measuring the Performance ofParallel Message-based Process Architectures,” inProceed-ings of the Conference on Computer Communications (INFO-COM), (Boston, MA), pp. 624–633, IEEE, April 1995.

[18] D. C. Schmidt and T. Suda, “Measuring the Impact of Al-ternative Parallel Process Architectures on CommunicationSubsystem Performance,” inProceedings of the4th Inter-national Workshop on Protocols for High-Speed Networks,(Vancouver, British Columbia), pp. 103–118, IFIP/IEEE, Au-gust 1994.

[19] D. C. Schmidt and T. Suda, “Experiences with an Object-Oriented Architecture for Developing Extensible DistributedSystem Management Software,” inProceedings of the Confer-ence on Global Communications (GLOBECOM), (San Fran-cisco, CA), pp. 500–506, IEEE, November/December 1994.

[20] D. C. Schmidt, “Reactor: An Object Behavioral Pattern forConcurrent Event Demultiplexing and Dispatching,” inPro-ceedings of the1st Annual Conference on the Pattern Lan-guages of Programs, (Monticello, Illinois), pp. 1–10, August1994.

16

[21] Bjarne Stroustrup,The C++ Programming Language, 2nd

Edition. Addison-Wesley, 1991.

[22] S. Saxena, J. K. Peacock, F. Yang, V. Verma, and M. Krish-nan, “Pitfalls in Multithreading SVR4 STREAMS and otherWeightless Processes,” inProceedings of the Winter USENIXConference, (San Diego, CA), pp. 85–106, Jan. 1993.

[23] Bjarne Stroustrup and Margret Ellis,The Annotated C++ Ref-erence Manual. Addison-Wesley, 1990.

[24] A. D. Birrell, “An Introduction to Programming withThreads,” Tech. Rep. SRC-035, Digital Equipment Corpora-tion, January 1989.

[25] J. Eykholt, S. Kleiman, S. Barton, R. Faulkner, A. Shivalin-giah, M. Smith, D. Stein, J. Voll, M. Weeks, and D. Williams,“Beyond Multiprocessing... Multithreading the SunOS Ker-nel,” in Proceedings of the Summer USENIX Conference, (SanAntonio, Texas), June 1992.

[26] M. Zitterbart, B. Stiller, and A. Tantawy, “A Model for High-Performance Communication Subsystems,”IEEE Journal onSelected Areas in Communication, vol. 11, pp. 507–519, May1993.

[27] J. C. Mogul and A. Borg, “The Effects of Context Switches onCache Performance,” inProceedings of the4th InternationalConference on Architectural Support for Programming Lan-guages and Operating Systems (ASPLOS), (Santa Clara, CA),ACM, Apr. 1991.

[28] Mats Bjorkman and Per Gunningberg, “Locking Strategiesin Multiprocessor Implementations of Protocols,” inProceed-ings of the Symposium on Communications Architectures andProtocols (SIGCOMM), (San Francisco, California), ACM,1993.

[29] M. Goldberg, G. Neufeld, and M. Ito, “A Parallel Approachto OSI Connection-Oriented Protocols,” inProceedings of the3rd IFIP Workshop on Protocols for High-Speed Networks,

(Stockholm, Sweden), May 1992.

[30] D. D. Clark and D. L. Tennenhouse, “Architectural Consid-erations for a New Generation of Protocols,” inProceedingsof the Symposium on Communications Architectures and Pro-tocols (SIGCOMM), (Philadelphia, PA), pp. 200–208, ACM,Sept. 1990.

[31] D. D. Clark, “The Structuring of Systems Using Upcalls,”in Proceedings of the10th Symposium on Operating SystemPrinciples, (Shark Is., WA), 1985.

[32] G. Booch and M. Vilot, “Simplifying the Booch Compo-nents,”C++ Report, vol. 5, June 1993.

[33] A. Garg, “Parallel STREAMS: a Multi-Process Implemen-tation,” in Proceedings of the Winter USENIX Conference,(Washington, D.C.), Jan. 1990.

[34] A. Tevanian, R. Rashid, D. Golub, D. Black, E. Cooper, andM. Young, “Mach Threads and the Unix Kernel: The Battlefor Control,” in Proceedings of the USENIX Summer Confer-ence, USENIX Association, August 1987.

[35] D. C. Schmidt, D. F. Box, and T. Suda, “ADAPTIVE: A Dy-namically Assembled Protocol Transformation, Integration,and eValuation Environment,”Journal of Concurrency: Prac-tice and Experience, vol. 5, pp. 269–286, June 1993.

17