OASIS: An Architecture for Dynamic Instrumentation of Enterprise DistributedReal-time and Embedded Systems

James Hill∗, Hunt Sutherland†, Paul Staudinger†, Thomas Silveria‡, Douglas C. Schmidt§, John Slaby‡ and Nikita Visnevski†∗ Indiana University-Purdue University Indianapolis

Indianapolis, IN USAEmail: hillj@cs.iupui.edu

† GE Global ResearchNiskayuna, NY USA

Email: {sutherland,staudinger}@crd.ge.comvisnevsk@research.ge.com

‡ Raytheon CompanyPortsmouth, RI USA

Email: {thomas silveria,john m slaby}@raytheon.com

§ Vanderbilt UniversityNashville, TN USA

Email: d.schmidt@vanderbilt.edu

Abstract—Instrumentation is a critical part of evaluating anenterprise distributed real-time and embedded (DRE) system’sperformance. Traditional techniques for instrumenting enter-prise DRE systems require DRE system developers to makedesign decisions regarding what metrics to collect during earlyphases of the software lifecycle so these needs can be factoredinto the system architecture. In many circumstances, however,it is hard for DRE system developers to know this informationduring early phases of the software lifecycle—especially whenmetrics come from many heterogeneous sources (such asapplication- and system-level hardware and software resources)and evaluating performance is traditionally an after-thought.

To address these issues, this article presents the designand performance of OASIS, which is SOA-based middlewareand tools that dynamically instruments enterprise DRE systemwithout requiring design-time knowledge of which metrics tocollect. This article also empirically evaluates OASIS in the con-text of a representative enterprise DRE system case study fromthe domain of shipboard computing. Results from applyingOASIS to this case study show that its flexibility enables DREsystem testers to precisely control instrumentation overhead.We also highlight open challenges in dynamic instrumentationfor next-generation enterprise DRE systems.

Keywords-dynamic instrumentation, enterprise DRE systems,service-oriented architecture, middleware, real-time instrumen-tation

I. INTRODUCTION

Enterprise distributed real-time and embedded (DRE)systems (e.g. power grid management, shipboard computingenvironments, and traffic management systems) are oftencomposed of loosely coupled applications that run atop dis-tributed infrastructure software (e.g., component-base mid-dleware and service-oriented middleware [26]) and deployedinto heterogeneous execution environments. In addition tomeeting their functional requirements, enterprise DRE sys-tems must also meet non-functional requirements, such as

The author would like to thank the Test Resource Management Center(TRMC) Test and Evaluation/Science and Technology (T&E/S&T) Programfor their support. This work is funded by the T&E/S&T Program throughthe Naval Undersea Warfare Center, Newport, RI, contract N66604-05-C-2335.

latency, response time, and scalability, so that applicationsachieve their expected behavior in a timely manner [32].Failure to meet non-functional requirements can cause thesystem to not meet its quality-of-service (QoS) guarantees.

To ensure enterprise DRE system QoS, DRE systemstakeholders, such as testers, developers, and end-users,monitor the system’s behavior and resources (e.g., CPUusage, memory allocation, and event arrival rate) and analyzecollected metrics. Conventional approaches for monitoringenterprise DRE system behavior and resources employ soft-ware instrumentation techniques [4], [24], [29], [32] thatcollect metrics of interest (e.g., busy time for an application,L2 cache memory usage, state of a component, and numberof heartbeats sent by a watchdog daemon) as a systemruns in its target environment. Performance analysis tools(such as testing and experimentation, distributed resourcemanagement, and real-time monitoring/reporting) then eval-uate collected metrics and inform DRE system stakeholdersif the system is meeting its QoS expectations, such ashonoring deadlines, processing events in a timely manner,not hogging to much memory. These tools can also identifyQoS bottlenecks and application components that exhibithigh and/or unpredictable behavior and resource usage [25].

Although software instrumentation helps ensure QoSguarantees, conventional techniques [32] for collecting met-rics are often tightly coupled to system implementations. Asa result, DRE system developers must decide what metricsto collect during the system design phase. These concernsare also factored into the verification and validation processfor system implementations [32]. DRE system developersand testers therefore are faced with the following drawbacksas a result of using a tightly-coupled approach to softwareinstrumentation:• Developers often do not know a priori what metrics

are needed to evaluate system QoS—especially sinceperformance is an afterthought [12] (i.e., evaluated latein the software lifecycle);

• Incorporating new metrics late in the software lifecycle

typically triggers verification and validation of thechanges system, which is costly [34];

• Developers continually reinvent instrumentation logicacross different application domains; and

• Developers apply ad hoc techniques, such as augment-ing existing code to inject the instrumentation hooks,without understanding the impact on overall systemdesign and maintainability.

To overcome the drawbacks with conventional enterpriseDRE system instrumentation techniques, there is need forimproved techniques that do not require DRE system stake-holders to make software instrumentation design decisionsduring early phases of software lifecycle. A promising ap-proach to realizing the goal is dynamic instrumentation [4],[31], where developers control what and how metrics arecollected from the system at runtime rather than during thedesign phase. When dynamic instrumentation methodologiesare combined with service-oriented middleware methodolo-gies, many challenges associated with dynamic instrumenta-tion, such as data collection and extraction, can be decoupledfrom—and resolved independently of—the core applicationbusiness logic. Moreover, these solutions can be reusedacross different application domains.

This article presents the Open-source Architecture forSoftware Instrumentation of Systems (OASIS) and providesthe following contributions to R&D on dynamic instrumen-tation of enterprise DRE systems:

• It describes the design challenges and OASIS-basedsolutions associated with realizing dynamic instrumen-tation middleware that is suitable for enterprise DREsystems;

• It shows how OASIS has been applied to a representa-tive enterprise DRE system case study from the domainof shipboard computing environments; and

• It analyzes empirical results that quantify how OA-SIS enables DRE system developers to dynamicallyconfigure the instrumentation middleware manually tominimize impact on end-to-end response time.

This article extends the seminal paper on OASIS [9] by:

• Presenting a more detailed overview of OASIS’s datacollection architecture;

• Discussing how the generalized architecture is cus-tomizable for different application domains, program-ming languages, and distributed technologies; and

• Presenting open challenges for future research on dy-namic instrumentation of enterprise DRE systems.

Article organization. The remainder of this article isorganized as follows: Section II motivates OASIS via acase study from the domain of shipboard computing; Sec-tion III describes the structure and functionality of OASIS;Section IV presents empirical results that evaluate OASIS’simpact on QoS requirements from the case study; Section V

compares work on OASIS with related efforts; and Sec-tion VI presents concluding remarks and lessons learned.

II. CASE STUDY: THE SPRING SCENARIO

This section motivates the need for OASIS via a casestudy based on the SPRING scenario. This scenario is arepresentative enterprise DRE system from the domain ofshipboard computing composed of programming languagesand technologies that execute in heterogeneous environmentswhere computers can run different operating systems, asshown in Figure 1.

!"#$%&'(

)*(

+,"-'.$/(

+,"-'.$/(

+,"-'.$/(

+,"-'.$/(

+012--"$/( +012--"$/( +012--"$/( +3.-4567/( +89"&#.$/(

+89"&#.$/(

+89"&#.$/(

+:2#";2</(

+67"$2#.$/( +67"$2#.$/( +67"$2#.$/(

=%&$.'.>(?@8A((

=.-%#.$(:BC(

DE,0?@8AF3GH(

)I=((

D3JJH(

0"$K.$!2-&"(E-21<'%'(A..1'(

8-#"$7$%'"()I8(2771%&2L.-(

D36I*E(3.!7.-"-#(=.M"1H(

D$"2M'H( D$"2M'H(

Figure 1. Overview of the SPRING Scenario

This scenario tests QoS concerns (such as throughput,latency, jitter, and scalability) of the underlying enterpriseDRE system infrastructure (e.g., hardware, middleware, op-erating system, and network) that will host it. For example,the scenario evaluates whether the infrastructure middlewarecan manage and meet the resource needs of applicationsin volatile conditions, such as changing arrival rates ofevents and dynamic resource usage. Detailed coverage ofthe SPRING scenario appears in [7].

Existing techniques for instrumenting enterprise DREsystems like the SPRING scenario assume software in-strumentation concerns (e.g., what metrics to collect andhow to extract metrics from the enterprise DRE appli-cation) are incorporated into the system’s design. Tightlycoupling system instrumentation with the existing softwaredesign, however, impedes flexibility in system design andimplementation throughout the software lifecycle, e.g., bycomplicating changes to the number/type of components andprecluding the refreshing of new technologies due to theeffort needed to revalidate QoS requirements.

In general, developers and testers of the SPRING scenariofaced the following challenges:• Challenge 1: Dynamic collection and extraction of

metrics without design time knowledge metric compo-sition. While the SPRING scenario is being developed itis hard for DRE system developers and testers in the early

phases of the software lifecycle to anticipate all the metricsthey will need to collect for performance analysis. Moreover,the SPRING scenario has many operating conditions (suchas high and low event throughput) and the QoS of itscritical path can be affected by instrumentation. DRE systemdevelopers and testers therefore need mechanisms that en-able them to (1) collect and extract metrics without makingsuch decisions at design time and (2) dynamically modifycollection to ensure instrumentation does not unduly perturbperformance. Section III-B1 shows how OASIS addressesthis challenge by using a multi-level packaging techniquethat enables it to adapt to different metrics being collected.

• Challenge 2: Dynamic discovery of metrics foranalysis without design time knowledge of metric compo-sition. Existing third-generation languages and distributionmiddleware technologies, such as J2EE [30], CORBA [21]–[23], DDS [19], and Microsoft .NET [13], support dynamictypes that enable transmission of data without knowledgeof its structure and quantify. Although dynamic types area plausible solution for this challenge, they are tightlycoupled to a particular programming languages and middle-ware. Since many enterprise DRE systems like the SPRINGscenario are composed of heterogenous technologies, DREsystem developers and testers need higher-level mechanismsthat enable them to discover metrics for performance anal-ysis tools without being bounded to specific technologies.Section III-B2 shows how OASIS addresses this challengeby using metadata to describe metrics being collected be-forehand so it is possible to discover the actual metrics later.

• Challenge 3: Real-time reporting of collected met-rics. The ability to dynamically instrument and collectmetrics without design time knowledge of what metrics arebeing collected is not complete without the ability to analyzesuch metrics in real-time. For example, performance analysistools may need to monitor system performance in real-timeand adapt the system’s behavior as needed, such as changingthe rate of collecting information without impacting systemresponse time or migrating software components to reduceCPU workload or improve fault tolerance [33]. DRE systemdevelopers therefore need architectural support that enablesthem to receive collected information in real-time for actu-ator purposes. Section III-B3 shows how OASIS addressesthis challenge using publisher-subscriber services that allowperformance analysis tools to register for a software probeand receive metrics in real-time (i.e., as it is collected).

These challenges make it hard for DRE system developersand testers to apply different performance analysis toolsto the SPRING scenario throughout the software lifecycle,particularly in early phases. Moreover, these challengesextend beyond the SPRING scenario and apply to otherenterprise DRE systems that need to collect and extractmetrics without knowledge of such concerns at design time.

III. THE STRUCTURE AND FUNCTIONALITY OF OASIS

This section describes the structure and functionality ofOASIS, focusing on how it addresses the key challengespresented in Section II that DRE system developers andtesters encounter when dynamically instrumenting enterpriseDRE systems.

A. Overview of OASIS

The challenges in Section II focus on the ability tohandle(e.g., collect, extract, and analyze) metrics withouta priori knowledge of metric details (e.g., their structureand quantity to the underlying middleware and systeminfrastructure). To address these challenges, the OASISservice-oriented middleware can instrument enterprise DREsystems to collect and extract metrics of interest withoutknowledge of their structure or quantity. Metric collectionand extraction in OASIS is also independent of specifictechnologies and programming languages, which decouplesOASIS from enterprise DRE system software implementa-tion details. DRE system developers and testers are thusnot constrained to make decisions regarding what metricsto collect for performance analysis tools during the designphase of the system.

!""#$%&'()*+(),-.,*

/01(2-*3!+*

4(56&7-*

87(9-*

:;/*<&)&=-7*

8-7>(7?&)%-*

!)&#@A$A*:((#*

Figure 2. Overview of OASIS

Figure 2 presents an high-level overview of OASIS. Asshown in this figure, OASIS consists of the following fiveentities:

Software probe is an autonomous agent [17] that collectsmetrics of interest (such as the current value(s) of an event,the current state of a component, or the heartbeat of acomponent or the node hosting the component) and actsindependent of the monitored application and other entitiesin OASIS. For example, a heartbeat software probe in theSPRING scenario has an active object that periodicallysends metrics representing a heartbeat, whereas an eventmonitor probe may send metrics each time a componentreceives/sends an event.

OASIS supports application-level and system-levelprobes. Application-level probes are embedded intoan application to collect metrics, such as the stateof an application component or number of events itsends/receives. System-level probes collect metrics (such

as current memory usage or the heartbeat of each host inthe target environment) that are not easily available at theapplication-level or may be redundant the application-level.Both application- and system-level probes submit theirmetrics to the Embedded Instrumentation Node (EINode)described below. Each software probe is identifiablefrom other software probes by a user-defined UUID andcorresponding human-readable name.

Embedded Instrumentation Node (EINode) is respon-sible for receiving metrics from software probes. OASIShas one EINode per application-context, which is a domainof commonly related data. Application-context examplesinclude a single component, an executable, or a single hostin the target environment. The application-context for anEINode, however, is locality-constrained for efficiency, i.e.,to ensure data transmission from a software probe to anEINode need only cross process boundaries—rather thannetwork boundaries. Moreover, the EINode controls the flowof data it receives from software probes and submits to thedata and acquisition controller described next. Each EINodeis uniquely identifiable from other EINodes by a user-definedUUID and corresponding human-readable name.

Data Acquisition and Controller (DAC) receives datafrom an EINode and archives it for subsequent acquisitionby performance analysis tools, such as querying the lateststate of component in the SPRING scenario. The DAC isa persistent database with a static location in the targetenvironment that can be located via a naming service [20].This design decouples an EINode from a DAC and enablesan EINode to discover which DAC it will submit data todynamically at creation time. Moreover, if a DAC failsduring at runtime the EINode can (re)discover a new DACto submit data. The DAC registers itself with the test andevaluation manager (described next) when it is created andis identifiable by an user-defined UUID and correspondinghuman-readable name.

Test and evaluation manager (T&E) is the main entrypoint for user applications (see below) into OASIS. TheT&E Manager gathers and correlates data from each DACthat has registered with it. The T&E Manager also enablesuser applications to send signals a software probe to alterits behavior at runtime, e.g., decreasing/increasing the hertzof the heartbeat software probe in the SPRING scenario. .

Performance analysis tools interact with OASIS byrequesting metrics collected from different software probesvia the T&E Manager. They can also send signals/commandsto software probes to alter their behavior at runtime. Thisdesign enables DRE system developers and testers andperformance analysis tools to control the effects of softwareinstrumentation at runtime and minimize the affects onoverall system performance, such as ensuring the SPRINGscenario’s critical path meets its deadline while under in-strumentation.

Figure 3 shows an example deployment of the OASIS

!"#$%&'()*'

+,!'-./.0&1'

23&1'

)44567.8$/'

()*'

!"#$%&'!"#$%&'!"#$%&'

!"#$%&'

23&1'

)44567.8$/'9&1:$1;./7&'

)/.5<363'+$$5'

!"#$%&'!"#$%&'!"#$%&'!=&/>'-$/6>$1'

91$?&'

!"#$%&'!"#$%&'!"#$%&'@$AB.1&'

91$?&'

C)@"@'($;.6/' )44567.8$/'($;.6/'+$$5'($;.6/'

!D&7>$1'*$;4$/&/>'

Figure 3. Deployment of OASIS in the SPRING Scenario

entities above in the context of an Effector componentfrom the SPRING scenario. This figure also shows howDRE system developers and testers write domain-specificsoftware probes—at either the application or system level—that collect metrics unknown to the OASIS middleware, suchas probe that monitors incoming events or the heartbeat ofa component. While the system under instrumentation isexecuting in its target environment, software probes collectmetrics and submit them to an EINode. The EINode, in turn,submits the data to the DAC, which stores the metrics untiluser applications request metrics collected from differentsoftware probes via the T&E Manager.

Figure 3 also highlights the application and networkboundaries of OASIS. Software probes do not submit dataacross different network boundaries directly; instead, theEINode localizes instrumentation overhead on the host.Likewise, the DAC and T&E Manager are designed toexecute on hosts separate from those running the instru-mented system to ensure data management concerns of theDAC and T&E Manager do not interfere with the system’sperformance.

The remainder of this section discusses how OASIS ad-dresses the challenges of enabling dynamic instrumentationwithout design-time knowledge of the collected metrics orbeing bounded to a specific technologies and programminglanguage.

B. Addressing the Dynamic Instrumentation Challenges inOASIS

1) Solution 1. Dynamic Collection of Metrics: As dis-cussed in Section III-A, OASIS has no knowledge of whatmetrics are collected during system instrumentation. Instead,DRE system developers and testers implement softwareprobes and register them with an EINode that ensuresits data is collected properly by the OASIS middleware.Figure 4 therefore shows OASIS’s architecture for facili-tating dynamic collection metrics from software probes. Asshown in this figure, the architecture is composed of severalinterfaces. Each interface, except for the SoftwareProbeinterface, represents a point where different implementationscan be inserted into the architecture to provide more domain-

!"#$%%&'()*+,-)..$

!"#"$%"&&'()

%%&'()*+,-)..$

*+,"(!"#"$%"&&'()

/001$

2*34&5)6$

2*34&5)6$

/001$

%%&'()*+,-)..$

-+./"0'10+2')

%%&'()*+,-)..$

-+./"0'10+2'3",#+04)

-*),()6$

/$

76)6$/$

/$

76)6$ 8,',9)6$

:;<$=,',9)*$

/001$

8,',9)6$

%%&'()*+,-)..$

567+8')

Figure 4. OASIS’s Architecture Diagram for Dynamic Collection of Metrics.

specific behavior, e.g., data channels that intelligently selectmetrics to transmit when dealing with software probes thathave high data rates [5].

Figure 5 shows the UML class diagram for theSoftwareProbe interface. As shown in this figure,each software probe implements an init() and fini()method to initialize and finalize a software probe, re-spectively. A software probe can optionally implement thehandleCommand() method, which allows it to processcommands sent from performance analysis tools via theT&E Manager. For example, if a software probe automati-cally sends updates to its host EINode, then a performanceanalysis tool can send a command to change its update rate,i.e., the software probes hertz. Such commands can also beused to change the software probe’s configuration based onthe state of the software system undergoing instrumentation.

!!"#$%&'()%**+

!"#$%&'(&")'*

,%$(-($(.+/$&"#0+

"#"$+1"#$+/$&"#0+23+(&0/4.+56"-+

7#"+156"-4.+56"-+

89/:+156"-4.+56"-+

:(#-;%<6,,(#-+1"#+/$&"#0+),-4.+56"-+

/%$=($(<:(##%;+1"#+>6)(;=($(<:(##%;+-)4.+56"-++

!!"#$%&'()%**+

!"#$%&'(&")'+%,-"&.*

)&%($%+156"-4.+?6@A(&%B&6C%+

Figure 5. UML Class Diagram for OASIS’s Software Probe Architecture

Each software probe implements a flush() methodthat allows the host EINode or application to force the

software probe to send its most recently collected metricsto its hosting EINode. The setDataChannel() methoddetermines the local data channel that the software probeuses to send collected metrics to the EINode. This methodtherefore is invoked by the hosting EINode when a softwareprobe is loaded into memory, but before the software probe’sinit() method in invoked.LocalDataChannel (see Figure 4 and Figure 7) is

an interface for a locality-constrained object. Its implemen-tation can therefore vary, depending on the needs of theenterprise DRE system undergoing dynamic instrumentation.For example, the local data channel implementation canaggregate collected metrics before sending them to theDAC to reduce network overhead. Each software probe hasa corresponding SoftwareProbeFactory to supportdynamic creation and linking into the application via theComponent Configurator pattern [28] and associating thesoftware probe with an EINode—via an implementation ofthe LocalDataChannel interface—to submit metrics.

!"#$%&!

'()%*&+$,,()%-./)-012/)3-4,%56-7$/%-

Figure 6. UML Class Diagram for OASIS’s EINode

Figure 6 presents the standard and lightweight objectstructure for an EINode. As shown in this figure, theEINode exports a single method to the software probe namedhandleCommand(), which receives commands from aDAC. The commands received via this method have theformat: [PROBE] [COMMAND], where PROBE is the nameof the target software probe that receives the command andCOMMAND is the command-line passes to the target probe,if found.

The EINode also implements the one or more

!!"#$%&'()%**+

!"#"$%"&&'()

,%#-.($(+/"#+0)$%$+12+-($(34+50"-+

!!"#$%&'()%**+

*+,"(!"#"$%"&&'()

Figure 7. UML Class Diagram for OASIS’s Data Channels

LocalDataChannel objects that are used by differentsoftware implementation probes. As shown in Figure 7,the LocalDataChannel has a single method namedsendData() used by the software probe to send packageddata in binary format based on the specification presentedin Table I to the EINode. As shown in this table, eachsoftware probe packages its metrics to include informationfor OASIS to learn the metric’s origin, i.e., the UUID of thesoftware probe, and the probe/metric’s state at collection,e.g., collection timestamp, sequence number of metrics,probe state, and metric’s data size. The remaining contentsof the BinaryData, such as the actual metrics, are outsideof the EINode’s concern and remain unknown.

Table IKEY ELEMENTS IN DATA PACKAGING SPECIFICATION FOR OASIS

SOFTWARE PROBES

Name (size) DescriptionprobeUUID (16) User-defined UUIDtsSec (4) Seconds value of timestamptsUsec (4) Micro-seconds value timestampseqeunceNum (4) Metric sequence numberprobeState (4) Probe-defined value (if applicable)dataSize (4) Size of metric data

After an EINode receives packaged metrics from a soft-ware probe, the EINode submits it to the DAC via an ex-posed DataChannel interface (see Figure 7). This versionof the data channel interface is designed to support networkcommunication, unlike the LocalDataChannel since theDAC and EINode reside on different machines (as shownin Figure 3). The behavior of an EINode for submittingcollected metrics to the DAC can vary between different im-plementations. Regardless of an EINode’s implementation,the specification in Table II is used to package data receivedfrom a software probe into binary data. The EINode theninvokes the sendData() method on the DataChannel,which sends collected metrics to the target DAC.

Table II shows the necessary header information theEINode prepends to binary data it receives from a soft-ware probe. This information allows OASIS to learn theapplication-context origin for collected metrics. The EINodethen submits the data to the DAC for storage. OASIS can

Table IIKEY ELEMENTS IN DATA PACKAGING SPECIFICATION FOR OASIS

EINODE

Name (size) DescriptionbyteOrder (1) Byte-order of dataversionNumber (2) OASIS version numberreserved (1) padding for word alignmenteinodeUUID (16) EInode unique id

dynamically collect metrics without knowing their details aslong as metrics sent to a DAC are packaged according toTable I and Table II. This approach also decouples OASISfrom any specific technology or programming languagesince data are a well-defined binary stream produceableusing any language or platform that supports sockets—thereby addressing Challenge 1 in Section II.

2) Solution 2. Discovery of Metrics for Analysis: Inmany cases, performance analysis tools requesting data fromOASIS via the T&E Manager will know at design timethe metrics being collected by software probes submittingdata to an EINode. In other cases, these tools may wantto request data for metrics not known at design-time. Forexample, in the SPRING scenario, DRE system developersand testers want to construct a generic GUI to monitor allmetrics collected while the system is instrumented in itstarget environment. Since the GUI does not know all thedifferent metrics collected by software probes at design time,the GUI needs mechanisms to learn metrics at runtime.

We address the challenge of discovering metrics foranalysis without design-time knowledge by requiring eachOASIS entity shown in Figure 2 to perform a registrationand unregistration process at startup and shutdown time,respectively. The registration process between an EINodeand a DAC, as well as a DAC with a T&E manager,involves understanding the composition of metric collection(i.e., the origin and path of metrics collected during systeminstrumentation). The registration process for a softwareprobe with an EINode is more critical, however, because thisis when the software probe notifies OASIS metric’s format.1 <? xml v e r s i o n =” 1 . 0 ” ?>2 <xsd : schema>3 <xsd :complexType name=” Component . S t a t e ”>4 <x s d : s e q u e n c e>5 <x s d : e l e m e n t name=”name” t y p e =” x s d : s t r i n g ” />6 <x s d : e l e m e n t name=” s t a t e ” t y p e =” x s d : b y t e ” />7 </ x s d : s e q u e n c e>8 </ xsd :complexType>9

10 <x s d : e l e m e n t11 name=” p r o b e M e t a d a t a ” t y p e =” Component . S t a t e ” />12 </ x sd : schema>

Listing 1. Example Registration File for Software Probe in OASIS

Listing 1 presents an example registration for a softwareprobe from the SPRING scenario that keeps track of a com-ponent’s state (e.g., activated or passivated). Each softwareprobe’s registration in this listing is a XML schema defini-tion (XSD) file. We use XSD since it provides a detailed

description of data types (such as quantity and constraints)that can help with optimizations and filtering/managing data.The element named probeMetadata defines the rootelement that describes the format of the metric collectedby a software probe. Likewise, the child annotation of theprobeMetadata element with the id named metadatadescribes information about the software probe, such as theuser-defined UUID and description of the software probe.

During registration, a software probe passes its XSD fileto the EINode when the create() method is invoked on itsSoftwareProbeFactory (see Listing 5). The EINodethen passes the XSD file to the DAC with which it is reg-istered. Performance analysis tools can request registrationinformation for a software probe via the T&E Manager,which includes the metadata describing its metric format.Using the metadata for a software probe, user applicationscan learn about metrics at runtime for analytical purposes—thereby addressing Challenge 2 in Section II.

3) Solution 3. Pub/sub for Real-time Reporting: In manycases, instrumentation of a software system is done to collectmetrics for postmortem analysis, such as debugging theapplication or locating data trends in performance properties.Moreover, dynamic software instrumentation provides theability to control at run-time the instrumentation’s behavior,which implies that performance analysis tools should havethe ability to control this behavior. Such control, however, istypically based on analyzing collected metrics at real-timeand invoking commands back into the system—similar toan actuator.

OASIS address this challenge problem by implementingthe Publisher-Subscriber [3] pattern that allows performanceanalysis tools to register for metrics collected by softwareprobes. When software probes send new data to the DAC, ifthe software probe has subscribers, the metrics are pushed tothe subscriber. OASIS implements the Publisher-Subscriberpattern using CORBA, unlike the rest of OASIS that is nottied to a platform or architecture. This design choice wasmade in the specification and architecture for the followingreasons:

• The technology-, architecture-, and platform-independence is needed only for extracting metricsfrom the enterprise DRE system. Once metrics reachthe DAC, it is outside the application-space of thesystem undergoing instrumentation.

• The performance analysis tool and the DAC typicallywill reside on different host machines, which impliesthat there will be network communication. CORBAtherefore simplifies inter-network communication con-cerns involving objects on residing on different ma-chines.

Listing 2 shows the publisher-subscriber interface imple-mented by the T&E Manager and DAC, respectively. Asshown in this listing the T&E Manager must implement

register() and unregister() methods. The regis-tration method requires an target DAC and software probe.Likewise, the performance analysis tool must implement aDataChannel interface—similar to the DataChannelimplemented by the DAC, which allows the performanceanalysis tool to provide a domain-specific implementationof the data channel, if necessary.1 / / P u b l i s h e r i n t e r f a c e f o r t h e T&E Manager .2 i n t e r f a c e T n E P u b l i s h e r {3 Cookie r e g i s t e r ( i n UUID dac ,4 i n UUID probe ,5 i n DataChanne l s u b s c r i b e r ) ;67 void u n r e g i s t e r ( i n Cookie c ) ;8 } ;9

10 / / P u b l i s h e r i n t e r f a c e f o r t h e DAC.11 i n t e r f a c e DACPublisher {12 Cookie r e g i s t e r ( i n UUID probe ,13 i n DataChanne l s u b s c r i b e r ) ;1415 void u n r e g i s t e r ( i n Cookie c ) ;16 } ;

Listing 2. Publisher-subscriber interface definition for the DAC and T&EManager.

The DAC also implements register() andunregister() methods. The DAC’s registrationmethod, however, requires only the target software probeand the provided DataChannel interface. The result ofsuccessful registration is a cookie, which is a subscriptionid for each client. This cookie is used to unregister thesubscribers.

After successful registration, performance analysis toolsbegin receiving updates via the provided DataChannelinterface. Based on collected data, the performance analysistool can send command to software probes at real-timevia the handleCommand() method. Performance analysistools thus have the ability to adapt software instrumentationautonomously based on the needs of the application atruntime—thereby addressing Challenge 3 in Section II.

C. Using a Definition Language to Define Software ProbesImplementing a software probe for OASIS requires the

following two steps:1) Creating the XML Schema Definition that defines the

software probes collected metrics, i.e., the softwareprobes metadata.

2) Implementing the software probe to package dataaccording to its metadata.

Software probes can be implemented in different program-ming languages, such as Java and C++, so it can be usedfor different application domains.

To simplify implementing software probes, OASIS pro-vides the Probe Definition Language (PDL), which is adomain-specific language for defining metric collected bya software probe. The language specification resemblesthe CORBA Interface Definition Language (IDL), but usesdifferent keywords to define data. Listing 3 shows the PDLfor the component state probe in Listing 1.

1 module Component {2 s t r u c t UnusedType {3 i n t 3 2 v a l u e 1 ;4 i n t 3 2 v a l u e 2 ;5 } ;67 [ uu id (0 A499B6B−7250−4B88−B9DC−360D32639081 ) ;8 v e r s i o n ( 1 . 0 ) ]9 p robe S t a t e {

10 s t r i n g name ;11 b y t e s t a t e ;12 } ;13 }

Listing 3. Specification of the component state probe using OASIS’s PDL.

As shown in this listing, PDL supports tuple definitions asstructures. A software probe is also defined by the keywordprobe. In addition, each probe has a set of attributesthat define its implementation properties, such as UUIDand version number. After a software probe is definedusing PDL, it is compiled into a separate XML SchemaDefinition and software probe implementation. The currentimplementation of OASIS generates software probes in C++,but can be extended easily to support other programminglanguages.

Table IIIDIFFERENT DATA TYPES SUPPORTED BY OASIS’S PDL.

Category KeywordInteger int8, int16, int32, int64Unsigned Integer uint8, uint16, uint32, uint64Decimal real32, real64String char, string

Table III list the different data types supported in PDL.Each data type can be specified as an array by using standardbracket notation: int8 v[N] where N is the number ofelements in the array. Likewise, ranges can be specified inthe bracket notation: int8 v[min, max] where min isthe minimum number of elements in the array and max isthe maximum number of elements in the array. If max =infinite, then there are unbounded number of elements inthe array, whereas if min = 0 and max = 1 the element isoptional—similar to XML Schema Definition.

IV. EVALUATING OASIS’S RUNTIME FLEXIBILITY ANDINSTRUMENTATION OVERHEAD

This section analyzes the results of experiments thatempirically evaluate the capabilities and performance ofOASIS in the context of the SPRING scenario presentedin Section II. The heterogeneity of the SPRING scenarioinfluenced the design of OASIS, as discussed in Section III.

A. Experiment Setup

A key concern of developers of applications and middle-ware for DRE systems is that dynamic instrumentation willnegatively impact existing QoS properties, such as response-time and utilization, of the instrumented system. Due to thedesign of OASIS (described in Section III), QoS properties

related to the Data Acquisition and Controller (DAC) and theTest and Evaluation (T&E) Manager do not impact existingQoS properties of the system undergoing instruction, suchas the application in the SPRING scenario. Instead, theEmbedded Instrumentation Node (EINode) and softwareprobes have more impact on QoS properties for instrumentedsystem since the DAC and T&E manager are not deployed inthe application domain. In contrast, the EINode and softwareprobes interact directly with the instrumented system.

Since the EINode and software probes have more of animpact on existing QoS properties for the system undergoinginstrumentation, developers and testers of the SPRING sce-nario were interested in determining if they could use OASISto monitor the application in real-time without causing theapplication in the SPRING scenario to miss its critical-pathdeadline. In particular, system testers wanted to monitor (1)as many events sent between each component and (2) theheartbeat of each application. Monitoring these two metricssignificantly impacts the end-to-end response time of theapplication in the SPRING scenario and is thus essential toevaluate OASIS’s capabilities.

System testers therefore used the CUTS system executionmodeling tool [8] to construct the application in the SPRINGscenario shown in Figure 1. CUTS was used to (1) model thebehavior and workload of each component at a high-level ofabstraction and (2) auto-generate source code for the CIAOarchitecture from constructed models. System testers thencompiled the generated source code on its target architectureand emulated the system on a cluster of computers runningin ISISlab (www.isislab.vanderbilt.edu). Each computer inISISlab used Emulab software to configure the Fedora Core8 operating system. Likewise, each computer is an IBMBlade Type L20, dual-CPU 2.8 GHz processor with 1 GBRAM interconnected via six Cisco 3750G-24TS networkswitches and one Cisco 3750G-48TS network switch. Theremainder of this section analyzes the results of the experi-ment.

B. Experiment Results

Section IV-A provided background information that thesystem testers of the SPRING scenario were interested inexecuting. In particular, the system testers wanted to under-stand how using the dynamic instrumentation capabilitiesof OASIS would affect end-to-end response time of theapplication in the SPRING scenario. Figure 8 shows asingle test run of the application in the SPRING scenarioin ISISlab, where system testers adjusted the hertz of theheartbeat software probe.

As shown in the figure, system testers initially started witha configuration of 1 hertz (or 1 event/sec) for each heartbeatsoftware probe embedded in a component. Since the end-to-end response time for the application’s critical path ofexecution between the Effectors and Sensors was under its500 msec deadline for this particular scenario, the system

!"#$#%# !"#$#&# !"#$#'# !"#$#(#

)*+,-./#0.12#34.5/+64#$#&77#894-#

Figure 8. Single Test Run of Adjusting the Heartbeat Software Probe’sHertz in SPRING Scenario’s Application Under OASIS Instrumentation.

testers decided to increase the heartbeat software probe’shertz to improve awareness of application liveliness. Afterincreasing the heartbeat to 5 hertz, the end-to-end responsetime began increasing above 500 msec, as shown in Figure 8.

Since the heartbeat probe negatively impacted end-to-endresponse time for the application’s critical path of execution,system testers decreased the heartbeat software probe’s hertzto 2. Figure 8 shows how this dynamic change enabledtesters to increase awareness of application liveliness and notnegatively impact end-to-end response of the application’scritical path of execution. System testers also increasedthe heartbeat software probe’s hertz to 3 to determine thenew configuration would impact end-to-end response time.Unfortunately, 3 hertz caused the end-to-end response timeto gradually increase with respect to time. System testerstherefore learned that they could configure the heartbeatsoftware probe to execute at 2 hertz without impacting end-to-end response time of the application’s critical path ofexecution.

C. Open Research Problems

The results above are just one of many different exe-cutions of the SPRING scenario. Experience gained fromrunning this scenario throughout the system lifecycle un-derscores the challenges of validating OASIS’s usage inenterprise DRE systems. Based on this experience, thefollowing is a list of open research problems in dynamicsoftware instrumentation of DRE systems.

Patterns for (dynamic) software instrumentation ofenterprise DRE systems. Different application domainsrequire different software instrumentation needs, which ispart of the reason that software instrumentation concernshave been tightly coupled with the system’s implementation.Once this tight-coupling is removed (as done with OASIS)it is possible to separate the core “business-logic” of soft-ware instrumentation from the more domain-specific aspects,such as collection, aggregation, and extraction techniques.

Moreover, this decoupling will force DRE system developersand testers to discover patterns for instrumenting distributedsystems, similar to patterns of software architectures [1]–[3],[10], [28].

Optimization techniques for general-purpose dis-tributed system instrumentation middleware. When aconcern, such as software instrumentation, is separated fromthe overall application it is typically over-generalized andhighly configurable. For example, distributed middlewarearchitectures, such as CORBA, Microsoft.NET, and J2EE,removed complexities associated with network communica-tion, through abstraction and generalization. Although theseabstractions generally improve reuse and software quality,many times the generalization does not fit every applicationdomain. Similar to general-purpose distributed middleware,DRE system developers and testers must develop differentoptimization techniques for software instrumentation of en-terprise DRE systems that try to minimize resource usage,such as network bandwidth, CPU overhead, memory, whilenot impacting existing qualities of the system undergoinginstrumentation.

Verification and validation of DRE systems that utilizedynamic instrumentation middleware. Verification andvalidation of dynamic enterprise DRE systems is still anactive research area [6], [11], [18], [27]. Dynamic instru-mentation middleware adds another level of complexity tothe problem because these concerns are not tightly coupledto the design and can change at run-time. These run-timechanges, in turn, can impact the integrity of the system ifthe such changes force the system to comprise its QoS guar-antees. DRE system developers and testers therefore needto research methodologies that support V&V of enterpriseDRE systems that use dynamic instrumentation middlewarecapabilities.

V. RELATED WORK

This section compares and contrasts our work on OASISwith related work.

Instrumentation techniques. Tan et al. [32] discussmethodologies to verify instrumentation techniques for re-source management. Their approach decomposes instrumen-tation for resource management into monitor and correctorcomponents. OASIS also has a monitor component (i.e.,software probes) and a corrector component (i.e., user appli-cations). OASIS extends their definition of instrumentationcomponents, however, to include those necessary to extractmetrics from the system efficiently and effectively (i.e., theEINode, DAC, and T&E manager).

DTrace [4] is a non-intrusive dynamic instrumentationutility for production systems that has similar conceptsas OASIS (such as application- and system-level softwareprobes). DTrace is locality constrained, however, whereasOASIS can collect data in a distributed environment. DTraceand OASIS can both collect metrics without design time

knowledge of what metrics are being collected via instru-mentation. DTrace also has the option of filtering instru-mentation data at the software probe level using predi-cates. OASIS can achieve similar functionality—and greaterflexibility—at the software probe, DAC, and T&E managerlevels if each component uses a software probe’s registeredmetadata to learn and filter collected metrics at runtime.DTrace and OASIS can complement each other to removethe locality constrained characteristics of DTrace.

Metric collection techniques. XML is the basis of severaltechniques for collecting metrics without a priori knowl-edge of their type. XML is used in technologies such asSimple Object Access Protocol (SOAP) for Web Services,XML-RPC, and XML Metadata Interchange (XMI). OASISlikewise enables collection/transmission of metrics withoutdesign time knowledge of their type. OASIS uses raw binarystreams to collect and transmit metrics, however, as opposedto verbose text strings as in XML that can impact theperformance of an enterprise DRE system. OASIS usesXML to describe metametrics, which are metadata thatdescribes the metrics structure, data types, and quantity,collected and transmitted at registration time (i.e., beforethe system is fully operational).

Generic data types, such as CORBA Any and Java Object,can be used to dynamically collect and transmit metrics.OASIS improves upon these techniques since it is notbound to a particular programming language or middlewaretechnology. Moreover, using generic types to collect andtransmit metrics makes it hard for receivers (such as user ap-plications) to learn about unknown types unless the languagesupports built-in discovery mechanisms, such as reflection.OASIS removes this complexity by storing metadata abouta software probe’s metrics (which is decoupled from theactual metrics) so programming languages and technologiesused to implement user applications in OASIS that do notsupport type discovery mechanisms can still discover andutilize metric types at runtime.

Distributed instrumentation middleware. The Testingand Training Enabled Architecture (TENA) [14]–[16] is adistributed service-oriented middleware that supports instru-mentation of distributed software applications—similar toOASIS. The main difference between TENA and OASISis that TENA focuses on interoperability between differ-ent application domains using CORBA, whereas OASISfocuses on separating and defining different concerns ofsoftware instrumentation so each can be implemented usingdomain-specific objects. Likewise, OASIS provides supportfor dynamic instrumentation and callback commands tosoftware probes, which provides the foundation for softwareactuators.

VI. CONCLUDING REMARKS

Conventional techniques for instrumenting enterpriseDRE systems and determining which metrics to collect

for performance analysis tools can limit their analyticalcapabilities. Moreover, deferring design decisions pertain-ing to instrumenting DRE systems and determining whatinformation to collect can limit the metrics available toperformance analysis tools since the instrumentation may notbe incorporated into the system’s design. To address thesedrawbacks, this paper presented OASIS, which is service-oriented dynamic instrumentation middleware that enablesenterprise DRE system developers and testers to collect met-rics via instrumentation without prematurely committing totightly-coupled design decision during early system lifecyclephases. The results of our experiments show how OASIShelps adapt the instrumentation needs of enterprise DREsystem developers and testers by enabling them to controlthe impact of instrumentation overhead at runtime.

The following are our lessons learned thus far based onour experience of applying OASIS to the SPRING scenario:

• Dynamic configuration of probes at runtime min-imizes probe effects. OASIS’s ability to alter thebehavior of software probes at runtime helps minimizeinstrumentation overhead, e.g., software probes haveessentially the same effects on QoS whether they arepresent or not. Enterprise DRE system developers andtesters thus have greater confidence they can incorpo-rate instrumentation into production systems and stillmeet system QoS requirements. Our future work isinvestigating techniques to automate minimizing probeeffects for dynamic instrumentation of enterprise DREsystems.

• Separating metadata from data improves discoverycapabilities. Since OASIS separates metadata (i.e., theXSD files) from the actual data (i.e., collected metrics)for each software probe, performance analysis toolsto utilize collected metrics at runtime without havingprior knowledge of its structure and quality. Our futurework is investigating techniques to optimize OASIS’sdata collection, archiving, and retrieval capabilities byleveraging this metadata.

• Data distribution services may improve publisher-subscriber architecture. The current OASIS specifi-cation and architecture relies on CORBA to realizeits publisher-subscriber requirements. A better solu-tion, however, may be to use QoS-enabled publisher-subscriber middleware, such as the Data DistributionService (DDS), to handle real-time notifications ofcollected metrics. Our future work is comparing DDSimplementations against CORBA implementations todetermine their relative value for OASIS.

OASIS is integrated into the CUTS system execution model-ing tool and both are available for download in open-sourceform from www.cs.iupui.edu/CUTS.

REFERENCES

[1] F. Buschmann, K. Henney, and D. C. Schmidt. Pattern-Oriented Software Architecture: A Pattern Language for Dis-tributed Computing, Volume 4. Wiley and Sons, New York,2007.

[2] F. Buschmann, K. Henney, and D. C. Schmidt. Pattern-Oriented Software Architecture: Patterns and Pattern Lan-guagesg, Volume 5. Wiley and Sons, New York, 2007.

[3] F. Buschmann, R. Meunier, H. Rohnert, P. Sommerlad, andM. Stal. Pattern-Oriented Software Architecture—A Systemof Patterns. Wiley & Sons, New York, 1996.

[4] B. Cantrill, M. W. Shapiro, and A. H. Leventhal. DynamicInstrumentation of Production Systems. In Proceedings of theGeneral Track: 2004 USENIX Annual Technical Conference,pages 15–28, June 2004.

[5] J. Chang, N. Park, and W. Lee. Adaptive selection oftuples over data streams for efficient load shedding. Interna-tional Journal of Computer Systems Science & Engineering,23(4):277–287, 2008.

[6] R. Feldt, R. Torkar, E. Ahmad, and B. Raza. Challenges withSoftware Verification and Validation Activities in the SpaceIndustry. In Third International Conference on SoftwareTesting, Verification and Validation, pages 225–234, April2010.

[7] J. H. Hill, T. Silveria, J. M. Slaby, , and D. C. Schmidt. TheSPRING Scenario: An Heterogeneous Enterprise DistributedReal-time and Embedded (DRE) System Case Study. Techni-cal Report TR-CIS-1211-09, Indiana University-Purdue Uni-versity Indianapolis, December 2009.

[8] J. H. Hill, J. Slaby, S. Baker, and D. C. Schmidt. ApplyingSystem Execution Modeling Tools to Evaluate EnterpriseDistributed Real-time and Embedded System QoS. In Pro-ceedings of the 12th International Conference on Embeddedand Real-Time Computing Systems and Applications, Sydney,Australia, August 2006.

[9] J. H. Hill, H. Sutherland, P. Staudinger, T. Silveria, D. C.Schmidt, J. M. Slaby, and N. A. Visnevski. OASIS: AService-Oriented Architecture for Dynamic Instrumentationof Enterprise Distributed Real-time and Embedded Sys-tems. In Proceedings of 13th IEEE International Symposiumon Object/Component/Service-oriented Real-time DistributedComputing (ISORC), Carmona, Spain, May 2010.

[10] M. Kircher and P. Jain. Pattern-Oriented Software Architec-ture, Volume 3: Patterns for Resource Management. Wileyand sons, 2004.

[11] O. Laurent. Using Formal Methods and Testability Conceptsin the Avionics Systems Validation and Verification (V&V)Process. In Third International Conference on SoftwareTesting, Verification and Validation (ICST), April 2010.

[12] D. A. Menasce, L. W. Dowdy, and V. A. F. Almeida. Perfor-mance by Design: Computer Capacity Planning By Example.Prentice Hall PTR, Upper Saddle River, NJ, USA, 2004.

[13] Microsoft Corporation. Microsoft .NET Framework 3.0Community. www.netfx3.com, 2007.

[14] J. R. Noseworthy. Developing Distributed ApplicationsRapidly and Reliably Using the TENA Middleware. In IEEEMilitary Communications Conference, volume 3, pages 1507–1513, October 2005.

[15] J. R. Noseworthy. The Test and Training Enabling Archi-tecture (TENA) Supporting the Decentralized Developmentof Distributed Applications and LVC Simulations. In 12thIEEE/ACM International Symposium on Distributed Simu-lation and Real-Time Applications, pages 259–268, October2008.

[16] J. R. Noseworthy. Supporting the Decentralized Developmentof Large-Scale Distributed Real-Time LVC Simulation Sys-tems with TENA (The Test and Training Enabling Architec-ture). In Distributed Simulation and Real Time Applications,IEEE/ACM International Symposium on, pages 22–29, 2010.

[17] H. S. Nwana. Software Agents: An Overview. KnowledgeEngineering Review, 11(3):1–40, 1996.

[18] R. Obermaisser, C. El-salloum, B. Huber, and H. Kopetz.Modeling and Verification of Distributed Real-time Systemsusing Periodic Finite State Machines. 23, 2008.

[19] Object Management Group. Data Distribution Service forReal-time Systems Specification, 1.0 edition, Mar. 2003.

[20] Object Management Group. Naming Service, version 1.3,OMG Document formal/2004-10-03 edition, October 2004.

[21] Object Management Group. The Common Object RequestBroker: Architecture and Specification Version 3.1, Part 1:CORBA Interfaces, OMG Document formal/2008-01-04 edi-tion, Jan. 2008.

[22] Object Management Group. The Common Object RequestBroker: Architecture and Specification Version 3.1, Part 2:CORBA Interoperability, OMG Document formal/2008-01-07edition, Jan. 2008.

[23] Object Management Group. The Common Object RequestBroker: Architecture and Specification Version 3.1, Part 3:CORBA Component Model, OMG Document formal/2008-01-08 edition, Jan. 2008.

[24] K. O’Hair. The JVMPI Transition to JVMTI. java.sun.com/developer/technicalArticles/Programming/jvmpitransition,2006.

[25] T. Parsons. Automatic Detection of Performance Designand Deployment Antipatterns in Component Based EnterpriseSystems. PhD thesis, University College Dublin, Belfield,Dublin 4, Ireland, 2007.

[26] M. Pezzini and Y. V. Natis. Trends in Platform Middleware:Disruption Is in Sight. www.gartner.com/DisplayDocument?doc cd=152076, September 2007.

[27] S. Poulding and J. A. Clark. Efficient Software Verification:Statistical Testing Using Automated Search. IEEE Transac-tions on Software Engineering, 36:763–777, 2010.

[28] D. C. Schmidt, M. Stal, H. Rohnert, and F. Buschmann.Pattern-Oriented Software Architecture: Patterns for Concur-rent and Networked Objects, Volume 2. Wiley & Sons, NewYork, 2000.

[29] A. Srivastava and A. Eustace. ATOM: A System for BuildingCustomized Program Analysis Tools. In PLDI ’94: Proceed-ings of the ACM SIGPLAN 1994 Conference on ProgrammingLanguage Design and Implementation, pages 196–205, 1994.

[30] Sun Microsystems. JavaTM 2 Platform Enterprise Edition.java.sun.com/j2ee/index.html, 2007.

[31] A. Tamches and B. P. Miller. Using Dynamic KernelInstrumentation for Kernel and Application Tuning. Interna-tional Journale High Performance Computing Applications,13(3):263–276, 1999.

[32] Z. Tan, W. Leal, and L. Welch. Verification of InstrumentationTechniques for Resource Management of Real-time Systems.J. Syst. Softw., 80(7):1015–1022, 2007.

[33] P. Troger and A. Polze. Object and process migration in.NET. 24, 2009.

[34] D. R. Wallace and R. U. Fujii. Software Verification andValidation: An Overview. IEEE Software, 6:10–17, 1989.