managing the configuration complexity of distributed applications in internet data centers

IEEE Communications Magazine • March 2006166 0163-6804/06/$20.00 © 2006 IEEE

NETWORK AND SERVICE MANAGEMENT

THE CHALLENGES OFMANAGING THE DEPLOYMENT AND

CONFIGURATION OFDISTRIBUTED WEB APPLICATIONS

Enterprises today rely on Web-based applica-tions to deliver critical services to their cus-tomers and partners. These applications aretypically assembled out of a set of communicat-ing components hosted over distributed and het-erogeneous platforms. Resources, organized indata centers, may be shared across applications,and potentially across customers. At present, thedeployment and configuration of Web-based

applications is an ad hoc, human-intensive pro-cess that poses significant costs and risks to ser-vice owners.

For example, deploying a J2EE-based Webapplication in a typical enterprise data centerrequires:• Installing and configuring Web modules,

EJB modules, and database tables acrossdifferent middleware containers

• Organizing middleware containers in appro-priately selected clusters and configuringload balancers

• Deploying messaging servers and configur-ing message topics and queues for commu-nicating EJB modules

• Deploying LDAP servers, and configuringroles and security policy rules

• Configuring J2EE data sources andinstalling clients compatible with the back-end enterprise information systems

• Configuring firewall access rules to protectprocesses and data from unauthorizedaccess, without interrupting required com-munication channels

These complex operations must be performedwithout disrupting existing applications.

Configuration dependencies cut across soft-ware stacks, network layers, and middlewarecontainer boundaries. Indeed, it is these inter-dependencies, which require the operator toswitch between servers, tools, and knowledgedomains, that govern the complexity of the con-figuration task [1]. The deployed applicationmust satisfy operational and business require-ments such as connectivity and performance,use available resources, and follow data centerpolicies and best practices (e.g., security poli-cies) [2]. Often, the same application must bedeployed in multiple environments, such asdevelopment, staging, testing, and production,with differing sets of requirements. Web appli-cations go through constant changes, and thiscomplicates the problem even further. In arecent study on J2EE productivity, one-third ofthe respondents reported that they deploy new

Tamar Eilam, Michael H. Kalantar, Alexander V. Konstantinou, Giovanni Pacifici, and John Pershing,

IBM T. J. Watson Research Center

Aditya Agrawal, The MathWorks, Inc.

ABSTRACT

In this article we examine the challengesfaced by data center administrators when deploy-ing and configuring Web applications. We dis-cuss how the configuration dependencies ofthese Web applications cut across softwarestacks, network layers, and middleware containerboundaries. We argue that the deployment andconfiguration process requires the combinedexpertise from multiple domains such as applica-tion, middleware, network, security, reliability,and performance. We review the model-basedtools available today to manage the configura-tion complexity of these applications and intro-duce a new tool that extends the existing state ofthe art by automatically generating actionabledistributed deployment models using modeltransformation techniques. The key idea behindthis new tool is the principle of separation ofconcerns: developers capture the logical struc-ture of the application in a model, best practicesexperts define deployment model transformationrules, deployers specify the required deploymentpatterns, and an operator provides a modeldescribing the data center resources. The toolautomatically finds solutions based on these fourinputs and executes the deployment.

Managing the Configuration Complexityof Distributed Applications in Internet Data Centers

EILAM LAYOUT 2/14/06 11:15 AM 66

IEEE Communications Magazine • March 2006 167

applications at least once every two months, and50 percent of the applications in production arelater rolled back [3].

While software engineers enjoy a mature setof methodologies and tools to develop and testapplications, operations personnel lack compre-hensive solutions to deploy and manage dis-tributed applications. Today, data centeroperators depend on manual processes involvingcustom scripts, low-level commands, and a largenonuniform set of tools to manage applicationlife cycles.

Furthermore, the knowledge required forsuccessful design and implementation of deploy-ment solutions is distributed across organization-al boundaries. In particular, the functionalrequirements of the application, well-understoodby the developers, are not communicated effec-tively to the operational personnel responsiblefor deploying it. As a result, more than 35 per-cent of testing time is spent on provisioning andconfiguring a testing environment. Configurationerrors are the major cause of application down-time and loss of revenue, and IT managementcost dominates the total cost of ownership.

In this article we outline a novel model-based approach to managing the complexity ofapplication deployment. Our methodology wasdeveloped on the principle of separation ofconcerns. We define an architecture for shar-ing deployment knowledge in the form of mod-els and model transformations defined bydomain experts. We express mechanisms andalgorithms for combining and transformingsuch models manually, as well as automatical-ly, to derive actionable deployment topologies,which can be used to provision applications.Specifically, we describe emerging technolo-gies in this field. We present our approach tomanaging distributed Web applications. Wediscuss a proof of concept implementation,and outline results obtained. Last, we offerconclusions and discuss future research direc-tions and open problems.

DEPLOYMENT ANDCONFIGURATION TECHNOLOGIES:STATE OF THE ART AND BEYOND

To address the challenges of configuring andmanaging increasingly complex and mission-criti-cal data centers, IT organizations are seekingtechnologies to automate the configuration man-agement of their data center resources, ratherthan relying on manual workflows, one-offscripts, and low-level commands. Several man-agement software providers have already incor-porated configuration automation technologiesinto their products (e.g., IBM’s Tivoli Provision-ing Manager [4], Sun’s N1 [5], HP’s Utility DataCenter [6], Microsoft’s DSI [7], Opsware [8], andBladelogic [9]). Administrators use these prod-ucts to automate the provisioning of servers andoperating systems, configure the network, anddeploy and configure applications. These existingprovisioning and configuration managementsolutions use either scripting or model-basedtools, or a combination of both. Scripts are pro-

grams that describe a sequence of fine-grainedconfiguration actions that must be executed toprovision an application. Scripts are specific tothe deployed application and the target environ-ment. The process of writing scripts is manual,complex, and error-prone. Techniques such astemplates and profiles are used to facilitate net-work provisioning script composition [11]. Othertechniques [10] attempt to generate softwareprovisioning scripts dynamically based ondetailed request-for-change structures thatinclude temporal dependencies.

Model-based tools define and externalize theapplication and data center configuration statein a model that IT administrators can use forvisualization, provisioning, configurationupdates, and configuration drift audits. Usingmodel-based tools, administrators provision andconfigure a new application by either replicatingthe configuration of a previously deployed appli-cation, or by manually describing a detaileddesired configuration state and driving provi-sioning from it. While the current state of theart limits model based application provisioningto single servers, some tools model distributedenvironments as composite resources capturedin hierarchical models that embed constraints[12]. To design and provision a new environ-ment, these tools automatically assemble com-posite resources dynamically out of smallerparts by solving a constraint satisfaction prob-lem. An emerging new class of model-basedtools aims to bridge the information gapbetween developers and deployers. These newtools capture the data center structure and con-straints in a model used to guide developmentand packaging of applications. The tools alsocapture the application’s structure and function-al requirements in models used to guide deploy-ers in deploying them [13]. See [14] for acomplete review of related work.

We believe that representing configurationknowledge in object models will offer significantadvantages and will be the basis for the nextgeneration of configuration management tools.Models provide easier visualization, conceptual-ization, extensibility, componentization, stan-dardization, and reuse. Most technologies in thisspace are moving toward model-based solutions.

In this article we discuss a new approachthat extends the current model-based tools toautomatically generate actionable distributeddeployment models using model transformationtechniques. In this approach deployment rulesand best practices are captured as a collectionof fine-grained model transformations. Weshow how this approach can leverage an auto-mated search algorithm that applies the trans-formation rules on an input model of theapplication to produce a complete, correct, andactionable deployment model in a distributedenvironment. Key to this approach is the princi-ple of separation of concerns: developers speci-fy logical application structures, best practicesexperts define deployment model transforma-tion rules, deployers specify the requireddeployment pattern, and an operator provides amodel describing the data center resources.The automated search finds solutions based onthese four inputs.

Model-based tools

define and

externalize the

application and data

center configuration

state in a model that

IT administrators can

use for visualization,

provisioning, config-

uration updates,

and configuration

drift audits.

EILAM LAYOUT 2/14/06 11:15 AM Page 167

IEEE Communications Magazine • March 2006168

MODEL-DRIVEN DISTRIBUTEDAPPLICATION PROVISIONING AND

MANAGEMENT

This section presents the details of the emergingmodel-driven approach for deploying and config-uring distributed Web applications. Figure 1depicts its architectural components and theirinteraction with different IT administrationroles. The developer of the application producesa logical application structure (LAS). The LAScaptures the functional requirements of theapplication, including hosting, communication,and collocation requirements. The structure ofthe LAS is described in detail later. A deployerwill use the tool to create a logical deploymentmodel (LDM). An LDM represents the desireddeployment pattern of the application, includinga mapping to hosting systems and isolationzones. To speed the construction of the LDM,the deployer can use predefined logical topologymodels (LTMs). LTMs capture common deploy-ment patterns. The structure of LDMs isdescribed in detail later.

A deployment topology generator consumesan LAS, or an LAS extended to an LDM, as

well as a representation of the data centerresources, capabilities, best practice transforma-tions, and constraints. It produces a set of physi-cal deployment topologies representing possibleways to deploy the application. A physicaldeployment topology includes complete softwarestacks, servers, configuration parameter values,and a detailed network architecture, includingfirewalls, ACLs, switches, and virtual LANs(VLANs). The functional requirements of theapplication, deployment constraints, resourceconstraints, and data center policies are satisfiedby every deployment topology in the producedset. Thus, it is guaranteed that the data centercan be brought from its current state to thedesired state described in a deployment topologywithout violating any policy or constraint.

The generator automatically produces thedeployment topologies by iteratively applyingsequences of model transformation rules to thelogical deployment model. Model transforma-tions are defined by domain experts. They repre-sent elemental, factual deployment knowledgeincluding resource capabilities and dependen-cies. Transformations are pluggable components;the library of transformations the generator usescan be extended to support multiple manage-

nnnnFigure 1. Model-driven application deployment architecture.

Provisioning engine

Deploymentplan

Planner

Optimizer

Physicaldeployment

topology

Physicaldeployment

topology

Transformationrules

Transformationrules

Logicalapplicationstructure


Deployment topologygenerator

(search engine)


Optimizationcriteria

Developer

Logicaldeployment

model

Physicaldeployment

topology

Physicaldeployment

topology

Deployer Businessuser

Logicaltopology

modelDeploymentexperts

Transformationrules

Datacentermodel

<<discovery>>

Domainexperts

Data centeroperator

A deployment

topology generator

consumes a LAS,

or a LAS extended to

an LDM, as well as a

representation of

the data center

resources,

capabilities, best

practice

transformations,

and constraints.

It produces a set of

physical deployment

topologies

representing possible

ways to deploy the

application.



ment domains (e.g., software, network, and stor-age) and technologies. The generator leveragesthe library of transformations as building blocksto produce deployment topologies for differentapplications in multiple data centers. See latersections for a detailed description of transforma-tions, and a description of transformation-basedsearch.

The set of valid physical topologies may befurther evaluated using an objective analyzerbased on, for example, cost, relative perfor-mance, or configuration time. The optimizationpolicies will be set by a business user.

The second stage of the process consists ofdriving automatic provisioning actions to config-ure the data center with the selected physicaldeployment topology. Our tool utilizes a plan-ning algorithm that consumes the physicaldeployment topology and automatically producesa deployment plan. The deployment plan consistsof a parallel workflow that the tool’s provision-ing engine executes to automate the deploymentand configuration.

Both search engine and planner use informa-tion from the data center model (DCM) to makedecisions. The DCM is an object relationshipmodel that describes the data center resources,their current configuration, availability status,and rules and constraints governing configura-tion changes. Data centers vary in the level offlexibility they allow in the configuration ofresources. For example, the number of networkinterface cards (NICs) attached to servers is ahard constraint. In some data centers the infra-structure may already be organized in a fixednetwork structure with isolation zones, separatedby firewalls. The generator uses the DCM toproduce deployment topologies that represent adesired state that can be reached from the cur-rent state without violating any of the con-straints. The planner produces a partial order ofbound provisioning operations that can be exe-cuted to implement the change.

In this article we focus on the first stage ofthe process: automatic generation of physicaldeployment topologies using model transforma-tion techniques.

LOGICAL APPLICATION STRUCTUREA logical application structure describes thecomponents of a distributed application togetherwith their properties and functional require-ments. Application developers create the LASwith tooling assistance. Application deployers

use the LAS to deploy the application (in differ-ent configurations). In this section we describe atypical LAS structure. Figure 2 shows an exam-ple LAS of a simple three-tier J2EE application.

An LAS is a model consisting of nodes andrelationships. Nodes represent application mod-ules and their requirements, while relationshipsexpress hosting and connectivity dependenciesbetween modules. In Fig. 2 four software mod-ules, TradeWeb, TradeEJB, TradeDS, and Trade-DB, represent, respectively, a Web component,an EJB component, a J2EE data source configu-ration, and a database.

We associate three types of requirementswith software modules in a LAS: hosting, com-munication, and collocation. Module A is hostedby a module B if B serves as A’s runtime con-tainer. In other words, B cannot exist as a run-time entity without A. The host association isrecursive. An EJB module must be hosted by anapplication server; an application server is, byitself, a module that must be hosted by an OScontainer. Every module has a container. Thecontainer of an operating system is considered tobe a server. In the LAS, we connect nodes tosoftware modules that represent their hostingrequirements.

A hosting requirement defines the type ofservice that must be provided by a hosting con-tainer. In Fig. 2 TradeEJB, TradeWeb, TradeDS,and TradeDB are associated with hosting require-ments of type ServletEngine, EJBContainer,JDBCProvider, and DB_Instance, respectively.The type of service provided by the containermay be further restricted by associating proper-ties with the hosting requirement. In Fig. 2 thedatabase container is restricted by vendor family(IBM DB2) and version (8). The LAS can alsospecify requirements on indirect containers.These requirements are defined by setting thehosting requirement’s deferred attribute to true.In the figure, TradeEJB states a requirement thatthe OS container, down the hosting stack, mustbe of type RH9.

Software modules need to communicate inorder for the application to function properly.We define communication requirements by con-necting the respective modules with a relation-ship of type communication. The type andproperties of the required communication canbe further specified. For example, in Fig. 2,TradeDS is connected via a communication rela-tionship to TradeDB. The LAS further specifiesthat the type of required communication is

nnnnFigure 2. Logical application structure example.

Communication

TradeWebTradeEjb

host req: OShost req: EJBContainerversion: 1.4

deferred: true

family: RH9

host req: ServletEngine

TradeDS

host req: JDBCProvider

Communication

TradeDB

host req: DB_ instancefamily: DB2version: 8

Communicationreq: DS2DB

Collocationtarget:EJBContainer



DS2DB. This requirement indicates that a struc-ture supporting a communication between aJ2EE data source and a database must be con-structed in any deployment of the application.Properties attached to the communicationrequirement nodes may further restrict the com-munication by requiring, for example, securityfiltering or encryption.

Software modules may need to be collocatedon the same container in order for the applica-tion to function properly. We define collocationrequirements by connecting the respective mod-ules with a relationship of type collocation. Therelationship further specifies the target containerfor the collocation in the containment hierarchy.For example, in Fig. 2, TradeEJB and TradeDSmust be collocated on the same EJB container.In other cases, modules may need to be placedon different containers for proper functioning ofthe application. We define this requirement,similar to the collocation requirement, with arelationship of type anticollocation.

Interdependent configuration properties, suchas the database host name and port, must alsobe set on modules and their direct and indirectcontainers.

LOGICAL DEPLOYMENT MODELA logical deployment model describes thedeployment policy requirements and constraintsof a distributed application. Deployers createLDMs as an extension of the LAS. A given LAScan be extended to an LDM by different deploy-ers in many ways, reflecting different deploy-ment choices. In creating an LDM, deployersmay choose to be very specific, selecting specificsoftware containers and servers, or may specifyhigh-level deployment choices, such as tiers,clusters, and zones, and let the tool resolve thedifferent choices using an automated search. Inthis section we describe the structure of an

LDM. Figure 3 shows an example LDM basedon the LAS described earlier.

A hosting environment is a software moduleor physical server that is capable of hostingother modules. In an LDM we define a hostingenvironment as a module that is connected to ahosting capability node describing the type ofhosting capability provided. Similar to the LAS,hosting environments may be associated withtheir own hosting requirements in addition toproviding hosting capabilities. In the LDM ofFig. 3 there are three nodes representing logicalserver hosting environments, all associated withhosting capability nodes of type NODE.

We define the requirements on the place-ment of modules in hosting environments usingtwo kinds of hosting relationships. The host rela-tionship prescribes direct containment of a mod-ule in the related hosting environment. ThedeferredHost relationship expresses an indirectcontainment requirement on a module down thehosting stack. This structure is semanticallyequivalent to defining in the LAS collocationand anticollocation relationships. For example,Fig. 3 shows the three LAS modules TradeWeb,TradeEJB , and TradeDB connected using adeferred-host relationship to three separatehosting environment instances of type Node. Asa result, the application modules will not be ableto share the same logical node or, by extent, thesame operating system or J2EE container. Themodel transformations described earlier and, inparticular, the software insertion transformationclass are responsible for enforcing this policy.

We define an isolation zone as a logical con-tainer for servers. In many cases, data centeradministrators define a fixed allocation of serversto zones, with policies governing communicationbetween zones. Firewalls are statically config-ured to filter the communication between pairsof zones based on these policies. In the LDM we

nnnnFigure 3. Logical deployment model example.

host cap: Node

deferredHost

Logicalserver

Zonename: Data

Communication

TradeWebTradeEjb

host req: OShost req: EJBContainer

version: 1.4

deferred: true

family: RH9

host req: Servlet engine

host cap: Node

deferredHost

Logicalserver

Zonename: Web

host cap: Node

deferredHost

Logicalserver

Zonename: App

TradeDS

host req: JDBCProvider

TradeDS

host req: DB_ Instancefamily: DB2version: 8

Communication Communication

req: DS2DB

collocationtarget:EJBContainer



define a mapping to zones by connecting hostingenvironments with nodes representing zones. InFig. 3 the leftmost logical server hosting environ-ment must be placed in a zone identified by thename Web.

DATA CENTER MODELThe DCM is an object-relationship instancemodel describing the data center resources, theircurrent configuration, availability status, andrules and constraints governing configurationchanges. Resources described in the DCMinclude physical devices such as servers, routers,and load balancers, installed and installable soft-ware assets with their hosting requirements andcapabilities, logical groupings such as securityzones, spare pools, and resource sharing policies.DCM resource instance types are associated withlogical provisioning operations, such as “createVLAN,” “add server to load-balancer,” and “setinterface IP address.” As mentioned earlier, datacenters vary in the level of configuration flexibil-ity they allow. Some data centers are organizedin a fixed network structure including firewallsand security zones. These constraints can bemodeled in the DCM by identifying and annotat-ing the fixed objects, attributes, and relation-ships. The search engine consults the DCM tofilter out physical deployment topologies thatcannot be realized subject to the DCM config-urability constraints. This can be done by vali-dating that the fixed parts of the models can bematched.

MODEL TRANSFORMATION RULESThe search engine applies transformation ruleson an input LDM to generate an output deploy-ment topology. At each step, the transformationrules define how elements in the logical deploy-ment model should be changed to be iterativelyconverted into elements representing resourcesthat can be physically configured and deployed.

Domain experts define transformation rulesto capture factual, fine-grained deployment andconfiguration knowledge and best practices. Dif-ferent sets of transformations can be defined tocapture knowledge pertaining to different man-agement domains and technologies. By leverag-ing a library of transformations, the searchengine can generate deployment topologies thatinclude the technologies captured by the trans-formations. For example, utilizing a library ofnetwork transformations, the tool will be able togenerate detailed network architecture from ahigh level server connectivity diagram. Adding tothe library a set of transformations representingsoftware products and their capabilities anddependencies will enable the tool to generatedeployment topologies that include dynamicallycomposed software stacks that satisfy the inputapplication requirements in addition to thedetailed network architecture.

Each transformation rule selects a subset ofnodes and relationships in the LDM and trans-forms the selected nodes by replacing them withnew model elements or modifying their proper-ties. A transformation may resolve some of theunsatisfied requirements, and may introducenew requirements that will have to be resolvedby subsequent transformations in order to gener-

ate a complete and valid physical deploymenttopology. For example, a transformation mayinsert a WebSphere Application Server moduleto satisfy the EjbContainer hosting requirementof an EJB module. This transformation will alsointroduce a new hosting requirement of type OSon the inserted WebSphere Application servermodule.

In this section we formally define the struc-ture of transformations. We first give a formaldefinition of transformation rules and then pro-vide a number of examples that show differenttypes of transformations and their effects on thelogical deployment model.

Let M = Min ∪ Mout ∪ Mint be the class of alldeployment models, including the class Min ofinput logical deployment models, the class Moutof output physical deployment models, and theclass Mint of intermediate models that are theresult of applying transformations to input logi-cal models, but are not yet valid physical deploy-ment topologies.

A transformer is a parameterized function τ(M, e1, … , en) that takes a model M ∈ M and asubset of model elements e1, … , en, where e1, …, en ∈ M, and it creates a new model by copying,adding, deleting, or altering elements in M.Every transform is associated with a precondi-tion ρ that must hold on the elements e1, … , enfor the transform to be valid. We term the pair(ρ, τ) a transformation rule, and we denote thatwith δ. A collection ∆ of transformation rules δ1,…, δm is used by the search engine to automati-cally produce physical deployment topologies.

Figure 4a is an illustrative example of a simpletransformation δj ∈ ∆. The precondition ρj statesthat model elements e1 and e2 are model nodesthat must be of types A and B, respectively. Fur-thermore, these nodes must be connected by arelationship e3 of type X. Finally, attribute f oo ofelement e1 must have the value bar. The effectof the transformer ρj is shown on the right ofFig. 4a. In this case the transform ρj has createda new node of type C and connected it to nodese1 and e2 through relationships of types Y and Z,respectively. The relationship e3 was deleted.Figure 4b shows the application of the transfor-mation rule δj to a particular model M. Notethat there are two possible bindings of modelelements to the transformation parameters. Theresult of applying the transformation for one ofthe bindings (e1 = a1, e2 = b1, e3 = x1) isdescribed on the right of Fig. 4b.

In the rest of this section we describe severalexamples of typical transformations, and showan application of a transformation sequence. Weinformally use graphical illustrations similar tothe example in Fig. 4 above to help illustrate thetransformations. The graphical representationonly partially describes the semantic specifica-tion of transformations. A graphical pattern isnot enough by itself to determine whether it isvalid to apply a transformation, and in particularhow to avoid recursion.

Figure 5a, shows an example of a class ofsoftware insertion transformations (insert-JDBC-Type2-Driver). Transformations in this classresolve hosting requirements of software mod-ules by inserting software modules that providematching hosting capabilities into the current

Resources described

in the DCM include

physical devices such

as servers, routers,

and load balancers,

installed and

installable software

assets with their

hosting requirements

and capabilities,

logical groupings

such as security

zones, spare pools,

and resource

sharing policies.


model. The new software modules may intro-duce new hosting requirements that will have tobe resolved by subsequent transformations. Iter-atively applying transformations in this set hasthe effect of dynamically composing the entiresoftware stack from the application modulesdown to the operating system and servers. In thisexample the tool adds a JDBC type 2 driver tohost a J2EE datasource. The transformationintroduces a new hosting requirement of typeEJB_CONTAINER. The tool has several trans-formations of this kind, one for each type ofsoftware that can be inserted. A more complicat-ed version of the software insertion transforma-tion is applied when the software module isconnected to a container through the deferred-Host relationship. In that case the transformfunction will take care of updating the deferred-Host relationship. Application of this transfor-mation will guarantee that the placementdecisions specified by the deployer are respect-ed.

Figure 5b shows an example of a class of soft-ware connectivity transformations (datasource-to-database-connectivity). Transformations in thisclass resolve communication requirementsbetween software modules by introducing neces-sary modules and configuration elements. Aswith the software insertion class of transforma-tions, they may also introduce new requirementsthat will have to be resolved by subsequenttransformations. In this example the transforma-

tion’s precondition identifies a structure thatincludes a database table hosted by a databaseinstance, and a data source hosted by a JDBCprovider of type 2, where the database and datasource are connected via a communication rela-tionship. The transformation resolves the com-munication requirement by introducing a newmodel node representing a database client, andtwo new requirements: collocation between thedatabase client and the JDBC provider, andTCP connectivity between the database clientand the database instance. A similar transforma-tion, not shown, describes an alternative, withouta database client, when the data source is hostedby a JDBC provider of type 4. As with the soft-ware insertion transformations, there is a soft-ware connectivity transformation for eachmechanism to implement communicationbetween two types of software.

The firewall-insertion transformation express-es a mechanism for physically implementing asecure logical connection between two servers(Fig. 5c). Other mechanisms for implementinglogical connectivity include an insert-vlan and aninsert-router transformation [14, 15]. The firstsimply places the two endpoints on the sameVLAN; the second inserts a router (and definesappropriate routes) between them.

Figure 5d shows an interface-consolidationtransformation. This transformation implementstwo required logical connections on a singleserver using a single physical network interface.


nnnnFigure 4. a) The structure of transformations; b) an example of a transformation application.

(b)

c1:C

M:

δj (M, e1, e2, e3):τj

ρj

(a)

foo = bar

e1 :A

e2 :B

e3 :X

e1 :A

e2 :B

:C

:Y

:Z

a1:Afoo=bar

a2:Afoo=bar

b1:B

x1:X x2:X

b2:B

c1:C

τj (M, a1, b1, x1):

c1:C

a1:Afoo=bar

a2:Afoo=bar

b1:Bz:Z

y:Y

x2:X

b2:B

The transformation

resolves the

communication

requirement by

introducing a new

model node

representing a

database client,

and two new

requirements:

collocation between

the database client

and the JDBC

provider, and TCP

connectivity between

the database client

and the database

instance.



The transformation rule introduces a new fire-wall to ensure that no unprotected connectionsare created as a side effect (the figure does notshow the rules defined on this firewall).

In the remainder of this section we describehow our tool applies transformations to an LASor an LDM. In this example we do not describehow the tool selects the transformation to apply;the heuristic-based selection is explained in thenext section. Consider the LAS defined earlier.We show a simplified version of this LAS in Fig.6a. In the figure we use a concise presentation,where, for example, we show hosting require-ments as text attached to modules rather than asa separate model node. Figure 6b shows thedeployment model obtained after we apply twosoftware insertion transformations: the insert-JDBC-Type2-Driver transformation describedabove and a similar transformation that satisfiesthe hosting requirement of the database table byinserting a database instance of type DB2. Notethat both transformations introduce new hostingrequirements that must be satisfied by subse-quent software insertion transformations. Figure6c shows the deployment model obtained afterwe apply the datasource-to-database-connectivitytransformation (described above) to the deploy-ment model. The transformation introduced anew model node representing a DB2 client, a

new collocation relationship between the DB2client and the JDBC provider, and two new com-munication relationships of type TCP.

To continue with this example, Fig. 6d showsa deployment model that contains full softwarestacks where we have resolved all hostingrequirements. We obtained this deploymentmodel by iteratively applying a sequence of soft-ware insertion and software connectivity trans-formations to the deployment model in Fig. 6c.For brevity, we omitted details such as resolvedhosting requirements. The model describes therequired communication pattern between thelogical servers. Figure 6e shows the deploymentmodel after two applications of a transformationtermed vlan-insertion (we do not show the soft-ware stacks because they are not affected by thenetwork transformations). Two model nodesrepresenting VLANs are inserted between logi-cal servers 1 and 2, and between logical servers 2and 3. These VLANs are necessary in order toimplement the required communication pattern.Because the required communication patterndoes not contain mapping to zones or securityrequirements, VLANs are sufficient. In a differ-ent situation, the firewall-insertion transforma-tion would have been applied. Now assume thatservers with two interfaces are not available inthe data center of this example. Then it is neces-

nnnnFigure 5. Example of transformations: a) the insert-JDBC-type-2-driver transformation; b) the data-source-to-database-connectivitytransformation; c) the firewall-insertion transformation; and d) the interface-consolidation transformation.

kind=DB_INSTANCE

kind=DATA

secure=true

kind=DB_INSTANCE

kind=DATA

kind=DB_CLIENT

req=TCP

req=TCP

target=OSkind=DATASOURCE

req=DS2DB

kind=JDBC_PROVIDERtype=2

kind=DATASOURCE

kind= DATASOURCEreq=JDBC_PROVIDERkind=JDBC_PROVIDERtype=2Req=EJB_CONTAINER

(a)

(c) (d)

(b)

θ

:Module

:Node

:Node

:LinkInterface

:Firewall

:LinkInterface

:SwitchPort

:Module

:Module

:Module

:Module

:Module communicationcollocation

:Module :Module

:Module

:Module

:Module

:Module

kind=DATASOURCEreq=JDBC_PROVIDER

kind=JDBC_PROVIDERtype=2

τ

θ

τ

θ

τ

θ

τ

com

mun

icat

ion

com

mun

icat

ion

com

mun

icat

ion

:Node :Vlan

:Node :Vlan :SwitchPort

:Vlan:Vlan :SwitchPort :LinkInterface

:Vlan:Vlan :SwitchPort :LinkInterface

:Node:Node :Firewall



sary to apply the interface-consolidation transfor-mation (described above). We show the result inFig. 6f where logical server 2 is now connectedto a single VLAN, and communication to logicalserver 1 is through a firewall.

TRANSFORMATION-BASED SEARCHWe use a search algorithm to automaticallytransform the input logical deployment topology

to a solution physical deployment topology. Thesearch iteratively applies transformation rules tothe current model, backtracking if necessary,until a physical deployment topology is reached.At each step the next transformation is selectedand applied. When all of the elements in themodel represent possible physical resources, thetool checks whether the deployment model isindeed implementable over the data center with-

nnnnFigure 6. Example of application of a sequence of transformations.

kind=JDBC type 2 Driverreq=EJB Container

kind=DB Tablereq=DB Instance

kind=DB Instancereq=OS

kind=Datasourcereq=JDBCProvider

TradeDS TradeDB

JDBCProvider DB2

TradeDB

communication communication

(a)

(b)

TradeWeb communication

kind=WebModulereq=ServletEngine

TradeEjb TradeDS

TradeWeb TradeEjbcommunication


communication communicationreq=DS2DBcollocation

target=EJBContainer

kind=EjbModulereq=EJBContainer

kind=JDBC type 2 Driverreq=EJB Container


kind=DB Instancereq=OS


TradeDS TradeDB

JDBCProvider DB2DB2 clientcollocation

communicationreq=TCP

target=OS

(c)

(d)

(e)

(f)

communicationreq=TCP

TradeWeb

TradeWeb TradeEjb TradeDS

JDBCProvider DB2 client

Linux RH9.0

WebsphereApp. server

Logicalserver 1

Logicalserver 1 VLAN Logical

server 2 VLAN Logicalserver 3

Logicalserver 1 VLAN

Switch port

Linkinterface

Switch port

LinkinterfaceFirewall

Logicalserver 2 VLAN Logical

server 3

TradeDB

Linux RH9.0

DB2

Logicalserver 3

Linux RH9.0

WebsphereApp. server

Logicalserver 2

TradeEjbcommunication


communicationcollocationtarget=EJBContainer

kind=EjbModulereq=EJBContainer

communicationcollocationtarget:EJBContainer

kind=EjbModulereq=EjbContainer

communicationkind:DS2DB



We use a search

algorithm to

automatically trans-

form the input

logical deployment

topology to a

solution physical

deployment

topology. The search

iteratively applies

transformation rules

to the current

model, backtracking

if necessary, until a

physical deployment

topology is reached.



out violating any of the data center constraints.This can be done by applying a matching algo-rithm that matches the fixed parts of both mod-els (e.g., servers and their NICs or, in othercases, fixed network structure) and ignores theparts that can be configured dynamically (e.g.,software stacks). In our prototypical implemen-tation the matching algorithm is a pluggablecomponent, where we implemented generic andoptimized matching algorithms. The searchreaches a dead end if no model transformationcan be applied and the current deploymentmodel cannot be mapped on to the infrastructureIn this case, the search backtracks and try toapply different transformations.

The search execution can be described as atree, where every tree node t corresponds to adeployment model Mt, and the links from a nodeto its descendants correspond to valid transfor-mation instances (w.r.t. Mt). A naive algorithmwill blindly attempt to select a transformation,bind, and execute it, backtracking if a dead endis reached. This approach suffers from poorsuperexponential performance. In this section wediscuss some of the challenges in designing effi-cient transformation-based search algorithms,and the approach we took in our prototype.

SEARCH HEURISTIC ALGORITHMSHeuristic search algorithms operate by rankingsearch branches based on the likelihood thatthey will lead to a solution. Search paths that areknown not to lead to new solutions are pruned.These algorithms take advantage of semanticknowledge of the set of transformations to makeranking and pruning decisions.

In our prototype implementation we useddependency knowledge between transformationsto make pruning decisions. For example, if weknow that to reach a physical topology one canapply all transformations of a certain type beforeall transformations of a different type, then allsequences of transformations that do not obeythis rule are pruned.

Thus, in our heuristic search we apply soft-ware insertion and software connectivity trans-formations before all of the networktransformations because they create require-ments on firewall rules that are handled by thenetwork transformations. Interface consolidationtransformations are applied before the serverselection and firewall selection transformationsbecause they change the server and firewallselection criteria. Firewall consolidation trans-formations are applied before firewall selectiontransformations for similar reasons. By fixing theorder between transformation categories andpruning all search paths that do not obey thisorder we improved dramatically the performanceover the naive approach.

The weakness of the approach we took in ourprototype implementation is that it assumesknowledge of the semantics of the transforma-tions. As a consequence, when the transforma-tion set is extended, the search logic will have tobe changed to leverage the new transformations.Ideally the search will be agnostic and indepen-dent of the set of transformations used. Someideas to achieve this property include analyticaldeduction of the dependencies between the

transformations in cases where transformationsare defined declaratively (using graphs, or logic),or, employing learning algorithms to examinesearch executions, to discover the dependencieseven in the case where transformations areopaque. Studying these approaches is a subjectof future work.

PROTOTYPE IMPLEMENTATIONTo study the feasibility of the model-basedapproach we have built a prototype of the modeltransformation search engine.

In our system transformations are definedusing a combination of declarative and impera-tive methods. A precondition of a transforma-tion rule can be defined declaratively as anobject-relationship pattern. Parameters of a validtransformation instance are bound to model ele-ments that isomorphically match the pattern.Our transformation engine declaratively evalu-ates preconditions by employing a graph isomor-phism algorithm. Transform functions areimplemented as a mix of models and Java code;models are used in part to easily define whatnew elements need to be inserted; Java is usedto define how existing elements need to be mod-ified. The Java portion of the transforms iscoded using a rich application programminginterface (API) provided by a modeler toolkitlayer. We implemented a few dozen transforma-tions capturing software and network deploy-ment and configuration knowledge.

A search engine controls the transformationprocess based on four inputs: a logical deploy-ment model, a model of the data center, a trans-formation set, and a search heuristic. The engineuses the modeler transaction API to checkpointand roll back model search states. It produces aset of deployment topologies that representdeployment and configuration solutions includ-ing infrastructure resource bindings. We experi-mented with a set of heuristic search algorithms,based on the principles outlined earlier.

A graphical user interface facilitates control

nnnnFigure 7. Search trees.

M

δ4(δ1(M))

δ1(M)

δ4

δ1 δ2δ3

Deploymenttopology 3



Deadend



and visualization of application deploymentdesign. The visualizer provides different views ofthe deployment topology. For example, in thesoftware view of a deployment topology mostnetwork details are hidden, and the user canchoose to show or hide configuration properties;in the network view the software stack is notshown, and the user can optionally hide suchdetails as routes and ACLs. The system alsoincludes a comprehensive set of tools to createand edit models, and debug transformations andsearch heuristic algorithms. These tools wereused extensively to implement the set of trans-formations and search algorithms. For a detaileddiscussion of our prototype see [14].

CONCLUSIONSDeployment and configuration management ofJ2EE applications is a complex, manual, anderror-prone process that poses significant risksto data center operators and owners. Configura-tion errors are the major cause of applicationdowntime and loss of revenue, and IT manage-ment cost dominates the total cost of ownership.The industry is rapidly adopting automationtechnologies to aid in the management of datacenters. There is a consensus in the industrytoday that such technologies must be driven bymodels rather than scripts.

In this article we outline a novel model-driv-en approach to deployment and configuration ofdistributed J2EE applications. Our main contri-bution is a method to automatically generateactionable distributed deployment models usingmodel transformation techniques. We captureknowledge of elemental deployment best prac-tices as a collection of model transformationrules. Then we provide an automated searchalgorithm that applies the transformation ruleson an input model of the application to producea complete, correct, and actionable deploymentmodel in a distributed environment.

Our approach offers the following advantagesover existing solutions. Automatic weaving ofconcerns reduces the complexity of applicationdeployment through the reuse of transformationsin the design of different applications, over vary-ing infrastructures. The process permits domainexperts to define transformations in their area ofexpertise, shifting complexity away from the endapplication deployer, resulting in reduced staffingrequirements, and improved separation of con-cerns and knowledge reuse. Model transforma-tions and planning replace the manual process ofwriting and adapting scripts with an automatedprocess driven by higher-level operational andpolicy constraints. Application service providersmay employ sophisticated objective function ana-lyzers to select revenue optimizing topologies outof the several alternative physical deploymenttopologies that satisfy all constraints.

ACKNOWLEDGMENTThe authors would like to thank Andrew Tross-man for providing insight and vision that helpedus shape our ideas. The authors would also liketo thank Hendrik Wagner, Harald Daur, andGerd Breiter whose early work on decouplingapplication representation from provisioning

logic led us to explore this research direction.Many thanks also to Guerney D. H. Hunt, LilyMummert, and Liana Fong, who have contribut-ed to this work with many comments and insights.

REFERENCES[1] A. B. Brown, A. Keller, and J. L. Hellerstein, “A Model of

Configuration Complexity and Its Applications to aChange Management System,” IEEE Int’l. Symp. Inte-grated Network Mgmt., 2005.

[2] P. Mehra, “Global Deployment of Data Centers,” IEEEInternet Comp., vol. 6, no. 5, Sept. 2002.

[3] T. Lanowitz, Mercury User Conference presentation,2004.

[4] IBM Tivoli Provisioning Manager (http://www- 306.ibm.com/software/tivoli/products/prov-mgr/)

[5] N1 — Grid Service Provisioning System Solution forApplication Management, http://www.sun.com/soft-ware/whitepapers/service provisioning/

[6] HP Utility Data Center. http://h71028.www7.hp.com/enterprise/cache/259007- 0-0-225-121.html

[7] DSI, applications of model-based management,http://www.microsoft.com/windowsserversystem/dsi/models.mspx

[8] Opsware, http://www.opsware.com[9] BladeLogic, http://www.bladelogic.com/company/

index.html[10] F. Shen and A. Clemn, “Profile-Based Subscriber Ser-

vice Provisioning,” IEEE/IFIP Network Ops. and Mgmt.Symp., 2002.

[11] A. Keller et al., “The CHAMPS System: Change Man-agement with Planning, and Scheduling,” IEEE/IFIP Net-work Ops. and Mgmt. Symp., 2004.

[12] S. Singhal et al., “Quartermaster — A Resource Utility Sys-tem,” IEEE Int’l. Symp. Integrated Network Mgmt., 2005.

[13] MSDN Magazine, “Bridge the Gap Between Develop-ment and Operations with Whitehorse,” http://msdn.microsoft.com/msdnmag/issues/04/07/whitehorse/default.aspxy

[14] T. Eilam et al., “Model-Based Automation of ServiceDeployment in a Constrained Environment,” Tech. rep.RC23382, IBM, Sept. 2004.

[15] T. Eilam et al., “Reducing the Complexity of Applica-tion Deployment in Large Data Centers,” IFIP/IEEE Int’l.Symp. on Integrated Mgmt., 2005.

ADDITIONAL READING[1] A. Felfernig and G. E. Friedrich, “UML as A Domain Spe-

cific Knowledge for the Construction of KnowledgeBased Configuration Systems,” 11th Int’l. Conf. Soft-ware Eng. and Knowledge Eng., 1999.

BIOGRAPHIESTAMAR EILAM ([email protected]) joined IBM Research as aresearch staff member in 2000, and she is currently a man-ager of the Model Driven Management TechnologiesDepartment. Her research focuses on developing model-driven technologies to deploy and manage distributedcomponents in constrained environments. She received herPh.D. from the Technion, Israel Institute of Technology,where she studied trade-offs between cost and efficiencyin communication networks.

ALEXANDER V. KONSTANTINOU ([email protected]) is a researchstaff member at the IBM T. J. Watson Research Center,Hawthorne, New York. His current research is focused onmodel-driven technologies for deployment of distributedmultitier data center applications. He has been awarded aPh.D. in computer science by Columbia University. His cur-rent research interests are in the areas of systems and net-work management, programming languages, tooling, andsecurity.

GIOVANNI PACIFICI [SM] ([email protected]) joined IBMResearch in 1995, and is currently a senior manager of theDistributed Systems Department. His research focuses onthe design, prototype, and evaluation of management sys-tems for cluster-based Web applications. He led theresearch team that pioneered the distributed resourcemanagement technology of IBM WebSphere ExtendedDeployment. Before joining IBM he was a research scientistat the Center for Telecommunications Research at ColumbiaUniversity. He received a Laurea in electrical engineering

The industry is

rapidly adopting

automation

technologies to aid

in the management

of data centers.

There is a consensus

in the industry today

that such technolo-

gies must be driven

by models rather

than scripts.



and a research doctorate in information science andtelecommunications from the University of Rome “LaSapienza” in 1984 and 1989, respectively. He was TechnicalProgram Co-Chair for IEEE INFOCOM 2001 and TechnicalProgram Co-Chair for the Fourth IFIP/IEEE InternationalConference on Management of Multimedia Networks andServices. He is an editor of IEEE/ACM Transactions on Net-working. He was an editor of Elsevier Science’s ComputerNetworks Journal, and a Guest Editor for two special issuesof IEEE Journal on Selected Areas in Communications. He isa member of ACM.

ADITYA AGRAWAL ([email protected]) is currentlyworking with The MathWorks on the Stateflow team. Here,he is working on the next generation semantics of State-flow and code generation technologies. Before joining TheMathWorks he worked at IBM T. J. Watson Research Cen-ter, where he researched the use of graph transformationsto automate the deployment of services on data centers.He received his B.E. in 2000 in computer engineering fromthe University of Pune, India. He received his M.S. andPh.D. in 2002 and 2004 in electrical engineering and com-puter science (EECS) from Vanderbilt University, Tennessee.His area of specialization is domain-specific modeling lan-guages. His dissertation was on graph transformation tech-

niques and their use in specifying model-to-model transfor-mations. In this endeavor he developed a graph transfor-mation-based language called GReAT (Graph Rewriting andTransformation Language) that has been used by academiaand industry to develop semantic translations.

JOHN A. PERSHING JR. ([email protected]) is a researchstaff member in the Internet Infrastructure and ComputingUtilities Department at the IBM T. J. Watson Research Cen-ter. He received his B.S. and M.S. degrees from the Mas-sachusetts Institute of Technology. He has been at IBMResearch since 1981, working on the systems aspects ofnetworking, system coupling, distributed systems, large-scale clustering, high availability, dynamic provisioning,and, most recently, utility computing and the on-demandinfrastructure. He is the author or co-author of severaltechnical articles and over 20 patents.

MICHAEL H. KALANTAR ([email protected]) has been withIBM Research since 1997. His research interests include dis-tributed systems, application provisioning, and automation.He received his B.Sc. from the University of Alberta in1989, and his M.Sc. and Ph.D. from Cornell University in1992 and 1995, respectively. He spent two years teachingcomputer science in China.

IEEE COMMUNICATIONS MAGAZINECALL FOR PAPERS

FEATURE TOPIC: NETWORK CENTRIC MILITARY COMMUNICATIONS

It has been several years now since the vision of Network Centric Warfare (NCW) first appeared on the horizon as the meansfor moving and using information within and among the militaries of the world. While the utility of NCW is self-evident,the complexity of communicating over mobile, tactical, inter-networked communication links built on inherently low-pre-dictability RF communication links demands fundamental advancement in the state-of-the-art of network engineering.While the relatively highly-predictable (at the physical layer) world of wired networks has been calling for the developmentof a theory of networks for some time, the challenges of military networked communications that demand robustness, scal-ability, and reliability globally, even in the face of hostile attacks, calls for a systematic, tractable theoretical foundation onwhich to build. While the application of existing commercial protocols are essential to migrate from the existing deployedcommunication systems of today, we are called to develop the next generation theory, protocols and architectures to fieldthe future systems that can meet the demands of tactical military communications. Suggested areas include (but are notrestricted to) the following subject categories:

Development of a Theory of Networks Distributed / Achievable QoSCross-layer optimization Freespace Optical CommunicationsCross-layer coherence length Distributed robust security certification

Designing for low predictability networks Robustness against hostile interferenceNetwork characterization Network adaptivityProtocol efficiency on bandwidth constrained links Dynamic spectrum allocation and managementNetwork mobility Rapid technological changeRobust signal provisioning and recovery Multi-national security issuesDispersive networks Airborne networksAlternatives to end-to-end network design Space-based networksDiversity in network security and encryption

SUBMISSION

Articles should be tutorial in nature and be of direct interest to those engineering military communication systems. Theyshould be written in a style comprehensible to readers outside the specialty of the article. Mathematical equations shouldnot be used (in justified cases up to three simple equations could be allowed, provided you have the consent of the GuestEditor. The inclusion of more than three equations requires permission from the Editor-in-Chief). Articles should not exceed4500 words. Figures and tables should be limited to a combined total of six. Complete guidelines for prospective authorscan be found at: http://www.comsoc.org/pubs/commag/sub_guidelines.html. Please send PDF (preferred) or MSWORD for-matted papers by April 1, 2006 to Manuscript Central (http://commag-ieee.manuscriptcentral.com), resister or log in, and goto the Author Center. Follow the instructions there. Select the topic “Nov 2006/Network Centric Military Communications”.

SCHEDULE FOR SUBMISSIONS

Submission Deadline: April 1, 2006Notification of Acceptance: July 1, 2006Final Manuscript Due: August 15, 2006Publication Date: November 2006

GUEST EDITORS

Torleiv Maseng Chris NissenForsvarets Forskningsinstitutt The MITRE CorporationEmail: [email protected] Email: [email protected]


managing the configuration complexity of distributed applications in internet data centers

Documents