Dynamic Resource Management in an eUtility

L.L. Fong, M. Kalantar, D. P. Pazel, G.S. Goldszmidt, S. Fakhouri, S. Krishnakumar IBM T. J. Watson Research Center

Abstract Océano is management software for a eUtility infrastructure capable of providing cost-effective, autonomic resource allocation for multiple customer or application domains, in response to existent performance and availability conditions. The Control Layer of Océano provides mechanisms to manage resources. This layer consists of a resource director and a set of resource managers. The resource director formulates resource configurations and coordinates their execution through resource managers. These resource managers maintain restore state and carry out detailed configuration tasks. This paper describes the Océano resource management model and its server and application deployment resource managers. A prototype of Océano has been developed and deployed on an 80 server platform, and has been tested with multiple domains and applications.

Keywords Computing utility, resource allocation, resource management, server management, operating system installation, application deployment

1 Introduction A computing utility, or eUtility, is a provider of computing services and resources on demand, subject to resource constraints. Providers and consumers of these services contract for a specific level of service[2]. Océano [1] is an umbrella project to build a system infrastructure for experimenting and developing eUtility technologies. In an Océano managed infrastructure, customers can contract for specific service quality and resource allocations, expressed as Infrastructure SLAs (ISLAs [7]), which include specifications of usage-based charges. Océano provides a cost-effective alternative to static resource allocation by providing dynamic resource allocation with automatic resource control. That is, allocations take place without human intervention in response to existent performance and availability conditions. For example, in a web hosting utility, Océano can dynamically allocate web servers based on workload fluctuation. Furthermore, Océano takes into account resource interrelations when making allocation decisions. For example, client requests

2

may be throttled due to database overload even though network bandwidth and web server capacity is plentiful.

Architecturally, Océano is composed of 3 software layers: an Infrastructure Layer, a Control Layer, and a Policy Layer. The Infrastructure Layer (See [8][15]) provides topology and configuration information, and a mechanism for detecting hardware failures. The Control Layer, which is the focus of this paper, provides the mechanisms to implement resource allocations, and to provide resource usage data. The Policy Layer (See [6][5]) is responsible for managing ISLAs and for determining when and what resources should be reassigned.

Océano implements a new resource management model that defines a new set of logical and physical resources relevant to a computing utility environment. In this model, a customer defines the scope of ISLA, and a customer segment defines resource ownership and secure protection boundary. A resource director [5] coordinates the dynamic allocation of available resources to a customer segment though resource managers (RMs) that encapsulate the automatic control and monitoring functions for resources with similar attributes. The resource director uses performance data collected through resource monitoring to identify when resources need to be assigned or reclaimed.

The rest of this paper is organized as follows. Section 2 describes the resource management model, Section 3 Server management, and Section 4 Application deployment. Section 5 describes the prototype implementation of Océano, Section 6 discusses related work, and Section 7 presents conclusions and future work.

2 Resource Management Model The goal of Océano resource management is to support dynamic allocation of resources enabling customer workloads to meet specific performance objectives and resource constraints. To effect allocation changes, resources are automatically reconfigured and instantiated for the target customer or application domains. Resources are also monitored for their usage and performance conditions. In the following sections, we will define the type of resources considered and their supported management functions for reconfiguration and monitoring.

2.1 Resources in a computing utility Resources in a computing utility depend on the type of utility considered. For example, in a storage utility the resources may include network storage devices while an application services utility might only consider application suites and their licenses. The Océano prototype considers resources generally present in a web hosting utility: web servers, application instances, request admission, VLANs, and backend database and file servers.

Resources have a set of attributes that describe their intrinsic characteristics. For example, the physical attributes of a server include processor speed, memory capacity, and local disk capacity. Resource attributes are

3

described by constant and varying values. For example, the attributes of a server include constant values like hardware characteristics and varying values like the operating system installed, the current CPU utilization, communication buffer usage, and online/offline status.

Resources may also be characterized as either physical or logical. Examples of physical resources include servers, storage devices, and network devices. A logical resource is an abstraction of non-physical entity, such as application instances and firewalls. For example, a logical disk drive might be instantiated on a disk partition of a physical disk, either local or remote.

Océano manages resources, both physical and logical, as either static or dynamic entities. When managed as a dynamic entity, the allocation, assignment, and the usage pattern of a resource can be automatically and dynamically changed. When managed as a static entity, a resource is statically allocated to a particular customer segment. For example, a web server may be used as a static resource allocated to a customer in a storage utility, or may be managed as a dynamic resource in a web hosting utility. In the latter case, the server allocated to one web site (customer segment) can be reassigned to another site (customer segment) later. In the Océano prototype, web servers, application instances, request admission, and VLANs are dynamic resources, while managed backend file and database servers are treated as static resources.

2.2 Resource Allocation In Océano, a resource director makes allocation decisions on the dynamic resources in response to operator commands or workload conditions[5]. Resources are assigned to a customer segment, defined to be a resource holder and security domain. Associated with each customer segment is a set of resource constraints and service objectives, which are expressed as ISLAs. A customer domain contains a set of customer segments. The domain may have a set of resource constraints and service objectives, which are inherited by customer segments. Customer domains are useful for recording resource usage. Figure 1 is a graphical representation of the relationships between a customer domain and its customer segments, each with their own set of ISLAs. The figure also illustrates a customer segment as a holder of resources that are, in turn, controlled by their respective resource managers.

2.3 Resource Managers Resource Managers (RMs) encapsulate the functions of allocating, configuring and monitoring resource instances sharing common attributes. Each RM is also intended to provide scalability by dividing the control of individual resources into groups. The RMs free the resource director to focus on meeting service objectives under dynamically changing workload conditions and the relational effects of allocating specific resource instances to different customer segments. Moreover, RMs enhance the extensibility of our resource model: new resource

4

types can be introduced by defining resource managers with appropriate supporting functions.

While RMs are passive entities that act upon receipt of external directives, they do have a certain degree of autonomy when satisfying these directives. For example, the server RM may choose which specific servers in a given pool to use in satisfying an allocation request.

In our current implementation, RM interactions are via asynchronous messages. We found that the following general form of interfaces suffice for the RMs we considered:

addRequest (targetCustomerSegment, resourceList | resourceAmount) addReply (returnCode [, resourceList]) removeRequest (CustomerSegment, resourceList | resourceAmount) removeReply (returnCode [, resourceList]) modifyAttributeRequest (attributeValuePair [,attributeValuePair]) modifyAttributeReply (returnCode)

For the prototype of Océano, we designed and implemented four RMs:

server, application deployment, network admission and VLAN resource managers. These resource managers functionally satisfied our objective of supporting dynamic allocation of servers and bandwidth with network security

Customer Domain

Customer segment

ConstraintsObjectives

ResourceDirector

Resources

Resource manager Resource managerResource manager

ConstraintsObjectives

Resource holder

ConstraintsObjectives

Resource holder

Figure 1: Customer segments hold resources (circles). These resources areallocated by the resource managers in response to resource director directives.The resource director takes into account system constraints and ISLA objectivesin making allocation decisions.

5

between customer segments in a web hosting utility. All four of the RMs manage dynamic resources.

The following scenario illustrates Océano’s interactions with RMs via the defined interfaces to carry out an allocation decision of adding a new server to a customer segment to alleviate overload on the existing servers. The resource director would send to the: 1. Server RM an addRequest for a server to a customer segment:

Server RM selects a server from the available server pool, prepares it with a fresh copy of the operating system, and starts the collection and publication of server utilization data.

2. Application Deployment RM an addRequest for instantiating an application instance on the selected server:

Application Deployment RM sets up application binaries and data on the server, and starts the collection and publication of application performance data.

3. VLAN RM an addRequest for the selected server: VLAN RM reconfigures the network switches to place the selected server into the appropriate VLAN.

4. Network admission control RM an addRequest for the selected server: Network admission control RM reconfigures the load balancing hardware or software for the added server, starts the collection and publication of data on request rate.

5. Network admission control RM a modifyAttributeRequest for admission rate:

Network admission control RM reconfigures the admission rates. The next two sections describe the design of the server and application

deployment RMs. Detailed description of the VLAN and network admission control RMs are not included due to lack of space, but are briefly described in Section 5.

3 Server Resource Manager The server RM supports allocation and deallocation of server resources to and from a customer segment. The server RM primes the allocated servers with the appropriate operating system environment and performs disk partitioning if necessary.

The server RM maintains information on the characteristics and current allocation status of all servers. When servers are requested for allocation, in either quantity or capacity increments, the manager identifies a suitable set of servers for the designated customer segment from its available pool. A request for servers may specify attributes such as hardware architecture, processing power, memory size, local disk capacity, and desired operating system type and release. If no appropriate servers are available for allocation, the server RM does not autonomously relocate servers allocated to other customer segments. Rather, it awaits directives from the resource director to identify those customer

6

segments from which servers can be relocated. This approach is necessary because the server RM is not aware of the possible performance and policy impacts of server relocation between customer segments.

When deal locating a server from a customer segment, the server RM selects a set of victim servers to be placed in a “dirty” pool. As an optimization, servers in the dirty pool can be reallocated to the same customer segment before their contents have been scrubbed. Most customers would desire no residual data and processes on relocating servers. Thus, servers in the dirty pool are scrubbed of their contents and a fresh copy of the operating system is installed before they are returned to the available “free” pool.

Automatic installation of the operating system and services is a key function for dynamic provisioning of servers. Customers may require different releases of system software on different hardware. Various hardware platforms and operating systems have different levels of support in automatic installation. System cloning and differencing[25] methods are popular and efficient for priming servers with pre-built system images in an operation with homogeneous hardware. To support Linux platforms, the Océano server RM relies instead on PXE[10], DHCP[17], and LUI[11], which provide support for all servers that support PXE. Using LUI functions, we dynamically bind a relocating server to an installation server and define a set of customized installation actions. Examples of customized installation actions include specifying the layout of disk partitions and optional service packages to install. Once the operating system is installed and in operation, the server RM initiates the collection and publication of server performance metrics, including processor utilization, memory usage, percentages of spare disk space, and communication buffer usage.

The server RM in Océano is currently implemented with a central controller, a thin agent on each deployable server, and one or more system installers. The controller coordinates the interactions with the resource director, and makes all the selection decisions. The thin agents initialize themselves at the end of a system reboot, and then “check-in” with the controller identifying information about their operating system type/release and last reboot time. When the check-in messages arrive, the controller will match them against its outstanding requests for OS priming. When a server is selected for allocation to a customer, the controller will direct an appropriate system installer to automatically install the operating system software and services. Figure 2 shows the relationships between the components involved in server installation. The server RM communicates with the installer and the local agents to prepare for a system install. The local agent causes the machine to reboot enabling a network installation of the operating system. Figure 2 also shows application deployment, described in the next section.

4 Application Deployment Resource Manager The application deployment RM is responsible for instantiating customer application software and data on servers. Obviously, the method of deploying an

7

application may be server and operating system dependent. We found it useful to functionally separate the deployment of applications from the installation of operating systems and services. This separation permits flexibility by allowing a variable number of application instances to be installed on the same server. More importantly, different techniques may be applied for operating system installation and application deployment. In our case, the operating system installation is based on LUI, as described above, while application deployment is based on a shared file system, described below.

Like the server RM, the application deployment RM is implemented using a central controller and local agents on each server. The controller interacts with the resource director to obtain application profiles for each customer segment. The appropriate profile is forwarded to a local agent, resident on the server on which applications and data are to be deployed. The local agent completes the installation.

Installing and configuring the applications at every deployment can be time consuming. Moreover, the automatic deployment of applications must be “silent”, not require human interaction. Finally, the deployment process needs to intelligently resolve dependencies on other applications that may reside either locally on the server or on back end servers.

server manager

multicast of cached databinaries

pxe/dhcp

application deployment manager

shared file server

installer

OS image

/proc data

Appl AgentApplication/Data

plugin

Monitoring Agent

Server Agent

OS image

/proc data

Appl Agent

Application/Data

pluginMonitoring Agent

Server Agentexisting server

OS image

/proc data

Appl AgentServer Agent

Monitoring Agent

Application/Data

plugin

Figure 2: Two new servers are allocated to a customer segment. Theinteractions of the server manager, the installer and the local server agent areshown. Further, the interactions of the application deployment manager, thelocal application deployment agent, the shared file server and the caches ofexisting servers in the customer segment.

8

The Océano application deployment RM addresses efficiency and automation issues by using a novel shared file system model. In this approach, the files that make up the applications are divided into two sets. One set is actively pushed to the server for local access. The other set resides on the file server. In practice, these files are typically read only and are cached by the servers. This technique can be viewed as a variation of the content delivery model in which application binaries and data are pushed to the client server from management servers.

Like the customer applications, data resides on the file server but will be cached by each server. The application deployment RM primes the local cache to avoid a large number of initial accesses to the file server. The success of priming depends largely on caching the right set of data. To do so, the application deployment RM uses the cache of peer servers. Both application and data files may be transferred in parallel (using multicast) to several new servers. Additional details on the application deployment RM are given in [9].

Figure 2 shows how the application deployment RM interacts with the local agents. These initiate installation of applications from a shared file server and the transfer of data from an existing server. In this example, two new servers are being prepared.

A part of application deployment is initiating appropriate monitoring of the applications. Monitoring application performance is necessary to identify bottlenecks and an application’s impact on a server’s capacity. For example, when performance conditions violate ISLAs, additional instances of an application may be deployed. Monitors may be internal or external to the applications. Some application and middleware software provides the ability to add code in the form of “plugins.” In Océano, we prototyped an Apache plugin that provided a number of application level monitors including the number of requests served, the output bandwidth utilized, the average response time for the requests, and average database access time per request. This data is collected and published as a list of (name, value) pairs.

5 Prototype Implementation A demonstration prototype of Océano has been developed and deployed on a testbed containing 80 Linux and AIX servers. The Linux servers are used as dynamic server resources, while the AIX servers are used as static server resources (for file systems and databases). These servers are interconnected using a 100 Mbps Ethernet switch. The prototype is written largely in Java. For testing purposes, the prototype implements a web hosting utility. Fictitious customers (web sites) were deployed in the test bed environment to illustrate the following possible applications: (1) static web pages, (2) dynamic web pages generated using Java Servlets accessing a back end database, (3) a streaming video server, and (4) the TPC-W benchmark [26][27].

9

5.1 Other Resource Managers We describe briefly the VLAN and admission control RMs. The VLAN RM controls network security by managing a set of VLANs to which a server may be connected. Each server may have several network connections. Each is a member of a VLAN allocated to a customer segment. The VLAN RM can automatically alter the VLAN membership of a server by programming the network switches. See [15] for additional details.

The Network Admission Control RM automates interactions with a request dispatcher. In the Océano prototype, its functions are implemented on top of Network Dispatcher [14]. This manager dynamically configures the appropriate instance of network dispatcher, adding and removing servers from a customer segment’s dispatch cluster as directed by the resource director. In addition, the manager implements a throttling function by either dropping or redirecting a portion of the incoming requests. In the prototype, redirected requests are sent to a specially configured “overflow server” which provides a simple error message.

Figure 3: Resource Allocation GUI.

10

5.2 Resource Manager Visualization A graphical user interface (GUI) depicts the activity of the entire system via a set of panels. The panel in Figure 3 shows all of the server resources as squares. They are arranged according to their assignment to either customer segments or to special pools. The Free List is the available servers pool. The Wharf contains servers that have failed and for which automatic recovery will be attempted. Finally, the Dry Dock contains servers that have been observed to have problems requiring human intervention. The “Admission” column shows the admission rate of requests for the customer sites.

Figure 4 shows the graphs of some of the metrics provided by the resource manager for customer segment “Mayflower”. The data provided by the server RM includes: the number of web servers allocated to the segment, the number of active TCP connections per server, the size of communication buffers (sk_bufs) used, the CPU utilization, the memory usage and the disk capacity use. The application deployment RM provides: the output bandwidth per server, the average response time per request per server as observed by the servers, the http Head response time (determined by an external monitor), and the average response time for database requests. In general, these data values are averages over all of the servers allocated to a customer segment. The horizontal lines

Figure 4: Sample performance data.

11

across the tops of some graphs are threshold values for those metrics. These threshold values can be used as triggers to cause allocation changes.

Our prototype is stable and a study of its performance is underway using the TPC-W benchmark. It focuses on the suitability of the measured metrics as triggers for resource allocation. As preliminarily steps, we measured the elapsed time between the resource director making a decision to allocate a server and the server being placed into operation. The total elapsed time was 130 seconds, including 115-120 seconds for the application deployment manger to deploy 120-160MB of data. In this case, the server RM did not install an operating system. For a 250MB Linux OS image, the server RM required about 420 seconds to network install a server. This includes the time for a required two system reboots. The first reboot is to transfer the new image from the installation server, while the second loads and executes the system from the new image on the local disk.

6 Related Work Océano customer segments have similarities to Resource Containers [3]. However, resource containers address only resources on a single machine. The Océano resource model is distinguished by the set of resource types considered, and the distributed nature of these resources. The class objects of Hosts and Vaults in the Legion project [24] encapsulated machine and storage physical resources. In contrast, we addressed both physical and logical resources. Moreover, Océano resource managers provide different configuration functions such as installing a fresh copy of operating system.

There has been much research in server matching and allocation algorithms, especially in the parallel computing area [23]. The algorithms developed by this work can be applied to the allocation functions of the server RM. Our work has the unique aspect of combining server allocation with automated installation of system.

Example techniques for silent application deployment are classified in [18]. These techniques included packaging managers like RPM [12], installers like InstallShield [19], application management systems like TME-10 Software Distribution, and Content Delivery like Castanet [20]. The application deployment RM presented here can be classified into the category of Content Delivery. Our approach based on offline installation and configuration using a file system is unique. Our multicasting of cached data using disk file images from one server to multiple servers during priming is also new. There are also emerging interests in the area of framework and protocol design for automatic configuration software deployment [22]. The work may ease the awkwardness of our offline installation and configuration process.

7 Discussion and Future Work To implement an efficient eUtility infrastructure, Océano requires automatic and dynamic resource allocation. To automate resource allocation, we developed a

12

resource model in which a set of resource managers is responsible for the allocation and deallocation of specific resources types, and a resource director that coordinates allocations and ensures that service objectives are met. We found that a relatively simple interface for all of the resource managers sufficed for an initial implementation. Applying this model to a larger set of resource types will be necessary to determine its generality. The design of an RM for network attached storage devices is a possible future extension. Moreover, our resource model currently focuses on web services. Applying the resource model to application services or mass storage services[16] would provide further insights into its generalization.

Océano currently manages the resources of a single server farm. Another direction for our future work involves applying Océano to multiple cooperating server farms. In a larger perspective, multiple server farms could interact in resource sharing. This “Hydroplex” scenario would make Océano a global autonomous distributed computing system. A number of control layer issues arise in this scenario, and new resource types would be needed. Also, the global and local control span of resource managers may require tiered structures

The topics of life-cycle management of application services and application-services deployment on tier architecture are of keen interest to us. In addition to the AFS based application deployment, we also pursue the approach that is based on FTP servers and application RPMs. While this method allows instantiation mechanism without the need for “offline” configuration onto a file system, it has the disadvantage of consuming additional time. Moreover, customer specific customization of standard packing will also need to be prepared offline.

8 Acknowledgements We are grateful to all the people who worked to design and develop Océano, including Karen Appleby, John Pershing, Tamar Eilam, Sandra Miller, Benny Rochwerger, and Arie Tal. Thanks also to Michael Frissora and James Norris who set up the test environment.

9 References [1] K. Appleby, S. Fakhouri, L. Fong, G. Goldszmidt, M. Kalantar, S.

Krishnakumar, D.P. Pazel, J. Pershing, and B. Rochwerger, “Océano - SLA Based Management of a Computing Utility,” In Proc. of the 7th IFIP/IEEE International Symposium on Integrated Network Management, May 2001.

[2] A. Balwin, M. C. Mont. “Policy Based Monitoring of a Web-based Services,” TR HPL-98-76, HP Laboratories Bristol, April, 1998.

[3] G. Banga, P. Druschel, and J. C. Mogul, “Resource Containers: A New Facility for Resource Management in Server Systems,” in Third Symposium on Operating Systems Design and Implementation (OSDI-99), February 1999, pp. 45-58.

13

[4] T. Kawamura, T. Hasegawa, A. Ohsuga, and S. Honiden, “Rental Application to Rental Service Development of Advanced ASP Framework,” in IEEE 4th International Enterprise Distributed Object Computing Conference (EDOC 2000), Sept. 2000, pp. 25-28.

[5] D. Pazel, T. Eilam, L. Fong, M. Kalantar, K. Appleby, and G. Goldszmidt, “Neptune: A Dynamic Resource Allocation and Planning System for a Cluster Computing Utility”, Submitted for Publication to CCGrid 2002.

[6] K. Appleby, G. Goldszmidt, and M. Steinder, “Yemanja – A Layered Event Correlation Engine for Multi-domain Server Farms,” in Proceedings of the 7th IFIP/IEEE International Symposium on Integrated Network Management (IM2001), May 2001.

[7] J. Cunha, F.Q.B. da Silva, G. Goldszmidt, and K. Appleby, “An Architecture to Define, Store, and Monitor ISLAs in Server Farm,” Proceedings of LANOMS, September 2001.

[8] S. Fakhouri, G. Goldszmidt, I. Gupta, M. Kalantar, and J. Pershing, “GulfStream - A System for Dynamic Topology Management in Multi-Domain Server Farms,” in IEEE International Conference on Cluster Computing (Cluster 2001), October 2001.

[9] A. Azagury, G. Goldszmidt, Y. Koren, B. Rochwerger, and A. Tal, “Khnum – Data Management for Océano,” IBM Technical Report RC200108, 2001.

[10] Intel Corporation. Preboot Execution Environment (PXE) Specification, Version 2.1, September 20, 1999. ftp://download.intel.com/ial/wfm/pxespec.pdf

[11] Linux Utility for Cluster Install (LUI). http://oss.software.ibm.com/developer/opensource/linux/projects/lui

[12] E. Bailey, Maximum RPM. Red Hat Press. 1997 [13] R. Campbell, Managing AFS: The Andrew File System. Prentice Hall PTR,

1998. [14] G. Goldszmidt and G. Hunt, “Scaling Internet Services by Dynamic

Allocation of Connections,” in Proceedings of the 6th IFIP/IEEE International Symposium on Integrated Network Management (IM 1999), May 1999, pp. 171-184.

[15] M. Kalantar and J. Fong, “Isolated Dynamic Clusters for Web Hosting,” in European Parallel Computing Conference (EUROPAR 2001), Manchester, UK, Aug 2001.

[16] J. Chuang and M. Sirbu, “Distributed network storage service with quality-of-service guarantees,” Journal of Network and Computer Applications, vol. 23, no. 3, pp. 163-185, July 2000.

[17] DHCP. http://www.isc.org/products/DHCP/ [18] A. Carzaniga, A. Fuggetta, R. S. Hall, D. Heimbigner, A. van der Hoek, and

A.L.Wolf, “A Characterization Framework for Software Deployment Technologies”, Technical Report CU-CS-857-98, University of Colorado, April 1998.

[19] InstallShield. http://www.installshield.com [20] Castanet of Marimba Inc. htpp://www.marimba.com

14

[21] TME-10 Software Distribution. http://www.tivoli.com/products/Courier [22] T. Poynor, “Automating Infrastructure Composition for Internet Services”,

Proc. of the 15th System Administration Conference (LISA), Dec 2001. [23] D. Feitelson, L. Rudolph, U. Schwiegelshohn, K. C. Sevcik, and P. Wong,

“Theory and Practice in Parallel Job Scheduling”, Proceedings of Job Scheduling Strategies for Parallel Processing, IPPS Workshop, April 1997.

[24] S. Chapin, D. Katramatos, J. F. Karpovich, and A. S. Grimshaw., “Resource Management in Legion”, Future Generation Computer Systems 15 (5-6), pp. 583-594 (1999), Special Issue on Metacomputing, Elsevier Science.

[25] “12 Steps to Cloning Windows NT System with SYSDIFF.EXE”. http://www.winntmag.com/Articles/Index.cfm?ArticleID=1

[26] “TPC-W”. http://www.tpc.org/tpcw/default.asp [27] “Java TPC-W Implementation Distribution.”

http://www.ece.wisc.edu/~pharm/tpcw.shtml