terms of service: data center addendum

New York State Chief Information Officer

and

New York State Office for Technology

Terms of Service:

Data Center Addendum

Revised 7/7/2009

Terms of Service: Data Center Addendum

Table of Contents

1.0 DESCRIPTION 5

2.0 TERMINOLOGY 6

3.0 FEATURES 7

4.0 MAINFRAME SERVICES 8

CONSOLE MONITORING 8

MAINFRAME SOFTWARE INSTALLATION/MAINTENANCE 9

MAINFRAME PERFORMANCE AND CAPACITY PLANNING 10

MAINFRAME TAPE SERVICES 11

IBM ONLINE REPORT VIEWING 12

PRODUCTION CONTROL SERVICES 13

ASSOCIATED PRINT SERVICES 14

IBM MAINFRAME DISASTER RECOVERY 15

5.0 OPEN SYSTEMS SERVER SERVICES 17

CENTRAL SERVER HOSTING 17

REVERSE PROXY 24

OPEN SYSTEMS BACKUP 25

TURN-KEY SERVER HOSTING 26

APPLICATION APPLIANCE HOSTING 29

6.0 OTHER SERVICES 30

PRINT SERVICES 30

MULTI-SITE NAME RESOLUTION 31

CUSTOMER ARCHITECTURAL CONSULTING AND REVIEW 34

DATA STORAGE SERVICES 35

FACILITIES MONITORING, OPERATIONS, PLANNING AND SETUP 37

DISASTER RECOVERY TESTING 38

OFFSITE STORAGE 39

Revised 7/7/2009 2 NYS CIO/OFT Data Center Addendum

MISCELLANEOUS SERVICES 41

7.0 AGENCY RESPONSIBILITIES AND TOOLS 42

8.0 PERFORMANCE MEASUREMENT AND METRICS 43

MAINFRAME SERVICES 44

CENTRAL SERVER HOSTING 46

PRINT SERVICES 47

DATA STORAGE SERVICES 48


Document Control Revision History Date Version Author Description September 05 1.0 Base Release

April 12, 06 1.1 Revised section on Central Server Hosting

April 2007 2.0 EF, MS, DC To continually provide Agencies with a clear explanation of services, revisions have been made throughout the document. Significant updates occurred in the following sections: Central Server Hosting, Server Monitoring – Non Resident (section deleted), Data Storage, Internal FTP (section added), and Customer Architectural Consulting & Review (section added).

August 2007 2.2 MS IBM Mainframe Disaster Recovery added

June 2008 2.1 CQ Clarification in Central Server Hosting for Basic Support

July 2008 2.2 PEH Augmented information on server hosting, added information on VMWare support, removed Acquisition of New Technology Section and added information relating to Multisite naming, added terms of service for turn-key VMWare servers

August 2008 2.3 PEH Clarified aspects of monitoring of servers

August 2008 2.4 PEH Removed references to surplusing customer equipment

February 2009 2.5 PEH Re-ordered many sections. Added information relating to Application Appliance Hosting

and Offsite Storage. Deleted references to Internal FTP Service and Model 204

databases.

April 2009 2.6 PEH Removed references to Tier 2 storage

July 2009 2.7 PEH Removed references to Tier2 applications


1.0 Description

The Data Center is a unified, centrally managed operation that provides a computing infrastructure in support of Agencies’ applications. Through the consolidation of the various computer operations, the State, the New York State Chief Information Officer and the Office for Technology (CIO/OFT), and State Agencies benefit from more cost effective use of technology investments. The Terms of Service documents define the services the CIO/OFT provides to State Agencies. The Core document details information regarding how CIO/OFT will provide services and includes responsibilities required of both CIO/OFT and customer Agencies.

The complete Terms of Service (TOS) documents consist of the following information.

• The Core Terms of Service base document, which defines the terms common to all CIO/OFT services, unless otherwise noted. It includes information such as terminology, ordering of services, and management of service outages.

• The Terms of Service Addendum documents supplement the Core Terms of Service document and define the terms specific to a CIO/OFT service for all Agencies. The documents provide target service levels and standard operating procedures between CIO/OFT and Agencies that use particular CIO/OFT services. If conflicting terms are described between the Core Terms of Service and a Terms of Service Addendum, the Addendum document will take precedence over the Core document.

• The TOS Services Provided Forms list the services CIO/OFT provides to a particular Agency. CIO/OFT provides the services as described in the Terms of Service documents.

• In special cases, when required, an Appendix is included. Appendices are limited in scope to describing a unique service not covered in the basic Addendum and are provided to a particular Agency. Appendices take precedence over Addendums and the Core document.

The effective date of this Agreement is the date service is rendered by CIO/OFT and the Agreement will continue in effect for as long as the service is provided to the Agency.

For more information on CIO/OFT services, contact Customer Service and Marketing at [email protected], or by phone at 518-473-2658.


mailto:[email protected]

2.0 Terminology

16x5

Shall mean 16 hours per day (7am-11pm), 5 days per week (Monday-Friday), except during a disaster.

24x7 Shall mean 24 hours per day, 7 days per week, 365 days per year and

include all State and federal holidays, except during a disaster. Enterprise Systems Management (ESM)

The practice implemented to support the CIO/OFT IT environment by developing and implementing processes, procedures, and tools to monitor and control the operation of equipment, system resources, and sytems utilities. ESM supports the CIO/OFT IT environment by integrating operation information into a consolidated system for reporting, monitoring and alerting IT staff to possible equipment and systems software problems.

FTP (File Transfer Protocol)

File Transfer Protocol (FTP), a standard Internet protocol, is the simplest way to exchange files between computers on the Internet. It is commonly used to download programs and other files to computers from other servers.

Infrastructure, Architecture and Technical Support (IATS)

The CIO/OFT operational organization responsible for technical support for IBM mainframes, Unisys mainframes, distributed systems, Data storage area network (SAN), and Enterprise backup.

Jobs Refers to a mainframe console action to start a run stream that may initiate one

or more program executions. Level 1 (Help Desk) Shall mean help desk staff that receives the initial call, triages the call and

refers unresolved problems to appropriate Level 2 & 3 organizations, which includes the State Data Center, CIO/OFT Network, vendors, and directly to state agencies for Agency specific applications. Provides service remotely.

Level 2 (Support) Shall mean staff dedicated to direct support of specific functional areas or

technologies such as data center support for open systems, IBM and Unisys; network support (LAN, WAN) including vendor supplied facilities; and Agency application support. May be required to provide hands-on solutions on site.

Level 3 (Expert) Shall mean expert staff who consults with Level 2 staff to address issues when

the level of knowledge and/or tool set is unavailable. This is generally available remotely to Level 2 staff.

OFT Enterprise Architecture (EA)

The explicit description and documentation of the current and desired relationships among business and management processes and information technology.

Open Systems Shall mean the grouping of servers identified as Windows, Unix, and Linux

administered by the Data Center and the storage and backup associated with the servers.


http://searchnetworking.techtarget.com/sDefinition/0,,sid7_gci212839,00.html

http://searchnetworking.techtarget.com/sDefinition/0,,sid7_gci211996,00.html

3.0 Features

CIO/OFT provides a variety of Data Center services in mainframe and open systems environments. CIO/OFT provides physical security and environmental protection to the extent approved by the Division of the Budget for equipment housed within physical areas CIO/OFT is responsible for maintaining. CIO/OFT provides services selected on the Agency’s Service Provided Form. Services include:

• Mainframe Services

Console Monitoring

Mainframe Software Installation/Maintenance

Mainframe Performance and Capacity Planning

Mainframe Tape Services

IBM Online Report Viewing

Production Control Services

Associated Print Services

IBM Mainframe Disaster Recovery

• Distributed Systems services

Central Server Hosting

Turnkey Server Hosting

Reverse Proxy

Distributed Systems Backup

Application Appliance hosting

• Other Services

Print Services

Multi-Site Name Resolution

Customer Architectural Consulting and Review

Data Storage Services

Facilities Monitoring, Operations, Planning and Setup

Disaster Recovery Testing

Offsite Storage


Miscellaneous Services

Details of the services and availability are described in the following tables.


4.0 Mainframe Services

The CIO/OFT Data Center hosts mainframe environments for NYS customer agencies, providing the following services:

Data Center Service

Service:

Console Monitoring

General Description: The service ensures an Agency’s mainframe computer systems are effectively monitored and that appropriate agreed upon actions or escalations are taken. CIO/OFT console operators will check for system availability; check tape and printer resources; answer system messages; execute scripts to monitor system resources and system health; and perform system checklists. Operators will start up or shut down systems as necessary to respond to both normal and abnormal situations based upon defined procedures or under the direction of CIO/OFT technical staff. For any hardware problem, vendor staff will be dispatched and the performance monitored. Operators will address specific questions from key users and work with the Enterprise Help Desk to address severity-1 and severity-2 problems. Operators will participate in developing and identifying mainframe abnormalities in the daily shift reports. The reports will be sent via e-mail to an Agency in the morning every 24 hours, Monday through Friday, non-holidays. CIO/OFT will notify an Agency of scheduled and unscheduled downtime events. Notification of users and interface agencies of these events will be mutually agreed upon by the Agency and CIO/OFT.

Offered To: The service is available to any Agency using IBM, Unisys 2200, and Unisys A mainframes in the CIO/OFT Data Center.

Availability: Base supported hours are 16x5; the Agency may increase coverage either by approving overtime or additional personnel. The hours of coverage for an Agency shall be documented in the Services Provided Form for the Agency.


Data Center Service

Service:

Mainframe Software Installation/Maintenance

General Description: CIO/OFT installs and maintains a wide selection of software to meet an Agency’s needs. CIO/OFT will meet regularly with Agencies to discuss possible upcoming operating system and software upgrades that may impact Agencies and negotiate a schedule that is mutually agreeable. CIO/OFT systems programming staff will review, configure, generate, install, and maintain all mainframe operating system software. CIO/OFT will install the operating system, libraries, and components and apply patches and upgrades as applicable. CIO/OFT systems programming staff will review, configure, generate, install, and maintain vendor-supplied software products and all mainframe-based telecommunications software unless specifically agreed otherwise with the Agency. Software product customization responsibilities are based on agreed upon responsibilities with an Agency. To request new software, an Agency should submit a formal request to the Data Center. Agency requests for new software will be analyzed for appropriateness to the overall mainframe architecture and the CIO/OFT Enterprise Architecture. CIO/OFT will review the cost impact, if any. CIO/OFT and the Agency will do appropriate testing of modifications prior to, when possible, and during, implementation of modifications. The testing plan will be agreed to on a project by project basis. CIO/OFT will diagnose software problems based on agreed upon responsibilities with the Agency, acquire any associated software fixes, apply the fixes, and work with the Agency to test any applied fixes.

Offered To: The service is available to any Agency using IBM, Unisys 2200, and Unisys A mainframes in the CIO/OFT Data Center.

Availability: Technical support is available during Standard Business Hours. The service is specific to a particular mainframe box, LPAR, or site. This activity will occur during an agreed upon time period scheduled with the Agency based on workload. CIO/OFT will work with the Agency to minimize downtime.


Data Center Service

Service:


General Description: CIO/OFT will monitor the performance and capacity of mainframe systems to ensure systems perform effectively. To meet mutually defined performance level goals of CIO/OFT and the Agency, CIO/OFT technical support staff will monitor system use and perform tuning efforts to ensure mainframe system resources are available to Agency environments. CIO/OFT will periodically solicit information from an Agency regarding new initiatives or changes in processing that may cause a deviation in resource consumption beyond historical trends. System configurations will be adjusted as necessary to accommodate these changes and provide performance levels mutually agreed upon to meet Agency needs. The Agency is responsible for notifying CIO/OFT of projected major changes in resource consumption, including changes that have a significant affect on system use, such as tape, print, batch, processing flow, new programs or modifications in transaction volumes.

Offered To: The service is available to any Agency using IBM, Unisys 2200 and Unisys A mainframes in the CIO/OFT Data Center. For Unisys mainframes, measurements are available by Agency. For IBM mainframes, measurements are available by individual virtualized Agency LPARs as well as aggregate system capacity for each individual mainframe system.

Availability: The service is available during Standard Business Hours.


Data Center Service

Service:

Mainframe Tape Services

General Description: CIO/OFT provides multiple services to handle usage and management of mainframe tape media. CIO/OFT will establish and publish supported media. Additional fees may apply for non-standard media. CIO/OFT will provide at least 6 months notice to an Agency when medium is out-of-date and will no longer be supported. Manual Tape Processing The service includes responding to all system, application, job, and user requests for manual tape mounts and dismounts on a variety of tape drives. As tapes are removed or deleted from the system, they are degaussed and/or destroyed according to defined security procedures. Round reel tapes will require conversion to a cartridge tape at a central facility for processing and will be reconverted after processing. Robotic Tape Processing The service includes adding physical and virtual volumes and ejecting volumes to/from the Virtual Tape Systems or Automated Tape Libraries as needed or requested. Data recoveries due to physical tape problems are responded to accordingly. Inquiries are performed to verify the availability and status of storage volumes. Tape Library Tapes are received and tracked as they enter the Data Center. Foreign tapes may be accepted from other sources established with the Data Center by the Agency. Periodically, foreign tapes are reviewed and returned to the Agency after processing. Tapes sent offsite are tracked for eventual return to the library. Tapes are recycled and historical reports are maintained per Agency standard. Transportation of tapes is the responsibility of the Agency with the exception of Agencies who participate in the CIO/OFT Iron Mountain contract. Full Backup Library Services The services include: ensuring Agency backup procedures are run; identifying tapes to be sent offsite; using offsite system vaulting software to enter tapes being sent offsite and requesting tapes to be returned; tracking vaulted tapes and ensuring that the correct generations are offsite and that the correct generations are returned to the Data Center; and ensuring that long-term vaulted tapes are returned to the Data Center at correct intervals for copy or creation of new tape. Cost for the service is included in the manual tape mounting fee. Limited Backup Library Services Limited services include: ensuring Agency backup procedures are run; identifying tapes to be sent offsite; and packaging and returning tapes to the Agency. Cost for this service is included in the manual tape mounting fee.

Tape Ejection The service consists of entering ejection commands to the robotic tape drives and manually removing tapes to be sent to offsite storage.

Offered To: The services are available to any Agency using a mainframe located in the CIO/OFT Data Center.

Availability: The services are available based on the Agency’s agreed upon mainframe console hours of operation.


Data Center Service

Service:

IBM Online Report Viewing

General Description: With the service, an Agency receives the benefit of fully managed report viewing and distribution environment. CIO/OFT will manage the operating system, storage subsystem, database management system, and network connectivity. CIO/OFT will also provide and manage the report viewing and distribution subsystem components allowing the Agency to focus support efforts on managing report requirements. The Agency will be responsible for defining reports to the system, including report attributes and retention criteria. The Agency will also provide access and training to the user community and administrative support (add users, delete users, and reset passwords). The Agency will define backup criteria and archival needs.

The Report Viewing and Distribution service offering provides:

• Integration of data created by application programs into an online, electronic information archive and retrieval system.

• Controlled and reliable access to all of an organization’s reports.

• High volume capture of computer output.

• Advanced ready-to-use client viewing applications for both desktops and standard Web browsers.

• Automated storage management.

• Advanced search and report-mining capabilities.

CIO/OFT Data Center staff will be available to report and correct performance problems and concerns.

Offered To: Limited to Agencies that use the IBM mainframe Data Center.

Availability: The service is available 24x7.


Data Center Service

Service:

Production Control Services

General Description: Schedule and Set Up The service is available from 7:00am to 3:15pm, Monday through Friday, non-holidays only. The service includes setup and scheduling of daily, weekly, monthly, quarterly, annual, and ad-hoc jobs. Jobs that ‘error’ or require quality control (e.g. validate hash totals) will be addressed with Agency programming staff during Standard Business Hours. The job is performed using detailed knowledge of an Agency’s work, job control language, and applicable scheduling software. CIO/OFT staff has a working knowledge of the relationships among various programs and Agency applications. Batch Monitoring and Error Resolution The service is only available to an Agency receiving Production Control services. CIO/OFT staff will monitor the execution of production batch schedules on off-shifts. Job errors will be addressed as follows: by modifying JCL/ECL (job control language/executive control language), using Agency escalation procedures, or following job documentation. Schedules will be modified as necessary to insure timely completion of Agency workloads. Control totals will be monitored. The job is performed using detailed knowledge of an Agency’s work, job control language and applicable scheduling software. CIO/OFT staff has a working knowledge of the relationships among various programs and Agency applications. Batch Schedules Console staff manually kicks-off batch schedules and the service includes starting a maximum of thirty (30) schedules per day. Some critical checks (fewer than 10 jobs) are performed from one job to another and limited totals are validated; the critical batch jobs are monitored and addressed per documentation.

Offered To: Limited to Agencies that negotiated the service at time of transfer of function. Expansion of the service to new Agencies is not available at this time.

Availability: See General Description above.


Data Center Service

Service:

Associated Print Services

General Description: CIO/OFT provides multiple services to support printing needs: Agency Stock Storage

The CIO/OFT print center will maintain limited storage space for Agency print stock. Up to a two-week supply can be stored. If additional space is needed, there may be a cost based on square footage and requirements for secure/non-secure storage.

Burst/Trim/Decollate

Bursting is the process that separates continuous form paper into individual sheets. Trimming separates pin feed strips from continuous form paper. Decollating separates multiple part forms into individual continuous parts.

Inserting

The service consists of the insertion of single or multiple letters, coupons, checks or other documents into envelopes.

Packaging

Packaging includes any special print separation, sorting and breakdown of output, shrink-wrapping and placing output in envelopes or delivery boxes.

Mailing

The service includes preparation of print for the U.S. mail or courier services including addressing envelopes/boxes.

Delivery

The delivery service moves packaged print from the print center to an Agency destination at determined frequency.

Forms Design

The service provides support to the user community in the design of laser-generated forms, fonts, and logos. CIO/OFT staff will create job source and form source language, utilize print and form design languages, scan source documents and place the forms in server folders. Forms are edited for print quality and tested in conjunction with the user; once approved the documentation is created and placed into the documentation library. The documentation contains the form number, name, edition date, paper size and color along with printing and finishing instructions.

Offered To: Services are limited to existing Agencies who negotiated this service at time of transfer of function; expansion to new Agencies is not available at this time.

Availability: Availability of services will continue for Agencies using this service, as agreed upon between CIO/OFT and the Agency. The service is grandfathered in at the time of transfer to CIO/OFT. It may not be expanded or modified. Components or particular jobs can be deleted and once removed cannot be replaced.


Data Center Service

Service:

IBM Mainframe Disaster Recovery

General Description: The service provides a production ready mainframe environment at a local CIO/OFT designated Recovery Data Center facility. In the event of a disaster at the production site, recovery shall occur within a Recovery Time Objective of 48 hours. The prioritization of participating Agencies will be determined at the time of the disaster and will remain the discretion of CIO/OFT Executive Management. CIO/OFT will provide the following mainframe resources and support services within 24 to 48 hours after CIO/OFT and the participating Agency officially declare a disaster.

• Mainframe computer resources at the Recovery Data Center

o Equivalent CPU and memory capacity

o Equal production DASD Capacity

o Equivalent Tape Capacity

• Infrastructure support services

o Redundant networking infrastructure required to support mainframe execution at the Recovery Data Center

• Network Services to the Mainframe LPAR

o Connection to the Agency’s current entry point to the Home and Recovery Data Center

o Connection to the Agency’s current critical business constituents from the Home and Recovery Data Center

• An operational mainframe environment in an Agency-familiar configuration

o Recovery LPARs containing the appropriate hardware resources (CPUs, Memory, co-processors, etc.) predefined and maintained at the Recovery Data Center

o OS and operational software (including production components, configuration, programs, and customization) maintained at the Recovery Data Center

o Restoration of user programs and data with a recovery point to the last successful replication

• Operational and Technical support of the Agency’s mainframe resident environment available at the Recovery Data Center

• Facilities supporting the Recovery environment maintained by CIO/OFT

• A mainframe specific DR Plan particular to the Agency maintained and exercised

An CIO/OFT mainframe DR technical coordinator will be assigned to each Agency to:

• be the point of contact between CIO/OFT and the Agency.

• coordinate exercises.

• be responsible for maintaining CIO/OFT’s mainframe recovery documentation.

CIO/OFT will maintain a Recovery environment consisting of:

• sufficient DASD to support the largest subscribing Agency. For the Agency Disaster Recovery Test, CIO/OFT will Flash Copy the Mirrored IBM DASD to isolate the test environment from the Production mirrored data at the Recovery Data Center.

• connectivity to the Mirrored DASD. The Flash Copy DASD will be established and maintained by CIO/OFT.

• operations and technical support for an Agency Disaster Recovery Test.


Participating Agencies are responsible for the following aspects of their disaster recovery operations: • Development and implementation of their DR plan and processes to recover at the Recovery Data Center

• Designation and management of necessary DR Resources at Agency locations needed to support the recovery strategy

• All end user and non-Data Center vendor coordination

• Customer-managed network failover configuration and activation during a disaster to the Recovery Data Center

• Application and data recovery validation (e.g. recovery of databases, data to sync-point, in-flight transaction recovery)

• Interaction with dependent systems (including vendors)

• Management of non-virtual and offsite tape data entities (e.g. off-site storage, retrieval and delivery to the recovery site are the customers responsibility; Non-virtual native tapes retained solely at the primary site will not be accessible to the recovery site)

• Application Data Backups at the Recovery Site

• Designating an Agency DR Coordinator who will:

o be the point of contact between the Agency and CIO/OFT

o coordinate exercises

o be responsible for maintaining Agency recovery documentation

To participate in this program, an Agency must consent to the following terms of service:

• The preceding list of Agency responsibilities

• A minimum commitment of 3 consecutive years to the service

• Mirroring all production data, including System Volumes, on IBM DASD at a designated Recovery Data Center

• A commitment to participate and perform disaster recovery tests at least every 9 months during the service offering period

The customer Agency will be charged a one-time setup fee for the service and a fixed fee for each DR test; the testing fee will be reviewed annually as part of the CIO/OFT rate determination process and is subject to change. There will be a monthly charge for the ongoing Agency mirrored DASD and an additional charge for ‘Flash-copied’ DASD Storage during the month of the conducted test. The cost of any unique, non standard equipment needed exclusively for the particular Agency is also the Agency’s responsibility.

Offered To: The service is available to any Agency using the IBM mainframe in the CIO/OFT Data Center.

Availability: Disaster Recovery services are available when a formal disaster is declared based upon the escalation procedures agreed to by CIO/OFT and the Agency. Recovery System availability will be based on the Agency’s agreed upon mainframe console hours of operation.


5.0 Open Systems Server Services

Data Center Service

Service:


General Description: CIO/OFT hosts and monitors servers in the CIO/OFT Data Center. CIO/OFT will coordinate and schedule the installation, configuration and maintenance of hardware and software for open systems servers residing in the CIO/OFT Data Center, including any problem diagnosis associated with this environment. The focus is on providing a stable environment where servers reside, in addition to providing appropriate levels of administration and monitoring to meet specific needs of each Agency. CIO/OFT offers hosting of Agency owned servers as well as hosting servers on SUSE Linux on IBM mainframes. All support for SUSE Linux servers on IBM mainframes is the same as provided for physical servers with some modification to the Environmental Management functions described below. Environmental Management functions related to IBM mainframes are performed for the entire system and not for individual SUSE Linux guest servers. Two levels of support are offered. Basic Support is available for certain non-production environments and requires the Agency to assume responsibility for certain software and application management tasks. Standard Support is the broadest level of support. CIO/OFT provides the following with Basic and Standard Support:

• Preparation and submittal to the Agency of a Project Support Proposal (PSP) outlining the server work unit being applied (which implies either Basic or Standard Support) and support costs associated with the servers. Work will not proceed until the Agency’s IT and Finance Directors have approved the proposal.

• Environmental Management including power, cooling, physical security, coordination of installation, and cabling of new equipment in the CIO/OFT Data Center.

• Technical consulting – limited by staff availability – including design and integration of hardware and software infrastructure and implementation of the design.

• Comprehensive Operating System (O/S) administration and maintenance including monitoring and application of patches and upgrades as available, security upgrades, server hardening, virus protection, and enterprise back-up client installation.

• CIO/OFT will notify the Agency when patches need to be applied. CIO/OFT will review patches with the Agency and jointly schedule the work. CIO/OFT will perform the patch upgrades.

• Change control management for hardware, Operating Systems, networking equipment, ESM, and backup software.

• Changes are performed during standard maintenance windows or at another time as mutually agreed to by the Agency and CIO/OFT.

• Changes to environments shared by CIO/OFT customers will be communicated to all users of the shared environment and will be coordinated to minimize impact to all customers of the shared environment. Changes to shared environments will follow the standard change processes as described in this document.

• Full support for CIO/OFT provided shared services such as, but not limited to, DNS and LDAP. • Full Data Center network support for all networking components within the defined Data Center network. • Establishment and maintenance of approved firewall rules, routing and VPN tunnels to ensure application

connectivity and ensure a secure network environment. • Load balancing services based on documented rules for use. • Backup and recovery as defined in the Open Systems Backup TOS and as stated in the Open Systems Backup

and Recovery Guidelines Document. • Data Center network support staff will work with Agency network support staff, or CIO/OFT Customer Network

Services if the customer network is maintained by CIO/OFT, to establish network connectivity. • Prior to implementation, CIO/OFT will meet with the customer to create troubleshooting scripts to expedite and

route issues to the appropriate group for resolution. • Based upon troubleshooting scripts provided, refers issues related to hardware or operating system to CIO/OFT


IATS resolvers. • Based upon troubleshooting scripts provided, refers issues related to the application to designated Agency

resolvers. • Based upon troubleshooting scripts provided, refers issues related to Data Center network components to NYS

CIO/OFT DCN and refers issues related to customer network components to Agency network support staff or to CIO/OFT Customer Network Services if the customer network is maintained by CIO/OFT

• CIO/OFT Based upon troubleshooting scripts provided, refers issues related to the other CIO/OFT shared services to designated CIO/OFT resolvers.

CIO/OFT establishes the following guidelines related to Basic Support:

• For requested non-production environments, the Agency may request administrative privileges on the server. Basic Support is not available for production servers.

• CIO/OFT will be responsible for the provisioning the Operating System and other CIO/OFT services as appropriate. The functionality will be confirmed prior to release to the customer.

• The Agency assumes all responsibility for installation and maintenance of Agency required software components.

• Upon release of server to the customer, CIO/OFT troubleshooting support will be limited to restoring server to date specified by the Agency. Restorations will be serviced as a next business day request.

CIO/OFT also provides the following with Standard Support: • Enterprise Systems Monitoring of servers using a CIO/OFT-defined Tivoli architecture supported by the SDC

operational staff available 24x7. • Conduct initial triage of problems, run diagnostics, ping servers, reboot servers, recycle applications and

generate printouts as directed or prescribed. • Performance monitoring and analysis. • Installation, configuration, and ongoing application software maintenance above the O/S including

patches, fixes, and upgrades not administered at the Agency level. • Installation and configuration of ESM agents to facilitate monitoring. • Defines and implements appropriate system management alerts and notifications to CIO/OFT and

Agency application and database staff in conjunction with Agency staff. • Provide Level II and Level III response, as appropriate, to ESM alerts related to hardware and the

operating system. • Technical consulting and project team participation, limited to staff availability.

Standard Tivoli Monitoring for Production Servers Tivoli monitoring is a required service for all production servers in the CIO/OFT Data Center. Through the use of Tivoli monitoring software, the data center infrastructure that end user applications run on is continually monitored and, when necessary, alerts are generated to initiate corrective or pro-active actions.

The Tivoli products utilized for the service include:

• Tivoli Enterprise Console (TEC) – Functions as event correlation engine and operations interface. All defined enterprise significant events are sent to the TEC for correlation and further action. Performance monitoring and analysis limited to staff availability.

• IBM Tivoli Monitoring (ITM) – Used to monitor software components of UNIX and Windows servers. • IBM Tivoli Netview – Used to monitor all data communications components of the data center network for

availability. • IBM Director – Used to monitor all hardware for IBM X86 servers.

OFT is responsible for: • Providing base level monitoring and alerting on all servers. • Providing hardware monitoring for all IBM X86 windows servers. • Providing availability monitoring for data communications equipment. • Providing a consolidated core infrastructure that all enterprise significant alerts will be integrated into

for consolidation and to initiate corrective actions.


• Working with CIO/OFT system administrators and Agency subject matter experts to modify or enhance monitoring. Note: Performance monitoring and analysis limited to staff availability.

• Providing a centralized infrastructure to collect and display utilization information for servers. (This functionality requires ITM 6, which is in the process of being rolled out. Once ITM 6 is available for all environments, this functionality will be standard.)

Participating Agencies will be responsible for the following:

• Purchase of Tivoli Licenses required for monitoring. • Working with CIO/OFT to set up Agency and application specific monitoring.

Base monitoring for Windows Servers Includes: • Processor (CPU) Monitoring

o Measures overall system processor utilization; generates alerts when the CPU’s are too busy. • Memory Monitoring

o Measures memory usage and paging, generates alerts as follows: Low cache read hits Low available memory Excessive paging Application leaking memory System leaking memory System Drivers leaking memory

• Physical Disk Monitoring o Measures Physical Disk response times; generates alerts when response times are long. o Measures performance of each physical disk; generates alerts as follows:

High percent disk time Possible fragmentation Slow physical Drive

• Logical Disk Monitoring o Measures the performance of each logical disk; generates alerts as follows:

High percent disk time Low logical space Possible fragmentation Slow logical drive

• Service Monitoring o Ensures that key services are running; generate alerts as follows:

Service failing Service stopped

• Process Monitoring o Measures process CPU usage; generates alerts when individual processes consume excessive

amounts. • Windows Event log monitoring

o Captures all “error” logs that are posted to the Windows System log, and forwards them to the Tivoli Enterprise Console (TEC) server

As specific logs are identified as being insignificant, they are excluded o Captures all ‘error’ logs that are posted to the Windows Application log, and forwards them to

the Tivoli Enterprise Console (TEC) server As specific logs are identified as being insignificant, they are excluded

Base monitoring for UNIX servers includes:

• Processor (CPU) Monitoring o Measures overall system processor utilization and processor run queues, generates alerts when

the CPU’s are too busy and queues are too long. o Monitors presence of individual CPUs.

• Memory Monitoring o Measures memory usage and swapping, generates alerts as follows:

Low memory Low swap space


Thrashing (excessive swapping) • File System Monitoring

o Measures available resources: Available space Available Inodes Detects file system fragmentation

• Security Monitoring – key files o Ensures the presence of key system security files o Generates a notification when they are updated

• Security Monitoring o Checks for the following items that can indicate security breaches:

Missing files Property changes for key files Suspect Superuser Invalid root account Duplicate User accounts Excessive log-ins

• Process Monitoring o Ensures that key Processes are running, generate alerts when processes are missing

The following defines Agency responsibilities for Basic Support:

• Optional administrative rights • Full database administration, including database backups to disk and database restorations • Data Security and Integrity • Installation, configuration, and maintenance of all software above the O/S including the application of

patches and upgrades • Performance monitoring, analysis, and design • Management scripting • Application migration support • Capacity planning in conjunction with data administration groups • Primary Level II and Level III technical support for problem resolution • Participation in the CIO/OFT Change Control Management process • Maintain current maintenance contracts on all hardware and software owned by the Agency and

managed by the CIO/OFT The following defines Agency Responsibilities for Standard Support:

• Full database administration, including database backups to disk and database restorations • Data Security and Integrity. • Testing and Quality Assurance of application code prior to migration to production environments. • Provide at least one pre-production environment to be managed by the Data Center to be used for

migrations and testing of patches. • Provide application testing in the event of scheduled infrastructure downtime. • Provide Agency resolver support and response, as appropriate, to ESM alerts related to applications. • Maintain current maintenance contracts on all hardware and software owned by the Agency and

managed by the CIO/OFT. CIO/OFT will issue a proposal for central server hosting within thirty (30) days of receiving a request for the service. Patch Management A technical review will be conducted and a schedule for critical and high security alerts will be developed and distributed within seventy-two (72) hours of receiving the critical or high alert from the vendor community. The actual schedule for distribution to servers hosted by the Data Center will be based on the results of the technical review and will be scheduled over the course of four available maintenance windows. Patches identified as less than critical will be bundled and scheduled for deployment during normal maintenance windows as resources permit. The process will be escalated upon direction from the CIO/OFT ISO or if a critical patch poses a specific risk to the environment.


Offered To: This service is available to any Agency with servers physically located in the CIO/OFT Data Center.

Availability: Basic Support: operational and on-site technical support is provided during Standard Business Hours. Standard Support: operational support for Enterprise Systems monitoring including Level II support is provided 24x7. On-site technical support is provided during Standard Business Hours as indicated in the Core Terms of Service Document.


Data Center Service

Service:

Server Virtualization using VMWare

General Description: Virtualization technology offers CIO/OFT customers a way to drive improvements in capital and operational efficiency. By leveraging VMWare virtual servers, customers can better manage server capacities and often streamline server provisioning processes. CIO/OFT provides the following services with Basic and Standard Support:

• VMWare Host (ESX) Configuration - The implementation of a VMWare Host system consistent with the Terms of Service for Central Server Hosting.

• VMWare Guest (Virtual Machine) Configuration - CIO/OFT will make use of Virtual Center and other tools to configure and provision requested Virtual Guests. The Virtual Guest will be configured based upon the requirements set forth in the Project Support Proposal (PSP) and other service request documentation consistent with the Terms of Service for Central Server Hosting.

• Manual VMWare Guest Relocation (VMotion) - In the event that a virtual guest must be relocated to another ESX server in response to maintenance or capacity issues, CIO/OFT will schedule the movement of that guest and facilitate the relocation via VMotion. Any customers wishing to make use of VMotion will be asked to provide detailed configuration instructions based upon their specific server requirements. CIO/OFT will configure VMotion based solely on the configurations set forth in these instructions. Any request to initiate VMotion should begin with an Open Systems Work Request or, if more immediate action is required, and Enterprise Help Desk Ticket.

• Distributed Resource Scheduling (DRS Load Balancing) – DRS is used to balance load by dynamically distributing virtual guests across multiple physical servers. Any customers wishing to make use of DRS will be asked to provide detailed configuration instructions based upon their specific server requirements. CIO/OFT will configure DRS based solely on the configurations set forth in these instructions. Any request to initiate DRS should begin with an Open Systems Work Request or, if more immediate action is required, and Enterprise Help Desk Ticket.

• VMWare High Availability (HA) – In the event that a VMWare host experiences a failure, guest servers can be configured to failover onto a different physical server. Any customers wishing to make use of HA functionality will be asked to provide detailed configuration instructions based upon their specific application requirements. CIO/OFT will configure HA based solely on the configurations set forth in these instructions. Any request to initiate HA should begin with an Open Systems Work Request or, if more immediate action is required, and Enterprise Help Desk Ticket.

Additional Information:

In addition to the VMWare-specific services listed above, please reference the Centralized Server Hosting section of the Data Center Addendum for more details on centralized server hosting and management services. CIO/OFT Data Center technical support staff must be involved with, and approve the design of any such specific services.

Design Criteria and Considerations:

• Capacity Planning and Load Testing - With one of the principle goals of virtualization being the better use of hardware, it becomes even more important to fully understand the capacity requirements of applications. The capacity planning and load testing should be specifically designed to address the special requirements of a virtualized environment including the impact it will have on a network interface.

• DRS and HA Planning – In the event that DRS or HA configurations will be used anywhere within the VMWare cluster, it is critical that detailed planning be performed by the application owners to assess the impact a server move will have on that server and any others which share the same hardware.


• Storage Requirements - In addition to any normal disk storage that an application may require, disk capacity will also be required for the OS along with any memory that will be configured for use on the VMWare guest. Each guest operating system requires 15GB of Enterprise Storage.

Offered To: The service is available to any Agency with servers physically located in the CIO/OFT Data Center.

Availability: Basic Support: operational and on-site technical support is provided during Standard Business Hours. Standard Support: operational support for Enterprise Systems monitoring including Level II support is provided 24x7. On-site technical support is provided during Standard Business Hours as indicated in the Core Terms of Service Document.


Data Center Service

Service:

Reverse Proxy

General Description: CIO/OFT will provide the Agency with Reverse Proxy Services by identified Universal or Uniform Resource Locator (URL). Reverse Proxy Services enable an application that is not architected to terminate user sessions in Tier 1 of the network, to be implemented as if it were. The Reverse Proxy layer is added in front of the user presentation layer creating a buffer between more accessible Tier 1 facing the Internet or NYeNet, and more secure presentation and business logic layers in either Tier 2 or the Legacy Tier. The service enables our customers to migrate many more applications to the Internet, intranet and the new Data Center Network three tier architecture than they would be able to without this service. If an application is re-architected to no longer require the Reverse Proxy Service, the Agency simply needs to contact CIO/OFT to terminate the service for the specified URL.

Offered To: The service is available to any Agency who is a central server hosting or mainframe hosting customer.

Availability: The service has the same availability as our Standard Support under Central Server Hosting: operational support for Enterprise Systems monitoring including Level II support is provided 24x7. On-site technical support is provided during Standard Business Hours.


Data Center Service

Service:

Open Systems Backup

General Description: CIO/OFT will provide the centralized infrastructure (hardware & software) necessary to satisfy Agency’s data backup and recovery requirements for data used in the Open Systems environment. CIO/OFT will provide complete management of the backup and recovery infrastructure, which includes:

• Configuration: CIO/OFT technical support staff will install and configure the server backup application software. Staff will install or assist in the installation of the client agent software on CIO/OFT supported operating environments.

• Monitoring: CIO/OFT will monitor daily backups of Agency server environments. CIO/OFT will monitor and restart failed backups. CIO/OFT will work with server administrators and the Agency to resolve persistent backup related problems. CIO/OFT will actively monitor backup infrastructure (server, tape library, and storage network) and make sure resources are available to meet Agency requirements.

• Recovery: CIO/OFT technical support staff will provide individual or complete server recovery services. • Capacity Planning: CIO/OFT technical support staff will monitor backup infrastructure resource use

(server, tape library, and networks) and reconfigure, tune or expand the infrastructure as necessary to make resources available to meet Agency requirements.

• Media Management: CIO/OFT will provide media management services including recycling and replacement of tapes and development of an offsite tape service.

Offered To: This service is available to any Agency that uses CIO/OFT central server hosting services.

Availability: Most backup functions are automated and occur during the Agency’s off-hours of operation based upon agreement with the Agency. Support for backup is available during Standard Business Hours for technical support staff. Emergency restoration will be handled through the standard CIO/OFT Enterprise Helpdesk process.


Data Center Service

Service:

Turn-Key Server Hosting

General Description: Virtual server environments are provided to our customers on shared CIO/OFT hardware through Turn Key Server Hosting. By making use of this service, our customers can realize more rapid application deployments without the capital expenditures involved to purchase new physical servers. Service Considerations:

• Operating System and any other software is not included

• Turn Key Servers require a minimum of 15GB of storage prior to application requirements

• Enterprise backup services are available but not included

CIO/OFT provides the following with all support levels:

• Preparation and submittal to the Agency of a Project Support Proposal (PSP) outlining the server work unit being applied (which implies either Basic or Standard Support) and the support costs associated with these servers. Work will not proceed until the Agency’s IT and Finance Directors have approved the proposal.

• Technical consulting – limited by staff availability – including design and integration of hardware and software infrastructure and implementation of the design.

• Comprehensive Operating System (O/S) administration and maintenance including monitoring and application of patches and upgrades as available, security upgrades, server hardening, virus protection, and enterprise back-up client installation.

• CIO/OFT will notify the Agency when patches need to be applied. CIO/OFT will review patches with the Agency and jointly schedule the work. CIO/OFT will perform the patch upgrades.

• Change control management for shared hardware, Operating Systems, networking equipment, ESM, and backup software.

• Changes are performed during standard maintenance windows or at another time as mutually agreed to by the Agency and CIO/OFT.

• Changes to environments shared by CIO/OFT customers will be communicated to all users of the shared environment and will be coordinated to minimize impact to all customers of the shared environment. Changes to shared environments will follow the standard change processes as described in this document.

• Based upon troubleshooting scripts provided, refers issues related to hardware or operating system to CIO/OFT IATS Level II.

• Based upon troubleshooting scripts provided, refers issues related to the application to Agency Level II.

• Backup and recovery as defined in the Open Systems Backup TOS and as stated in the Open Systems Backup and Recovery Guidelines Document.

• Full Data Center network support for all networking components within the defined Data Center network.

• Based upon troubleshooting scripts provided, refers issues related to Data Center network components to CIO/OFT DCN and refers issues related to customer network components to Agency network support staff.

• Data Center network support staff will work with Agency network support staff or CIO/OFT Customer Network Services if the customer network is maintained by CIO/OFT, to establish network connectivity and to troubleshoot problems.

• Establishment and maintenance of approved firewall rules, routing and VPN tunnels to ensure application


connectivity and ensure a secure network environment.

• Load balancing services based on documented rules for use.

OFT also provides the following with Gold and Platinum Support:

• Enterprise Systems Monitoring of servers using CIO/OFT-defined Tivoli architecture supported by the SDC operational staff available 24x7.

• Conduct initial triage of problems, run diagnostics, ping servers, reboot servers, recycle applications and generate printouts as directed or prescribed.

• Performance monitoring and analysis. • Installation, configuration, and ongoing application software maintenance above the O/S including

patches, fixes, and upgrades not administered at the Agency level. • Installation and configuration of ESM agents to facilitate monitoring. • Defines and implements appropriate system management alerts and notifications to CIO/OFT and

Agency application and database staff in conjunction with Agency staff. • Provide Level II and Level III response, as appropriate, to ESM alerts related to hardware and the

operating system. • Technical consulting and project team participation, limited to staff availability.

Standard Tivoli Monitoring for Gold and Platinum Servers

Tivoli monitoring is a required service for all Turn-Key Servers with Gold or Platinum Support in the CIO/OFT Data Center. Through the use of Tivoli monitoring software, the data center infrastructure that end user applications run on is continually monitored and, when necessary, alerts are generated to initiate corrective or pro-active actions.

For more information on ESM services, please reference the section on Standard Tivoli Monitoring for Production Servers under the Centralized Server Hosting section of the Data Center Addendum. The following defines Agency responsibilities for Silver Turn-Key Servers:

• Data Security and Integrity • Installation, configuration, and maintenance of all software above the O/S • Performance monitoring, analysis, and design • Management scripting • Application migration support • Capacity planning in conjunction with data administration groups • Primary Level II and Level III technical support for problem resolution • Participation in the CIO/OFT Change Control Management process

The following defines Agency Responsibilities for Gold and Platinum Turn-Key Servers:

• Data Security and Integrity • Testing and Quality Assurance of application code prior to migration to production environments • Provide at least one pre-production environment to be managed by the Data Center to be used for

migrations and testing of patches • Provide application testing in the event of scheduled infrastructure downtime • Provide Agency Level II and Level III response, as appropriate, to ESM alerts related to applications • Maintain maintenance contracts on software owned by Agency and managed by CIO/OFT

CIO/OFT will issue a proposal for turn-key server hosting within thirty (30) days of receiving a request for this service. Patch Management: a technical review will be conducted and a schedule for critical and high security alerts will be developed and distributed within seventy-two (72) hours of receiving the critical or high alert from the vendor community. The actual schedule for distribution to servers hosted by the Data Center will be based on the results of the technical review and will be scheduled over the course of four available maintenance windows. Patches identified as less than critical will be bundled and scheduled for deployment during normal maintenance windows as resources permit. The process will be escalated upon direction from the CIO/OFT ISO or if a critical patch poses a specific risk to the environment.


Offered To: The service is available to any New York State Agency.

Availability: The service is available with three different technical support options:

• Silver Support – Technical support is available from 7am until 5pm Monday through Friday excluding state holidays. Servers with Silver Support are not monitored and initial response to issues is on a best-effort basis.

• Gold Support – Technical support is available from 7am until 5pm Monday through Friday excluding state holidays. Servers with Gold Support are monitored and initial response to Severity 1 and Severity 2 issues is within two (2) hours.

• Platinum Support – Technical support is available 7/24 including state holidays. Servers with Platinum Support are monitored and initial response to Severity 1 and Severity 2 issues is within two (2) hours.


Data Center Service

Service:

Application Appliance Hosting

General Description: CIO/OFT hosts and monitors application appliances in the CIO/OFT Data Center. CIO/OFT will coordinate and schedule the installation, configuration and maintenance of hardware application appliances residing in the CIO/OFT Data Center, including any problem diagnosis associated with this environment. The focus is on providing a stable environment where application appliances reside, in addition to providing appropriate levels of administration and monitoring to meet specific needs of each Agency. The following defines the CIO/OFT responsibilities for the hosting of application appliances:

• Conduct an architectural review to determine appropriate placement • Coordinate ERB process and follow-through • Networking form (Firewall, IP, CSS, etc.) submittal, coordination, and follow-through • Submit hardware support calls to the maintenance vendor and follow-through • Respond to Tivoli alerts, if applicable, related to connectivity and hardware. If Tivoli monitoring is not

possible then respond to EHD tickets related to the appliance's functioning. (Customer will be responsible for alternative monitoring solution - see below).

The following defines Agency responsibilities for the hosting of application appliances:

• Involve IATS in the decision to pursue purchase of the appliance • Provide detail requirements for the proper functioning of the appliance • Provide detailed vendor maintenance information • Configure the appliance's "application" and provide all support thereto related to the functioning of that

application • Utilize alternative monitoring solution if Tivoli is not possible and submit EHD ticket for IATS to respond to

if related to hardware or connectivity. Application problems to be addressed by the Customer. • Provide support for the service that the appliance provides

Offered To: Available to existing and potential CIO/OFT Data Center customers.

Availability: Service available upon request.


6.0 Other Services

Data Center Service

Service:

Print Services

General Description: CIO/OFT offers print services that provide high volume and high speed laser printing. Standard Laser Print service includes the following laser paper types:

• White 8.5 x 11 cut sheet. • White roll paper. • White laser standard labels (Standard label size is 1” x 1 7/16”.) • 3-hole punch white 8.5 x 11 cut sheet. • 3-hole punch white roll paper. • Non-Standard Laser Print: • A set up fee will be required for any non-standard laser paper.

• Impact Print • A set-up fee and additional equipment fee will be applied for any print directed to the impact printers.

CIO/OFT Operations staff will ensure that printers are properly maintained and monitored. Print consoles are used to direct output to the appropriate printer. Printer output is visually scanned for print quality and alignment. Documents are re-queued if necessary. CIO/OFT will interface with a maintenance provider for maintenance and problem resolution.

Offered To: The service is available to any Agency using CIO/OFT Data Center server or mainframe services.

Availability: Based upon hours of operation of the console for the Agency.


Data Center Service

Service:

Multi-Site Name Resolution

SERVICE DESCRIPTION Overview Global Load Distribution and Failover services allow an application to reside in two NYS CIO/OFT virtual data centers for the purpose of multi-site high availability (H/A) or disaster recovery (DR). By using the service, CIO/OFT customers can mitigate risk by reducing the impact a site or component issue may have on application availability. The Global Load Distribution and Failover Services are available for servers residing in:

• Tier 1 (NYeNet or Internet) • DCN Legacy • Limited Customer Center • Limited Tier 2x networks

SERVICE COMPONENTS Global Site Selector (GSS) The core component of the service is a Global Site Selector or GSS. The device acts as an authoritative source for DNS inquiries and can alter the DNS response for load distribution or failover purposes. Note: The GSS does not function in any way to maintain session state or other connection-specific information. If a user is switched from one site to another due to failure or DNS expiration, session state will be lost unless handled by the application. Content Services Switch (CSS) While not absolutely required to use the GSS Service, the CSS is considered the best tool to gauge application health. Since the CSS is generally in use to provide single-site load balancing, it is generally the first point of entry into an application architecture and the best place to check application health. GSS SERVICE OPTIONS Load Distribution The GSS can be configured to direct traffic to each site in a Round-Robin fashion. This means that when one request comes in, the GSS will direct them to Site A whereas the next request will be directed to Site B. This process is repeated to essentially distribute the traffic to each of the sites configured. Automatic Site Failover One of the core strengths of the GSS is the ability to automatically perform a site failover when a site outage has occurred. The function is most useful when the secondary site is fully capable of serving the application without manual intervention. Some customers, however, may choose to implement this even when manual intervention is required simply to reduce the number of tasks which must be performed at the time of a site failure or disaster. Declared Site Failover Due to the complexity of many applications and the numerous dependencies that may exist, fully automatic site failover may not be feasible or desirable. In these cases, the GSS device is preconfigured with all the secondary site addresses but is not configured to perform health checks1 or perform site failover during an outage. The preconfigured rules on the GSS are used for Disaster Recovery Testing and for Disaster Failover when a customer management decision has been made to initiate the failover. At that time, CIO/OFT will be notified by

1 In some cases, it may be desirable to perform health checks if only for the purpose of displaying a Sorry Page during an application outage. This configuration is optional and can be very complex to implement.


the customer to disable all production DNS rules and to enable all DR DNS rules as previously established. Once initiated, the DR rules will be in place until it is later decided by customer management to fail-back to the production configuration. Upon notification, CIO/OFT will revert the rules back to production state. DESIGN CRITERIA AND CONSIDERATIONS The implementation of Automatic Site Failover or Load Distribution will rely on the ability of the GSS to ascertain application health at each of the available sites. Application Health Checks A standard GSS implementation will be configured to communicate with the CSS to ascertain application health. While GSS could be configured to communicate with another device such as the application web server, this should not be done unless a CSS is not being used for single-site load balancing. The CSS will be configured to initiate an HTTP GET to a specific server or server grouping. The response (or lack of response) this server returns will determine how the CSS reports application health back to the GSS.

1. The GSS will ask the CSS what the health is of aspecific application.

2. The CSS will send an HTTP GET request to a server that is configured to provide a health check response. Note: More than one server could be configured to provide this response.

3. Once the server has ascertained the application health, it will respond back to the CSS with a specific HTTP return value.

4. When the CSS receives this value, it will respond to the GSS based on predetermined CSS rules.

Since each application will have different criteria for measuring application health, it is up to the application owner to design this health check and implement it into a simple HTTP response.

While HTTP is used by CSS to collect application health information, the back-end health check process is not restricted to this protocol. Some applications may not currently use web services but could still be supported by GSS/CSS systems through the configuration of a specialized health check system or process.

Example Health Check: The CSS is configured to check URL:http://www.somesite.com/test (testing frequency is configurable based on application requirements) 1) The CSS is configured to expect the response of “Site Available”

2) The http://www.somesite.com/test URL is configured with a customer-provided script that performs the following actions

a) Call a specific script on the application server

b) The application server script will log into the application through SiteMinder

c) The application server script will then query the database for known information

d) The database server responds with the known information

e) The application server tests to see that the provided information is as expected

f) The application server responds back to the web server with a good result

g) The web server responds back to CSS with a “Site Available” message on the page.

GSS

CSS

SERVER 1 SERVER 2(OPTIONAL)

SERVER 3(OPTIONAL)

1

3

4

2


http://www.somesite.com/test

3) If CSS receives the “Site Available” message, it reports good health back to GSS and that site remains in the load distribution group

4) If any of the functions in item three (3) fail, the CSS will not receive the correct message and will report bad health back to the GSS which will pull that site out of the load distribution group

5) If the CSS does not respond to the GSS, site health is assumed bad and the site is removed from the load distribution group

If neither site responds as healthy (or not at all), GSS can be configured to redirect all responses to a Sorry Page2. 1 In some cases, it may be desirable to perform health checks if only for the purpose of displaying a Sorry Page during an application outage. This configuration is optional and can be very complex to implement. 2 A Sorry Page is generally a simple HTML page that reports an application issue to end users attempting to connect. Where possible, this page should be hosted on a network that isolated from problems that would generally affect the primary and secondary sites.



2 A Sorry Page is generally a simple HTML page that reports an application issue to end users attempting to connect. Where possible, this page should be hosted on a network that isolated from problems that would generally affect the primary and secondary sites.


Data Center Service

Service:


General Description: CIO/OFT will provide the Agency with Customer Architectural Consulting and Review services based on CIO/OFT staff availability. Customer Architectural Consulting and Review Services are available to existing or potential CIO/OFT Data Center customers where the infrastructure would be hosted in a CIO/OFT Data Center. The services can include review of a customer agency or vendor supplied architecture, participation in the architectural design team, and interfacing with the CIO/OFT Enterprise Architecture Board for review and disposition of any potential exceptions. The service is currently limited by staff availability but will be expanded in the future.




Data Center Service

Service:


General Description: CIO/OFT will provide the Agency with storage resources (Storage Area Network [SAN] and Disk) required to meet business needs. CIO/OFT will provide complete management of the SAN and Disk resources.

• Provisioning: Add or remove open systems servers from the SAN and/or allocate and reallocate disk volumes from servers or mainframe systems as required. Configure and expand storage network and disk resources as required.

• Monitoring and Problem Resolution: CIO/OFT will actively monitor the storage network and disk infrastructure for impending failures. CIO/OFT will resolve storage infrastructure problems and coordinate problem resolution with vendors when appropriate. CIO/OFT will work with vendors to upgrade firmware and patches associated with the SAN and Disk Subsystems as required.

• Performance: CIO/OFT will review the storage infrastructure performance within the SAN and Disk Subsystems. CIO/OFT will work with Agency and system administrators to identify storage related performance issues and resolve or recommend solutions.

• Capacity Planning: CIO/OFT will review storage infrastructure resource use and reconfigure, tune or expand the infrastructure as necessary to make resources available to meet Agency requirements.

• Architecture and Design: CIO/OFT will architect and design cost effective SAN and tiered disk solutions based on best practices for security, performance, availability, and scalability.

• Reporting and Billing: CIO/OFT will track disk resource use by Agency and provide monthly billing reports.

CIO/OFT offers the following types of Data Storage Services at specific rates:

Data Storage – IBM Tier 1 IBM Tier 1 is designed for the highest availability and performance requirements. It is built on completely redundant components with features for online firmware upgrades to minimize downtime. It uses the most current disk technology and is based on a cache-centric design that provides fastest response times while sustaining a large number of data transfers.

Tier 1 is most often selected for Business critical 24x7 Databases, fileservers and e-mail applications as well as for I/O intensive applications such as data warehouses. Tier 1 is available to most Open Systems servers and to IBM mainframe operating environments.

Data Storage - Tier 3c Tier 3c is designed as inexpensive magnetic storage used for long-term storage of fixed content, archived or reference data. The storage is normally characterized by much lower performance than and Tier 1 storage. Data in Tier 3c is automatically mirrored and does not require backup to tape.

Tier 3c is most often selected for storage of images, documents, static content, archived data and reference data such as GIS data. It is recommended as the replacement for optical storage and for use with applications such as FileNet, Documentum, and IBM Content Manager.

This tier is available to most Open Systems servers.

Data Storage - Tier 3i Tier 3i is designed as inexpensive magnetic storage used for test, quality assurance, development, or production application server environments. The storage is normally characterized by lower performance than Tier 1. It provides for the use of less expensive ATA disk drives and ethernet interfaces using the iSCSI


protocol or as Network Attached Storage (NAS – NFS only). The Disk Subsystem uses a private, secure, and redundant Ethernet Data LAN rather than fiber channel as used by Tier 1 and Tier 2 disk. Some uses may be as disk for Windows and Unix applications, SQL databases, Windows file servers, and e-mail applications. Tier 3i provides a highly resilient platform through the use of 24x7 predictive self-monitoring with vendor phone-home, RAID, battery backup, redundant power supplies, hot-swappable power and cooling, ethernet trunking/link aggregation, network interface port failover, data mover failover and internal environmental monitoring. It also directly interfaces with our EMC Control Center (ECC) SAN management and monitoring system. This tier is available to most Open Systems servers.

EMC SAN EMC SAN is designed for high availability requirements specifically for the Unisys mainframe operating environments. EMC SAN is available to Unisys 2200 and A-Series operating environments.

Mainframe-only Storage Management Services: • File Management (Unisys 2200 environment): CIO/OFT operations staff provides file management

services that include the allocation, monitoring, backup, and recovery of disk storage. Backup and recovery services apply to both daily operating functions and Business Continuity/Disaster Recovery. Staff will track storage usage, manage disk and file usage, develop offsite backup procedures and, in conjunction with technical support, tune the system file environment for efficiency. For new Agencies, it is the Agency’s responsibility to perform backup and disk usage monitoring activities on Unisys mainframes.

• Storage Management (IBM Mainframe systems): CIO/OFT technical staff will provide maintenance of storage classes and SMS constructs as necessary to manage the allocation and storage of data in DASD and Tape media. The service includes management of storage volumes as appropriate and providing criteria and guidelines to the customer Agency Information Security Officer where applicable to manage security access permissions.

Offered To: The service is available to any Agency that uses either CIO/OFT mainframe or central server hosting services. The exceptions are File Management for Unisys 2200 and Storage Management for IBM Mainframe systems, which are limited to Agencies who negotiated the service at time of transfer of function and are not available to new Agencies.

Availability: The service is available during Standard Business Hours.


Data Center Service

Service:

Facilities Monitoring, Operations, Planning and Setup

General Description: CIO/OFT will monitor and operate the CIO/OFT Data Center site facilities. This service includes operating, monitoring and maintaining the environment to ensure availability. It includes reacting appropriately to abnormal environmental situations. This includes Heating, Ventilation and Air Conditioning (HVAC), electrical power with Uninterrupted Power Supply (UPS) protection, smoke detection, cooling, fire suppression and water leak detection. Some sites also include diesel generator backup in case of commercial power failure. Physical security of the equipment will also be ensured as all sites have highly secure access protection and procedures. CIO/OFT will coordinate with hardware vendors for repairs as needed. CIO/OFT will place CIO/OFT-owned equipment properly in the CIO/OFT Data Centers to ensure a robust physical environment for delivering services to the Agency. Costs for this service are embedded in overall fees. CIO/OFT will also assist the Agency in the planning and placing of Agency–owned equipment that is fully managed by CIO/OFT, in CIO/OFT Data Centers. This includes site preparation, installation planning, and ongoing facilities maintenance. CIO/OFT will charge a fee for these services. For each piece of Agency-owned equipment placed in the CIO/OFT Data Center, a fee will be charged at the time of initial deployment to cover the facilities costs to support it. An additional fee will be charged for every piece of equipment for HVAC/Electrical and rent (if appropriate). CIO/OFT installs and maintains a wide selection of hardware to meet the Agency’s needs. CIO/OFT will notify the Agency when hardware the Agency is operating on may be impacted by an upgrade and negotiate a schedule that is mutually agreeable. Agency requests for new hardware will be analyzed for appropriateness to the overall architecture.

Offered To: The service is available to any Agency that houses their equipment in a CIO/OFT Data Center that uses either CIO/OFT mainframe or central server hosting services.

Availability: The physical environment is maintained 24x7. Planning and setup is available during Standard Business Hours.


Data Center Service

Service:

Disaster Recovery Testing

General Description: The Agency is responsible for developing a disaster recovery/business continuity plan and the associated staffing and funding. Agencies can take advantage of the resilience offered in the Enterprise Infrastructure; however, they are responsible for the implementation of Disaster Recovery capabilities. CIO/OFT can assist the Agency in answering questions. CIO/OFT is reviewing options for future potential disaster recovery services. Currently, CIO/OFT Operations and Technical Services staff will participate in live Disaster Recovery tests to meet Agency specific needs. Technical support staff will be provided to support the system IPLs/reboots and for general technical support to diagnose and resolve technical issues arising during the test process. Cost for staff will be determined and charged as a special service. In addition, travel, lodging, and meals will be reimbursed by the Agency. CIO/OFT will work with Agency staff to resolve technical issues during each test.

Offered To: The service is available to any Agency that uses either CIO/OFT mainframe or central server hosting services.

Availability: Conduct testing at prescribed time interval.


Data Center Service

Service:

Offsite Storage

Overview Offsite storage services are defined as secure transport and off-site storage of site mission critical computer information systems records in the form of magnetic media. CIO/OFT has contracted with a vendor to provide this optional service to NYS agencies to facilitate the offsite storage of critical backup tapes for disaster recovery. CIO/OFT's vendor manages and maintains an underground facility which is environmentally controlled, secure, and located more than 50 miles from Albany. Detail CIO/OFT signed a contract for offsite media storage in February 2006. The winning vendor employs a fleet of environmentally controlled vehicles to securely transport media. The media is then stored in state-of-the-art environmentally controlled vaults. Business needs of the customer determine the frequency and numbers of volumes that are sent and returned off-site. Services are available 7 days a week. If a Data Center CIO/OFT customer agency elects to take advantage of these services, CIO/OFT will perform the day-to-day logistics of moving the mainframe tape based on the agencies defined media backup plan. Agencies must provide CIO/OFT Finance Office a cost code and authorization to encumber sufficient funds to pay for courier, storage and media handling by the vendor. Standard service includes transportation, secure storage, and scheduled return of tapes. There is an option to store multiple tapes in containers which costs less than storing and racking individual tapes. Transportation costs will be divided amongst the agencies that participate. Tapes may be recalled from the storage facility as needed for data recovery. Benefits IT industry research analysts recommend offsite tape storage for critical data as a best practice and have identified the following guidelines for selecting a tape storage provider:

• A minimum distance of 10 miles between the data center and the tape storage facility. • Regional risks (e.g., flood plains, transportation centers) around the storage facility should be evaluated

and deemed acceptable. • Facilities management that includes: • Consistent environmental controls (e.g., temperature, humidity, fire suppression)

o Secure container storage (optional) o Published service level guidelines o Annual facility audit and tape retrieval audit o The winning vendor provides offsite tape media courier and storage services to meet these

standards. Audience Offsite Storage services are available to any NYS agency. The CIO/OFT Data Center staff will manage the movement of media for customers using IBM, Unisys Voyager, and Unisys Libra mainframes in the CIO/OFT Data Center. Agencies that are not CIO/OFT Data Center customers can still take advantage of the CIO/OFT offsite storage contract, but the customer will manage the movement of the media. This contract also provides for secure transport of media to a disaster recovery site defined by the customer. Cost


There are the items that incur charges during the first year of the contract (i.e. scanners, transport/storage cases, etc.), and there are transaction costs based on the actual frequency and number of items sent and stored offsite. Please contact your Customer Relations Manager for the cost of these items: Shared expenses by all the participating agencies:

• Daily Courier Services – The courier costs are allocated to customer agencies in the following manner: Agency monthly courier cost = total cost of courier charges for the month X (number of agency trips/total number of trips)

o For non Data Center agencies, the costs are calculated differently, please call CIO/OFT Customer Relations for specific information.

• Optical Scanners - used for verifying tape and box shipments Individual expenses based on Agency usage:

• Storage - Boxes hold approximately 60 tape cartridges • Handling Transaction Fee - A transaction is defined as the movement of tapes/boxes "into off-site

storage" or "out of off-site storage" • On Demand (Emergency) Courier Services • Hot Site Deliveries • Hot Site Courier Services • Plastic Storage Containers

All offsite media will require bar-coded labels to allow optical scanning. Additional items at minimal or possibly no cost are identifying labels for the boxes and cardboard inserts for transporting the tape cartridges within the boxes (supplied by the vendor). Order Process Any agency interested in learning more about this service should contact their CIO/OFT Customer Relations Manager at [email protected] or 1-866-789-4OFT (4638). The Customer Relations Manager will schedule an informational session with the interested agency and CIO/OFT offsite storage experts to present this service offering, address agency questions, provide the agency with a template for estimating costs, and discuss agency offsite storage needs. To order this service:

1. Customer Agency defines its offsite storage backup procedures, schedules, jobs, and retentions. The Customer Agency completes a copy of the CIO/OFT Cost Estimator and CIO/OFT Estimated Offsite Storage Service Proposal documenting the agency’s estimated annual offsite storage services cost, the agency’s appropriate cost center code for this service and the signatures of the agency’s IT Director, ISO Officer and Fiscal Director agreeing to these services.

2. Customer Agency contacts its CIO/OFT Customer Relations Manager in writing indicating its intent to establish offsite storage services through CIO/OFT Contract number C000124.

3. The Customer Agency’s Finance Office faxes a copy of the completed Cost Estimator and Estimated Offsite Storage Service proposal to the CIO/OFT Finance Office and the agency’s Customer Relations Manager.

4. CIO/OFT Finance Office encumbers the Customer Agency’s funds and notifies the CIO/OFT Customer Relations Manager, who will update the Customer Agency.

5. CIO/OFT Customer Relations Manager contacts the appropriate site manager to initiate service provision.

Offered To: The services are available to any Agency using a mainframe or server located in the CIO/OFT Data Center.


Data Center Service

Service:

Miscellaneous Services

General Description: Network Monitoring by Operations

Console operators will monitor an Agency’s network. Procedures and monitoring tools will be supplied by the Agency along with appropriate instructions.

Help Desk – non-prime shift coverage

Console operators will monitor an Agency’s help desk phones during the off shifts. The Agency will provide detailed procedures as to any follow-up action that is required, such as placing trouble calls with vendors, calls to Agency personnel and/or problem resolution attempts.

CICS Support

CIO/OFT technical support staff will install, customize, and maintain an Agency’s CICS environment. The process includes ongoing day-to-day subsystem operational support, subsystem migration to vendor supported levels, and the definition/activation of new transaction processing environments. CIO/OFT will provide level-2 and level-3 technical support to the Agency as appropriate. The support consists of subsystem performance monitoring and tuning, subsystem architecture design, and level-2 application problem determination.

DB2 Support

CIO/OFT technical support staff will install, customize, and maintain an Agency’s DB2 environment. The process includes ongoing day-to-day subsystem operational support consisting of: backup and recovery of subsystem control structures, subsystem problem determination and resolution, subsystem migration to vendor supported levels, and the definition/activation of new DBMS environments. CIO/OFT will also perform subsystem performance monitoring and tuning, ensuring the Agency’s DBMS environments operate efficiently.

Database Software Support (Unisys Voyager systems)

CIO/OFT technical support staff will review, install, configure, and maintain all database software (DMS, RDMS, UREP, SFSW, QLP, and IRU). They will also diagnose database problems and monitor all UDS systems in order to maintain optimum performance of the software and the applications using it.

Offered To: The service is available to Agencies who negotiated this service at time of transfer of function. Expansion of these services to new Agencies is not available at this time.

Availability: Monitoring services are available based on the Agency’s agreed upon mainframe console hours of operation. Technical support services are available during standard business hours.


7.0 Agency Responsibilities and Tools

Network/Communications

The Agency and its subcontractors are responsible for remote network/communications that extends outwards from a jointly agreed upon demarcation point. Any network analysis and problem investigation is the joint responsibility of CIO/OFT and the Agency, including subcontractors. The notification to CIO/OFT of a Network/Communication problem may be made by opening a helpdesk ticket or other agreed upon process, depending on the Agency agreement with CIO/OFT. CIO/OFT can support remote network/communications beyond the demarcation point if Agencies request CIO/OFT Customer Networking Services, at an additional cost.

System Access/Data

The Agency will be responsible for implementing access security protections in the Agency’s defined logical partition (LPAR). CIO/OFT will assist the Agency with their testing efforts to validate Agency and CIO/OFT security protections limit or prevent LPAR resource access as desired. Such testing may include contracting with outside vendors specializing in mainframe/network security planning and penetration.

Documentation The Agency will be responsible for supplying documentation of standard operating procedures for any services provided by the Data Center. CIO/OFT will maintain and update documentation supplied by the Agency.

Run Books

The Agency will be responsible for supplying run books documenting all jobs run and monitored by the Data Center, including procedures for error handling.

Backup

The Agency will be responsible for defining and monitoring their logical backup needs and any payment for third party off-site storage that may be incurred.

Application Development Support

Application development, support and problem diagnosis support for application issues is the sole responsibility of the Agency.


The Agency is responsible for notifying CIO/OFT of projected major changes in resource consumption including changes that have significant effect on system use, such as tape, print, batch, processing flow, new programs or modifications in transaction volumes.

Central Server Hosting, Basic and Standard Support

See Central Server Hosting Service description.


8.0 Performance Measurement and Metrics

Service Quality

CIO/OFT recognizes that the services being provided are critical to the Agency. CIO/OFT is also dedicated to providing a reliable, robust environment, with a focus on service quality. There is substantial complexity to the technology and services being provided. CIO/OFT provides performance measures and metrics as detailed in this section.

Service Problem Communication

Unlike most other CIO/OFT services, when using Data Center Services, the initial point of contact is the Data Center. The Data Center will create a help desk ticket, verify with the Agency as needed, and then forward to the Enterprise Help Desk for distribution and escalation.

Mainframe Maintenance

An agreed upon schedule is followed for mainframe maintenance.


Data Center Service Performance Metric

Service:

Mainframe Services

Reporting: CIO/OFT will provide monthly reports included detailing the usage by the Agency of services, including:

• The amount of mainframe CPU time used the past month, and a graphical comparison of usage for each month of the previous year

• Any special services used, and the total quantity used • The number of manual tapes mounted the past month, and a graphical comparison of usage for each

month of the previous year • The number of robotic tapes mounted the past month, and a graphical comparison of usage for each

month of the previous year • The rate for each service provided • The total cost for each service provided • The total amount due and any prior overdue payments

Additionally, the Data Center can individualize specific reports for services performed for each Agency. The reports include:

• IPLs/reboots • System availability reports (sample provided in Table 1 below) • Monthly schedules • Technical request tables • Weekly change summaries • Explanations of changes

Service Level Availability: The service level objective for mainframe uptime is: For Standard Business Hours: 99% annualized platform availability, reported monthly, calculated by the formula:

Percent availability = (Standard Business Hours platform actually available per month x 100) / (total Standard Business Hours per month during that time period – hours of scheduled downtime and acts of force majeure)

For Non-standard Business Hours: 97% annualized platform availability, reported monthly, calculated by the formula:

Percent availability = (Non-Standard Business Hours platform actually available per month x 100) / (total Non-Standard Business Hours per month during that time period – hours of scheduled downtime and acts of force majeure)

Service Problem Addressed: For 97% of Severity 1 and Severity 2 production problems, investigation and triage will begin within 3 Standard Business Hours.


Table 1 Sample system availability table, Unisys operations (May 2004)

Albany Production System

NYC Production System

Albany End-user System

SED Production System

Civil Service Production System

Prime (1) Non-prime (2) Overall

100% 98.9% 99.2%

100% 98.6% 99.0%

100% 98.7% 99.4%

100% 100% 100%

100% 99.5% 99.6%

Notes: 1) Prime 2) Non-prime

Mon—Fri 08:00—18:00 18:00—08:00

Mon—Fri 08:00—18:00 18:00—08:00

Mon—Fri 07:00—24:00 23:00—07:00

Mon—Fri 06:00—20:00 21:00—06:00

Mon—Fri 07:00—19:00 19:00—07:00



Service:


Reporting: Upon request, CIO/OFT will provide monthly reports included with the Agency’s monthly bill detailing the usage by the Agency of services, including:

• The number of work units charged for the month with a detail report that identifies each Agency server with equated work units

• Any special services used, and the total quantity used • The rate for each service provided • The total cost for each service provided • The total amount due and any prior overdue payments • Monthly server availability report

Service Level Availability: The service level objective for central server hosting is: For Standard Business Hours: 99% annualized platform availability, reported monthly, calculated by the formula:


Service Problem Resolution: For 97% of Severity 1 and Severity 2 production problems, investigation and triage will begin within 3 Standard Business Hours.



Service:

Print Services


• The number of pages printed the past month, and a graphical comparison of usage for each month of the previous year

• Other print services charges • The rate for each service provided • The total cost for each service provided • The total amount due and any prior overdue payments



Data Center Service Performance Metric Service:



• The amount of data storage used the past month • The rate for each service provided • The total cost for each service provided • The total amount due and any prior overdue payments

Service Level Availability: The service level objective for Data Storage Services is: For Standard Business Hours: 99% annualized platform availability, reported monthly, calculated by the formula:


Service Problem Resolution: For 97% of Severity 1 and Severity 2 production problems, investigation and triage will begin within 3 Standard Business Hours.

terms of service: data center addendum

Documents

mainframe services

data storage services

mainframe tape services

central server hosting

open systems server

server monitoring

associated print services

production control services