911 data center operations - denver · we have completed an audit of the 911 data center operations...

Dennis J. Gallagher

Auditor

Office of the Auditor

Audit Services Division

City and County of Denver

911 Data Center Operations Performance Audit

June 2010

The Auditor of the City and County of Denver is independently elected by the citizens of Denver. He is

responsible for examining and evaluating the operations of City agencies for the purpose of ensuring the

proper and efficient use of City resources and providing other audit services and information to City

Council, the Mayor and the public to improve all aspects of Denver’s government. He also chairs the

City’s Audit Committee and oversees the City’s Comprehensive Annual Financial Report (CAFR)

The Audit Committee is chaired by the Auditor and consists of seven members. The Audit Committee

assists the Auditor in his oversight responsibilities of the integrity of the City’s finances and operations,

including the integrity of the City’s financial statements. The Audit Committee is structured in a manner

that ensures the independent oversight of City operations, thereby enhancing citizen confidence and

avoiding any appearance of a conflict of interest.

Audit Committee

Dennis Gallagher, Chair Robert Bishop

Maurice Goodgaine Robert Haddock

Jeffrey Hart Bonney Lopez

Timothy O’Brien

Audit Staff

Audrey Donovan, Deputy Director, CIA

Stephen E. Coury, IT Audit Supervisor, CISA

Robert Pierce, Lead IT Auditor, CISA

Aaron Pratt, Senior IT Auditor, CISA

Brandon Blomquist, Staff IT Auditor

You can obtain free copies of this report by contacting us at:

Office of the Auditor

201 W. Colfax Avenue, Dept. 705 Denver CO, 80202

(720) 913-5000 Fax (720) 913-5026

Or view an electronic copy by visiting our website at:

www.denvergov.org/auditor

http://www.denvergov.org/auditor

To promote open, accountable, efficient and effective government by performing impartial reviews and other audit

services that provide objective and useful information to improve decision making by management and the people.

We will monitor and report on recommendations and progress towards their implementation.

City and County of Denver 201 West Colfax Ave., Dept. 705 Denver, Colorado 80202 720-913-5000 FAX 720-913-5247 www.denvergov.org/auditor

Dennis J. Gallagher

Auditor

June 17, 2010

Ms. Molly Rauzi, Chief Information Officer

Technology Services


Mr. Alvin J. LaCabe, Jr.

Manager of Safety


Dear Ms. Rauzi and Mr. LaCabe:

Attached is the Auditor’s Office Audit Services Division’s report of their audit of the 911 Data

Center Operations for the period January 1, 2009 through January 31, 2010. The purpose of the

audit was to assess the efficiency and effectiveness of controls related to operating the 911

Data Center, such as managing software changes, patching systems, and providing disaster

recovery capability.

The audit revealed that while many advances have been made at the data center, procedural

improvements are needed to maintain system reliability.

If you have any questions, please call Kip Memmott, Director of Audit Services, at 720-913-5029.

Sincerely,

Dennis J. Gallagher

Auditor

DJG/ap

cc: Honorable John Hickenlooper , Mayor

Honorable Members of City Council

Members of Audit Committee

Ms. Roxane White, Chief of Staff

Mr. Claude Pumilia, Chief Financial Officer

Mr. David T. Roberts, Chief Services Officer

Mr. David Fine, City Attorney

Mr. L. Michael Henry, Staff Director, Board of Ethics

Ms. Lauri Dannemiller, City Council Executive Staff Director

Ms. Beth Machann, Controller

Mr. Mel Thompson, Deputy Manager of Safety


To promote open, accountable, efficient and effective government by performing impartial reviews and other audit

services that provide objective and useful information to improve decision making by management and the people.

We will monitor and report on recommendations and progress towards their implementation.

City and County of Denver 201 West Colfax Ave., Dept. 705 Denver, Colorado 80202 720-913-5000 FAX 720-913-5247 www.denvergov.org/auditor

Dennis J. Gallagher

Auditor

AUDITOR’S REPORT

We have completed an audit of the 911 Data Center Operations for the period January 1, 2009

through January 31, 2010. The purpose of the audit was to assess the efficiency and

effectiveness of controls related to operating the 911 Data Center, such as managing software

changes, patching systems, and providing disaster recovery capability.

This performance audit is authorized pursuant to the City and County of Denver Charter, Article

V, Part 2, Section 1, General Powers and Duties of Auditor, and was conducted in accordance

with generally accepted government auditing standards. Those standards require that we plan

and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis

for our findings and conclusions based on our audit objectives. We believe that the evidence

obtained provides a reasonable basis for our findings and conclusions based on our audit

objectives.

The audit revealed that while many advances have been made at the data center, procedural

improvements are needed to maintain system reliability. Specifically audit work determined that

internal controls for workstation patching, antivirus updates, offsite storage of archive backups,

documentation of change management procedures, and periodic review of building and

system access all need to be improved.

We extend our appreciation to the personnel who assisted and cooperated with us during the

audit.

Audit Services Division

Kip Memmott, MA, CGAP, CICA

Director of Audit Services


TABLE OF CONTENTS

EXECUTIVE SUMMARY 1

Procedural Improvements Needed to Maintain System Reliability 1

INTRODUCTION & BACKGROUND 2

SCOPE 3

OBJECTIVES 3

METHODOLOGY 4

FINDING 1 5

Workstation Patching and Antivirus Updates Not Performed or

Monitored for Successful Installation 5

FINDING 2 5

Archive Backups Not Stored Offsite 5

FINDING 3 6

Informal Change Management Procedures 6

FINDING 4 6

No Formal Procedure or Periodic Review for Building and System Access 6

AGENCY RESPONSE 8

P a g e 1

OOffff iiccee ooff tthhee AA uuddiittoorr

EXECUTIVE SUMMARY

Procedural Improvements Needed to Maintain System Reliability

Over the past two years significant improvements have been made for the 911

Communications Center and its data center operations. A new Recovery Operations

Center has been established which supports complete offsite recovery of both the 911

call taking and the data center in the event of a disaster at the main facility. Control

over the installation of software changes have been enhanced, along with many

building infrastructure improvements.

Although many advances have been made at the data center, formalization and

improvements to strengthen existing controls are needed. For example, audit work

revealed that the installation of critical security patches and antivirus updates were not

monitored to ensure that all required updates were applied. Audit work also found that

important data archives were not stored offsite, that production software change

management procedures were informal and do not produce evidence of necessary

approvals, and that access lists were not periodically reviewed for who has access to the

physical building or to critical computer systems. These issues could affect overall system

reliability, inhibit the ability to recover important City data after a disaster, or allow the

interference or disruption of critical operations.

P a g e 2

CCiittyy aanndd CCoouunnttyy ooff DDeennvveerr

INTRODUCTION & BACKGROUND

Advances at the 911 Communications Center

The Denver 911 Communications Center serves as the Public Safety Answering Point

(PSAP) for 911 telephone calls into the City and County of Denver. Personnel at this

center dispatch police, fire, and medical personnel (ambulances) in response to citizen

calls for emergency assistance.

Over the past two years significant improvements have been made for the 911

Communications Center and its data center operations. Specifically, a new Recovery

Operations Center (ROC) has been established which supports complete offsite recovery

of both the 911 call taking and the data center in the event of a disaster at the main

facility. Controls over the installation of software changes have been enhanced,

including the separation of the test, quality assurance, and production environments.

Building improvements include enhancements to ventilation, physical access security,

and cabling infrastructure.

Funding to establish the Recovery Operations Center came from the federal government

for the 2008 Democratic National Convention held in Denver. Federal safety and

security requirements mandated that the convention have its own dedicated PSAP to

serve the area immediately surrounding the convention site. As such, the City received

federal grant monies for the purchase and upgrade of hardware and software and staff

training. The grant allowed the City and County of Denver to retain all of the physical

improvements funded by the convention, thus providing the City with an ongoing

recovery capability as critical servers and call taker workstations are backed up at the

ROC through duplicate hardware and software configurations. The systems at the ROC

are routinely tested to ensure they remain operable and current.

Challenges to Keeping Software Up to Date

The establishment of the ROC has allowed the City to better address challenges related

to software updates. For example, a regular use of the ROC is to host 911 operations

while system upgrades are periodically performed on the Computer Aided Dispatch

(CAD) system. While the main systems are taken down for maintenance and upgrades,

the 911 operations center staff operates from the ROC. This process provides for a

controlled and uninterrupted transfer of operations with minimal to no impact on the

City’s ability to answer 911 calls.

In addition to CAD software updates, servers and workstations undergo regular

maintenance, patching, and updating. In order to minimize the impact on critical

operations, special consideration must be given to both the testing of updates and the

time of day for when updates are applied. The ROC provides the City with additional

flexibility for this purpose.

P a g e 3


Although a rare occurrence, both operating system software and antivirus vendors have

released defective updates that have caused system outages to their respective

customer bases.1 Before operating system software updates are applied to the CAD

servers, they must be tested and approved by the vendor company that supports the

Computer Aided Dispatch System. Before antivirus updates are applied, they must be

proven to be stable. The timing for when updates are applied needs to be coordinated

so that critical workstations and servers are not re-booted while being used during a

production shift.

SCOPE

The audit examined and assessed the efficiency and effectiveness of controls over data

center operations for the 911 Communications Center. The audit period extended from

January 1, 2009 through January 31, 2010.

OBJECTIVES

Audit objectives were to ensure:

Change controls provided for: the separation of processing environments for test,

quality assurance, and production; the separation of duties for the roles

performed by software developers, system testers, and end users; and that

changes are authorized, tested and approved before being implemented into

production;

Security settings limited access to authorized individuals for Computer Aided

Dispatch (CAD) systems at the application, database, operating system, and

physical security levels;

Access management controls limited employee access to specific job functions

and that access to City systems and data is removed when individuals terminate

their employment with the City;

Operational controls provided for system backup and recovery capability for the

CAD systems;

All relevant security patches were installed on all 911 computers; and

Antivirus definitions were up to date on all 911 computers.

1 For example, on April 21, 2010, many PCs within the City were not usable due to a defective antivirus update file.

P a g e 4


METHODOLOGY

We utilized multiple methodologies to achieve audit objectives. These evidence

gathering and analysis techniques included, but were not limited to:

Interviewing personnel in Technology Services and reviewing selected policies

and procedures related to CAD and its infrastructure;

Utilizing Computer Assisted Auditing Techniques (CAATs) to compare 10,204

employees terminated since 2005 to the population of 1,510 individuals with

active user accounts within the CAD system;

Directly observing physical access controls in place at both the main and

recovery data centers and verifying that individual access to the data center

facilities was granted to current authorized employees;

Directly observing environmental controls in place at the data centers supporting

the CAD systems through onsite inspection and examination of maintenance

records;

Examining evidence for backup and offsite storage of media;

Obtaining access to Active Directory Users and Computers (ADUC) for examining

login account access and information;

Reviewing Windows Server Update Services (WSUS) reports for security patch

status;

Reviewing McAfee ePolicy Orchestrator reports for antivirus updates; and

Reviewing change management processes and procedures for CAD software

modifications.

P a g e 5


FINDING 1

Workstation Patching and Antivirus Updates Not Performed or

Monitored for Successful Installation

Technology Services utilizes automated software tools to apply critical system patches

and antivirus software updates to City computers. Our testing identified computers with

missing updates and others that had not been updated for several years.

For the computers missing updates, we found that the software tools did not accurately

report their update status and that there was no management follow-up process to

ensure that all patches and antivirus updates were being applied successfully. For the

computers that were not updated for several years, responsibility for who was to perform

the updates had not been established.

Workstations that are not patched against known system vulnerabilities and/or do not

have up to date antivirus software could be susceptible to malicious computer software

that may disrupt normal operations and facilitate unauthorized access and the

subsequent disclosure, misuse and/or destruction of sensitive City information.

Recommendation

1. Technology Services should establish responsibility for applying tested and approved

security patches and stable antivirus updates for all computers at the 911 center and

implement a formal follow-up process to ensure the updates are being applied

successfully. The timing of updates to critical servers and workstations should be

performed during scheduled maintenance periods as to not interfere with critical

production shifts.

FINDING 2

Archive Backups Not Stored Offsite

Two important data archives are backed up to enable the recovery of important

historical information related to 911 calls. One data archive is backed up on a real time

basis to optical media and the other is backed up on a daily basis to tape media.

However, due to incomplete procedures, neither of these two data archives have

backups stored offsite. Without offsite storage of backup media, there is an increased

risk that important historical data will be lost in the event of a data center disaster.

Recommendation

2. Technology Services should develop formal procedures to store important data

archive backups at an offsite location, such as at the Recovery Operations Center

through the physical transport of media or through remote backup technology.

P a g e 6


FINDING 3

Informal Change Management Procedures

Change management procedures for moving proposed changes into production are

not documented and do not provide formal evidence of approvals. Currently, proposed

changes to production software are tested from both a systems and end-user

perspective. System testing takes place in a development environment and

acceptance testing by end-users is completed in a training environment. When both

systems personnel and end-users agree to implement the proposed change, the vendor

is allowed to install the change on the production server.

Although this process provides for separation of testing environments and separation of

testing roles, the overall process is not documented and approvals are provided on a

verbal basis rather than being formally documented. Without a formally documented

and monitored change management process there is an increased risk that

unauthorized changes may go undetected which could lead to unintended application

downtime or processing errors.

Recommendation

3. Technology Services should formally document its production software change

management policies and procedures for the 911 center, including its separation of

testing environments and separation of testing duties. The procedures should also

provide formal evidence of authorization, testing results, and approvals, including

user sign-offs.

FINDING 4

No Formal Procedure or Periodic Review for Building and System

Access

The 911 Communications Center was undergoing a major upgrade to its building security

access system during the audit that corrected several discrepancies with the former

system. Audit work confirmed that only current and authorized employees have access

under the new building access system.

In addition to building security, we reviewed system user access and identified individuals

with inappropriate or unnecessary access. Discrepancies for both building and

computer system access are a result of inconsistent procedures, a lack of clear authority

for granting access, and the absence of a periodic review of access lists.

Without standardized procedures, there is an increased risk that access may not be

consistent with employee job functions which may result in employees or former

P a g e 7


employees retaining access for which they are no longer authorized. The use of

unauthorized access could be used to interfere with or disrupt critical operations.

Recommendation

4. Technology Services and Communications Center Management should formalize

building and system access procedures to ensure that access is authorized and

granted according to employee job function, adjusted when employee roles

change, and removed when an employee transfers out of the Communications

Center or terminates employment with the City. Procedures should ensure that IDs

are unique in order to maintain accountability for both individual building and system

access. Both building and system access should be periodically reviewed, perhaps

on a quarterly basis, to ensure they remain accurate.

P a g e 8


AGENCY RESPONSE

P a g e 9


P a g e 10


P a g e 11


911 data center operations - denver · we have completed an audit of the 911 data center operations...

Documents