diamond management technology c na, inc diamond cook county clerk sequoia... · diamond management...

DIAMOND MANAGEMENT & TECHNOLOGY CONSULTANTS NA, INC.

REPORT ON THE USE OF VOTING TECHNOLOGY IN THE NOVEMBER 7TH COOK COUNTY ELECTION

PREPARED FOR

SEQUOIA VOTING SYSTEMS AND

THE COOK COUNTY CLERKS ELECTION REVIEW PANEL

JANUARY 7, 2007

This is a confidential report issued to Sequoia Voting Systems and the Office of the Cook County Clerk whereby Diamond has acted as an independent contractor. This report sets forth certain findings and opinions of Diamond and should be used and/or cited accordingly.

Cook County Clerks Election Technology Audit - CONFIDENTIAL

2006 Diamond Management & Technology Consultants NA, Inc i

TABLE OF CONTENTS 1 Executive Summary.............................................................................................. 1

1.1 Overview ......................................................................................................................................1 1.2 Our Findings................................................................................................................................1 1.3 Summary Recommendations ....................................................................................................3

2 Introduction........................................................................................................... 7 2.1 Background.................................................................................................................................8

3 Context ................................................................................................................ 10 3.1 Voting Machine Transmission from the Precincts ................................................................10 3.2 Voting Machine Transmission from the Receiving Stations................................................12 3.3 Voting Machine Input at the County Office ............................................................................13

4 Our Approach for the Review of Cook County Voting Technology ............... 14 4.1 Overview ....................................................................................................................................14 4.2 Methodology..............................................................................................................................14

5 Timeline ............................................................................................................... 16 6 Technical Architecture Assessment ................................................................. 21

6.1 Description ................................................................................................................................21 6.2 Assessment of Technical Architecture ..................................................................................22

7 Hypotheses and Tests........................................................................................ 24 7.1 Hypotheses................................................................................................................................24 7.2 Tests...........................................................................................................................................24

8 Findings & Recommendations .......................................................................... 40 8.1 Findings .....................................................................................................................................40 8.2 Recommendations....................................................................................................................41

9 Appendix 1: Data Sources ................................................................................. 45 10 Appendix 2: Architecture Assessment ............................................................ 46

10.1 Overview of DEADONS+I and PARTS Frameworks ..............................................................46 10.2 Assessment of Election Results Transmission, Tallying and Reporting Technology ......48

11 Appendix 3: Comments from Sequoia Voting Systems ................................. 60


2006 Diamond Management & Technology Consultants NA, Inc 1

1 Executive Summary 1.1 Overview

The Cook County Clerks Office selected Sequoia Voting Systems in June 2005 to provide electronic voting equipment, software and services to support Cook County elections. The Clerks decision to pursue electronic voting was largely in response to several legislative drivers, including The Federal Help America Vote Act (HAVA) and the states Help Illinois Vote Act (HIVA), both of which were designed to address concerns arising out of the 2000 presidential election and issues pertaining to ballot access among disabled voters. This legislation, prompted by difficulties during the 2000 presidential election, was intended to insure all votes were counted by minimizing the error rate inherent in punch-card voting and to improve the accessibility of voting equipment for the disabled community. It was not the primary intent of election legislation to improve the speed of processing election results, but rather to reduce the undervote experienced in previous elections.

Due to HAVAs requirement related to ballot accessibility, each county precinct was required to have two types of voting machines, and potentially multiple units within each precinct. As such, the amount of equipment required for elections is greater than in the punch-card era. This, coupled with a legal requirement to tally votes in the precinct and the newly instituted early voting option, created a unique and more complex set of system requirements for todays electronic voting environment.

The County Clerks Office worked with Sequoia to implement its electronic voting system, which included, among other features, an electronic means of capturing votes at 2,370 election precincts, wireless transmission of election results to a central processing center at the County, a secondary process to transmit election results from regional receiving stations in the event of transmission failures at the precincts and a software application to tally and report results. This application and the associated election process were first used by both Cook County and the City of Chicagos Board of Elections for the March, 2006 election primary and again for the November 2006 federal, state and local elections.

On Election Day November 7, 2006 voters cast ballots in several hotly contested races. Only 56% of precincts were able to successfully transmit results over the wireless network to the Countys office, where votes were to be tallied. Upon failing to transmit, polling workers were instructed to deliver election equipment to one of 19 regional receiving stations throughout the County, where further technical problems were encountered, as well as confusion resulting from inaccurate information being posted on the election website, essentially rendering the secondary transmission process ineffective. Election workers then were instructed to bring election equipment to the County Clerks office for consolidation and tallying, a process which was 90% completed by about 10:44 a.m. Wednesday, November 8. This result was inconsistent with performance prior to the March primary and did not meet the expectations of Sequoia, the County Clerks Office, the candidates and the public.

1.2 Our Findings Diamond Management & Technology Consultants conducted an independent and objective rapid assessment of the process and technology supporting elections in Cook County. Our analysis indicates a confluence of events led to Election Night results falling short of expectations. Our overarching finding is that the end-to-end system as configured on election night, November 2006, was not able to meet the processing expectations of the County. This is supported by the findings below, each of which is presented in detail in the following report:



1.3 Summary Recommendations Diamond believes that a number of recommendations, if correctly implemented, will significantly improve the process of tallying and reporting election results. These recommendations have been reviewed by the Countys Election Review Panel.

Our recommendations are focused on ensuring the February 2007 election and subsequent elections are more successfully executed. We suggest broad implementation timeframes for our recommendations based on the election timetable.



Technical Recommendations

Diamond has identified twelve technical recommendations based on an assessment of the technical architecture and structured tests which were carried out using election technology and data. Our recommendations span the entire technical architecture from individual components to the complete end-to-end system. The emphasis in our recommendations is on:

Risk mitigation through the use of well tested technologies known to work reliably together

Special application design and configuration considerations for wireless applications

Improving the performance and usability of individual technology components

Improving the performance of the overall end-to-end system

Some of these recommendations can be implemented prior to the February 2007 election. Other recommendations may require certification and consequently are targeted for longer term implementation.



1.3.1 Process Recommendations Diamond has identified eight process recommendations based on our understanding of events on the night of the election, the Election Judge Manual and an assessment of the preparation undertaken for the election. The emphasis in our process recommendations is on:

Ensuring process and technology fit together in a complementary and coherent way

Risk mitigation through structured and auditable testing

Reducing the cycle time to complete the vote count

Our process recommendations do not require any major changes to the way the election process works today and also do not require any form of certification. Many of them can be achieved by a simple re-ordering or re-scheduling of activities. Therefore we suggest that many, if not all, can be implemented prior to the February 2007 elections.



1.3.2 Governance Recommendations Diamond has identified six governance recommendations based on our understanding of best practices in implementing and operating enterprise systems. The emphasis in our governance recommendations is on:

Setting, managing and communicating expectations

Risk mitigation through structured and auditable change control

Ensuring the appropriate span of control for technology management

Based on our assessment of the November 2006 election, we feel it is critical that key governance recommendations concerning expectations management and program management are implemented before the next election. However, governance also needs to be forward looking and seek to plan and control the future direction of election technology within the County.



2 Introduction Diamond Management & Technology Consultants NA, Inc. (Diamond) is a Chicago-based, publicly traded (symbol: DTPI), international consulting firm that works with public and private sector clients to address their most challenging problems, typically those that sit at the intersection of business and technology issues. Diamond employs more than 400 employees internationally.

Diamond was engaged to perform a rapid assessment of the business process and technology that support the transmission of election results from each County precinct to the Countys central office where votes are tallied. The focus of Diamonds effort was not to investigate the result or accuracy of the election, which is not in dispute. Rather, our assessment centered on the processing of election results, which on November 7, 2006, performed in a manner that fell short of expectations.

To determine the root cause of the processing delays, we sought to answer the following questions:

What factors caused the precincts to experience transmission failure of results after polls closed?

What then caused the secondary/backup process to also fail?

What other factors compounded the delays?

In addition to these questions we sought to determine what steps can be taken to prepare for future elections to improve performance.

Diamonds assessment team included business process, technology and public sector experts. Our team consulted with a panel of independent experts, appointed by the Cook County Clerk. The panels membership consisted of:

Hon. Abner Mikva, Panel Chair, Visiting Professor & Senior Director of the Mandel Legal Aid Clinic, University of Chicago; former Chief Justice U.S. Court of Appeals for the District of Columbia; former White House Counsel

Ms. Tricha Anjali, Ph.D, Assistant Professor, Dept. of Electrical and Computer Engineering, Illinois Institute of Technology

Mr. Xiaoping Jia, Ph.D, Director, Institute for Software Engineering, DePaul University

Mr. Ian Robertson, Senior Director of Global IT Americas, William Wrigley Corp.

Mr. Cyrus Walker, Founder & President, Data Defenders, Inc.; Dept. Head, Computer Forensics & Information Security, Wilbur Wright College

Mr. V.N. Venkatakrishnan (Venkat), Ph.D., Assistant Professor in Computer Science, Univ. of Illinois Chicago

The scope of our effort was to produce the following:

A Situation Timeline reconstructing events leading up to and through the November 7, 2006 election

An Architecture Assessment analyzing all aspects of the election infrastructure

After-the-fact Tests attempting to reproduce the results of the November election and diagnose their causes

A Final Report detailing results of the above, as well as recommendations for improvement



Throughout the course of our assessment, we interacted closely with County Clerk employees, Sequoia Voting System employees, and other vendors to the County. We met frequently with the independent panel. We were free at all times to examine any aspect of the election results process. Since accuracy of the vote itself was not in question, any technology related to the capturing of the vote, logic to calculate results and accuracy of those results was expressly outside of our scope.

Prior to submitting the final copy of our report, we briefed the independent panel on our findings and recommendations, which were reviewed by the Panel. We also provided a draft copy of our report to Sequoia Voting Systems for comment, with no commitment to modify our findings and recommendations based on Sequoias comments. If requested by Sequoia, Sequoias comments have been included as an attachment to the final report. At the same time a draft copy of the report was provided to Sequoia, a draft copy was also provided to the independent panel for review and comment.

We take great pride in the explicit objectivity and independent nature of our work. We have taken these steps, agreed to at the onset by Sequoia with endorsement from the County, to ensure that our work has focused on the facts and we have produced our own informed and unbiased recommendations.

2.1 Background The Office of the Cook County Clerk serves as the chief election authority for all of Cook County, although elections in the City of Chicago are overseen by a separate entity, the Chicago Board of Elections. The Office of the Cook County Clerk oversees election technology and processes for both primary and general elections in the County. There are approximately 1.4 million registered voters within Cook County and they vote at approximately 2,370 polling precincts, which are geographically distributed throughout the County.

Historically, elections in Cook County and the City of Chicago were based on a punch-card system with ballot counters. Prior to 2000, results from these ballot counters were transmitted manually, but in 2000, the County and the City equipped punch card ballot counters with modems that allowed for faster transmission and tallying of votes from the precincts. By all indications, this process was highly successful and set a high expectation for the speedy delivery of election results to the media. For example, in the March, 2004 primary, 95% of all precincts had reported results by 8:20 pm on election night one hour and twenty minutes after the polls had closed.

As a result of the widespread media attention given to the contested 2000 Presidential Election, attention which was focused on the inherent problems of punch-card voting systems, a national consensus was formed regarding the need to move from paper-based, punch-card ballots in favor of electronic voting equipment. This movement was further fueled by urging from the disabled community to ensure that the voting process was accessible to all. Federal law, in the form of the Help America Vote Act (HAVA) and local law, in the form of the Help Illinois Vote Act (HIVA), were promulgated mandating a variety of changes in local election procedures and equipment. HAVA mandated that all election jurisdictions have at least one piece of handicapped accessible voting equipment in each polling place by the first election of 2006, and the HIVA, among other things, mandated that electronic voting equipment produce a voter verifiable paper audit trail. The HIVA requirement for a paper trail was driven in large measure by concerns expressed by some that electronic voting equipment is subject to manipulation. These changes were designed to increase the reliability of the election results, not necessarily to improve the time in which election results are reported.

These federal and state mandates, in conjunction with business processes that the County had cultivated over years of conducting elections, resulted in requirements for election equipment and software that were unique. The Office of the County Clerk required both electronic voting equipment that would meet accessibility standards and equipment that would produce a paper trail. Furthermore, wireless transmission of results was required to meet results processing standards that had been perfected in recent years.



In response to these requirements, the Office of the Cook County Clerk, in conjunction with the Chicago Board of Elections, issued a Request for Proposals (RFP) in June of 2004, with a subsequent re-issuance of the RFP in December due to an inadequate response to the initial solicitation. The City and the County selected the solution proposed by Sequoia Voting Systems in June of 2005, and the Sequoia solution was, to a certain extent, customized to meet the unique requirements of the County and the City.

Implementation of the Sequoia solution began in late June of 2005, and was completed in time for early voting to commence, for the first time, in February of 2006. The full system was first put to widespread use in Cook County and the City of Chicago during the March 2006 primary.

As was to be expected, problems were reported during this first-time use of the system. These problems ranged from confusion among the voting public regarding how to use the new equipment to problems encountered by polling workers operating the equipment for the first time. Additionally, problems were reported with the process of transmitting election results from the polling place to the Office of the Clerk. Only 58% of the precincts were able to successfully transmit their results that night.

To better understand the problems that occurred during the March primary, the County hired Freeman Craft McGregor Group, to perform an evaluation. The firm proposed several changes to the equipment used for elections, including changes to the optical scan system and the Hybrid Activator, Accumulator and Transmitter (HAAT) device which consolidates and transmits election results. The report further found that inadequate transmissions resulted from user error, and a thorough training of election judges was recommended, and ultimately conducted, in preparation for the November 7th election.

Prior to the general election on November 7, 2006, several tests were conducted on new and existing equipment, including tests on the new touch screens and the optical scan ballot during a mock election and back-up test using Cicero township. Certification of the changes made, a requirement prior to implementing significant alterations to voting equipment, software and processes was provided by the State on October 14, 2006, and early voting began on October 16, 2006.

During the November 7, 2006 election in Cook County, significant delays were experienced in the processing of election results. Nearly half of Cook County precincts failed to transmit results after polls closed. By 10:00 pm on election night, only 45% of precincts had reported results. By 10:44 am November 8, 90% of precincts had reported results



3 Context

In order to fully understand the circumstances surrounding the November 2006 elections, as well as their causes and ramifications, it is important to set context regarding the processes, hardware and software that make up the Cook County election environment. This environment, broadly speaking, consists of three distinct components, each with its own set of processes and technology: the precinct, the secondary receiving stations and the Cook County Clerks office. This section will provide a descriptive overview of how each of these components was designed to function.

3.1 Voting Machine Transmission from the Precincts Citizens had the choice of casting their votes using one of two types of voting machines at the polling places. The first option was a touch screen voting machine called an Edge 2 Plus (or E2P). The second option allowed voters to use a marker to identify their selected candidates on a paper ballot which was then scanned into an optical voting machine called an Insight Optical Scan Voting Machine. In the case of E2P machines, the votes were stored on a memory stick (a USB compatible flash drive) and in the case of the Insight Optical Scan Voting Machine, results were stored on a memory pack. Each polling place had at least one of each type of each voting machine, a legal requirement and one that increased the amount of voting equipment at each polling place.

The polling places were distributed across 2,370 precincts. Within each polling place, there was at least one precinct and in some cases more than one precinct, each with its own set of voting machines.

Each precinct had one HAAT device (Hybrid Activator Accumulator Transmitter). The HAAT served 3 purposes: to initialize the cards each voter using touch-screen voting required, to accumulate the votes for a given precinct and to transmit those votes to a centralized server located at the office of the Cook County Clerk. For transmission of results, the HAAT used 1XRTT (Interexchange Radio Transmission Technologies) wireless modem technology. This is an always on technology, so no dial-up was required.

Prior to opening the polling place for voters at 7:00 am on Election Day, poll workers were required to switch on their HAAT and transmit a configuration file. This was used as an early warning signal to determine if any HAATs had transmission problems, such as a failure to obtain a wireless connection or a problem with the modem. From the time the polls opened, the HAATs were used throughout the day to activate cards for the Edge 2 Plus machines. By contrast, the transmit function was intended only to be used outside of the polling period (7:00 am to 7:00 pm) for transmission of the configuration before the polls opened and transmission of the election data after the polls had closed.

Once the polls closed at 7:00 pm, the equipment judge at each precinct was required to consolidate the cartridges and memory packs for each of the polling places within the precinct using the HAAT. The purpose of the consolidation was to accumulate polling place data into a single dataset that would be transmitted to the County office. This process consisted of inserting results cartridges (either in the form of a memory stick or memory pack) into the HAAT. On completion of the consolidation, the HAAT would then transmit the consolidated data using the HAATs wireless transmission capability. Once the transmission of the vote was complete, a tape was printed by the HAAT with summary election results and a message indicating the transmission was successful.

The precincts election data was then carried on the Verizon wireless network to the Countys frame relay service (provided by AT&T). Once the data was moved past the County firewall, it was distributed via a load balancer to two Linux servers referred to as HAAT Listeners. These HAAT Listeners acted as a repository of all data and transmission information from the HAATs. The voting data was then transferred from the HAAT Listeners to the HAAT Database on the WinEDS server. The voting cartridge and memory pack data was then transferred by the Migration Job from the HAAT Listener Database to the WinEDS database.



WinEDS is the core application and database used to manage the election process. Its functionality covers Election Set-Up (Including Contests), Precinct Assignment, Ballot Management, Cartridge Management and Tally. At the database level, WinEDS includes the following Microsoft SQL databases:

Profile Database that stores the base election information (precincts, contests, candidates, etc)

Election Database which contains the election results including cartridge data and ultimately tallied election results

Two databases used to temporarily store reporting data

One database used to permanently store reporting data

The Profile Database was set up before the election and maintained a master inventory of controlled election equipment, such as cartridges (but not HAATs)1.

On election night the WinEDS Migration Job moved cartridges from the HAAT Listener Database and placed them in a queue. WinEDS waited for the queue to build up and processed cartridges in groups of five batches with one hundred records in each batch. The output of the WinEDS process came in the form of tallied election results which were stored in the Election Database. A separate job transferred these tallied results to the Reporting Database from which the Reporting Application reported the election results.

Figure 3.1: Voting Machine Transmission from Precincts

1 WinEDS stores cartridge information, but not HAAT information. This reflects the origin of these products from different sources in a two company merger.



3.2 Voting Machine Transmission from the Receiving Stations There were 19 receiving stations distributed throughout Cook County on election night. These receiving stations served two purposes: (1) serve as a staging point for the collection of paperwork, voting machine cartridges and memory packs from the precincts and (2) transmit voting machine cartridges and memory packs that had not successfully been transmitted from the precincts.

Each receiving station was equipped with two laptops. The intended operating mode was to use the Cook County Clerks web-site as a guide to which cartridges and memory packs had successfully been transmitted and then select those that had not been successfully transmitted and retransmit them. The transmission mechanism required the laptops to use a client version of the WinEDS system, connecting with a wireless card over Verizons EVDO wireless network to the Cook County Clerks secured network via the Countys VPN (Virtual Private Network). These three components were all unique to the transmission mechanism used at the receiving stations.

The WinEDS Client is a client server application built using Powerbuilder application development technology. The Countys VPN is provided over a dedicated T3 line. EVDO (Evolution-Data Optimized) is a wireless broadband technology adopted by CDMA (Code Division Multiple Access) mobile phone service providers and is designed to provide a faster level of service than the 1XRTT solution used at the precincts .

Once a transmission was sent from the receiving station, the data was carried over Verizons EVDO data network to the Countys T3 internet connection. It passed through the Countys firewall and a VPN concentrator (Cisco AS5510 switch) before reaching the WinEDS server.

The HAAT Listener is not involved in transmissions from the receiving stations. Receiving station transmissions went directly into WinEDS.

Subsequent processing steps were identical to those used of the precinct voting machines

Figure 3.2: Voting Machine Transmission from Receiving Stations



3.3 Voting Machine Input at the County Office The County Office, at 69 West Washington Street in Chicago, used a mechanism to input voting machine cartridges and memory packs that was similar in concept to the receiving stations. The key difference is that the County Office mechanism was local-area-network- (LAN-) based and did not involve wireless technology.

The workers at the County Office selected cartridges that had not been successfully transmitted as they arrived from the receiving stations. These were entered directly to WinEDS via the WinEDS client software.

Subsequent processing steps were identical to those used of the precinct voting machines

Figure 3.3: Voting Machine Input at the County Office



4 Our Approach for the Review of Cook County Voting Technology

4.1 Overview The purpose of Diamonds review was to determine what caused delays on Election Day and to recommend changes in the process and technology that, if implemented, will prevent similar delays in the future, particularly in the upcoming elections in February and April of 2007. This review was focused on the systems and processes designed to transmit and process election results as implemented by Sequoia Voting Systems and Cook County on Election Day.

Diamonds approach focused on three core deliverables which are essential to building a solid fact base and a clear set of recommendations. These three core deliverables include (1) a timeline of the events to understand exactly where delays came from and why, (2) an architecture assessment to examine the systems overall design and (3) a series of hypothesis driven, after the facts tests to understand the root causes of performance problems on Election Day. Diamond expects that these three deliverables, included in this report, will provide a fact base and show the way forward for more efficient elections in the future.

4.2 Methodology Each major deliverable used an appropriate methodology to structure the approach, analyze scenarios, and develop conclusions. Diamond has frequently and successfully used these methods in hundreds of client engagements across a wide range of industries over the past 14 years. More detailed descriptions of the methodologies used for each deliverable are presented in the sections that follow:

4.2.1

4.2.2

Timeline of Events The starting point for the timeline of events was provided by Cook County, in the form of a detailed description of the events leading up to the election of November 7, 2006. This information was verified and supplemented with information obtained from interviews with poll workers, County staff, Sequoia employees, and other interested parties. Furthermore, Diamond reviewed hundreds of pages of documents related to the election, including the proposal submitted by Sequoia in response to the Countys RFP, the contracts between service providers and documented results of tests performed. Most importantly, the timeline was tied to actual time stamps from log files and formal records to determine at what state the system was in at different points before, during and after Election Day. The timeline was ultimately vetted with all parties, and adjusted to encapsulate the widest range of relevant activities on Election Day. This timeline will serve as a solid baseline to judge performance in future elections.

Architecture Assessment Diamond has conducted hundreds of architecture assessments for Clients ranging from the United States Department of Justice to Goldman Sachs. The purpose of our architecture assessment framework is to assess the design, implementation and performance of complex, enterprise-wide technology tools.

In reviewing the County Clerks technology for electronic voting, Diamond utilized its DEADONS+I framework. DEADONS+I is a structured framework which divides an organizations architecture into eight areas for assessment Data Architecture, Execution Architecture, Application Architecture, Development Architecture, Operations Architecture, Network Architecture, Security Architecture, and Integration Architecture. For the County, Diamond undertook interviews with technical experts, code analyses, log reviews and detailed testing to gather a fact base for each of these eight areas of the electronic voting architecture. Diamond then assessed the architecture using the PARTS framework. Good software PARTS (Portability, Performance, Adaptability, Availability, Reuse, Reliability, Testability, Supportability and Scalability) serve as key objectives to drive and measure the success of each



architecture area, and Diamond used this framework to render a judgment regarding the suitability of each component of the architecture.

4.2.3 Hypothesis-driven Approach and Testing Diamond developed and executed the tests for our review using a traditional hypothesis driven inquiry. Diamond has used this approach many times to solve complex problems and diagnose failures. The emphasis of this approach is on hypothesis generation, followed by disciplined testing and analysis to prove or disprove those hypotheses. This method is designed to get to the leading causes of a problem very quickly and provide a set of tangible steps to move forward in a short amount of time. The major steps in this process are presented in the graphic below:

For this review, a list of over 30 hypotheses for the causes of the election delay were brainstormed in discussions with Diamond experts, Sequoia personnel, Cook County Personnel, and the Countys Election Review Panel to make sure that all of the most likely causes were understood and vetted. In consultation with the election review panel, nine hypotheses were selected as the most likely root causes of the delay.

Diamond then evaluated the validity of each hypothesis, using two main mechanisms. First, a series of analyses were structured to collect primary data from log files, interviews, and configuration settings that describe the conditions on Election Day. These primary sources were analyzed to determine what actually happened on Election Day, and what can be learned from extant information.

In addition, a series of tests were designed, implemented, and evaluated to simulate conditions on election night, and isolate components of the system to search for causes of delays and new ways to reduce those delays in the future. These tests were conducted in collaboration with County and Sequoia personnel, and supervised by the Election Review Panel. Ultimately, these tests support or contradict the hypotheses developed by the group, and are very useful in determining the best path forward.

All three deliverables were then considered in developing a set of key findings and high level recommendations to clearly understand the issues with the election on November 7, 2006, and ultimately to move forward with better expected performance for upcoming elections.



5 Timeline

Prior to Election Day [Aug 01 Nov. 7]

Prior to Election Day, the key activities consisted of testing the election process and technology and the Pre-LAT (Pre-Election Logic and Accuracy Testing) process. The major portion of testing took place between September 14 and 21. The County and Sequoia jointly conducted testing of the election process. This included testing the processes for ballot preparation, audio files, early voting, absentee voting, preparation of voting machines, preparation of media (i.e. memory packs and cartridges), help desk, repair center, inspection and pick-up of equipment from the warehouse, delivery of equipment and materials to a test precinct, and all the Election Day processes. Transmissions for both the precinct and receiving stations were also tested. However, a scheduled stress test did not take place because some HAAT devices were incorrectly set up with the wrong Election Day. A smaller scale test did subsequently take place.

Much of the pre-election preparation is logistical in nature. The HAAT devices were assigned identifying numbers to track which HAAT is assigned to which precinct. 2 Cartridges were assigned to precincts and the HAATs configured to ensure they would only accept those cartridges assigned to their respective precincts.

The Pre-LAT process is one that is required by the Illinois State Board of Elections and is intended to be a comprehensive preparation process for all equipment to be used in the election. It covers testing the ballot creation process, testing of memory packs, cartridges and printed election tapes, testing of E2P, and Optical Scan voting machines and the HAAT. The HAAT testing has two parts: (a) ensure cartridge consolidation can be performed and (b) ensure the HAAT can transmit its configuration file successfully. An inventory of all materials used by election judges is performed. During the Pre-LAT process, the election equipment was prepared for use in the election. This is primarily a matter of configuration to ensure election data and events are treated and recorded differently from pre-election data and events. The election equipment is put into election mode during the Pre-LAT process.

Verizon also performed testing of the signal strength at the 19 receiving stations and made technical modifications to boost the signal at 7 of the stations. In another test in August, Verizon tested 56kb backup lines at the receiving stations. Although these lines were available, they were not used in the November 7 election. While the signal strength was not tested by Verizon at the 2,370 precincts, Verizon did provide hand-held devices to the County to test signal strength. 3

During Election Day [Nov. 7]

Poll workers were required to have their precincts HAAT transmit its configuration file to the HAAT Listener before opening polls at 7:00am. The majority of HAATs (97%) succeeded in doing this, although many only succeeded after 7:00am.

During the normal course of Election Day, the HAATs were used mainly to perform card activation for touch-screen voting machines. If a poll worker had difficulty with a voting machine or a HAAT, he/she could call a help desk for assistance. The help desk was staffed by County personnel with access to voting equipment similar to the precincts. If a HAAT failed, a new HAAT configured for the precinct would replace it.

The HAATs utilized the standard approach used by Windows based computers to connect to Verizons 1XRTT network The expected behavior of HAATs was to connect to the HAAT Listener once in the

2 [These HAAT identifiers are not derived from the serial numbers assigned to the devices at time of manufacture. The HAAT identifier that appears in the HAAT Listener is therefore sometimes different to the identifier relating the HAAT to a precinct]. 3 Verizon (John Merriwether and Tim Wong) conference call with County CIO on November 27.



morning before polls open and once after polls have closed to transmit the precincts votes. During Election Day over 18,400 calls were logged from the MDNs (Mobile Directory Numbers) assigned to the HAATs. This was more than three times the expected call volume, assuming two calls per day per HAAT. Figure 5.1 illustrates the number and duration of modem transmissions at each hour during Election Day.

Figure 5.1: Call Frequency & Duration

Election Night (from 7:00pm)

Polls closed at 7:00pm. Election workers were then required to perform a variety of activities to close down the precinct. The sequence of actions required were as follows4: close down the optical scanner, store voting, duplicate, spoiled and unused ballots, close the touch screen voting machine, consolidate cartridges, transmit the vote and close the HAAT (Card Activator), disassemble the voting machines, record the provisional ballots and create a statement of ballots.

The system is designed so that election results are transmitted to the HAAT listener, and from there, they are placed into a queue for tallying by WinEDS. The first HAAT transmissions from the precincts to the listener are recorded in the HAAT listener at 7:16pm. In all 1,336 HAATs out of a total of 2,370 (56.4%) transmitted data over approximately a three hour period on election night.

In many cases, a HAAT consolidates multiple cartridges, so there are many more cartridges than HAATs. By 7:30pm, 241 voting cartridges from precincts had been transmitted. The majority of the cartridges that successfully transmitted were transmitted between 7:30pm and 8:30pm with the cumulative total reaching 3,384 by 8:30pm. In all, 3,848 (53%) cartridges were successfully transmitted from the polling places by HAATs on election night out of a total of 7,170. As depicted in the accompanying Timeline (Figure 5.4),

4 For more detail on poll closing procedures see Election judge manual for General Election November 7, 2006 published by the Office of the Cook County Clerk.



HAAT transmission activity was minimal after 10:00pm (3 cartridges) and had completely ceased by 11:00pm of election night.

Figure 5.2: Key Events Across Precincts, Receiving Station and Election Central on Election Night

Elec

tion

Cen

tral

Precinct specific web pages provided and communicated

Cartridges begin arriving from

Receiving Stations

Rec

eivi

ng S

tatio

n

Receiving Stations start accessing

website

Slow and unexpected

information on web-page

Stations asked to stop uploading

cartridges

Stations resume uploading cartridges

Stations asked to bring cartridges

to Election Central

Slow and unstable WinEDS process

7:00 8:00 9:00 10:008:30 9:30 11:00 12:0010:30 11:307:307:00 8:00 9:00 10:008:30 9:30 11:00 12:0010:30 11:307:30

Prec

inct

s

Poll Judges attempt consolidation and

transmission of results

7:00 8:00 9:00 10:008:30 9:30 11:00 12:0010:30 11:307:30 10:00

Manual cartridge loading and tallying

of approximately 4000 cartridges

Poll Judges transfer cartridges to Receiving

Stations

Stations stop uploading cartridges

HAAT cartridges receivedEarly Vote Processing

At the County office, as planned, WinEDS started processing the early vote at 7:00pm. Note that the City of Chicago opted for a different approach; the early vote for the City was tallied in WinEDS separately from the Election Day vote and prior to the transmission of data from the polling places. The processing of the early vote for the County into the WinEDS tallying system proceeded slowly, with individual cartridges having processing times as high as 45 seconds.

By 7:40pm, only 48 cartridges had been tallied, but 772 cartridges were present in the HAAT Listener database. This disparity between the number of cartridges transmitted to the HAAT Listener Database but waiting to be tallied and the number already tallied became a cause for concern and was referred to as a logjam.

After closing down their site, poll workers were instructed to bring all of their materials to the appropriate receiving station, and receiving station personnel were charged with transmitting cartridges that had not successfully transmitted from the polling place. At 8:10pm, the receiving stations began transmitting cartridges. To track the progress of transmission, the workers at the receiving stations used the Cook County Clerks web site, which was designed to indicate which cartridges had been transmitted. At the



start of the night, however, the web site erroneously showed data from a previous test of the system, not live data from that nights election. Furthermore, when the site was finally corrected, it displayed cartridges that had been processed in WinEDS rather than those that had been received by the HAAT Listener. The delay between receipt of the transmission by the HAAT Listener and the processing of cartridges in WinEDS caused receiving station personnel to retransmit some cartridges that had already been received by the HAAT Listener. The first duplicate cartridge was transmitted at 8:10pm and the last at 11:13pm. The last transmission from a receiving station was at 11:51pm. Evidently, the receiving stations were not clear on what had already been successfully transmitted and continued to transmit duplicates.

Problems at the receiving stations were complicated by reports of difficulties maintaining network connectivity. Receiving station staff reported problems with the WinEDS client, as well as problems maintaining connectivity with the internet.

Though the receiving stations operated for over three and a half hours, on average each receiving station transmitted fewer than 15 cartridges. The most cartridges transmitted by any receiving station was 33.

The following diagram contrasts the expected and actual usage of primary, secondary and tertiary transmission systems. As indicated, the County Clerk expected the vast majority of cartridges to be transmitted from the HAAT at the precinct to the County Clerks office via the wireless network. In reality, only 56% of cartridges transmitted in this fashion. A manual process including handling at the receiving stations and the County Clerks Office, expected to account for less than 10% of all cartridge entry, actually accounted for 44% of all cartridge entry.

Figure 5.3: Expected & Actual Transmission Rates5

In response to the concern caused by the logjam, the tally process in WinEDS was stopped by Sequoia personnel at approximately 8:30pm and the early vote processing was terminated in an incomplete state. At about the same time, Sequoia asked that the County direct the receiving stations to suspend transmission of cartridges for 30 minutes to allow the logjam issue to be resolved. At 9:30pm, the receiving stations were told to resume transmissions. For the most part, the receiving stations complied with this request and only 13 cartridges were transmitted during this period. However, the receiving stations could have been allowed to continue to transmit during this period of time, even if the migration job and WinEDS were stopped and started.

As a result of the continued slow pace of transmission from the receiving stations, the 19 stations were contacted at approximately 10:45pm and told to bring their cartridges to the County building at 69 West Washington. Of the 19 receiving stations, only 3 continued to transmit after 11.22pm.

The time between 10:45pm and 12:45am was largely devoted to transporting cartridges to the County building, sorting those cartridges as they arrived and setting up the workstations available to input the cartridges to WinEDS.

The County staff continued to input cartridges to WinEDS throughout the night and achieved 90% completeness at approximately 10:44am on November 8.

5 Expected transmission rate of primary (HAAT) system from Noah Praetz, Election Preparation & Planning Manager at the Cook County Clerks Office. Actual transmission rates taken from HAAT Listener Database and WinEDS Database.



The timeline in figure 5.4 shows the progress in transmitting and processing cartridges from the point when the polls closed on Election Day (7.00pm November, 7) to 7:00am on the morning of November 8. The data in figure 5.4 is derived from the logs of the HAAT Listener database and the WinEDS database. The graphs are based exclusively on factual data that are independently verifiable.

Figure 5.4 shows progress in five-minute intervals. The bar charts show the number of cartridges received by a database in each five minute interval. The line graphs show the cumulative number of cartridges received in a database. The figure shows progress processing different types of ballots in WinEDS based on the source of the cartridge (e.g. HAAT, Receiving Station, Manual Entry to WinEDS and the Early Vote).

The logjam encountered early on election night is clearly visible as cartridges build up in the system between 7:20pm and 8:40pm.

Figure 5.4: Timeline of Cartridge Processing from 7.00pm November 7 to 7.00am November 8



6 Technical Architecture Assessment 6.1 Description

By definition, a technical architecture that must, within a short window of time, accept and tally ballots of multiple different types from 2,370 locations that are staffed by temporary workers is complex. To meet these and other requirements, the County Clerk and Sequoia chose to pursue a distributed, parallel execution architecture that contemplates three distinct types of locations and operations:

Precincts (2,370 with voting and vote transmission equipment)

Receiving Stations (19 with laptops and vote transmission equipment)

County Headquarters (1 with all voting and transmission equipment as well as tallying software and hardware)

A graphic that depicts this architecture in greater detail is presented below:

1 2

3 4 8

5

6

7


2006

1 Within precincts electronic votes are consolidated onto HAAT devices.

Results are transmitted to the Cook County Office of the Clerk using cellular 1XRTT network connections (Verizon) utilizing SOAP calls over HTTP 2

HAAT Listeners at the Cook County Office of the Clerk receive the results 3WinEDS processes the precinct results to determine the election results. Three jobs constitute WinEDS functionality: migrate, tally and report. 4

a. Migrate moves data from the HAAT Listener database into working tables.

b. Tally calculates the results analyzing cartridges in groups of 100.

c. Report provides election result reports. Early votes are also processed at the clerks office.

Once precincts close and have transmitted their results, the cartridges containing the electronic votes along with other election materials are brought to receiving stations.

If atra

If aCo

In lat

6A dprerelSt

5

Diamond Management & Technology Consultants NA, Inc 22

precinct has trouble transmitting results to the Cook County Office of the Clerk, another attempt to nsmit can be made when the cartridges are brought to the receiving station.

6 Cartridges are loaded onto a WinEDS client laptop

7 The WinEDS client transmits results to the Cook County Office of the Clerk over cellular EVDO network (Verizon)

8 The WinEDS server receives the results

receiving station has trouble transmitting results, the cartridges can be physically brought to the Cook unty Office of the Clerk and loaded into a WinEDS client in that office.

the unusual circumstance where the database is unavailable the election results are cached locally for er transmission.

.2 Assessment of Technical Architecture etailed assessment of the technology based on the DEADONS+I and PARTS frameworks are sented in the Appendix. Below is a summary of findings based on that assessment organized by

evant elements of the DEADONS+I framework: rengths

Execution: The underlying hardware, operating systems and data base server capabilities of the voting system are adequately sized to handle the anticipated processing load

Application: The back-end systems (HAAT Listener, WinEDS Server) were properly designed to minimize coupling with each other, thereby enabling scaling of each component independently

Operation: Detailed Pre-LAT tests which modeled election night processes were conducted to verify the correct operation of specific components of the overall solution

Network: The Countys LAN and WAN connections, except for the shared T3 connection to the Internet, is dedicated to the voting system limiting the impact of concurrent events within the Cook County network on election night traffic.



Weaknesses Application: Specific components of the voting system software set, namely the WinEDS server,

WinEDS client and HAAT devices, are architecturally ill-suited to the countys performance expectations for the following reasons:

o The HAAT embodies limited adaptability to wireless cellular data environments directly impacting its robustness

o The WinEDS client is unsuitable for use over a cellular data network which was the selected access network in the Receiving Stations

o The WinEDS client displayed serious database management problems including the creation of duplicate keys and improper rollback of disrupted operations due to network errors

o The WinEDS servers tally jobs design adopts a batch processing model instead of a greedy algorithm which tallies data as and when it becomes available, thus introducing delays in processing votes

o The overall performance of the WinEDS tally process appears to be particularly slow relative to the complexity of the tasks it performs

Operation: While significant pre-election testing was performed as part of the Pre-LAT process,

holistic tests, which replicated the entire set of technologies used in the end-to-end path for the primary, secondary and tertiary data transmission mechanisms with clear test objectives, measurements and analysis should have been undertaken. Such a test likely would have prepared the county to expect a log-jam at the WinEDS server

Operation: The implications of 2,370 devices, distributed across specific precincts, attempting a connection following a policy of Connect once and give up should have been discussed with Verizon and Sequoia. Furthermore, while it was believed that the HAAT was configured to connect three times to the Verizon network, it was actually configured to attempt to connect only once and was configured to attempt to transmit three times once it connected. The system should have been configured appropriately based on such discussions or failure rates estimated reliably to design the secondary system appropriately.

Operation: The operation of the WinEDS client over EVDO should have been tested thoroughly to establish stable and high-speed operation before selecting EVDO as the access network from Receiving Stations

Operation: High success rates for HAAT transmissions in the overall process design should not have been assumed when no prior history existed to establish a baseline for its performance

Network: The existing voting system precinct and receiving station network architectures use a single wireless carrier, creating a single point of failure



7 Hypotheses and Tests 7.1 Hypotheses To drive a fact-based, methodical approach, the following hypotheses were used as the basis for investigating the likely bottlenecks within the overall vote consolidation, transmission and tallying process.

7.2 Tests To validate each of the hypotheses, the testing methodology adopted utilized election night data from various system logs and created and conducted experiments to test operation when necessary.

7.2.1 Hypothesis 1: Wireless Network Rationale behind hypothesis and test objective

In the path between the HAAT to the HAAT Listener, the wireless network represents the weakest link in terms of reliability and capacity. Given that 2,370 HAATs attempted to set up a connection with the network over a short time window, the investigations focus was to determine if (a) the network refused some of these connection requests or generated errors or packet losses due to the volume of requests, resulting in the HAATs inability to send results to the HAAT Listener and/or (b) the HAAT applications intrinsic design or implementation was responsible for the observed issues in transmission of election results from the precincts.

Approach and Process

Given that it would be infeasible to recreate the wireless network environment from election night, the verification process was based primarily on an analysis of the logs created by the HAAT, the logs provided by the wireless network provider, Verizon, and the logs created by the HAAT Listener. However, in order to confirm that the HAATs which did not transmit on election night do not suffer from some



fundamental issue affecting their ability to transmit we performed a test with a sample of 39 HAATs. The test set up and observations are presented in the figure below:

Transmission Performance Test Objective: Determine if HAATs, as stand alone

devices, under limited network load perform without problems

39 HAATs which failed to transmit on election night were set up to transmit pre-consolidated cartridges

All 39 successfully transmitted Takes 20-40 seconds per transmission Brief visual indication if transmission is successful No obvious visual indication if transmission fails

Description & Observations

Listener DB

WinEDS DB

Reporting DBVerizon

AT&T

. . .

Listener 1

Listener 2

Router

Test Setup

Des

crip

tion

Obs

erva

tions

39 HAATs

All 39 of the HAATs tested transmitted successfully, and the observed time to transmit was between 20 and 40 seconds; it should be noted, however, that this time includes the time required to establish and terminate a connection. Given that there were no observed transmission failures, we could not conclude anything substantial regarding the HAAT or the Verizon networks role in a potential transmission failure situation on the basis of this experiment.

Analysis of the records provided by Verizon suggested that 835 HAATs did not attempt to make a connection with the Verizon network after 7 P.M. on election night, despite successfully connecting earlier in the day. This observation presented a point of inflection in our analysis while we were proceeding with the assumption that the HAATs attempted transmission but for some reason were unsuccessful, this observation unequivocally indicated that from the networks perspective 835 HAATs did not even attempt to connect.

We then approached the problem from three distinct perspectives captured by the following hypotheses and problem solving approaches:

o Correlation led analysis:

o Hypothesis 1.1: There were specific regions of the county where the Verizon Wireless network lacked adequate capacity to handle the aggregate connection requests from HAATs in precincts located in the region. Hence, there was some correlation between the location of a HAAT and the probability of successful transmission.

o Structural analysis:

o Hypothesis 1.2: The HAAT software was not designed to adapt to the vagaries of the wireless network. Specifically, the logic for establishing a connection was not adaptive enough to increase the probability of getting a connection with the network. Hence, the HAAT only had a limited probability of successfully connecting to the network in a single attempt.

o Hypothesis 1.3: The HAAT suffers from a subtle problem in the way it utilizes the devices resources such as the memory or communication buffers which impact its ability to communicate with the network. In other words, the problem with the HAAT is not necessarily confined to its communication module, but potentially elsewhere impacting the stability of the HAAT software.



Hypothesis 1.1: Regional weaknesses in the wireless network

Approach and process: We analyzed the logs from 848 HAATs which did not transmit successfully on election night and 1,186 HAATs which did transmit. The first step in this analysis was to develop an exhaustive data table that captured the HAATs serial number, the precinct it was assigned to, the street address of the location, the time of the transmission attempt and the status of the transmission. Once this data table was developed, we then mapped the location of all 2,034 HAATs along with the status of each HAATs transmission. This is presented in the figure below, with HAATs that failed to transmit indicated in red:

Conclusion: A visual inspection of the distribution of HAATs across the map shows a roughly uniform distribution of successful and failed HAATs across the precincts. Specifically, there was no obvious clustering of failed HAATs. Such a clustering would have called for additional investigation of Verizons network state on election night in that location.



Hypothesis 1.2: Adaptability of the HAAT to network state

Approach and process: In order to determine the robustness of the HAAT in the face of varying network conditions, we analyzed that part of its source code that deals with establishing a connection with the Verizon network and transmitting data (configuration, logs or consolidated precinct results). The pseudo-code for the section of the software which performs the connect and transmit functions is provided below:

PROGRAM.CS

MainFormClass_Load ()MAINFORMCLASS.CS

main ()

TRANSMITCONTROLLERCLASS.CSTransmissionClass Transmit (TransmitOperationType)

Set server location to emptyTry to connect to the HAAT Listener via the LANIf unsuccessful connect via remote access services

Disconnect the modem: Disconnect ()Initialize the modem: Initialize ()For each phone number while the number of attempts not exceeded

Attempt connection: RasConnect()Get server location by trying to connect: TryToReachSomeServer()

Open a socket connection: IsReachable()If an errors occurs throw a socket exception

Server location remains emptyIf server is unreachable disconnect: Disconnect()

End ForIf the server location is still empty

throw an ImpossibleToEstablishConnection exception

1. If the HAAT cannot connect to Verizon, an ImpossibleToEstablishConnection exception is captured in the HAAT log2. If the HAAT cannot connect to the HAAT Listener, an ImpossibleToEstablishConnection exception and a socket

exception are captured in the HAAT log

Conclusions: The HAAT software follows the standard procedure adopted by connectivity software provided by cellular carriers for use over the Windows OS. From that perspective, we do not think the approach to establishing connectivity was flawed in any significant manner. However, while the procedure adopted by network connectivity software can expect a human to adapt to observed connectivity problems (such as choose to reconnect, move to a different location with better signal strength, etc.), the HAAT code did not include adaptation techniques to respond to the condition of the network.

There are some aspects to the code which could contribute to lowering the odds of a HAAT successfully connecting with the network:

o The HAAT, as configured on election night, attempts to connect to the Verizon network once. If this attempt was not successful, it does not attempt again.

o The code is designed to attempt a fresh connection to the network ahead of each transmission. Such a model increases the number of times a HAAT attempts a connection request, as opposed to a model where a HAAT maintains its connection after establishing it once. The technique adopted by the HAAT, as programmed on election night, correspondingly increases the probability of encountering connectivity problems.


2006 Diamond Management & Technology Consultants NA, Inc

Hypothesis 1.3: System stability issues with the HAAT

Background: Our review of the logs of 848 HAATs that had unsuccessful transmissions indicated that in 64 cases, the election judges in fact attempted to transmit cartridges several times. In all but two cases, HAATs which failed the first time, failed every time. In one cases, it tried for as long as an hour and 15 minutes across 22 consecutive attempts with no success. Furthermore, in one of the two cases where the HAAT was subsequently successful, the HAATs power was cycled, which typically resets the state of a computer system.

Serial number Tx Time Tx Status Number of attempts Start Finish819 7:51:00 PM Failed 22 7:51:00 PM 9:05:00 PM6336 7:57:00 PM Failed 10 7:57:00 PM 8:30:00 PM3063 8:06:00 PM Failed 9 8:06:00 PM 8:44:00 PM61 8:15:00 PM Failed 8 8:15:00 PM 8:49:00 PM

3249 7:55:00 PM Failed 8 7:55:00 PM 8:28:00 PM6312 7:32:00 PM Failed 8 7:32:00 PM 8:05:00 PM2296 7:37:00 PM Failed 7 7:37:00 PM 8:11:00 PM4026 7:19:00 PM Failed 7 7:19:00 PM 7:44:00 PM3930 7:54:00 PM Failed 5 7:54:00 PM 8:20:00 PM5052 7:45:00 PM Failed 4 7:45:00 PM 8:09:00 PM6195 8:14:00 PM Failed 4 8:14:00 PM 8:37:00 PM858 8:49:00 PM Failed 3 8:49:00 PM 9:18:00 PM3234 7:53:00 PM Failed 3 7:53:00 PM 8:24:00 PM4350 8:20:00 PM Failed 3 8:20:00 PM 8:41:00 PM5067 7:50:00 PM Failed 3 7:50:00 PM 8:12:00 PM5228 7:43:00 PM Failed 3 7:43:00 PM 8:05:00 PM5283 8:14:00 PM Failed 3 8:14:00 PM 8:33:00 PM5362 7:42:00 PM Failed 3 7:42:00 PM 7:59:00 PM5455 7:49:00 PM Failed 3 7:49:00 PM 8:00:00 PM5536 8:46:00 PM Failed 3 8:46:00 PM 9:11:00 PM6034 7:30:00 PM Failed 3 7:30:00 PM 7:58:00 PM6183 8:28:00 PM Failed 3 8:28:00 PM 8:49:00 PM6236 7:31:00 PM Failed 3 7:31:00 PM 7:49:00 PM6294 7:38:00 PM Failed 3 7:38:00 PM 7:44:00 PM86 8:52:00 PM Failed 2 8:52:00 PM 9:07:00 PM490 7:17:00 PM Failed 2 7:17:00 PM 7:29:00 PM591 7:29:00 PM Failed 2 7:29:00 PM 7:46:00 PM622 7:29:00 PM Failed 2 7:29:00 PM 7:44:00 PM872 7:44:00 PM Failed 2 7:44:00 PM 8:14:00 PM959 7:47:00 PM Failed 2 7:47:00 PM 8:04:00 PM2295 7:58:00 PM Failed 2 7:58:00 PM 8:15:00 PM2352 8:25:00 PM Failed 2 8:25:00 PM 8:43:00 PM2393 7:30:00 PM Failed 2 7:30:00 PM 7:45:00 PM2417 8:08:00 PM Failed 2 8:08:00 PM 8:33:00 PM2438 7:24:00 PM Failed 2 7:24:00 PM 7:39:00 PM3044 8:02:00 PM Failed 2 8:02:00 PM 8:40:00 PM3044 8:40:00 PM Success** 23086 7:36:00 PM Failed 2 7:36:00 PM 7:59:00 PM3187 8:01:00 PM Failed 2 8:01:00 PM 8:21:00 PM3272 7:21:00 PM Failed 2 7:21:00 PM 7:37:00 PM3302 7:27:00 PM Failed 2 7:27:00 PM 7:53:00 PM3921 7:30:00 PM Failed 2 7:30:00 PM 7:47:00 PM3990 7:53:00 PM Failed 2 7:53:00 PM 8:13:00 PM4049 8:12:00 PM Failed 2 8:12:00 PM 8:29:00 PM4207 8:03:00 PM Failed 2 8:03:00 PM 8:19:00 PM4221 7:33:00 PM Failed 2 7:33:00 PM 7:51:00 PM4252 7:35:00 PM Failed 2 7:35:00 PM 7:51:00 PM4277 7:57:00 PM Failed 2 7:57:00 PM 8:14:00 PM4300 8:05:00 PM Failed 2 8:05:00 PM 8:09:00 PM4330 7:17:00 PM Failed 2 7:17:00 PM 7:40:00 PM4347 8:48:00 PM Failed 2 8:48:00 PM 9:13:00 PM5061 8:15:00 PM Failed 2 8:15:00 PM 8:32:00 PM5090 6:22:00 PM Failed 2 6:22:00 PM 7:56:00 PM5133 7:47:00 PM Failed 2 7:47:00 PM 8:01:00 PM5185 7:47:00 PM Failed 2 7:47:00 PM 8:07:00 PM5185 8:07:00 PM Success 25211 7:14:00 PM Failed 2 7:14:00 PM 7:34:00 PM5433 7:53:00 PM Failed 2 7:53:00 PM 8:25:00 PM5438 7:49:00 PM Failed 2 7:49:00 PM 8:06:00 PM5518 8:06:00 PM Failed 2 8:06:00 PM 8:23:00 PM6103 7:25:00 PM Failed 2 7:25:00 PM 7:40:00 PM6109 7:53:00 PM Failed 2 7:53:00 PM 8:14:00 PM6207 9:00:00 PM Failed 2 9:00:00 PM 9:12:00 PM6272 8:15:00 PM Failed 2 8:15:00 PM 8:33:00 PM6383 8:03:00 PM Failed 2 8:03:00 PM 8:19:00 PM6404 7:45:00 PM Failed 2 7:45:00 PM 8:03:00 PM
> 5 attempts > 2 attempts 2 attempts
28



Of these 62 HAATs, 49 of them were not seen on the Verizon network on election night according to the call logs provided by Verizon. In other words, even though the election judges were pressing the Print/Transmit button several times on these HAATs, the HAATs were actually not even attempting to connect to the network. This observation led us to suspect that for some reason the HAATs get into a state which essentially renders them unable to transmit. If this were true, it would indicate that the origin of the problem lies in the HAATs as opposed to the Verizon network. The remaining 13 never even appear on the Verizon logs, which mean that the Verizon Wireless network does not recognize these as valid numbers on the system.

Break down of 62 HAATs which failed to transmit successfully following multiple attempts

Not a valid Verizon number

21%

Not seen on election day

18%

Not seen after 7:00 pm

61%

We therefore decided to experimentally recreate a scenario which would potentially give us access to a HAAT which fails to transmit. Specifically, we suspected that recreating the entire sequence of events that a HAAT is subjected to on election day, starting with powering it on in the morning, to using it as a card activator several hundred times and consolidating cartridges may lead us to a HAAT which then suffers from transmission problems. Our objective was to test if a HAAT which does not transmit the first time is able to transmit successfully in subsequent attempts.

The test was organized in three parts the pre-activation phase, the activation phase and the post-activation phase. In the pre-activation phase, we attempted to consolidate and transmit cartridges from the HAATs prior to exercising the card activation function on the HAAT. In the activation phase, we activated between 150 to 400 cards on the HAATs, attempting to simulate the variation in the number of cards a HAAT would be required to activate. In the post-activation phase, we attempted to consolidate and transmit cartridges from all the HAATs after they had been used for card activations.

Pre-activation phase:

1. We started with a collection of 8 HAATs which were reset and prepared to work on official election mode.

2. We then consolidated cartridges on each HAAT and attempted a transmission

3. One out of the 8 HAATs did not successfully transmit its results

a. We then made 6 successive attempts to transmit from this HAAT across a 30-minute period. It failed to transmit each time.

b. Our attempt to prepare the HAAT that did not transmit failed and upon running a diagnostic, we found that the HAAT could no longer detect its modem. The diagnostic report clearly indicates this status as Modem test: Failed

4. The remaining 7 HAATs transmitted successfully during this phase

Activation phase:



5. We reset the HAATs and brought them back to official election mode

6. We then activated 150 to 400 cards on each HAAT to simulate election day activities

7. With the rare exception of about 3 cards out of a collection of 300, all the others were activated without any issues

Post-activation phase:

8. We then consolidated cartridges and attempted a transmission

9. All of the HAATs successfully transmitted, including the HAAT that previously failed to transmit in the first phase of the test. This confirmed that the HAAT did not suffer from a modem which was incapable of transmitting.

Conclusions

For the first time in our testing process, we observed a HAAT that failed to transmit, and, as in at least 63 cases on election night, failed to transmit on subsequent attempts. Furthermore, this failure to transmit occurred when the election tape indicated that the modem was active. These observations lead us to believe that the origin for at least some of the transmission related problems lies in the HAATs themselves, as opposed to the network. It appears unlikely that the network would selectively reject a specific HAAT several times, when 7 others transmitted successfully. It seems more likely that, as the Verizon logs from election night indicate, the HAATs that fail to transmit do not even make an attempt to connect due to problems within the HAAT.

It is important to note that, while Diamond was able to observe this condition, the root cause of this failure has not been uncovered as of the writing of this report. Further testing and code analysis should be conducted to determine the root cause of this observed failure.

Summary of conclusions for Hypothesis 1

Based on our testing and analysis of election night HAAT and Verizon logs, Diamond believes that it appears more likely that the origin for the majority of transmission-related problems on election night lies within the HAAT as opposed to the wireless network. Diamonds belief is based on the following points:

1. There appears to be no spatial or time correlation to the pattern of election night HAAT transmission failures, ruling out the hypothesis that there was a localized or time-specific problem due to load on the network

2. The HAATs which failed to connect repeatedly on election night did successfully connect at pervious points during the day. The fact that these HAATs permanently ceased transmitting over the wireless network on election night diminishes the possibility that transmission problems were due to the network

3. The Verizon network did not receive a registration request from 835 HAATs on election night despite successfully establishing connections from the same locations during the day. This suggests that the HAAT can transition to a state in which it no longer is able to connect to the wireless network

We have not been able to find any evidence to support the theory that an aggregation of HAATs increases the probability for the network to drop or refuse connections. The problem therefore appears to be rooted in a structural defect in the HAAT as opposed to the collective behavior of several HAATs attempting simultaneous transmissions.



7.2.2 Hypotheses 2 and 8: Concurrent VM Processing with WinEDS and Solution Capacity

Rationale behind hypothesis and test objective

The WinEDS experienced what was termed as a log jam on election night. Specifically, it was noticed that while several thousand cartridges had arrived in its processing queue, it was actually processing only a few of them at any given point of time. This led to unexpected waiting delays in getting intermediate results from the WinEDS reporting system. Our objective was to determine if WinEDS, as designed and configured, was capable of performing reliably when subjected to loads from multiple sources (HAAT Listener, Receiving Stations and Early Vote cartridge processing stations) and secondly to calibrate its performance under the best to least favorable loading conditions. Specifically our goal was to determine if the operating performance of WinEDS was responsible for the log jam condition.


In order to systematically study the impact of varying load sources on WinEDS, a modular testing approach bringing in various load sources in a sequence modeled after election night was adopted. The test set up and sequence is presented in the figure below:

Test Setup

HAAT Listener primed with 3,845 cartridges from election day HAAT transmissions.

Enable data migration for loading WinEDS database with data from Listener Database

38 Laptops utilized in receiving stations send cartridges to the WinEDS server

20 computers utilized to load data into WinEDS server (to simulate manual loading of cartridges of early votes and last resort for failed transmissions)

Description

Listener DB

WinEDS DB

Reporting DB

Local WinEDS for early votes, last resort for election day votes

Remote WinEDS clients

. . .

. . .

Early Vote + HAAT Listener DB + Direct Entry WinEDS

Concurrent WinEDS load with Early Vote, Precincts & Receiving Stations transmitting

Test Runs Replicates

Test

ing

Sequ

ence

30 60 90 120 150 1800

HAATEarly Vote

Remote WinEDS Clients

Local WinEDS Clients

minutes HAAT Listener DB +

Direct Entry WinEDS Concurrent WinEDS load

with Precincts & Receiving Stations transmitting

HAAT Listener DB + Direct Entry WinEDS Precincts only transmitting

We performed a series of four tests focused on the WinEDS environment. In each test, the Listener database was populated with cartridges from election night data. The test set up is described in the figure below, and the test configurations were as follows:

1. WinEDS Server simultaneously processing early votes, HAAT data, receiving station data, and manual county data

2. WinEDS Server simultaneously processing HAAT data, receiving station data, and manual county data

3. WinEDS Server simultaneously processing early votes, HAAT data, receiving station data, and manual county data while executing the two custom queries created by Cook County Staff



4. WinEDS Server processing HAAT data only.

Conclusions

The average processing time of the WinEDS varies from 1.13 seconds when it only has to process Election Day cartridges to 5.88 seconds when it is faced with simultaneous loads including Early Vote cartridges. Early vote cartridges, notably have a significant impact in slowing down the efficiency of the WinEDS.



A detailed summary of observations under different loading scenarios is presented below. It was also observed that running the two queries on the WinEDS that were run on election night had no impact on the operation of the WinEDS.

Comparision of average processing times for test on (Early Votes + Precincts + Receiving Centers) and (Precints + Receiving centers)

0.00

2.00

4.00

6.00

8.00

10.00

12.00

14.00

16.00

5 m

ins

10 m

ins

15 m

ins

20 m

ins

25 m

ins

30 m

ins

35 m

ins

40 m

ins

45 m

ins

50 m

ins

55 m

ins

60 m

ins

65 m

ins

70 m

ins

75 m

ins

80 m

ins

85 m

ins

90 m

ins

95 m

ins

100

min

s

105

min

s

5 minute intervals

Seconds

Early votes + Precints +Receiving stations

Precincts + Listener logs



7.2.3 Hypotheses 3 and 5: WinEDS Client and VPN Connectivity Rationale behind hypothesis and test objective

The election workers at the Receiving Stations experienced numerous difficulties in transmitting data from the WinEDS client software installed on laptops at these locations. The laptops utilized a higher bandwidth wireless connection (EVDO access technology) relative to the HAATs. Since the client was communicating directly with the WinEDS server on the county network, the laptop computers utilized a VPN to gain access to the WinEDS server. The objective of this test was to evaluate the robustness of the WinEDS client when it is required to operate over an EVDO network, using a VPN to connect to the county network. Since hypotheses 3 and 5 are closely related, one test was constructed to evaluate both.


In order to isolate the effect of layering different elements of the networked environment, the test was conducted in a modular fashion. As a benchmark to evaluate the stability of the WinEDS client under the best of networking conditions, the test was first performed on a high speed, highly reliable and available Ethernet connection instead of EVDO. However, the network (Ethernet) was configured as a subnet outside the countys network, forcing the laptops to utilize a VPN to reach the WinEDS server. The same test was repeated with EVDO as the physical network technology instead of Ethernet. In each case, the test was conducted as two phases: the first phase tested for the stability of VPN over the underlying physical network and the second phase added the WinEDS client to test its performance over VPN and the underlying network. The test utilizing EVDO was conducted once with all laptops located in the county building and another time with the laptops distributed across 10 Receiving Stations throughout the County. The laptops used the same configuration as those used on election night.

Test over EVDO from county building at 69 W.Washington Avenue, Chicago

When we conducted the test over EVDO, in the first phase when all we had was the VPN running over EVDO, we observed that the VPN connection was unstable. Specifically, monitoring the VPN concentrator revealed that clients were getting disconnected from the VPN concentrator every few minutes. Upon observing that starting a ping process on the network stabilized the VPNs from failing, we stabilized the VPNs on all laptops by pinging a server on the UUNET network and one in the county network. After establishing stable VPN connections, we then entered into Phase 2, which was to test the performance of the WinEDS clients. The test set up is described in the figure below:

Test Setup

Test EVDO performance Open connection Ping computer on public Internet

(198.6.1.3) Test VPN performance

Start VPN client Observe stability of connection Ping computer on county network

(172.16.122.103) Test WinEDS + VPN + EVDO performance

Start WinEDS client Enter election vote cartridges Record client side experience Record WinEDS server logs

Description

Listener 1

Listener DB

WinEDS DB

Reporting DB

32 operators

. . .

VPN concentrator

Verizon network

Internet



The observations from the test are provided below:

VPN Stability Evaluation:

As the 32 users of the laptops attempted to login to the VPN server, we observed a highly unstable connectivity situation with connections getting rejected or dropped as others were accepted. We recorded a ma

diamond management technology c na, inc diamond cook county clerk sequoia... · diamond management...

Documents