WaterDistributionSystemSCADASurveyReport

DevelopedbyTheUniversityofMissouri

Preparedforthe

NationalInstituteofHometownSecurity

368N.Hwy27

Somerset,KY42503

April16,2013

 

This research was funded through funds provided by the Department of Homeland Security, 

administered by the National Institute for Hometown Security Kentucky Critical Infrastructure 

Protection program, under OTA # HSHQDC‐07‐3‐00005, Subcontract # 02‐10‐UK 

1  

NIHS SCADA SURVEY REPORT Table of Contents

Acronyms……………………………………………………………………………………….. 2 Abstract ………………………………………………………………………………………… 2 Background …………………………………………………………………………………….. 2 The Survey……………………………………………………………………………………… 3 Implementation ………………………………………………………………………………… 4 Data Analysis …………………………………………………………………………………... 4 Results ………………………………………………………………………………………….. 5 A. Results for most common uses of SCADA …………………………………………. 5

B. Results directly supporting this project ……………………………………………... 7 Documents used in Preparation of Survey and Report ………………………............………… 8 Appendix A: Survey Document Appendix B: Survey Results

2  

Acronyms American Water Works Association (AWWA) Bristol Standard Asynchronous Protocol (BSAP) Contaminant warning systems (CWS) Department of Health and Human Services (DHHS) Digital Prototype Systems (DPS) Environmental Protection Agency (EPA) General Electric (GE) Institute for Public Policy (IPP National Institute of Hometown Security (NIHS) Nota Bene Technology (NBT) Short message service (SMS) Supervisory Control and Data Acquisition (SCADA) University of Missouri (MU) Vendor management system (VMS) Abstract This survey is part of the project Studying Distribution System Hydraulics and Flow Dynamics to Improve Water Utility Operational Decision Making funded by the National Institute of Hometown Security. It was intended to develop an understanding of the status of Supervisory Control and Data Acquisition (SCADA) system usage in water supply utility distribution systems around the country. This information could be used to provide an understanding of impediments to full development of SCADA to support and verify hydraulic and water quality modeling, and to the implementation of contaminant warning systems (CWS) by water supplies. With respect to information directly supporting this project, the intent was to learn:

how data sets are being analyzed with respect to operations management, security management, and incident response management;

how management is using their SCADA data;

what are the interfaces w/ distribution system sensors & hydraulic modeling;

what do utilities identify as the benefits from using SCADA;

how do utilities demonstrate economic benefits from using SCADA & distribution system models for management decision-making.

The survey activity was required to be limited to a total of nine respondents for compliance with federal paperwork reduction requirements. This number is not adequate to develop statistical analysis of the survey data. So the results simply identify a snapshot of the nine respondents’ use of this technology. However, this project developed the tools that could be applied to advance the use of SCADA data for distribution system modeling, as a component of a CWS, and other operational management activities of water supply utilities.

3  

Of this group of water distribution systems that responded to the survey, all have SCADA systems, some for 20 or more years. SCADA data is used primarily for operational purposes and security monitoring, with limited management decision making application. There were several major findings. One of the respondents applies water quality modeling software for demand forecasting. Eight use hydraulic modeling software for various purposes, with long term planning & design being the most common ones. Only one utility has developed a CWS. Background SCADA systems have been used in water supply for more than several decades. Anecdotal information and experience of the authors of this document indicate that SCADA is used primarily for day-to-day operational decision making and short term planning. The more common applications of these monitoring systems include:

security monitoring,

water quality monitoring,

equipment management,

data management,

process control,

alarm condition identification. The survey was organized using these generic applications. Other and more advanced uses of SCADA data include verification of distribution system hydraulic and water quality monitoring, as a component of a CWS, optimization of energy efficiency, operational report generation, and operational cost control. The focus of this project was the first three of these advanced uses. The Survey The intent of the survey was to determine the status of SCADA usage in water distribution systems, provide information so utility managers could compare their SCADA systems and usage of SCADA data to other utilities of similar size, and to provide data in support of this project. This would include how SCADA systems are being employed, including how management is using their SCADA data with respect to operational, security, incident response and data management, and a variety of related information about SCADA systems and usage. Specifically with respect to information supporting this project, the intent was to learn:

how data sets are being analyzed with respect to operations management, security management, and incident response management;

how management is using their SCADA data;

4  

what are the interfaces w/ distribution system sensors & hydraulic modeling;

what do utilities identify as the benefits from using SCADA;

how do utilities demonstrate economic benefits from using SCADA & distribution system models for management decision-making.

Survey Design The National Institute of Hometown Security (NIHS) SCADA survey was developed based on the six more common applications of SCADA systems currently used in support of water distribution system operations identified previously. A series of questions associated with each application were prepared. There were numerous questions, so a decision tree display logic survey was developed. In this method, based on the response to a question, the software selects the next set of questions. If the answer to a question was positive, an additional list of questions on that subject are provided. If an answer is negative, a question on a different subject is provided. Further information on this approach is available at http://www.qualtrics.com/university/researchsuite/basic-building/question-options/display-logic). Survey tool The survey questionnaire used in the design this instrument was reviewed by a survey expert in the University of Missouri (MU) Institute for Public Policy (IPP). The survey was developed using Survey software Qualtrics. It was an online survey hosted on the MU IPP server at https://umissouriipp.us2.qualtrics.com/SE/?SID=SV_bO9e1kZZLrO1aw4. Institutional Research Board (IRB) requirements The Campus Institutional Review Board (Campus IRB) is charged with assuring that all human subject research, regardless of sponsorship, conducted at the MU complies with the federal regulations (45 CFR 46) of the Department of Health and Human Services (DHHS), Food and Drug Administration (FDA), the Belmont Report, State and Local Laws, and Campus IRB policies and procedures. Because the survey was to involve humans, the required documentation on this project was submitted to the IRB for review. An Exempt Approval was received from the Board. Implementation To develop the survey, a web search of SCADA manufacturers and technologies was conducted. SCADA Survey content was drafted, and the survey content was entered on software package Qualtrics. The draft survey was posted online, and comments were requested from the Project team members and advisory board. Comments on the draft survey were processed and appropriate modifications were made to the questions. Upon final approval, the survey was

5  

advertised to the water supply utility sector and the survey was launched. An adequate number of online responses were not received, so face-to-face interviews of a number of water utilities were conducted. To assure consistency between the set of online responses and interview responses, the survey questions were used as the interview information collection tool. The survey activity was required to be limited to a total of nine respondents. This number is not adequate to develop statistical analysis of the survey data. The reviewing statistician identified that a sample of 30 is needed begin to have a representative sample. Samples short of that have an increasing chance of being skewed by outliers in the data. In other words, a sample of less than 30 simply does not produce enough statistical power for there to be confidence in the results of the survey (MU IPP, 2013). Data Analysis Data analysis was limited because of the small size of the survey population. Percentages of respondents and summaries of responses were developed. The respondents of the survey were categorized based on their size as small (<10000 customers), medium (10000> & <100000 customers) and large (>100000 customers) utilities. Respondents were also categorized based on source waters: ground water system, surface water system, or combination of both. Results A table of the survey questions is located in Appendix A. A table of the complete responses to the survey is located in Appendix B. Of the respondents to this survey, three were small systems, three medium and three large. Six of these utilities are using surface water supply, two are using ground water and one is has a combination of both. A. Results for the Most Common Uses of SCADA. The survey responses that follow are divided into the SCADA use categories used to design the survey. General Information Regardless of size all the participating utilities have been using SCADA to monitor or control their distribution system for a minimum of 5-10 years and majority of them over 20 years A majority of them received an upgrade in last 2 to 3 years.

6  

Since some cities have combined electric and water departments, several have more than one SCADA system to control/monitor different aspects of their operations. Of these respondents, one utility have more than one SCADA system to support their operations. Security Monitoring Eight of the nine respondents have security monitoring capabilities for their distribution systems. These include motion detection, glass breakage, fire/smoke, etcetera. Facility access control is monitored by SCADA, and all of the respondents have door/window or equipment access sensors in their distribution system facilities. Only one utility has developed a CWS, which is alternatively termed an event detection system. Water Quality Monitoring The responding utilities have hydraulic sensors including flow, pressure & depth, and water quality sensors including pH, chlorine & hardness placed at different locations in their distribution systems. The locations include pumping stations, water storage towers and storage reservoirs. While the majority of the hydraulic measurements are performed by SCADA, less than 50% of the water quality measurements are performed by SCADA. The number of security and measurement sensors placed in these distribution systems by utilities, and the percentage of measurements performed by SCADA appear to be dependent on the size of utility to some extent. Only one responding water utility has a CWS. Sensor placement optimization software was used to locate water quality sensors in that system. Equipment Maintenance Responses from our survey were almost evenly divided with respect to equipment maintenance and SCADA. Fifty six percent have SCADA provide them with equipment status monitoring like run-time, oil pressure and temperature and use this data for maintenance prediction or repair/replacement forecasting. Interview responses indicate that this provision is mainly depended on the resources a utility has at their disposal rather than a necessity. Data Management These water utilities use Windows, open-vendor management system (VMS) or In-touch Window ware as the operating system platform to run their SCADA. This decision has been mainly based on the preference of the SCADA technician and treatment plant managers. The results of the survey show that there are several SCADA software packages that utilities are using. The software employed by this group of utilities is a combination of multiple

7  

manufactures, with General Electric (GE), Iconics and Rockwell Automation being the preferred manufactures by these respondents. In addition to an existing SCADA system, only one of the respondents applies water quality modeling software for demand forecasting; that system includes InfoWater with a computer algorithm. Eight use hydraulic modeling software like KyPipe, EPANet, SynerGEE water and InfoWater for different purposes, with long term planning & design being the most common. Most of our participants either use wireless radio or fiber optic as their mode of SCADA telemetry, and a few still use leased line as their mode. The protocols that are being used for communications within the SCADA systems are Modbus, Ethernet, Genius Global, and Bristol Standard Asynchronous Protocol (BSAP). The data collected through the use of SCADA sensors in the distribution systems can be used for a variety of managerial purposes including historical trending of water quality, water usage and bill generation, regulatory reporting etc. Survey responses indicate that utilities are using the data from SCADA sensors mainly for historical trending of water quality/usage and water usage and bill generation. Most regulatory reporting is done using manually collected data or a combination of the two. The data collected by SCADA systems in these utilities is stored indefinitely. Remote access to this data is commonly available. Process Control Seven of participating utilities are using their SCADA system to remotely control processes including pumping system operations, chemical addition and water storage leveling. These features of their SCADA systems can be accessed from locations other than the primary SCADA control interface. A number can also access the systems using the internet. Alarm Handling Utilities who responded to our survey have numerous alarm notification capabilities within their SCADA systems, mainly for change in hydraulic characteristics, security, and water quality. Notification is usually by pop-up on screen or short message service (SMS) text messaging. Operators at these utilities consider about 20% of the alarms generated are nuisance. Alarm notifications are usually grouped into categories based on priority. Operators consider water quality change as the first priority. In most cases once the alarm is triggered the SCADA system does not have the capability to notify the emergency services like fire or police.

8  

Cost The respondents to the survey were not knowledgeable of costs or economics of the SCADA systems. However, they identified a number of areas of cost savings, including remote control of distribution system processes with control from more than just one central location, which reduces manpower and travel costs going to the individual sites in the distribution system to exercise equipment control. Another obvious cost savings results from using SCADA data to generate monthly user bills, which reduces the time and travel for water meter reading. These are obvious financial savings, although the respondents could not quantify them. B. Results Directly Supporting this Project Data sets are being applied for management of a number of operational activities. Those identified by respondents include:

remote control of pumping, chemical addition and water storage leveling,

generating monthly user bills,

notification of alarm conditions for changes in hydraulic characteristics in the distribution system, deterioration of water quality in the system, facility security.

Security management is supported by facility access sensor data. Sensor data identifies when someone or something enters a distribution system facility. Only one respondent uses SCADA data as a component of a CWS. This was the only incident response management data application identified.

Management obviously plays a role in the use of SCADA data for the operational applications identified above. In addition, management uses it for:

historical trending of water quality and water usage,

equipment repair and replacement forecasting,

maintenance prediction,

demand forecasting. None of the respondents identified electronic/software interfaces between distribution system sensors & hydraulic or water quality modeling. Information on methodology of manual application of SCADA data for hydraulic and water quality modeling was also not provided.

The benefits from uses of SCADA data that were identified by this set of respondents include:

reduced personnel time for controlling equipment,

monitoring equipment operations,

generating bills,

forecasting of equipment repair and replacement,

increase security of the distribution system facilities,

9  

notification of alarm conditions (excursions from normal operating conditions) identified above,

maintenance prediction,

more consistent knowledge of water quality in the distribution systems.

Respondents were not knowledgeable of quantified economic benefits from using SCADA & distribution system models for management decision-making. The cost reductions noted earlier were those identified. Documents used in Preparation of Survey and Report Allen M, Mudasser I, Srirangarajan S, Lim B L, Girod L, Whittle A. 2011. Real-time in-network distribution system monitoring to improve operational efficiency. American Water Works Association (AWWA). 103(7): 63-75. Bentley Systems Incorporated, 2004. The Fundamentals of SCADA – A white paper from Bentley Systems, Inc. Berry, Bob, 2009 SCADA Tutorial: A Fast Introduction to SCADA Fundamentals and Implementation. Digital Prototype Systems (DPS) Telecom, [cited 2011 May 23]. Available from: http://www.dpstele.com/white-papers/scada/ Berry, Jonathan W., et al. "Sensor placement in municipal water networks." Journal of Water Resources Planning and Management 131.3 (2005): 237-243. Chen A, Haught R C, Goodrich J, Pollack A. 1999. Options for Remote Monitoring and Control of Small Drinking Water Facilities. Columbus (OH): Battelle Press. 318 p. Clingenpeel, William H., and Don B. Rice. "Development of a water and wastewater SCADA system." Public Works 121.7 (1990): 60-62. Contamination Warning System CIPAC Workgroup Final Report. March 2012.Water Sector Coordinating Council and Water Government Coordinating Council. EPA Software that Models the Hydraulic and Water Quality Behavior of Water Distribution Piping Systems [Internet]. c2011. Environmental Protection Agency (EPA); [cited 2011 June 08]. Available from: http://www.epa.gov/nrmrl/wswrd/dw/epanet.html EPANet 2 Users Manual [Internet]. c2000. USEPA; [cited 2011 July 11]. Available from: http://www.epa.gov/nrmrl/wswrd/dw/epanet/EN2manual.PDF

10  

Garbrecht, Stephen D., “Invensys Wonderware, The Benefits of Component Object-Based SCADA and Supervisory System Application Development”. Undated. Available at http://www.wonderware.co.za/content/ComponentBasedArchi.pdf Hart, W., and Murray, W., Review of Sensor Placement Strategies for Contaminant Warning Systems in Drinking Water Distribution Systems. Journal of Water Resources Planning and Management. ASCE. Nov/Dec 2010. 611:619. Mounce, Stephen R., et al. "Sensor-fusion of hydraulic data for burst detection and location in a treated water distribution system." Information fusion 4.3 (2003): 217-229. Mulhauser G. [Internet]. 2010. The Benefits of Component Object-Based SCADA and Supervisory System Application Development. Lake Forest (CA): Wonderware.com; [cited 2011 June 24]. Available from: http://global.wonderware.com/EN/PDF%20Library /The_Benefits_of_Component_Object_Based_SCADA_and_Supervisory_System_Application_Development_White_Paper.pdf MullaneyJ. [Internet]. Introduction to SCADA and Telemetry. MSE Technology Applications Inc.: Tetragenics.com; [ cited 2011 June 24]. Available from: http://www.tetragenics.com/ pdf/presentations/scada_2.pdf Nota Bene Technology (NBT) Telemetry Equipment, A SCADA System Assessment, [cited 2011 May 21]. Available from: http://www.nbtinc.com/scada-system-assessment.html Pickard B C, Haas A J, Allgier S C. 2011. Optimizing Operational Reliability of the Cincinnati Contamination Warning System. AWWA. 103(1): 60-68. Pipe2010: KYPipe (Steady State Analysis) [Internet]. c2009. KYPipe; [cited 2011 June 08]. Available from: http://www.kypipe.com/kypipe SCADA [Internet]. c2011.Wikipedia; [cited 2011 May 23]. Available from: http://www.en.wikipedia.org/wiki/SCADA Serjeantson B, McKinney S, Van Buskirk. 2011. Leverage Operations Data and Improve Utility Performance. Opflow. 37(2): 10-15. Software that Models the Hydraulic and Water Quality Behavior of Water Distribution Piping Systems [Internet]. c2011. USEPA; [cited 2011 June 08]. Available from: http://www.epa.gov/nrmrl/wswrd/dw/epanet.html

11  

Strategic Metering [Internet]. 2009. Redwood Shores (CA): Oracle; [cited 2011 May 21]. Available from: http://www.oracle.com/us/industries/utilities/046596.pdf US-CERT: United States Computer Emergency Readiness Team [Internet]. c2011. US Department of Homeland Security; [cited 2011 June 14]. Available from: http://www.us-cert.gov/control_systems/ Van Hemert G. [Internet]. So You Have a SCADA System What is Next?. [cited 2011 May 23]. Available from: http://downloads.vertmarkets.com/files/downloads/c1809894-df8a-46ce-b191-e807e13ae4d5/schneiderelectric_scada.pdf

12  

Appendix A: Survey Document

13  

SCADA Survey Questions

SCADA General Information

1. Are you currently using any form of SCADA or Distributed Control System (DCS) to monitor or control your distribution system?

2. Approximately how many years has your facility been utilizing some form of SCADA or remote monitoring and control technology?

3. Which of the following operating system platforms are used to run your SCADA software? (More than one answer may be selected.)

4. Which of the following companies manufacture the SCADA software program are you currently using? (More than one answer may be selected.)

5. Do you use more than one SCADA system to control/monitor different aspects of your operations? For instance, one system for security and another for treatment or distribution operations.

6. How many separate SCADA systems are in use at your utility?

7. Do you use hydraulic modeling software (e.g., EPANet, WaterCad, or KYPipe) to support water distribution system operations?

8. Which of the following hydraulic modeling software programs do you use in your distribution system? (More than one answer may be selected.)

9. How recently has your SCADA system received upgrades?

14  

10. If known, please specify the main SCADA component (e.g., software, sensors, PLCs, etc.) that was upgraded

Security Monitoring

11. Do you have security monitoring capabilities for your distribution system?

12. Which of the following protections are incorporated into your water distribution security monitoring system? (More than one answer may be selected.)

13. Are you using SCADA to monitor security system features within your distribution system?

14. Of the following security system features, which of the following are monitored by your SCADA software interface?

15. Which of the following security sensors are you currently using within your distribution system? (More than one answer may be selected.)

16. Does your utility use a contamination warning/event detection system such as Whitewater Security's Blue Box or Hach Guardian Blue within the distribution system?

Ambient Air / Water Quality Monitoring

17. Please indicate the approximate percentage of water quality and hydraulic measurements that are performed by SCADA sensors.

18. The following table contains a list of common hydraulic and water quality sensor types. Please choose location(s) that correspond to the sensor types in use within your distribution system that are currently monitored by SCADA.

15  

19. Has your utility used programs such as InfoWater SLM (Sensor Location Manager) or EPA's TEVA-SPOT to assist with selection of optimal locations for sensor placement in your distribution system?

Equipment Management

20. Does your SCADA system provide equipment status monitoring such as run-time, oil pressure, or temperature?

21. Is data collected from equipment sensors used for maintenance prediction or repair/replacement forecasting?

22. Which of the following sensor technologies are used with your SCADA system for predictive maintenance or replacement forecasting? (More than one answer may be selected)

Data Management

23. What mode/modes of SCADA telemetry (data transmission) are used to transmit information from distribution system SCADA components to your SCADA system?

24. Which communication protocols (e.g., IEC 60870, Profinet, Hart) are being used to communicate within your SCADA system? (More than one response is allowed, if unknown you may skip question)

25. Data collected manually or through the use of SCADA sensors can be used for a myriad of managerial purposes. The following list contains several common managerial uses of data with respect to the water industry and in general. Please indicate which are done with the assistance of manually collected data, SCADA data, or a combination of both.

26. Do you have data storage and analysis system (Historian/ODMS) that stores data collected from sensors, water meters, or the like?

16  

27. On average, how long is data collected by SCADA able to be stored before it is "dumped" or erased?

28. Do you have remote access (other than your primary control interface) to your SCADA data?

29. If applicable or known, how do you protect your system from internet attacks (e.g., Firewall, Owl Perimeter Defense Solutions, etc)? If known, please indicate either the type or brand name of the security product in the text box below.

Process Control

30. Is your SCADA system used to remotely control physical processes (e.g., Pumps, valves, etc.) in the distribution system?

31. What remotely controlled features are used in your distribution system?

32. Are process control features of your SCADA system able to be accessed from locations other than the primary SCADA control interface?

33. Are process control features accessible via the internet?

Alarm Handling

34. Please select any of the following areas that currently have alarm notification capabilities within your SCADA system. (Additional areas may be entered manually)

35. How does SCADA system notify operators/responders of an alarm? (Please check all that apply.)

36. Approximately what percentage of alarms generated are false or nuisance alarms?

17  

37. Do you group alarms into categories based on priority?

38. Please rank the following alarm categories by priority

39. If a security alarm is triggered, does SCADA system have the ability to notify emergency services such as fire or police?

Cost

40. What is the estimated replacement value of your entire SCADA system?

41. What are your approximate annual SCADA expenditures related to system upkeep/maintenance? (Please provide your best estimate of annual maintenance costs.)

42. What percentage of time would you estimate your SCADA system to be completely operational and not experiencing a "downtime" event due to hardware or software failures?

43. What is your utilities approximate annual budget? (If unknown leave blank)

44. Does the use of SCADA provide a noticeable cost savings (e.g., reduction in staff time to necessary to maintain operations, reduction in material/ energy consumption, etc) in annual operating expenses?

45. Rank the following SCADA functions that have provided a noticeable cost savings (e.g., reduction in staff time to necessary to maintain operations, reduction in material/energy consumption, etc) from greatest savings to least.

Opinion

46. Data collected by SCADA sensors is equally reliable as manually collected data.

18  

47. Continuous real-time monitoring of equipment status (vibration pattern analysis, temperature, oil pressure, etc.) should be a critical part of your maintenance program.

48. Do you believe cloud or internet based SCADA software/networks pose a serious security risk with respect to distribution operations?

49. Rank areas in which the use of historical data would provide the greatest benefit to your operations.

50. Do you believe there is any risk in allowing access to operational SCADA data over the internet?

51. Having too many alarm conditions could lead to an important alarm being missed.

52. Please rank the following SCADA functions in order of their importance to your operations with one being the most important.

53. Having a SCADA expert on staff is vital to maintaining a functional and safe SCADA system.

54. What are the biggest deterrents when deciding to implement new features to an existing SCADA system? (Choose any that apply.)

Utility and Respondent Information

55. What size utility do you represent?

56. What is your technical/professional background?

 

19  

Appendix B: Survey Results

20

SCADA General Information Are you currently using any form of SCADA or Distributed Control System (DCS) to monitor or control your distribution system? Options: Yes/No City Response System 1 Yes System 2 Yes System 3 Yes System 4 Yes System 5 Yes System 6 ----- System 7 Yes System 8 Yes System 9 Yes Approximately how many years has your facility been utilizing some form of SCADA or remote monitoring and control technology? Options: 0 to 5 years, 5 to 10 years, 10 to 15 years, 15 to 20 years, 20 years or longer City Response System 1 20 years or longer System 2 20 years or longer System 3 5 to 10 years System 4 20 years or longer System 5 10 to 15 years System 6 15 to 20 years System 7 20 years or longer System 8 20 years or longer System 9 20 years or longer Which of the following operating system platforms are used to run your SCADA software? (More than one answer may be selected.) Options: Windows, Linux, Mac, Unix, Other City Response System 1 Windows System 2 X-windows open-VMS System 3 Windows System 4 Windows System 5 In-touch window ware System 6 Windows System 7 Windows System 8 Windows System 9 Windows

21

Which of the following companies manufacture the SCADA software program are you currently using? (More than one answer may be selected.) Options: ABB, InduSoft, Advantech, Invensys, Allmax Software, Iskra, ARC Informatique, Multitrode, Bentek Systems, National Instruments, Control Micro Systems, Real Flex, CSI UCOS, Rockwell Automation, Data Flow Systems, Schneider Electric, Elutions, Siemans, Emerson Software, Horizons, Etap, Survalent, GE, Tascomp, Hac, Transdyn, Hydromantis, Trihedral, Iconcics, Yokogawa, Icx Technologies (FLIR), Other City Response System 1 GE System 2 QEI System 3 ------ System 4 Peragon, Interlusion, Iconcics System 5 ------ System 6 Rockwell Automation System 7 Honeywell System 8 Inconcics System 9 Rockwell Automation Do you use more than one SCADA system to control/monitor different aspects of your operations? For instance, one system for security and another for treatment or distribution operations. Options: Yes, No, Unsure City Response System 1 Yes System 2 No System 3 No System 4 No System 5 No System 6 No System 7 No System 8 No System 9 No How many separate SCADA systems are in use at your utility? Options: Two, Three, More than three City Response System 1 More than three System 2 ------ System 3 ------ System 4 ------ System 5 ------ System 6 ------ System 7 ------ System 8 ------ System 9 ------

22

Do you use hydraulic modeling software (e.g., EPANet, WaterCad, or KYPipe) to support water distribution system operations? Options: Yes, No, Unsure City Response System 1 Yes System 2 Yes System 3 Yes System 4 Yes System 5 Yes System 6 No System 7 Yes System 8 Yes System 9 Unsure Which of the following hydraulic modeling software programs do you use in your distribution system? (More than one answer may be selected.) Options: SynerGEE Water, Mike by DHI, KYPipe, InfoWater, HydrauliCAD, H20Net/H20Map, EPANet, WaterCad, Other City Response System 1 RTIS, Plank model System 2 InfoWater-MWH System 3 ------ System 4 ------ System 5 ------ System 6 ------ System 7 SynerGEE Water System 8 InfoWater-MWH, EPANet System 9 ------ How recently has your SCADA system received upgrades? Options: Less than one year, 1 to 2 years, 2 to 3 years, 3 or more years, System has never been upgraded, Unknown City Response System 1 Less than one year System 2 ongoing System 3 System has never been upgraded System 4 2-3 years System 5 Less than one year System 6 Less than one year System 7 Less than one year System 8 Less than one year System 9 2-3 years

23

If known, please specify the main SCADA component (e.g., software, sensors, PLCs, etc.) that was upgraded City Response System 1 HMI software System 2 RTU/PLC System 3 ------ System 4 ------ System 5 ------ System 6 2 computers that are used as client/server machines System 7 PLC, Hardware, Software, Historical database System 8 Computer, PLC System 9 Total redo, PLC, RSVue, Servers, Computers Security Monitoring Do you have security monitoring capabilities for your distribution system? Options: Yes, No, Unsure City Response System 1 Yes System 2 Limited System 3 Yes System 4 Yes System 5 Yes System 6 No System 7 Yes System 8 Yes System 9 No Which of the following protections are incorporated into your water distribution security monitoring system? (More than one answer may be selected.) Options: Video Surveillance, Intrusion Detection (Motion detection, glass break detection, etc.), Facility Access Control (e.g., Card Readers, Keypads, etc.), Fire/Smoke Detection, Equipment Tampering, Other (Please Specify) City Response System 1 Intrusion detection, facility access, fire/smoke detection, equipment

tampering, security & remote monitoring in DS System 2 Facility access control, intrusion detection System 3 Intrusion detection, facility access control, equipment tampering System 4 Video surveillance, intrusion detection, facility access, fire/smoke detection System 5 Fire detection System 6 ------ System 7 Video surveillance, intrusion detection, facility access, fire/smoke detection System 8 Video surveillance, intrusion detection, facility access, fire/smoke detection System 9 ------

24

Are you using SCADA to monitor security system features within your distribution system? Options: Yes, No, Unsure City Response System 1 Yes System 2 Limited System 3 Yes System 4 Yes System 5 Yes System 6 No System 7 Yes System 8 Yes System 9 No Of the following security system features, which of the following are monitored by your SCADA software interface? (More than one answer may be selected.) Options: Video Surveillance, Intrusion Detection (Motion detection, glass break detection, etc.), Facility Access Control (e.g., Card Readers, Keypads, etc.), Fire/Smoke Detection, Equipment Tampering, Other City Response System 1 Intrusion detection, facility access, fire/smoke detection, equipment tampering System 2 Limited intrusion detection System 3 Intrusion detection, facility access, equipment tampering System 4 Video surveillance, intrusion detection, facility access, fire/smoke detection,

equipment failure System 5 Fire detection System 6 ------ System 7 Intrusion and fire/smoke detection System 8 Intrusion detection, facility access System 9 ------ Which of the following security sensors are you currently using within your distribution system? (More than one answer may be selected.) Options: door/window or equipment access contact switches, proximity/motion sensors, glass break detectors, pressure sensors (mats, plates, etc.), Water (Leak) Detectors, Other City Response System 1 Door/window switches, proximity sensors, pressure sensors, water leak detectors System 2 Door/window switches System 3 Proximity sensors System 4 Door/window access switches, proximity sensors, glass break detectors, water

leak detectors System 5 Door/window access switches, pressure sensors System 6 ------ System 7 Door/window switches System 8 Door/window switches System 9 ------

25

Does your utility use a contamination warning/event detection system such as Whitewater Security's Blue Box or Hach Guardian Blue within the distribution system? Options: Yes, No, No idea City Response System 1 Yes System 2 No System 3 No System 4 No System 5 No System 6 No System 7 No System 8 No System 9 No Ambient Air / Water Quality Monitoring Please indicate the approximate percentage of water quality and hydraulic measurements that are performed by SCADA sensors. City Water Quality

Measurements

Hydraulic Measurements

System 1 30 60 System 2 Only Continuous data 60 System 3 ------ 30 System 4 >90 50 System 5 20-30 60 System 6 <25 0 System 7 75-100 <25 System 8 <25 75-100 System 9 25-50 75-100 The following table contains a list of common hydraulic and water quality sensor types. Please choose location(s) that correspond to the sensor types in use within your distribution system that are currently monitored by SCADA. If a sensor is placed in a location that is not listed please specify the sensors relative location in its corresponding text box. Any sensor types in use within your system that are not listed in the table may be manually entered in the "Other" field at the bottom of the table. This question is not mandatory, but if you are able to please answer to the best of your ability. Note: Responses are marked with System number. Parameter Water Tower Pump Station Storage

Reservoir Water Main

Flow 1, 2, 3, 9 1, 2, 3, 4, 7, 8 1, 2, 4 1, 2, 5 Level 1, 2, 3, 5, 6, 7, 9 1 1, 2, 4, 8 1 Pressure 1, 3, 9 1, 2, 3, 4, 5, 7, 8 1, 4, 5 1, 2, 5 Depth 1, 5, 7 1 1, 4, 5 1 pH 1, 3, 9 1, 3, 7, 8 1 1, 2

26

Temperature 1, 3, 4, 6 1, 3, 7, 8 1, 4 1, 2 Total organic Carbon

------ 7 ------ ------

Turbidity 1, 3, 4, 9 1, 3, 4, 7, 8 1, 4, 5 1, 5 Dissolved Oxygen

------ 7 ------ ------

Suspended Solids

------ 7 ------ ------

Dissolved Solids ------ 7 ------ ------ Hardness ------ 7 ------ ------ Chlorine 1, 3, 6 1, 2, 3, 4, 7, 8 1 1, 5 Chloramines ------ 7 ------ ------ Ammonia 3 2 ------ ------ Iron ------ 7 ------ ------ Nitrate ------ 7 ------ ------ Fluoride 3 3, 7 ------ ------ COD ------ 7 ------ ------ Sodium ------ 7 ------ ------ Conductivity/ Resistivity

6 6, 7 ------ ------

ORP ------ ------ ------ ------ Ozone ------ ------ ------ ------ Phosphate ------ ------ ------ ------ Potassium Permanganate

------ 7 ------ ------

Copper ------ 7 ------ ------ Particle Counters ------ 7 ------ ------ Manganese ------ 7 ------ ------ Corrosion ------ 7 ------ ------ Hydrazine ------ ------ ------ ------ Salinity ------ ------ ------ ------ Oil in water ------ ------ ------ ------ Other ------ ------ ------ ------ Other ------ ------ ------ ------ Has your utility used programs such as InfoWater SLM (Sensor Location Manager) or EPA's TEVA-SPOT to assist with selection of optimal locations for sensor placement in your distribution system? Options: Yes, No, No idea City Response System 1 Yes System 2 Convenience of the station System 3 No idea System 4 No idea System 5 No System 6 No System 7 No idea System 8 No System 9 No

27

Equipment Management Does your SCADA system provide equipment status monitoring such as run-time, oil pressure, or temperature? Options: Yes, No, No idea City Response System 1 No System 2 No System 3 No System 4 Yes System 5 No System 6 No System 7 Yes System 8 Yes System 9 Yes Is data collected from equipment sensors used for maintenance prediction or repair/replacement forecasting? Options: Yes, No, No idea City Response System 1 No System 2 No System 3 No System 4 Yes System 5 No System 6 ------ System 7 Yes System 8 Yes System 9 Yes Which of the following sensor technologies are used with your SCADA system for predictive maintenance or replacement forecasting? (More than one answer may be selected) Options: Temperature, Vibration, Current/Voltage, Oil Pressure, Other City Response System 1 ------ System 2 ------ System 3 Temperature System 4 Temperature, vibration, current voltage, oil pressure System 5 ------ System 6 ------ System 7 Temperature, vibration, current voltage, oil pressure System 8 Run time hours System 9 Temperature, current voltage

28

Data Management What mode/modes of SCADA telemetry (data transmission) are used to transmit information from distribution system SCADA components to your SCADA system? (Choose all that apply.) Options: Wi-Fi, Leased Line / Dial-up phone line, Cellular, Internet, Wireless, Radio (VHF, UHF, etc.), Satellite, Fiber Optic, No idea City Response System 1 Wi-Fi, Wireless radio, Fiber optic, Leased line, Internet, AT&T-T1 System 2 Fiber optic, leased line – back up System 3 Wireless radio, Fiber optic System 4 Wireless radio, Leased line System 5 Wireless radio System 6 Wireless radio, Fiber optic, Leased line System 7 Wi-Fi, Wireless radio, Fiber optic System 8 Wi-Fi, Wireless radio, Fiber optic System 9 Wireless radio, Fiber optic Which communication protocols (e.g., IEC 60870, Profinet, Hart) are being used to communicate within your SCADA system? (More than one response is allowed, if unknown you may skip question) City Response System 1 Ethernet, Modbus, Genius global, SQL, EM, FW, WEB System 2 Modbus, SNP-GE System 3 Modbus System 4 Modbus System 5 ------ System 6 ------ System 7 ------ System 8 BSAP, IP System 9 ------

29

Data collected manually or through the use of SCADA sensors can be used for a myriad of managerial purposes. The following list contains several common managerial uses of data with respect to the water industry and in general. Please indicate which are done with the assistance of manually collected data, SCADA data, or a combination of both. Note: Responses are marked with System number. Managerial Uses Manually

collected data Data collected by SCADA

Combination of two

N/A

Historic trending of water quality 3 1, 2, 5, 8 4, 6, 7, 9 ------Water usage & bill generation 2, 4, 9 1, 5, 7, 8, 9 3 ------Regulatory reporting 2, 3, 4 3, 7 1, 6, 8, 9 Financial reporting 2, 6 1, 3 4, 5, 9 Incident response management 2, 6 1, 5 ------ ------System performance optimization ------ 7, 8 1, 4, 6, 9 ------ Equipment maintenance scheduling & performance analysis

2, 6 1, 7 8, 9 4

Plant production reporting 2 1, 5, 7, 8 3, 6, 9 ------Historical trending of water usage 2, 6 1, 5, 7, 8 9 ------Event prediction 2, 4, 6 1, 5, 8 7, 9 ------Material consumption reporting/analysis

2, 4, 5, 6 1, 8, 9 7 ------

Calibration of hydraulic model 2, 3, 4 1, 8 7 5, 6, 9 Optimizing alarm handling & response procedures

6 1, 5, 7 ------ 2, 4, 9

Do you have data storage and analysis system (Historian/ODMS) that stores data collected from sensors, water meters, or the like? Options: Yes, No, No idea City Response System 1 Yes System 2 No System 3 No idea System 4 No System 5 Yes System 6 No System 7 Yes System 8 Yes System 9 No idea

30

On average, how long is data collected by SCADA able to be stored before it is "dumped" or erased? Options: 6 months to a year, indefinitely, 3 to 6 months, not stored at all, 1 to 3 months, No idea, Less than 30 days City Response System 1 Indefinitely System 2 45 days System 3 3 years System 4 3 or more years System 5 3-6 months System 6 No idea System 7 Indefinitely System 8 Indefinitely System 9 6 months to year Do you have remote access (other than your primary control interface) to your SCADA data? Options: Yes, No, No idea City Response System 1 Yes System 2 Limited System 3 Yes System 4 Yes System 5 Yes System 6 Yes System 7 Yes System 8 Yes System 9 Yes If applicable or known, how do you protect your system from internet attacks (e.g., Firewall, Owl Perimeter Defense Solutions, etc)? If known, please indicate either the type or brand name of the security product in the text box below. City Response System 1 Firewall EM - industry defense System 2 Not hooked to internet System 3 ------ System 4 ------ System 5 ------ System 6 ------ System 7 Firewall, Internal System 8 Firewalls System 9 Firewalls, Unplugging the outside line

31

Process Control

Is your SCADA system used to remotely control physical processes (e.g., Pumps, valves, etc.) in the distribution system? Options: Yes, No, No idea City Response System 1 Yes System 2 Yes pump station & wells System 3 Yes System 4 Yes System 5 Yes System 6 No System 7 Yes System 8 Yes System 9 No What remotely controlled features are used in your distribution system? (Choose all that apply, extra features may be entered manually) Options: Flow control, Pumping systems, Chemical addition (disinfectants, tracers, etc.), Water storage leveling, No idea, Other 1 City Response System 1 Flow control, Pumping systems, Water storage leveling System 2 Pumping systems, Chemical addition System 3 Pumping systems, Chemical addiction, water storage leveling System 4 Flow control, Pumping systems, Chemical addiction System 5 Flow control, Pumping systems, Water storage leveling System 6 ------ System 7 Flow control, Pumping systems, Water storage leveling, Chemical addition System 8 Flow control, Pumping systems, Chemical addiction System 9 ------ Are process control features of your SCADA system able to be accessed from locations other than the primary SCADA control interface? Options: Yes, No, No idea City Response System 1 Yes System 2 Yes System 3 Yes System 4 Yes System 5 Yes System 6 No System 7 Yes System 8 Yes System 9 Yes

32

Are process control features accessible via the internet? Options: Yes, No, No idea City Response System 1 Yes System 2 No, fiber connected System 3 Yes System 4 Yes System 5 Yes System 6 --------- System 7 Yes System 8 No System 9 Yes Alarm Handling Please select any of the following areas that currently have alarm notification capabilities within your SCADA system. (Additional areas may be entered manually) Areas: Air

Quality Change in hydraulic characteristics

Security Energy consump- tion

Water quality

Equipment status

Other

System 1 x x x x x ------ ------System 2 ------ x ------ ------ x x System 3 x x x ------ x ------ ------System 4 x x x x x x ------ System 5 ------ x x ------ x ------ ------ System 6 ------ x ------ ------ x ------ x System 7 x x x ------ x x ------ System 8 x x x x x x x System 9 ------ x ------ ------ x x ------ How does SCADA system notify operators/responders of an alarm? (Please check all that apply.) Options: Page, Email, SMS (text message), Phone call, Pop-up or some other message on HMI/computer screen, No idea City Response System 1 Email, SMS, Pop up screen System 2 Pop up screen System 3 Pop up screen System 4 Page, SMS, Phone call, Pop up screen System 5 Phone call, Pop up screen System 6 Pop up screen System 7 Pop up screen System 8 Pop up screen, Plant intercom System 9 Email, SMS, Pop up screen

33

Approximately what percentage of alarms generated are false or nuisance alarms? City Response System 1 Not many System 2 <10 System 3 50 System 4 50 System 5 50 System 6 No idea System 7 10-15 System 8 10-15 System 9 15-20 Do you group alarms into categories based on priority? Options: Yes, No City Response System 1 Yes System 2 Only 2, so no priority System 3 ---- System 4 Yes System 5 Yes System 6 No System 7 Yes System 8 Yes System 9 Yes Please rank the following alarm categories by priority by dragging and dropping them in order with one being the most urgent. Areas: Air

Quality Hydraulic characteristics

Security Energy consumption

Water quality

Equipment status

System 1 4 3 2 5 1 6 System 2 ------ ------ ------ ------ ------ ------ System 3 4 2 3 ------ 1 ------ System 4 ------ 1 2 ------ ------ 3 System 5 ------ 2 3 ------ 1 ------ System 6 ------ ------ ------ ------ ------ ------ System 7 2 5 4 6 1 3 System 8 1 3 5 6 2 4 System 9 6 2 3 5 3 5

34

If a security alarm is triggered, does SCADA system have the ability to notify emergency services such as fire or police? Options: Yes, No, Unsure City Response System 1 No System 2 No System 3 No System 4 No System 5 Yes System 6 ------ System 7 No System 8 No System 9 ------ Cost What is the estimated replacement value of your entire SCADA system? (Please provide your best estimate for replacement of all hardware and software.) City Response System 1 300000-500000 System 2 1.2 Mil System 3 1 person System 4 250000 System 5 Not cheap System 6 ------ System 7 100000 or above System 8 100000 or above System 9 100000 or above What are your approximate annual SCADA expenditures related to system upkeep/maintenance? (Please provide your best estimate of annual maintenance costs.) Options: $0-$5,000, $5,000-$10,000, $10,000 or above, No idea City Response System 1 25000-SCADA, 50000- Instrumentation System 2 120000 System 3 >10000 System 4 No idea System 5 0-5000 System 6 10000 or above System 7 10000 or above System 8 10000 or above System 9 0-5000

35

What percentage of time would you estimate your SCADA system to be completely operational and not experiencing a "downtime" event due to hardware or software failures? Options: 90-100%, Less than 70%, 70-80%, 80-90%, Unsure City Response System 1 90-100 System 2 90-100 System 3 90-100 System 4 90-100 System 5 90-100 System 6 90-100 System 7 90-100 System 8 90-100 System 9 90-100 What is your utilities approximate annual budget? (If unknown leave blank) City Response System 1 30 Million System 2 ------ System 3 1 Million System 4 ------ System 5 ------ System 6 1.5 Million System 7 ------ System 8 350 Million System 9 ------ Does the use of SCADA provide a noticeable cost savings (e.g., reduction in staff time to necessary to maintain operations, reduction in material/ energy consumption, etc) in annual operating expenses? Options: Yes, No, Unsure, No idea City Response System 1 Yes System 2 Yes System 3 Yes System 4 No System 5 No System 6 No System 7 No System 8 Yes System 9 Yes

36

Rank the following SCADA functions that have provided a noticeable cost savings (e.g., reduction in staff time to necessary to maintain operations, reduction in material/energy consumption, etc) from greatest savings to least. Not all choices have to be ranked. Areas: Ambient

Air/water quality monitoring

Energy manage- ment

Data manage- ment

Security monitoring

Equipment manage- ment

Process control

Alarms

System 1 4 2 1 2 7 1 3 System 2 3 4 5 6 2 1 7 System 3 3 ------ ------ ------ 2 1 ------ System 4 ------ ------ ------ ------ ------ ------ ------ System 5 ------ ------ ------ ------ ------ ------ ------ System 6 ------ ------ ------ ------ ------ ------ ------ System 7 ------ ------ ------ ------ ------ ------ ------ System 8 5 6 3 7 2 1 4 System 9 4 6 2 7 5 1 3 Opinion Data collected by SCADA sensors is equally reliable as manually collected data. Options: Strongly disagree, Disagree, Neither agree nor disagree, Agree, Strongly agree City Response System 1 Strongly agree System 2 Can be System 3 Agree System 4 Agree System 5 Agree System 6 Agree System 7 Strongly agree System 8 Agree System 9 Strongly agree Continuous real-time monitoring of equipment status (vibration pattern analysis, temperature, oil pressure, etc.) should be a critical part of your maintenance program. Options: Strongly disagree, Disagree, Neither agree nor disagree, Agree, Strongly agree City Response System 1 Disagree System 2 Disagree not ideal System 3 Agree System 4 Strongly agree System 5 Agree System 6 Neither agree nor disagree System 7 Strongly agree System 8 Agree System 9 Agree

37

Do you believe cloud or internet based SCADA software/networks pose a serious security risk with respect to distribution operations? Options: Strongly disagree, Disagree, Neither agree nor disagree, Agree, Strongly agree City Response System 1 Strongly agree System 2 Strongly agree System 3 Disagree System 4 Neither agree nor disagree System 5 Agree System 6 Neither agree not disagree System 7 Neither agree not disagree System 8 Strongly agree System 9 Agree Rank areas in which the use of historical data would provide the greatest benefit to your operations.

Areas: Reporting Trending Analysis System 1 1 1 2 System 2 1 2 3 System 3 2 3 1 System 4 2 1 3 System 5 1 2 3 System 6 ------ ------ 1 System 7 1 2 3 System 8 2 1 3 System 9 3 2 1

Do you believe there is any risk in allowing access to operational SCADA data over the internet? Options: Not applicable, Extreme risk, Some risk, Neutral, Slight risk, No risk City Response System 1 Extreme risk System 2 Extreme risk System 3 No risk System 4 Not applicable System 5 Some risk System 6 Some risk System 7 Neutral System 8 Extreme risk System 9 Some risk

38

Having too many alarm conditions could lead to an important alarm being missed. Options: Strongly disagree, Disagree, Neither agree nor disagree, Agree, Strongly agree City Response System 1 Disagree System 2 Agree System 3 Disagree System 4 Strongly agree System 5 Disagree System 6 Agree System 7 Disagree System 8 Strongly agree System 9 Agree Please rank the following SCADA functions in order of their importance to your operations with one being the most important. Not all choices have to be ranked. Functions Air/water

quality monitoring

Energy manage- ment

Data manage- ment

Security monitoring

Equipment manage- ment

Process control

Alarms

System 1 2 1 1 4 7 1 2 System 2 4 3 2 6 5 1 7 System 3 2 6 4 3 5 1 4 System 4 7 6 2 4 5 1 3 System 5 1 6 4 7 5 3 2 System 6 ------ ------ 4 ------ 2 1 3 System 7 1 7 6 5 4 2 3 System 8 4 5 3 7 6 1 2 System 9 2 6 ------ 7 5 1 3 Having a SCADA expert on staff is vital to maintaining a functional and safe SCADA system. Options: Strongly disagree, Disagree, Neither agree nor disagree, Agree, Strongly agree City Response System 1 Strongly agree System 2 Strongly agree System 3 Strongly agree System 4 Strongly agree System 5 Disagree System 6 Agree System 7 Strongly agree System 8 Strongly agree System 9 Agree

39

What are the biggest deterrents when deciding to implement new features to an existing SCADA system? (Choose any that apply.)

Deterrent Multi- vendor compati- bility

Availab- ility of spare parts

Establish- ing new procedures

Reliability concerns

Financial constraint

System down- time

Operator training

Other

System 1 ------ x ------ x ------ ------ x ------ System 2 x x ------ ------ x ------ ------ ------ System 3 x ------ ------ ------ x ------ x ------ System 4 ------ x ------ x ------ x ------ ------ System 5 ------ ------ ------ ------ x ------ x ------ System 6 ------ ------ ------ ------ ------ ------ ------ x System 7 ------ ------ ------ x ------ ------ ------ ------ System 8 x ------ ------ ------ ------ ------ ------ ------ System 9 ------ ------ ------ x ------ ------ ------ ------

Utility and Respondent Information What size utility do you represent? Options: less than 10,000 customers, greater than 10,000 customers but less than 100,000 customers, greater than 100,000 customers City Response System 1 >100000 System 2 >100000 System 3 <10000 System 4 10000<X<100000 System 5 <10000 System 6 <10000 System 7 10000< X <100000 System 8 >100000 System 9 10000< X <100000 What is your technical/professional background? City Response System 1 Control systems System 2 Process control engineering System 3 BS-Aviation System 4 Operations System 5 Farmer/ Jack of all trades System 6 Civil/Environmental Engineering System 7 Operations System 8 Operations System 9 Operations