inoc - art meierdirk presentation from utc april 15, 2014
Post on 25-Jun-2015
84 Views
Preview:
DESCRIPTION
TRANSCRIPT
© 2013 Utilities Telecom Council
Transforming Critical Infrastructure
The Cost of Modernization: Perspectives on Effectively Managing Today's Advanced Technology
Assets without Breaking the Bank
• Art Meierdirk, Senior Director, Business Services – INOC • Dan Belmont, Director, Energy & Utilities - West Monroe Partners • Gary Sanders, Telecom Manager – Sun Prairie Utilities
Utility & Advanced Technology Overview
Dan Belmont, Director, Energy & Utilities
Utility & Advanced Technology Overview
3
• Utilities – Drivers for Advanced Technologies • The Telecom “Tipping Point” • Advanced Technologies – Applications vs. Telecom Capabilities • Creating a Roadmap • Cyber Security and Telecom Synergy • Lesson’s Learned and Path to Success
Utilities – Drivers for Advanced Technologies
4
• Mission? • Over the next 10-15 years, we will transform our delivery
infrastructure into a model we call the “Utility of the Future” which will:
– Empower customers to control electricity usage and costs
– Improve service, enhance reliability and cost controls
– Improve the customer experience
– Increase efficiency of our operations
– Enable Distributed Generation
– Networks and Microgrids
Utilities – Drivers for Advanced Technologies
5
GRID Modernization
End Use Programs
Enabling Infrastructure Investments
Information Technology
Core Telecommunications
Service and Data Center,
Substation, and SCADA
Communications
Transmission, Distribution, and
Substation Automation
(DA/SA)
Advanced Metering
Infrastructure (AMI)
Mission Critical Voice and Data
Systems Backhaul
The Telecom Tipping Point
6
AT&T, Verizon, and LECs announce discontinued analog circuit support in 2013 and service in 2018
NERC CIP version 5
removes the non-
routable protocol
exclusion for critical assets
NISTiR 7628 Smart Grid cyber security standards must
address attacks from disgruntled employees,
industrial espionage, and terrorists ….
95% of Utility companies are going to experience copper theft this year.
Cooper theft caused 456,000 outage minutes, 52 injuries and 35 deaths in one year.
Utilities with AMI have more managed devices per square mile than LECs,
Cable, or Cellular Carriers
Applications vs. Telecom Networks Capabilities
7
Smart Metering & Home Area
Network
Distribution Automation
Substation Protection and
Automation
Network Backhaul
Enterprise Communications
Mobile Voice/Data
Communication Applications New challenges for processes and people
Applications vs. Telecom Networks Capabilities
8
Distribution Automation
Substation Protection and
Automation
§ Coverage of >99% of Meters (preferably Electric, Gas and Water)
§ Higher Latency, Excellent Coverage § Typically Need Mesh Capability
between Meters and HAN Devices
Network Backhaul
Smart Metering & Home Area
Network
Enterprise Communications
Mid-Tier Backhaul
Mobile Voice/Data
Telecom Backbone
AMI and HAN Communications
Mobile Voice/Data
Applications Communication Systems
Communication Systems Characteristics
8
§ Mission Critical Voice/Data § Wide Band Mobile Data § Coverage across Service Territory § Power Generation Back-up
§ Low Latency Broadband § Good Uplink Capacity for Video § Coverage across Service Territory § Power Generation Back-up
§ Mission Critical Voice/Data § Wide Band Mobile Data § Relay Teleprotection Services § Coverage across Service Territory § Power Generation Back-up
Applications vs. Telecom Networks Capabilities
9
Tier OneCore MPLS
Tier Three Access Collectors
Tier TwoBackhaul
Explore combinations of Tier 2 solutions with Tier 3 networks to minimize net cost while maintaining Cyber Security and IP design
standards
Creating a Telecom Roadmap
10
Microwave/Fiber Backhaul • Backbone IP Communica0ons • Connec0vity to AMI comm. • Connec0vity to substa0ons • Connec0vity to 2-‐Way Radio
IT “Hand-‐Off”
ENTERPRISE BUS • Customer Billing/CRM • Load Control Management System • Meter Data Management System • Outage Management System • Distribu0on Management System • Energy Management System
Service Center
AMI • AMI Communica0ons • Smart Meters • Distribu0on Automa0on
• Power Quality Meters • Vehicular Radios • Portable Radios
2-‐Way Voice & Mobile Data
• Microwave or Fiber Based • SCADA to Substa0ons • Communica0ons to AMI • Relay Teleprotec0on services • Distribu0on Automa0on
SubstaLon & DistribuLon CommunicaLons
Home Area Network (HAN) • Home Energy Display • Programmable Thermostats • Load Control Relays • Renewable Energy Genera0on • PHEV/EV Smart Charging
Cyber Security and Telecommunications Synergy
11
Cyber Security and Telecommunications Synergy
12
MPLS service with four forwarding classes
§ Reserve 2 Mb/s for voice and 4 Mb/s for video
§ Enforce 14 Mb/s PIR for overall service
§ Allow critical and best-effort traffic to burst up to 14 Mb/s if bandwidth is available
Less overall bandwidth required § lower overall cost
Priority and best-effort traffic are equally well served § Voice and video do not always consume
all the reserved bandwidth
CIR=0 PIR=4 Mb/s
PIR=max
PIR=max
PIR=2 Mb/s CIR=2 Mb/s
CIR=4 Mb/s
CIR=4 Mb/s
Overall SLA=14 Mb/s
Voice
Video
SCADA FE
Normal Data
Change Management and Training
13
= Aware = Knowledgeable = Expert
Telecom/Cyber Security will
Require Training across the Utility
• Create a holistic business case and roadmap
• Design cyber security in from the ground up
• Tight Quality Control, Quality Assurance, and Audit
• Carefully Orchestrated Cutover/Migration
• Business Process Design
• Organizational Change Management
Lesson’s Learned and Path to Success
REB REBMEB
Ping MEB to verify power (From ALU
gear)
Verify in GS which MEB(s)/REB(s) are communicating with
Ping 7705 to verify power (Can be done within SW or with a device on the same subnet/network). Also
check SW for downed equipment alert
Ping 6855 from 7705 to verify power. (Can be done within SW or a device on the same subnet/network).
Also check SW for downed equipment alert
Ping 7705 from 6855 to test connectivity and
power
Check SAM to ensure MEB port is up and active on
either 6855 or 7705
Check success rates in G.S than PI to identify other possible issues
Still up?No
Refer to site master to verify what equipment is on site
If a 77
05 is o
n-‐site
If 6855
is on-‐si
te
MEB
Fail ping
Then
Pass ping
Pass Ping
If all else fails:On-‐site visit by qualified and identified group through process of
elimination. Worst case scenario. (SSN, BIG, ALU, Standby, MJ, CED)
Pass ping
Pass ping
Alert/Remedy Ticket from SSN that MEB is
down
Find out alerting tool for AC power loss, DC backup activated, Backhaul issues. That way if other steps “fail” we can
start a new process flow
Pass Ping
Fail Ping
Fail Ping
Fail Ping
Yes
Gary Sanders Telecom Manager
The Sun Prairie Utilities Experience
Sun Prairie Utilities
16
• Who and what we are – Municipality
– 100+ years old
– Population of 30,000
– Electric / Water / Internet / WAN
Municipal Entry Into Telecom
17
• Approached by Sun Prairie Area School District in 1998
• They had been quoted a price from the local ILEC for $1.2 million to furnish WAN services to School District
• Cable company was not interested in furnishing services at the time
• City of Sun Prairie got involved • Formed study committee between the three entities
• They used a map of the city to lay out 10 school sites, 5 city sites and 12 Sun Prairie Utilities sites.
• The result was a natural overhead loop on our electric distribution poles.
• At that point, we did an engineering study which was funded by the three Committee members for $5,000 each.
• Also, at this point our Utility Commission became involved and was very supportive.
• Obtained CLEC license • Saving to the SPASD & City exceeds $1M
The Committee Met…
18
Launching Fiber WAN
19
• Simple P2P roll out for School District , City & Utility sites for SCADA
• Began offering business solutions • Evolving expectations / scope • Transition from ATM to Ethernet • Engineering, design & support needs • Partner with NOC provider
– 24x7x365 Support – Reduced staff & tools (monitoring / ticketing)
costs
• At the time customers were not happy with services & availability offered by Incumbent providers
• Temporary solution – FTTH technology had not matured enough at the time. We wanted to offer the community something until we found a FTTH product that we were comfortable with
• Wireless solution would ultimately complement FTTH technology
Why Wireless?
20
Muni Wireless Hype Cycle
21
Enhanced Fiber WAN
22
• Growing services and support • Required outside resource / expertise • Higher level of expectations from subscribers
Partner with NOC provider – Network engineering, design &
deployment – 24x7 Support – Reduced staff & tools (monitoring /
ticketing) costs • Multi-node 10Gb ring architecture • Serving MDU’s as way to expand
infrastructure and grow network cost effectively
Lessons Learned
23
Fixed Wireless • Line of sight / unlicensed spectrum interference • Technology / hardware life cycles • Complimentary to fiber infrastructure – Not end-all
solution Fiber • High demand • Symmetrical bandwidth • Reliability • Scalability
• GPON • High demand • Economic Development • Smart Grid • Quality of Life • Customer Service • Public Private Partnerships • Political challenges at local & state levels
2014 FTTX Initiative 2.0
24
The Network Operations Center On-going Support
Art Meierdirk Senior Director, Business Services
On-going Support – Service Level Agreement (SLA) Based
26
• Define the “product” • Facilities / power / environmental
• Transport / switching
• Connectivity • Set Service Level Requirements
• Metrics (alarm / call handling / accuracy)
• Service availability (uptime)
• Time to entry
• Restoration response time (time to action)
• Quality Assurance metrics & reports
Value Proposition Elements
Value Converted to SLAs
Meeting Business / Communication Expectations
27
• Build a solution for 24x7 support to meet SLA requirements: • Action on Monitored Infrastructure Alarms
• Answer Incoming Calls / Messages
• Notifications of Clients / Customers / Users
• Escalation and/or Dispatch for Restoration
• Time to Restoration
• Documentation of all Actions for Reporting SLA Performance
Tools Process People Integration
Solution Elements
28
• Tools – Monitoring, Workflow, Documentation & Reporting
• Process – Best-in-class (ITIL) / Focus on SLA
• People – Skill Set to Match Functional Requirements
• Professional (Consultation) Services – Integration of Tools and Process-based on SLA Requirements
”Structured NOC”
Tools
29
• Monitoring, Workflow Documentation & Reporting • Device / EMS / NMS / Dry Contacts – Alarm
Aggregation from all Infrastructure Systems
• Workflow Management / Ticketing System with Time-stamp as Work Progresses or Escalation of Work is Delayed
• Portal / Web Access for Customer Access to their Information
Comprehensive (Single) View
30
NOC Monitoring
System (Alarm Aggregation)
Secure Connection
Security Systems
HVAC Systems
EMS / NMS
Devices Ideal: Monitoring System to
Aggregate ALL Alarms into “Single Pane of Glass”
Process
31
• Process (ITIL) – Documented, Standards-based and Focus on SLA Fulfillment
• Process Flow Mapping • Work Instructions for Every Process • Expected Results • Escalation when Required • Closure and Documentation • Reporting • Real Need for Best-in-class Process
Development
People
32
Skill Set to Match Functional Requirements • Service Desk • Tier 1 Support • Advanced Support • Subject Matter Expertise
Quantity
Time
Staf
fing
Wor
k Loa
d
Staffing Limitations: • Budget Constraints • Staff Additions Often Lag Growth • Multiple Responsibilities - Staff for Highest Job
Requirements
Structured NOC
33
ACTIVITY TYPE BY TIER: Tier 1 Tier 2/3PERIODIC REVIEW 3% 10%DOCUMENTATION 2% 3%INCIDENT MANAGEMENT 7% 18%24x7 EVENT MONITORING 39%CALLS/E-MAILS 14% 4%
Tier 1 Activities65%
Tier 2/3 Activities35%
• Tiered Jobs • Resource Pool • Maintain Focus • Meet SLAs • Control Costs • Right Sized” Solution
Knowledge Base / Runbook
34
• Documentation for all Aspects of the Business
• Architecture • Organization • Process Flow • Work Instructions
• Contingencies
• Business Continuity Plan
Other Considerations
35
Service Turn-Up Team • Dedicated service turn-up team • Initial set-up and changes – process and
procedure, ticketing configuration, equipment monitoring, etc.
Systems & Development
• Continued development of monitoring tools • Integration with third-party/client systems • Custom monitoring, reporting
Project Management & NOC Run Book Development
• Development and documentation of operational support processes for clients
• Advanced Runbook including comprehensive client-specific process flow diagrams and detailed work instructions for Incident Management, Problem Management and Change Management
Customer Advocacy • Represents customer’s interests • Reports, QA, QC programs • Ongoing reviews with client team
Transforming Critical Infrastructure
Questions / Open Discussion
• Art Meierdirk, Senior Director, Business Services – INOC • Dan Belmont, Director, Energy & Utilities - West Monroe Partners • Gary Sanders, Telecom Manager – Sun Prairie Utilities
top related