ip/mpls performance and path analytics for … · ip/mpls offers a variety of protection...

TECH BRIEF

IP/MPLS PERFORMANCE AND PATH ANALYTICS FOR

TELEPROTECTION IN POWER UTILITIES

Copyright © 2018, Packet Design, LLC of 14

Table of Contents

Introduction 3

Adapting legacy technologies and devices to an IP/MPLS network 4

IP/MPLS networks introduce new challenges 4

IP/MPLS path and performance reporting with Packet Design 6

A utility use case 9

Time-stamped alerts for tunnels 11

Evolving the network 13

IP/MPLS Performance and Path Analytics for Teleprotection in Power Utilities


Introduction

Power utilities face major cost and technology challenges when building out and modernizing their communication networks. New IP applications in the Smart Grid need to be rolled out and the existing power grid that uses old TDM technologies must be integrated with modern technologies, such as IP/MPLS networks. Utilities can’t afford to operate parallel networks – such as one SONET/SDH network and another IP/MPLS network – to carry both legacy and new applications.

Maintaining reliability of supply and human competence are other key factors that drive the migration to IP/MPLS-based transport infrastructures. At the same time, utilities must continue to comply with strict specifications for TDM-based equipment, such as E1/C37.94, to maintain electric power delivery and control of grid equipment under all circumstances.

This paper explains how to monitor and simulate the behavior of highly flexible IP/MPLS networks that carry legacy E1/C37.94 information and Smart Grid applications from a performance and routing redundancy perspective. It is essential to monitor performance metrics, such as IP/MPLS latency, jitter and packet drops, in Inline Differential Protection power systems that use E1/C37.94 equipment. It is also critical to monitor redundancy in various MPLS Traffic Engineering protection schemas, such as Fast Reroute (FRR), Secondary Tunnels and Secondary Paths, to ensure the reliable transport of E1/C37.94 time-sensitive information from power grid protective relays.



Adapting legacy technologies and devices to an IP/MPLS network

Traditionally, utility companies have relied on TDM technologies, such as SONET/SDH. These technologies deliver carrier-class performance and support the deterministic traffic critical for grid operations. However, they no longer adequately support the long-term needs of utilities. TDM infrastructures are often built and operated for specific applications, thereby creating siloed solutions.

Utilities are introducing new IT systems and Smart Grid applications. These new systems cause traffic patterns in the network to change rapidly, requiring packet-switched networks that provide much more flexible network topologies based on multipoint connections (Figure 1).

IP/MPLS networks introduce new challenges

Packet-based networks offer more and better functions and higher levels of service for the new IP applications while continuing to deliver reliability and deterministic (real-time) traffic support for old TDM applications and devices. However, packet-based networks use a statistical multiplexing technology (store-and-forward) that may introduce delay and delay-variations ( jitter) that can severely affect legacy TDM applications. These new IP/MPLS networks need to be closely monitored and properly designed to continue to allow for proper operation of the integrated TDM equipment.

One example of a critical TDM application with very stringent requirements of the IP/MPLS network is “Inline Differential Protection.” This technology uses relays that connect E1/T1 (electrical) or C37.94 (Optical TDM) interfaces and measure and compare the AC current of the two remote ends of a power line (Figure 2). When comparing the two ends, it is critical that they are synchronized in timing. If not, this could lead to currents from different time instances being compared, producing a phase angle shift that ultimately leads to degraded protection sensitivity and incorrect tripping of the power line.

Figure 1. A new network for the Smart Grid



Correct tripping of protected lines is a good thing since it isolates failures, preventing them from “spreading” across the power grid.

If the IP/MPLS network introduces too much latency variation ( jitter), it reduces protection sensitivity. So, it is critical to measure the jitter caused by the IP/MPLS network. Delay that is constant and symmetrical usually does not introduce any functional issues, but it does add to the time it takes for the relays to convey a failure and trip the line. Therefore, constant delay is an important metric to measure. The relay specifications for latency tolerance and jitter are vendor specific, but some typical values are:

– 50ms maximum delay tolerance; 4ms latency asymmetry; 0.5 percent error rate. – 3-4ms jitter

To handle latency asymmetry, many implementations now clock the relays locally using the Precision Time Protocol (PTP) derived from the IEEE 1588-2008 specification. Still, maximum delay and jitter need to be handled properly over the IP/MPLS communication channel.

Another complexity introduced by the IP/MPLS network is the transport and protection of the TDM traffic. In SONET/SDH technologies, there is a built-in, carrier-class protection using a 1+1 protection scheme. IP/MPLS offers a variety of protection mechanisms, like FRR protection of nodes and links, secondary pre-signaled tunnels, secondary path-options and/or fallback to native IP routing, to mention some of the more commonly used techniques. Once these TE-tunnels and protections are introduced, there is a critical need to monitor the behavior of the primary and protected tunnels. In a large network, there may be several thousand tunnels, creating complexity and making troubleshooting very difficult. This is where route analytics technology can be very helpful.

Figure 2. Inline Differential Protection of Powerlines



IP/MPLS path and performance reporting with Packet Design

To monitor TE-Tunnels in IP/MPLS networks, Packet Design’s Route Explorer™ physical or virtual machine appliance delivers real-time, historical and simulated insights into the network control plane. By connecting to OSPF/IS-IS using a passive neighbor relationship, all routing messages for both the plain IPv4 and the Traffic Engineering LSA/LSP types are captured and stored in the Route Explorer database. This database is then used to create reports in real time and for any point in time.

Reports start at an overview level (Figure 3), summarizing the network topology and its primary and protected tunnels, and enable drill-ins to very detailed views about TE attributes, such as ERO objects, Affinity Bits and Reserved Bandwidth, to mention a few (Figure 4). Comparison reports on FRR and Secondary Tunnels and Paths are easily available and graphed on the HTML5-based Web interface (Figure 5). Dashboards showing TE tunnel information can be easily customized on an individual user basis.

Figure 3. TE Tunnel Overview



Figure 4. TE Tunnel Details



Reporting on paths, their symmetry and protection is important to operations and engineering teams within utilities. In addition, performance metrics, such as latency, jitter and errors, overlaid on the IP/MPLS information, are also invaluable to network engineers. The Packet Design Performance Explorer collects SNMP latency metrics using Cisco IP SLA tests and the equivalent for other major router vendors. This information is then projected on top of the routing and path topology, producing a coherent picture of the most important network performance attributes for a utility company (Figure 6).

Figure 5. Comparing an active Primary Tunnel (blue) with a stand-by Secondary (yellow), here also showing dynamic delay information for each link on both paths.



A utility use case

A regional utility operator in the Nordics uses Route Explorer to monitor its entire L3 routing infrastructure, including OSPF and RSVP-TE tunnels. The operator encountered stability issues when integrating the TDM systems into the new IP/MPLS network – the line protection application (the relays monitoring and comparing AC current at substations) proved to be very sensitive to “partial” or bad connectivity. The relays can handle “no connectivity” or “full connectivity” but not asymmetries.

After conducting thorough testing using the various RSVP-TE options, the engineering team configured all tunnels in the network to use a multiple path-option statement to achieve redundancy, instead of more traditional FRR techniques. This proved to mimic the legacy SDH network better than other FRR configurations. With “path-option,” the protection tunnel (or more precisely, the higher path-option tunnel) is not pre-signaled, but rather it is set up on a new explicit path, once the primary path has failed.

Figure 6. A Primary TE-Tunnel crossing a high-delay link



An example configuration using the path-option statement is shown here:

interface Tunnel1234 ip unnumbered Loopback0 mpls ip tunnel source Loopback0 tunnel mode mpls traffic-eng tunnel destination 1.2.3.4 tunnel key 1234 tunnel mpls traffic-eng path-option 1 explicit name PATH1 tunnel mpls traffic-eng path-option 2 explicit name PATH2 ! ip explicit-path name PATH1 enable <------Primary Path-Option index 1 next-address 1.2.3.4 ! ip explicit-path name PATH2 enable <------Redundant re-signaled Path-Option index 1 next-address 2.3.4.1 index 2 next-address 2.3.4.2 index 3 next-address 2.3.4.3 index 4 next-address 2.3.4.4 index 5 next-address 2.3.4.5 index 6 next-address 2.3.4.6 ! pseudowire-class LDS-1234_PW encapsulation mpls preferred-path interface Tunnel1234 disable-fallback <-------- don’t route using vanilla IP if tunnel fails



Time-stamped alerts for tunnels

Route Explorer can alert whenever a tunnel goes down and is re-signaled on the second path-option, giving the utility operator exact time-stamped information on tunnel down events (Figure 7). This information is logged locally in the Route Explorer database, displayed on a dashboard and sent via SNMP/Syslog/email to external systems. NOC staff and engineers can focus on the time of the event simply by clicking the alert message (Figure 8) to easily and quickly troubleshoot and monitor for symmetry. Before route analytics were implemented, engineers had to sift through thousands of lines of router logs to try to find relevant events.

Figure 7. Time-stamped alert with URL link in message for drill-down into failed tunnel

Figure 8. Drill-in to alert URL to see details of tunnel and compare to IGP if need be



Figure 9. View of critical jitter measurements over 48 hours



Evolving the network

With rapidly changing requirements on the network to enable more functions in the Smart Grid, the regional utility operator is looking at evolving network functionalities, including the ability to monitor and analyze new technologies using Packet Design’s Explorer products. Among the planned changes are:

– Performance measurements - latency, jitter (Figure 9) and packet drops, projecting IP SLA information on paths and routes, to assure the inline differential protection mechanisms for substations.

– L2 pseudo wires, L3 VPNs (using MP-BGP) in combination with labeled BGP, creating a seamless MPLS transport network.

– Segment Routing.

Packet Design’s Explorer Suite already supports these technologies, greatly easing design and operations in the ever-evolving network.



To learn more about Packet Design and the Explorer Suite, please visit www.packetdesign.com

ip/mpls performance and path analytics for … · ip/mpls offers a variety of protection...

Documents