©2006 pjm 1 kevin j. komara p.e. project manager pjm interconnection ems users group conference...

©2006 PJMwww.pjm.com 1

Kevin J. Komara P.E.Project ManagerPJM InterconnectionEMS Users Group ConferenceSeptember 24, 2007

A Discussion of Secure Field Device Data over the Internet

in Today's Environment


Secure Internet Communications

Why Use The Internet ?• Lower Communications Costs to Directly Connected Sites

– Only option used to be Frame Relay (PJMNet)• All PJM RTU Communications is DNP3 over TCP/IP over Frame Relay• 5 Years of experience of DNP over TCP/IP

– PJMNet (Per Installation)• $1500-$3000 Per Month Recurring Communications Costs• $7500-$15000 in Capital Costs

– Internet• $0 Per Month Recurring Communications Costs

– Uses Customers Internet Connection• $1500 in Capital Costs

• Reduce Communications Setup Time– PJMNet

• Approximately 90 Days to Install and Configure– Internet

• Rapid Deployment• Instant ON

• Pervasive– It’s EVERYWHERE !



What are the Critical System Objectives ?• Support standard DNP over TCP/IP Communications

– No modifications to PJM CFE– Customer uses any TCP/IP or Serial DNP Meter/RTU

• PJM does not own the end equipment

• Easy Installation– Modular systems approach

• Bi-Directional Real-Time– 10 Second Scans– Allows communications to and from customer.

• Support AGC Data Objects• Support Revenue Data Objects

– Real-Time over narrow bandwidth

• Secure– Triple Des Encrypted Messaging– Authenticated Message Sequencing



System Solution• Formed Joint Project with Comverge Technology and

Arcom Controls– Existing PJM vendors

• Comverge supplied a Gateway Device (DCMS Router) installed at PJM to Authenticate/Encrypt Standard DNP over TCP/IP Traffic to/from CFE– CFE at PJM not modified

• Arcom supplied a Gateway Device (Director) installed at Customer site to Authenticate/Encrypt Standard DNP over TCP/IP Traffic to/from Customer Equipment (RTU/Meter/etc.)– Customer Equipment not modified



Hurdles to Deployment – Now the FUN starts !• Security, Security, Security…Did I mention

Security ?• PJM Security very nervous about Internet data into EMS

network• System needed to meet PJM’s SAS70 level 4 Audit

requirements• Separate DEV/Test/Stage/Prod East/Prod West/BUCC

Systems and Networks• Project requirements changed as project evolved

– Hurdle after Hurdle…(2+Years)…after Hurdle after Hurdle

• Security and Operations required project to be implemented in phases



Hurdles to DeploymentImplementation Phase I• Limited to max of 20 pilot installations• Limit of 10 MW • Originally split DCMS into 2 components

– WEB services on server in WEB DMZ– Application services on server in APP DMZ– Firewall between 2 DCMS components and on front and back of

both– Allowed much tighter management of critical data paths and

connections• Redundancy was not included• System required a separate internal IP address for each

Internet RTU connection– PJM Network people NOT happy !



Hurdles to DeploymentImplementation Phase II• Still limited to max of 20 pilot installations• Still limited to 10 MW • All DCMS functionality on single server

– WEB s and Applications services on same server in WEB DMZ– Firewall on front and back

• Added Redundancy– DCMS stateless– Used existing Load Balancers

• Modified system to use only 1 Internal IP Address/Port– DCMS now DNP aware– Supports up to 65534 Unique DNP RTU’s on single IP/Port



Production• Removed limit of 20 Internet RTUs• Implemented Backup Control Center Instance

– Non redundant

• Implemented completed Test system• Supports Internet connections from PJM East

and PJM West RTUs• Raised MW limit to 50 MW – may raise limit

even higher in the future


Wired Internet Communications Overview



Device Configuration Method • Prior to Remote Device Installation

– Generate unique Triple-DES Master Key/Secret Key Combination (at PJM)• Use Comverge Key Generation Tool• Communicate Remote Device Serial Number to Arcom Controls• Communicate Remote Device Master Key to Arcom Controls

• Define new customer in DCMS Router Database (at PJM)– Remote Device Serial Number– Remote Device Secret Key– Remote Device DNP Address (Unique)– Remote Device Internal IP address (Common)– Remote Device Internal TCP Port (Common)

• Define new customer in PowerCC CFE Database (at PJM)– Define Customer RTU

• Remote Device DNP Address (Unique)• Remote Device IP Address (Common)• Remote Device TCP Port (Common)

• Configure Arcom Director Encryption Information (at Arcom Controls)– Remote Device Serial Number (Provided by PJM)– Remote Device Master Key (Provided by PJM)– Ship Director to Customer



Device Authentication Sequence• Enable Communications to Field Device in CFE (Remote Device does not need to be Connected)

– CFE attempts to communicate to Remote Device every 60 seconds.– Communications from CFE to Remote Device is through DCMS Router as Gateway to all Internet Remote

Devices.– DCMS inspects TCP Packet Payload and identifies DNP address for Remote Device– If Remote Device has not authenticated to DCMS

• TCP Socket is closed to the CFE by DCMS.– CFE Continues Cycle

• Power Up Remote Device (Arcom Director)– Director initiates TCP/IP connection to DCMS Router on Power Up.– Director identifies itself to the DCMS Router (Using Remote Device Serial Number)– DCMS verifies that Remote Device Serial Number is valid.

• If invalid DCMS closes TCP Socket.– If VALID

• DCMS stores external IP address of Remote Device.• DCMS Responds with encrypted message generated using internal unique Secret Key for Remote Device.

– Remote Device Receives Encrypted Message from DCMS.– Remote Device Decrypts message from DCMS using its unique Master Key for Remote Device.

• If Invalid Remote Device closes TCP Socket.– If VALID

• Remote Device Encrypts Session Key Request Message using Master Key and sends to DCMS.– DCMS Receives Encrypted Message from Remote Device and Decrypts using Secret Key

• DCMS sends New Session Key embedded in message Encrypted with Secret Key.– Remote Device Receives Encrypted Message and Decrypts using Master Key– Bi-Directional Encrypted/Authenticated TCP/IP Communications Now Established using

Unique Session Key



Device Authentication Sequence (Continued)• Communications to Field Device in CFE (Remote Device

Authenticated)– CFE attempts to communicate to Remote Device every 60

seconds.– DCMS inspects TCP Packet Payload and identifies DNP

address for Remote Device– Remote Device currently authenticated to DCMS

• TCP Socket is left open.

• DCMS NATS TCP Socket from CFE with TCP Socket from Remote Device

– Bi-Directional DNP Communications Now Established through Encrypted TCP/IP Tunnel



Key System Features• CFE and Customer equipment not affected by Internet Communications

– Authentication/Encryption Transparent to existing equipment– Retained all current capabilities in CFE

• Works with any kind of Internet Transport– Dial-Up/Cable Modem/DSL/Cellular/Satellite/Smoke Signals

• Supports fixed and non-fixed IP addresses from Remote Devices– Greater number is ISPs supported

• Allows pre-definition of IP address and port in CFE.– Asynchronous CFE Database/DCMS Database/Remote Device configuration

• Supports up to 65534 unique DNP RTU’s on a Single internal PJM IP address and Port combination

– Made my IT people VERY happy !• DCMS High availability uses standard Load Balancer Techniques

– DCMS is Stateless– Allows for n-number of redundant DCMS routers

• Director re-authenticates automatically



Key System Features• PJM Manages Keys• Efficient Session encryption allows communications over extremely

narrow bandwidth.– Dial-up internet speeds of 9600 Baud Supported– Random In-Band Session Key exchange ensures strong encryption.

• Rapid Deployment– Typical 90 Days for PJMNet– Instant ON with Internet !

• Extremely easy to use and maintain.• Extreme Cost Savings

– 31 Production Sites – increasing at about 2-4 site/month– Approx. $47K/Month or $560K/Year recurring communications costs– Approx. $248K in Capital Costs– Project has saved over $750K in 3+ years of production


Applications

• Applications– Small Distributed Generation– Vehicle to Grid– Battery to Grid– Fly Wheel to Grid


Internet SCADA Using Cellular Wireless

Technology


Wireless Internet SCADA

• Has All the Features Of Wired Internet SCADA• Extends Current Internet SCADA

Communications With Cellular Leg.• Lowers Communications Costs Even Farther

– Nextel - $13/Month for Real-Time Data (Un Limited)– Verizon High Speed – Approx $50/Month (Limited)

• Easy Installation– Modular systems approach.


Wireless Internet Communications Overview


Demonstration

• Live Demonstration of Bi Directional Encrypted DNP Communications over the Internet Using Verizon High Speed Wireless Router– PJM CFE already configured to communicate to Internet RTU– 2 separate CFE systems will communicate simultaneously (PJM

TEST CFE and PJM BUCC CFE) – Director authenticates automatically on Power Up

• Authenticates to 2 separate Comverge DCMS Gateways

– Real-time DNP communications established with 2 systems without human intervention

– Supports Simultaneous Analog Output from both CFE Systems to single RTU


PJM Small Generation Interconnection Working

Group Application of Internet SCADA Communications for Generators of 20 MW or Less


Current Data Transfer Method

Problem:• Utility Requirements Not standardized across PJM.• Transmission company requires Generator owner to

purchase and install proprietary Transmission company RTU.

• Requires Generator owner to purchase expensive 4 wire leased Telco Circuit.

• Automatic Generation Control Signals from PJM not usually supported.

• Generator owner my have to purchase second RTU to support AGC from PJM.


Turning the Tide

Time to Do Something Different !• Proposed Eliminating Utility RTU for all PJM Generators

of 20 MW or less.• Customer would install PJM Internet Solution• Use Existing Customer Internet Transport• PJM would send required telemetry to Utility using

existing ICCP connection.


SGIWG Internet System Requirements

Solution Requirements:• PJM collects required Real-Time Data from Generator

Site using encrypted Internet communications and makes data objects available to Utility over existing ICCP Link.

• PJM collects required Revenue Data from Generator Site using encrypted Internet communications and makes data objects available to Utility over existing ICCP Link.

• The utility must be able to TRIP the Generator offline by directly controlling the Generator Circuit Breaker using an encrypted Internet method.

• The availability of the equipment must have an availability rate of 99.8% or less than 16 hours per year outage.


PJM EM S/SC AD A

EM S

PRI NT

HELP

ALPH A

SH IFT

ENTERRUN

DG ER FI

AJ BK CL

7M 8N 9O

DG DG DG

DG T 3U

0V .WX Y Z

TAB

% U TI LIZAT IO N

HUB/ MAU NIC

2BNC4 M b/s

PPL M eterSiem ens 2510

Generator BreakerSuperv isory C ontrolover T ransm iss ionOw ner Serial R T U

Protocol

Pro to type SG IW G S tandard Da ta T rans fe rM ethod W ith G enera to r C ircu it B reake r C on tro l

a t F rey F a rm

Generator D ata (M W /KV/C B/M W H /etc . )

and Autom aticGenerat ion C ontrol

PPL Elec tricU tilit ies

PJM

Frey FarmGenerator Site

Generator D ata (M W /KV/C B/M W H /etc .)

over ex is t ing IC C P/PJM N et D atalink

Ex is t ing PPLEULocal R TU

Ex is t ing internal PPLU Einterface to Local R T U

D ataProbe C ontac tover EthernetOutput D ev ice

D ataProbe C ontac tover EthernetInput D ev ice

Generator Breaker

Superv isoryC ontrol

(T rip Only )

Superv isoryC ont rol

AESEncrypted

T C P/IPover

Internet

N ex tel W irelessN etw ork

Arcom W irelessEdge R outer

Encrypted D N P3 TC P/ IPover Internet

H IT AC H I

Arcom D irec torSeries I I I

Data Flow Diagram


SGIWG Prototype RTU Installed at Frey Farm

DataProbeCOE-8O

24VDC to 5VDC Converter

18”x18”x6” Enclosure

Arcom Controls

WER1500

CB Simulator Relay

Arcom Controls

Director Series III

Power Switch

DIN Rail IO Connector

RS-485 Connector


DataProbe COE-8I Installed in Local RTU and PPLEU


PJM Frey Farms SCADA Display (Production with MWH Delta)


Results

• Real-Time Data objects required by utility available from PJM over ICCP (MW/MVar/KV/Amp/etc.) - Completed

• Revenue Data objects required by utility available from PJM over ICCP (MWHRec/MWHDel) - Completed

• Utility has direct capability to Trip Generator Breaker - Completed

• Reliable data transfer method – Achieved 100% availability from PJM Production on April 7 2006 to current.


Conclusion

• New PJM Small Generation Interconnection Working Group data transfer method met ALL requirements of the acceptance criteria.

• Utility RTU no longer needed at PJM Generation Sites of 20MW or less.

• Unanimously Accepted by PJM Stakeholders on September 29, 2006.


Applications

• Applications– Small Distributed Generation– Vehicle to Grid– Battery to Grid– Fly Wheel to Grid


If You Have Questions

Contact Kevin J. Komara P.E.Senior Engineer

Operations Development DepartmentPJM

• (610) 666-4751 Phone• (610) 666-4282 FAX• [email protected] Email

©2006 pjm 1 kevin j. komara p.e. project manager pjm interconnection ems users group conference...

Documents