the implication of faulty sensors to asset...
TRANSCRIPT
©2013 Doble Engineering Company. All Rights Reserved
The Implication of Faulty
Sensors to Asset Analytics
Oct. 21-22, 2013
Matt Garber
Doble Engineering Company
Don Angell
Doble Engineering Company
Lee Margaret Ayers
Doble Engineering Company
©2013 Doble Engineering Company. All Rights Reserved Asset Management Meeting, dobleARMS™ Consortium 2
• Implication of faulty
devices to the Analytic
Process
• The importance of data
validation within the
Analytic Process
• Where/how this is being
tested today
• Tangible examples
Key Takeaways
©2013 Doble Engineering Company. All Rights Reserved Asset Management Meeting, dobleARMS™ Consortium 3
• Analytic results give us insight into asset health.
• Writing an analytic is relatively easy. – E.g. calculating abnormal OLTC activity
– The result should also inform us about:
• Who, what, where, when, and how long do I have to respond?
• The challenge is communications and monitoring errors can result in false positives – Diverts $$$ from key areas that need
attention
• Analytic results – Too many alerts, people will stop listening
– Not enough alerts, people will not trust the system
– Storing a “bad” result muddies our understanding of asset history and analytics for determining asset life
Determining Asset Health?
• Woman: I'm not a witch! I'm not a witch!
• Valdimir: ehh... but you are dressed like one.
©2013 Doble Engineering Company. All Rights Reserved Asset Management Meeting, dobleARMS™ Consortium 4
• Supervisory Control and Data Acquisition (SCADA)
– Very reliable data source
– w/ 1-2 sec timestamps
– SCADA represents an amalgam of sources:
• RTU’s/IED’s/Relays/Transducers
• When linked with SCADA, data sources are typically dropped
– SCADA might see “DGA Alarm,” but not whether the alarm
is from a Delphi, a Severon, or a Hydran, etc. • 1:n vs. 1:1 relationship
• “n” represents a variety of reliable and non-reliable devices in the field.
What happens if the sensor behind a good data
source, is bad?
©2013 Doble Engineering Company. All Rights Reserved Asset Management Meeting, dobleARMS™ Consortium 5
• Those with tribal knowledge learn to ignore bad sources of
data
• Those with less knowledge, but with responsibility must
respond
• Can we program/teach an automated system to do this?
– And can the automated system be used to debug
questions humans cannot?
Humans Filter
©2013 Doble Engineering Company. All Rights Reserved Asset Management Meeting, dobleARMS™ Consortium 6
4-A—Data Pathways
6
Member Data
4-A NOC & Data Center
©2013 Doble Engineering Company. All Rights Reserved Asset Management Meeting, dobleARMS™ Consortium 7
Data Pathways and Relationships
DNP 3 4-A
1:1 relationship DGA Monitor Vendor A (via cellular/DNP) directly into AAA
DNP 3 RTU/IED/
Relay/ Gateway
SCADA 4-A Vendor OPC Historian OPC
1:N relationship DGA Monitor Vendor A to IED/RTU/Relay/Gateway to SCADA to AAA
RTU/IED/Relay/
Gateway SCADA DNP 3 4-A Vendor Vendor
Custom DB
Vendor
N:N relationship DGA Monitor Vendor A to ED/RTU/Relay/Gateway to SCADA to The PI System to AAA DGA Monitor Vendor B to ED/RTU/Relay/Gateway to SCADA to Custom DB to AAA
DGA A
DGA A
DGA A
DGA A
©2013 Doble Engineering Company. All Rights Reserved Asset Management Meeting, dobleARMS™ Consortium 8
Data Sources Currently Feeding Analytics
Conditional Alerts
Criticality & Impact
Risk
Error Trapping/ Validation
Analytics
Asset Registry/ CIM/ IEC 61850 Co
nd
itio
nal
Eve
nts
Co
nd
itio
nal
Eve
nts
M5300
Sweep Frequency
M4000
Insulation Analyzer
Partial Discharge
PDS100
2. Online Diagnostics
IDD
Moisture in Oil
Domino
Continuous Monitoring
PD-Guard
dobleDATABASE™
SCADA/ Relay/
Gateway
Oil Sample
**Inspection
1. Offline Diagnostics
3. Real-time
5. Maintenance
4. Lab
IEDs, RTUs, Transducers
1:1
1:N
Delphi
Bushing DGA
**Manual Errors
6. Apparatus test DATABASE
©2013 Doble Engineering Company. All Rights Reserved Asset Management Meeting, dobleARMS™ Consortium 9
• Complex, multilayered process – Math, Science & Art
• Establish behavior
• Error trapping, Points of failure – From Source to Target
– Device/System, Hardware/Software, Communications/Network
• Validation
• Analytics-Basic & Advanced
• Complex Analytics – Tap changes SCADA sees versus
TAP changes reported during physical inspection.
• Rules for how we handle bad data
The Analytics Before the Analytics
• Health & Risks • Rolling up results • Relevance • Confidence • Complex Algorithms • Rules • Math/Algorithms • Data Validation • Error Trapping • Behavior • Data sources
Math?
©2013 Doble Engineering Company. All Rights Reserved Asset Management Meeting, dobleARMS™ Consortium 10
Tuning the Data: Establishing Expected Ranges & Behavior
• Timestamps (**need common)
• Update Rates (.5 sec. to 5 yrs.)
• Expected Range (0-span)
• Exception (when data changes)
• Compression (when stored)
• Engineering units (W or MW)
• Scaling (x10)
• Type-digital, integer, float
3500 points
72 points
No sense going any further if this
is not well understood!
©2013 Doble Engineering Company. All Rights Reserved Asset Management Meeting, dobleARMS™ Consortium 11
Deviations were tied to filtering on relay
• Station to Station with PT in between
• Same filter on back-end, different filters on the front-end
• We can resolve during the tuning phase
©2013 Doble Engineering Company. All Rights Reserved Asset Management Meeting, dobleARMS™ Consortium 12
Front-End: Points of Failure or Delay
•Source, intermediary, target (& then source, intermediary & target)
•Hardware shut down, or hardware/ interface not rebooted
•Memory, CPU, Speed?
•Comms loss or network delay
•Ability to generate alerts—hardware, comms, & network
•Watchdog tags = 0 or 1
©2013 Doble Engineering Company. All Rights Reserved Asset Management Meeting, dobleARMS™ Consortium 13
• Devices: Transducers/Meters/RTU’s/IED’s can be faulty from the start
– Missing data
– Spikes (is the new meter under warranty?)
– Data not changing or are out of range
– Data are changing, within range, but abnormal read
– Improper reconfiguration of relay
• Known to degrade over time
– Unreliable with intermittent good and bad values
• Are the data good or bad?
– And what should we write to database if bad—policies?
– If data source is available, but a point is not = Not a Number (NaN)
• Analytic has to watch for initiation
Once We Establish Consistent Source, Validate the Data
©2013 Doble Engineering Company. All Rights Reserved Asset Management Meeting, dobleARMS™ Consortium 14
Temperature Scenario
• Three temperature monitors (bad (blue), went bad (yellow)& good (red)
• Identification of faulty sensor (blue & yellow trend) is straightforward
• Business must decide how we handle the alert- built into analytic rules
©2013 Doble Engineering Company. All Rights Reserved Asset Management Meeting, dobleARMS™ Consortium 15
• Data-source dependent (uses behavior constraints)
Validation Rules
©2013 Doble Engineering Company. All Rights Reserved Asset Management Meeting, dobleARMS™ Consortium 16
Summer Peak
• A=data.rt.scada.3Phs.mva.meas
• B=data.rt.SummerPeak
• [if badval(B/A) then NaN else B/A]
– NaN= Not a Number
Front-end Validation Routines
• A “zero” value would be confusing
• Not an error. Data source is available, but not yet!
• Analytic must initiate when data is available
• Must consider viewer in the whole data validation process
If data.rt.scada.3Phs.mva.meas.val = good, then calculate summer
Peak….
©2013 Doble Engineering Company. All Rights Reserved Asset Management Meeting, dobleARMS™ Consortium 17
Advanced Analytic: Overly active OLTC:
SCADA Analytic Results 1.
A look at the actual data
©2013 Doble Engineering Company. All Rights Reserved Asset Management Meeting, dobleARMS™ Consortium 18
Correlation Using Maintenance Records
• Can we correlate with maintenance period?
• Compare number of operations
• Or maybe compare Average daily operations
2. Compare with
maintenance
records
start_date end_date type counter_ltc ltc_position_max
7/23/2012 9:07 7/23/2012 9:43 Patrol 50105 6R
6/6/2012 8:46 6/6/2012 9:14 Patrol 49284 4R
4/23/2012 11:39 4/23/2012 12:07 Patrol 48547 7R
2/29/2012 8:22 2/29/2012 8:47 Patrol 47761 4R
1/11/2012 8:34 1/11/2012 9:07 Patrol 47219 3R
11/4/2011 9:09 11/4/2011 9:57 Patrol 46522 4R
08/03 07/23 08/09 06/06
©2013 Doble Engineering Company. All Rights Reserved Asset Management Meeting, dobleARMS™ Consortium 19
• Front-end error-trapping and validation processes are essential to a producing valuable analytic – Over time we will have a history of characteristics for data sources to add to
the historical database (80 years, 30M records)
– Additional Tuning—need analytics for expected variations & pattern recognition
– **Some sources can have a “Quality bit”
• Codes are custom and range from 1 to 200
• Correlation Analytics will improve Confidence – Users will likely tune their Alerts to see higher Confidence.
• Bad data = Alerts – Provides a good test that analytics are working
– Start-up is typically about correcting configuration, comms issues or data errors
– Asset analytics infrastructure offers the same opportunity to prioritize, sort and address data errors as it does asset health issues.
Summary
©2013 Doble Engineering Company. All Rights Reserved Asset Management Meeting, dobleARMS™ Consortium 20
• Company preferences and procedures need to become
part of analytic rules when alerting on data errors
– Do we send an alert to repair, ignore or flag issue?
• Ad hoc tools to query & perform analysis of like devices
– Only the trained few should only see comms alerts
– People tend to trust data, so users should see: “Watchdog reports
an error on this page”
• Asset Management offers a real business case for
standards • When a data source normalizes the origin of data (turns 1:n into 1:1
relationships), standards (CIM/IEC 61850) give us a path forward
• Re-infuse the Asset Registry with intelligence during integration
Summary
©2013 Doble Engineering Company. All Rights Reserved Asset Management Meeting, dobleARMS™ Consortium 21
• Today data origins have to be manually re-infused in the asset model (if it is an option).
• Example – Before: data.rt.scada.3Phs.mva.meas
– After: data.rt.pi.scada.SEL287E.3Phs.mva.meas • Allows us to troubleshoot comms issues and points of failure
• PI-to-PI between ARMS has inherent comms troubleshooting
• Can correlate other data issues related to the relay
– Before: data.rt.scada.DGAAlarm.status • Is it a high alarm or low alarm?
• Does the alarm refer to a specific gas?
• Analytics can only be broad.
– After: data.rt.pi.scada.IDD.Delphi.DGAAlarm.status • Asset Analytics can specific
Proposal—Extending Data Path Names to the
Asset Risk Management System
©2013 Doble Engineering Company. All Rights Reserved Asset Management Meeting, dobleARMS™ Consortium 22
Matt Garber
Lee Margaret Ayers
+1-808-333-9845
Don Angell
+1-508-887-1945
Contacts
22