Session 403

Supercharging Smart Meter Big Data Analytics with Microsoft AzureJason Wilhite, Manager, Enterprise BI Services, SRP

Kirk Nason, Director, BI Solutions, Neudesic

Salt River Project (SRP)

SRP is one of the nation's largest public power utilities. Providing electricity to more than 984,000 retail customers across three Arizona counties

SRP is an integrated utility, providing generation, transmission and distribution services, as well as metering and billing services.

SRP's water business is one of the largest raw-water suppliers in Arizona. We deliver about 800,000 acre-feet of water annually to a 375-square-mile service area and manage a 13,000-square-mile watershed

Neudesic in the Utility Market

History & Experience• Half a decade in the domain space

• 10+ dedicated technologists developing solution & IP

Solutions & Intellectual Property• Smart Meter Analytics Solution:

Big Data Hybrid Analytics Solution for the Utility Industry

• Neudesic BI for SAP: Integrates SAP data into the Microsoft Data Platform

• Neudesic BI BIG Data Framework

SRP’s Goal & Business Challenges

• Massive amounts of data

• Disparate data sources

• Market alternatives are cost prohibitive

Maximize grid reliability, power delivery & service through Big Data analytics

Rising Infrastructure Challenges

• Must solve the infrastructure challenge to attack the business problems

Smart Meter Analytics Solution

• Big Data Business Intelligence solution

• Processes data collected from Smart Meters

• Built on the Microsoft data platform

• Built using Neudesic BI framework

• Augmented by Neudesic BI for SAP

Visualization

Domain Model

ETL

Connectivity

Meter Data and other Line of Business Data Sources

V1 Smart Meter Arch at SRP

850,000+ Residential Meters• 15-minute interval reads• 96 reads per customer per day• 81.6M meter records added per day• 2.6B rows a month = 160GB data• Reprocess 30 day rolling window

nightly

New IT challenge: ETL processing Service Level Agreement (SLA)began to slip

Smart Meters Meter Data Management System

ETL integrationBI Reporting & Analytics

Customer data (CRM)MDMs Data

Store

BI Staging

Reporting and AnalyticsSQL Server BI

Input files

V1 On Premises Data Prep SLA

6:00 PM 8:45 AM7:00:00 PM 8:00:00 PM 9:00:00 PM 10:00:00 PM 11:00:00 PM 12:00:00 AM 1:00:00 AM 2:00:00 AM 3:00:00 AM 4:00:00 AM 5:00:00 AM 6:00:00 AM 7:00:00 AM 8:00:00 AM

6:00 PMMDMS Load Start

1:59 AM - 3:47 AMValidation (1:48)

9:33 PM - 1:43 AMDerived Facts/file extract (4:10)

1:43 AM - 1:59 AMCube Processing (00:16)

6:00 PM - 9:33 PMStaging and Base Facts (3:33)

8:00 AMSLA Availability

SLA Window is 6pm to 8am = 14 hoursSmart Meter Fact Processing • Staging & prep base facts

• 81.6M meter records added daily• Reprocess 30 day rolling window nightly

• 2.6B rows a month = 160GB data• Create derived gap facts

• 188K rows daily = 20MB• Load new facts into Analysis Services

Derived facts are now taking 6 to 12 hours

Requirements to address the SLA Challenge

Maximize use of existing architecture

Minimize new CapEx expenditures

Enable Scale on Demand

Lowering overall storage costs

Protect Personally Identifiable Information

V2 Addressing the SLA Challenge

Azure Hybrid Smart Meter Solution

Smart Meters Meter Data Management System

Base Facts and Confidential data BI Reporting & Analytics

Reporting and AnalyticsSQL Server BI

Customer data (CRM)MDMs Data

Store

BI StagingBlob Storage

HDInsightInput: Base Facts

Extract meter reads from data mart and upload to blob

storage

Download derived facts

Output: Derived Facts

Input files

2.6B Rows160GB

188K Rows20MB

81.6M Rows3.5GB

Azure HDInsight Catalyst for Success

Hyper Scale on demand

Example: • 32-Node cluster running 1hr/day • 15TB Storage & Data movement• ($357.12+$354.41)=$700/month

($8,400/year)

Lowest Cost of Storage

Smart Meter Analytic Solution Content

Solution Contents:• SQL Server EDW Utility Schema

• SSAS OLAP Cube Utility Schema

• SSIS ETL Packages

• Azure HDInsight: Hive Queries & Scripts

• Neudesic BI Framework

• SharePoint Server BI Portal

• Excel Reports

• Implementation Services

Q&A

Contact Info:Kirk NasonDirector, BI SolutionsKirk.Nason@Neudesic.com(949)789-2683

Jason WilhiteManager, Enterprise BI ServicesJason.Wilhite@srpnet.com(602) 236-5528

Thank You