big data meets renewable energy: building a real-time

Big data meets renewable energy: Building a real-time asset management platform

Stamatis Stefanakos, D ONE

About

▪ D ONE: Based in Zurich, established in 2005

▪ Data - Data - Data

▪ International & Swiss clients

▪ Dogfooding: Investor for data-driven startups

Power generation mix (EU; 2017)

Data source: Agora Energiewende and Sandbag (2018): The European Power Sector in 2017. State of Affairs and Review of Current Developments.

Renewables are bigger than you thought?

Wind

Solar

Biomass

Hydro

11%

6%

4%

9%

Net electricity installations (in GW; EU; 2000—2017)

Data source: Wind in power 2017, WindEurope, windeurope.org

Wind and solar are leading in new investments

So, all is great! What is the need?

▪ Unreliable performance → renewables have no switch!

▪ Deregulation → more risks, less return

▪ Relative secrecy → hard access to data

Actually: Investors in renewables are having a hard time

Making renewable investment more attractive through the power of data analytics

IMPROVE PERFORMANCE&

MAXIMIZE ROI&

FIT FOR MARKET

DATA POWER&

BRAIN POWER

WIND&

PV&

HYDRO&

BIOMASS

The “True Power Platform”Integrated data platform to make renewables predictable

Performance analysis

Performance benchmark

Portfolio aggregation

Production forecast

▪ Expected power vs as is

▪ As a function of weather and technology

▪ Availability & loss analysis

▪ Per turbine, park, portfolio

▪ Weather adjusted & normalised

▪ Trends over time

▪ Support strategic decisions (invest / divest)

▪ Bottom-up▪ Dispatching based on

plant condition

How a wind turbine generates electricity

GearGenerator

Tower

Blade

Rotor and pitch system

Nacelle

Yaw drive

Controller

Nacelle: Sits on top of the tower and contains all components.

Pitch system: Rotates the blades and can take them out of the wind.

Yaw drive: Orients the whole structure to face the wind depending on wind direction.

Gearbox: increases rotational speed from the low-speed rotor to the higher speed generator

Generator: Transforms mechanical energy into AC electricity.

Controller: Starts and stops the turbine from working.

The physics behind the data: Theory vs realityWhy does production deviate from the power curve?

Power*

Wind speed

Cut-in speed

Rated speed

Cut-out speed

Rated power

* Power as a function of wind-speed for a given air-density. Typically given at sea level for a fixed temperature.

The physics behind the data: Wind

Power available: Pin = 1/2 A 3

= wind speed [m/s] = air density [kg/m3]

A = 2 = radius of windmill wings [m]

Produced power: Pout = p Pin [W = 1J /sec]

p = efficiency constant provided by the manufacturer.

Betz’s law : p < 16/27 (~60%)

Typical wind turbines achieve at peak 75% to 80% of the Betz limit.

How much power can we achieve?

Pin Pout

A

How to estimate power expected?

▪ Estimate the meteorological parameters - Correct the wind speed (typically measured behind the blades). - Estimate air pressure (which depends on altitude but also on temperature). - Still relies on having an accurate efficiency constant.

▪ Learn the power curve from the data - Using e.g. neural nets. - Assumption: you have historic data with no actionable inefficiencies (or only sporadic that will average out). - Advantage: takes the efficiency constant from the OEM out of the equation.

Setting off to build the platform

▪ Start-up mode - Use cases well understood. However: pivoting is always a possibility. - Not much data to start with.

▪ People - End clients, energy professionals, data scientists. - Analysis of data with SQL-only skills required.

Our understanding of the requirements in the beginning of the project

▪ Data - Batch to start with (aggregated to 5-min). - 100 measurements per second. - Weather from turbine or park, weather from 3rd party, yaw, pitch, rotor, shaft, gear, generator.

▪ Algorithms - MVP first. - Fast-track & EoD.

Azure: Components for λ architectureMix & match: Multiple options even within one platform

File

Stream

DB

Batch

Stream

Ingestion Storage Processing

Ingestion Processing

Code Synchronization

Serving

Source: https://www.analytix-unboxed.com/mapping-microsoft-azure-analytics-components-to-lambda-architecture/

Data Factory

Sqoop

Kafka IoT Hub

Event Hub

Blob

HDFS

Data Lake Store

Data Lake Analytics

Batch

Hive | Spark SQL DW

Stream Analytics

StormSpark Streaming

Event Processor Host

SQL Azure

SQL DW

Document DB

Analytics Services

Redis Cache

Search

HBase

System architecture: λ (or φ?)Platform is built on Azure with three processing lines

Input processor & storage

Front-end layer

User application

Visuals

Streaming line

Fast-track batch

EoD batch

SQL ServerWeb application w/

embedded Power BI & R visuals

Custom made queuing(python)

PV, WT, weather measured & FC

File

OPC Stream

Azure Blob storage

A closer look: Analytical batchData vault and analytics in the core of the platform

Azure Power BI Embedded SQL Server + RAzure Blob Storage

Data LakeRawVault

BusinessVault

Power BI ServingAnalytics

Inbound over

polybase

data movement within DB

Deep Dive: Azure Storage Options

▪ Unstructured data with frequent and fast retrieval → Azure Blob Storage

▪ Analytics on stored data → Azure Data Lake

▪ Both frequent and fast data retrieval + analytics → Blob + SQL server (winji)

Azure Blob Storage vs Azure Data Lake Store

Azure Data Lake Store Azure Blob Storage

Purpose: Optimized for parallel big data analytics Not optimized for analytics

Access: Access via Azure Active Directory at file & folder level Access via Access Keys

Price: High storage, low operation price Low storage, high operation price

Suggestion

PolybaseWhy move data?


Data LakeRawVault

BusinessVault


Inbound over

polybase

▪ Simplicity Reduced complexity : - No need for external tools to access data - Access data using T-Sql

▪ Productivity Analysis without moving data before implementation

Rationale

Analytics with SQL Server + RAnalytics, all you can eat


Data LakeRawVault

BusinessVault


▪ Flexibility Broad analytical options

▪ Simplicity Ready to go R services

▪ Productivity Integrated R services allows fast ETL

Rationale

Deep dive: Data vaultEndless scalability & auditability


Data LakeRawVault

BusinessVault


Rationale

▪ Flexibility Parallel workloads and data ingestion allowing automation

▪ Scalability Extendable model without re-engineering

Deep dive: Data vaultInmon, Kimball, and now Linstedt

Associations

Details

Business Keys

Farm

Sat

Sat

WT

MeteoLink

Sat

Owner

Sat

Sat

LinkLink

Sat

WT Type

Sat

Data Vault

Dimensional

OwnerFarm

Meteo

WT Fact

WT Type

3NF

Farm

WT Fact

WT LI

Owner

MeteoWT Type

Deep dive: Data vault“A repeatable modelling technique”

LINK_FARM_WT

PK Hub_Farm_WT_Key Char(32)

UF Hub_Farm_Key Char(32)

UF Hub_WT_Key Char(32)

Hub_Load_Dts Datetime

Hub_Record_Src Varchar(50)

Farm_Name Varchar(50)

WT_Code Varchar(25)

HUB_WT

PK Hub_WT_Key Char(32)

U WT_code Varchar(25)



HUB_FARM

PK Hub_Farm_Key Char(32)

U Farm_Name Varchar(50)



SAT_FARM

PF Hub_Farm_Key Char(32)

P Hub_Load_Dts Datetime


HASH_DIFF Char(32)

Farm_Local_Id Varchar(50)

Manager Varchar(50)

Installation_date Datetime

SAT_WT

PF Hub_WT_Key Char(32)

P Hub_Load_Dts Datetime


HASH_DIFF Char(32)

WT_name Varchar(50)

Installation_date Datetime

- Increase in the number of data objects- Complexity is shifted downstream- Contra-trend?

Hubs : Unique list of Business Keys

Links : Unique List of Associations / Transactions

Satellites : Descriptive Data for Hubs and Links (Type 2 with history)

Critics

Delivering an IoT project

▪ Data Not a given in win-ji’s case: need partners and early projects to develop solution.

▪ Technology & Architecture Important to have flexibility (e.g., Azure: pick & play, multiple solutions for different parts of the puzzle). But: Finding the best architecture can be complicated (tools, compatibilities) - this should not be delaying you.

▪ Team You need the right mix with data science, system engineering, and most importantly: subject-matter expertise aiming high in the value chain.

Learnings

Delivering an IoT project

▪ Top management buy-in is crucial A given in win-ji’s case.

▪ From “blind” to data-driven: Short time to value 5-10% increase in power output in most cases realistic without any sophisticated ML.

▪ Are you in a fast changing environment? Be prepared to adapt. In the case of renewables the market is rapidly changing due to deregulation.

Learnings (cont’d)

Questions? Feel free to contact us:

[email protected]

[email protected]

big data meets renewable energy: building a real-time

Documents