realization of big data platform for pmu · realization of big data platform for pmu - real-time...

© Hitachi, Ltd. 2014. All rights reserved.

Realization of Big Data Platform for PMU - Real-Time Data Processing & Historical Data Analysis -

March 27, 2014

Yutaka Kokai Shinji Fujiwara

Hitachi America, Ltd. Hitachi, Ltd.


1.Technical requirements for Control Center

2.High-Speed Data Access Platform for Control Center

3.Live Demo ~ PMU Application for Control Center ~

Contents

1


1. Technical requirements for Control Center

2


1-1 PMU Installation in North America

3

PMU installation in North America

The number of installed PMUs are growing in North America

Source: NASPI Oct 22, 2013

0

200

400

600

800

1000

1200

March 2012 March 2013

Num

be

r

About 500

1,126

Source: DOE Synchrophasor Technologies and their Deployment

in the Recovery Act Smart Grid Programs, August 2013

In the case of WECC,

481 PMUs are installed.

Source: DOE Synchrophasor Technologies and their Deployment

in the Recovery Act Smart Grid Programs, August 2013

The number of installed

PMUs are increased.

Now the usage of PMU data

needs to be considered.


Platform for Big Data

Analysis is required

Information Storage

& Retrieval System

for Big Data Analysis

(PMU data)

1-2 Database Requirement for PMU Data

4

In the case of 500 PMUs are installed ……

PMU data will be accumulated about 40 TB / year! And the number of PMUs are expected to increase more and more in the future...

・・

・・

・

・

・・

PDC

PDC

PMU

PMU

PMU

PMU

Operator

94[bytes] 500 31536000 [seconds/year]

≒ 40 TB (8 phasors, 4 analogs, 2 digitals)

30 [messages/sec]

PMU data size × Number of PMUs × Message rate × Time


1-3 PMU Applications for Control Center

5

Historical Data Analysis Real-Time Data Processing

Wide-area monitoring

Voltage stability analysis

Oscillation analysis

Islanding

Mode meter

State estimation

Power plant model validation

Frequency response analysis

Post event analysis

Baselining

Fault location

・

・

・

・

・

・

To realize applications using PMU data, both “Real-Time Data

Processing” and “Historical Data Analysis” are required.


2. PMU Data Analysis Platform for Control Center

6


2-1 PMU Data Analysis Platform for Control Center

7

PDC PMU

PMU

PDC PMU

PMU

Super

PDC

Stream Data

Processing

In-memory

Data Cache

Analysis

Applications

Send

Sensor Data (every a few sec)

Online Power System Analysis

Load

Sensor Data

(every 5 to10 min)

Model updates

Model

DBMS

Archive

Files Access

Historical Data

(in 2 to 3 sec)

Real-Time Data Processing

Historical Data Analysis

Off-line Analysis

Historical Data Analysis: Quick Data Loading and High-Speed data access to historical data, comfortable data viewing and quick power system analysis.

Real-Time Data Processing: Online power system analysis using the PMU data in real time.


2-2 History of Hitachi’s Database Management System

8


Hitachi Advanced Data Binder Platform *1 (HADB) - x100 fast Database Engine *2

- Configured with Hitachi server and storage system □ High-speed Data Loading for Huge Data □ Multi-Table JOIN over Multiple Data Sources

Ad-hoc Data Analysis

High-speed Database Engine

Hitachi Server

Hitachi Storage

2-3 High-speed Data Access Platform for Control Center

9

What’s the New Technology ?

ＢＩ Customer Value

Business Improvement

MicrosoftⓇ

Excel

High-speed Data access Platform

Hitachi Advanced Data Binder Platform *1 The outcome of “Development of the fastest database engine for the era of very large database, and Experiment and evaluation of strategic social services enabled by the database engine” project (Principle Investigator: Prof. Masaru Kitsuregawa, University of Tokyo and also Director-General, National Institute of Informatics), supported by the Japanese Cabinet Office’s FIRST Program (Funding Program for World-Leading Innovative R&D on Science and Technology).

*2 Compared with Hitachi conventional technology. The performance was measured based on standard analytical benchmarks. The speeding-up effect on various queries are different. We compared the performance of searching a fixed amount of data which met a certain condition in the database.


2-4 Innovative Technology of Database Engine

10

“Out-of-Order Database Engine” technology *3

- SQL is automatically divided into thousands of task and each task issues asynchronous I/O to get data from disk - These tasks are executed by Out-of-Order control method Thousands of Asynchronous I/O improves the SQL processing

従来型データベースエンジン

時間

決定的な処理順序

同期入出力発行

超高速データベースエンジン

時間

超大量

非同期入出力発行

非決定的な処理順序

Conv. DB Engine （In-Order） Out-of-Order DB Engine

Most of CPU time is I/O wait SQL is executed by thousands of task

without waiting I/O completion

Sparse I/O Dense I/O

*3 A principle proposed by Prof. Kitsuregawa (Univ. of Tokyo and also Director-General, National Institute of Informatics) and Project Associate Prof. Goda (Univ. of Tokyo)

Small number of

Sync. I/O

Conv.

DB Engine In-order

processing

Out-of-Order

DB Engine Out-of-Order

processing

Large number of

Async. I/O

time time


Search processing (μs)

Synchronous I/O processing (ms)

Server

Storage

Server

Storage

[Conventional Technology] In-order exec.

[New Technology] Out-of-Order exec.*4

Task assignment

Search processing

Wait for I/O completion

Disk I/O

Key point

DB engine automatically divides search process into thousands of tasks and executed them asynchronously.

I/O wait time causes long proc. time

•Search process is divided into thousands of tasks, and they are carried out in parallel.

•Using asynchronous I/O, DB server issues as many I/O as possible.

•DB server continues execution of each task in I/O completion sequence.

A B C D

E

B

C

D

E

A

B

C

D

E

A

11

*4 A principle proposed by Prof. Kitsuregawa (Univ. of Tokyo and also Director-General, National Institute of Informatics) and Project Associate Prof. Goda (Univ. of Tokyo)

2-5 Technology in Detail


2-6 HADB broke TPC-H Benchmark Record


2-7 Demonstration

13

Please watch a Live Demonstration


2-7 Demonstration (Cont’d)

14


2-7 Demonstration (Cont’d)

15


3.Live Demo ~ PMU Application for Control Center ~

16


3-1 PMU Application for Control Center

17

Imports measurement data generated at a speed of tens of thousands per second without delay. Retrieves requested data within a few seconds from 100 TB scale historical data.

Technological Merits

HADB: Hitachi Advanced Data Binder

PDC: Phasor Data Concentrator

PMU: Phasor Measurement Unit

STATCOM: Static Synchronous Compensator

IS & R: Information Storage and Retrieval

Hydro power plant

Thermal power plant

STATCOM

Wind farm

PDC

Historical

Data

PMU

PMU

PMU

Information Storage

& Retrieval System

Transmission Network Operator

Operator Real-time

Check

1. Fault

2. Fault event

Sensor data are sent every a few seconds

Real-time problem analysis

HADB

Power Grid

3. Analysis

PMU

Visualize historical data

High-speed data import

(45 MBytes/sec; every 5 minutes)

Retrieves data in a few seconds

from a 100TB-scale historical data


3-2 Demonstration System Configuration

18

Client PC PMU Application for Control Center

Data source : Time series data from Phasor Measurement Units (dummy) - # of PMUs: 500 - Measurement interval: 33.3 milliseconds Data import operation: Imports data in every five minutes (400 MBytes) Size of historical Data: 1.2 Tbytes (14.5 Billion records, 11 days) DB server : HA8000/RS220 (12 cores/24 threads) + AMS2500 (64 HDDs) Client PC : Notebook PC DBMS : Hitachi Advanced Data Binder Platform

: Open Source DBMS

Voltage Trend of PMUs

PMU : Phasor Measurement Unit

HADB

DB Server

HA8000/ RS220

AMS2500

Historical Data

Application Server Sensor Data

Sensor data are imported periodically

Open Source DBMS

Historical Data

Traditional DBMS based IS & R

HADB based IS & R

1st Demo

2nd Demo

NW


3-3 Demonstration

19

1st Data Import Demonstration

2nd Data Retrieve Demonstration


3-4 Summary - Data Import Performance

20

4.5 MBytes/sec

45.0 MBytes/sec

x 10.0 Faster


HADB based IS & R


3-5 Summary - Data Retrieve Performance

21

14.5 sec 1.1 sec

x 13.2 Faster


HADB based IS & R


March 27, 2014

Hitachi America, Ltd.

END

22

•Infrastructure Systems Company, Power Information & Control Systems Division

•Information & Telecommunication systems Company, IT Platform Division Group, IT Platform R & D Management Division

Hitachi, Ltd.

Realization of Big Data Platform for PMU - Real-Time Data Processing & Historical Data Analysis -


Trademarks

23

• Microsoft and Excel are either registered trademarks or trademarks of Microsoft Corporation in the United

States and/or other countries.

• TPC Benchmark and TPC-H are trademarks of the Transaction Processing Performance Council.

• Other company and product names mentioned in this document may be the trademarks of their respective

owners. Throughout this document Hitachi has attempted to distinguish trademarks from descriptive terms

by writing the name with the capitalization style used by the manufacturer, or by writing the name with

initial capital letters. Hitachi cannot attest to the accuracy of this information. Use of a trademark in this

document should not be regarded as affecting the validity of the trademark.

realization of big data platform for pmu · realization of big data platform for pmu - real-time...

Documents