smap 2.0: integrating and managing physical data stephen dawson-haggerty computer science division,...
Post on 21-Dec-2015
233 views
TRANSCRIPT
sMAP 2.0: Integrating and Managing Physical Datahttp://code.google.com/p/smap-data
Stephen Dawson-Haggerty
Computer Science Division, University of California, [email protected]
sMAP Redesign
• Design iteration based on 2 years’ experience• Case studies
– Building management system integration– Residential monitoring– Retrofit system– Repository integration– Transient instrumentation
• Ecosystem – Real-time– Slicing– Analysis
January 9, 2012 Local Winter Retreat 2012 2
Services Communicating with sMAP
January 9, 2012 Local Winter Retreat 2012
Archiver
RDBMS TSDB
sMA
PsM
AP
sMA
P
con
trol
web
6lowpan networks
RS-458 bus
BacNET/IP
mo
de
ls
mg
mt
Many deployments share common infrastructure
Points of decoupling
3
sMAP Principles
• Decoupled, multi-site design– Instruments, archive, and front-ends can all
cross administrative boundaries• Common representation of Timeseries
– Between all parties• Import all data and metadata
– Don’t be tied to particular schemas– Allow for slicing and dicing
• Fix common pain points once
January 9, 2012 Local Winter Retreat 2012 4
Why Not an Existing System?
• We are built on existing systems– postgresql, django, twisted, bdb
• Many existing systems are vertical stovepipes or proprietary – OSISoft, BMSs, BI software
• Open source software is still struggling to take timeseries seriously– OpenTSDB, Flume/FlumeBase– PubSubHub, pachube
• Be modular and efficient and share code in an open ecosystem– Decoupled architecture allows independent development,
optimization
January 9, 2012 Local Winter Retreat 2012 5
SMAP SOURCE CASES
January 9, 2012 Local Winter Retreat 2012 6
Case 1: Building Management System
• Sutardja-Dai (SDH) Hall– Managed with Siemens Apogee
System• Shared serial bus, polled by master• Large number of points, high data
volume• Similarly complex data
relationships
January 9, 2012 Local Winter Retreat 2012 9
SDH Apogee System
• 1358 control settings– Set points, Relays (lights, pumps, etc), Schedules
• 2291 meters/sensors– Power (building, floor, lights, chiller, pumps, etc)
• Current, voltage, apparent, real, reactive, peak
– Temp (rooms, chilled water, hot water)– Air volume– Alarms, Errors
• 2165 control outputs– Dampers, valves, min/max flow, fan speed, PID
parameters• 72 other
January 9, 2012 Local Winter Retreat 2012 10
Case 2: Residential Plug-Loads
Dense plug-load metering using wireless mesh networks
Goal: understand the use patterns of appliances
• Deployment methodology for inventory and stratified sampling of loads
• Takeaways: unreliability of home internet, NATed operation
January 9, 2012 Local Winter Retreat 2012
Edge Router
(With LBNL MELS group)
12
Network Complexity & Reachability
January 9, 2012 Local Winter Retreat 2012 13
(Re)design Takeaways
• Integrating heterogeneous communication substrates is [still] a must– Was a goal of sMAP 1– Better support is possible
• Intermittent connectivity is unavoidable– Don’t lose data when not necessary– Design for reliability
• Existing repositories are unavoidable– Efficient import/export of data (same as a trickle load)– Well known requirement of existing BI/decision support
products• Accept metadata from existing systems when available
January 9, 2012 Local Winter Retreat 2012 14
Local Winter Retreat 2012
sMAP 1 Overview
/ # list resource under URI root [GET] /data # list sense points under resource data [GET] / [sense_point] # select a sense point [GET] /meter # meters provide this service [GET] / [channel] # a particular channel [GET] /reading # meter reading [GET] /format # calibration and units [GET/POST] /parameter # sampling parameter [GET/POST] /profile # history of readings [GET]
Represent units, sampling parameters of the underlying device, for each channel
Expose over HTTP
January 9, 2012 15
Change 1: Represent Timeseries
• Objects are Timeseries and Collections• Timeseries are durably identified
– A collection of structured key-value pairs, data– Data are (time, value[, seqno]) vectors
• Collections group timeseries with shared metadata
January 9, 2012 Local Winter Retreat 2012
{ "/" : { "Contents" : ["sensor0"], “Metadata” : { “SourceName” : “Example sMAP Source” }, }, "/sensor0" : { "Contents” :["channel1"] }, "/sensor0/channel0" : { "uuid" : "a7f63910-ddc6-11e0-8ab9-13c4da852bbc", "Readings" : [ [1315890624000, 12.5 ] ] }}
colle
ctio
nstim
esei
res
16
/true_power
Properties/ReadingType: doubleProperties/Timezone: America/Los_AngelesProperties/UnitofMeasure: kW
/ABC
Extra/Phase: ABC
Change 2: Simple Metadata
• Each stream can be “tagged” with key-value pairs• Easy to tag related collections of points• Natural way to represent “subset of” relation
January 9, 2012 Local Winter Retreat 2012
/
Location/Campus: UCBLocation/Building: Cory Hall
gpe
Instrument/Manufacturer: Dent IndustriesInstrument/Model: PowerScout 18Instrument/SamplingPeriod: 20
/elt-A
Extra/DentElement: elt-AExtra/Circuit: 1Extra/System: lightingExtra/ServiceDetail: lighting floors 0-4
17
OSS sMAP Source Implementation
• Implementation of new design– Focus on ease of writing new source
• Reliable delivery with local buffering• Multiple configuration options for adding metadata,
setting up drivers• Growing driver library• Documentation, tutorials, etc.• Focus on minimum effort to add new source
January 9, 2012 Local Winter Retreat 2012
Den
t ci
rcui
t m
eter
Tem
pera
ture
/PA
R/TS
R
sMA
PsM
AP
http://code.google.com/p/smap-data/
18
Example Teaser
caiso.ini[/]uuid = dd6a4ae4-b17c-11e0-9d53-0026bb56ec92Metadata/SourceName = CA ISO price and consumption feed
[/demand]type = smap.drivers.caiso.CaIsoDriver
[/oakland]type = smap.drivers.caiso_price.CaIsoPrice# this is passed to the driverLocation = OAKLAND_1_N001
January 9, 2012 Local Winter Retreat 2012 19
INFRASTRUCTURE
January 9, 2012 Local Winter Retreat 2012 20
sMAP Ecosystem Architecture
January 9, 2012 Local Winter Retreat 2012 21
Services Communicating with sMAP
January 9, 2012 Local Winter Retreat 2012
Archiver
RDBMS TSDB
sMA
PsM
AP
sMA
P
con
trol
web
6lowpan networks
RS-458 bus
BacNET/IP
mo
de
ls
mg
mt
Many deployments share common infrastructure
Points of decoupling
22
Archiver
• Application server functionality on top of back-end databases– Add data and metadata from sMAP sources– Query stream data and metadata– Enforce security policy
January 9, 2012 Local Winter Retreat 2012
/add/KEY # add new data to system/republish # data firehose/api/query # look up timeseries by tags/api/[data,next,prev] # retrieve timeseries data/api/tags # return tagsets
23
High-performance Storage Service
January 9, 2012 Local Winter Retreat 2012
Page CacheLock
Manager
Key-Value Store
Storage Alloc.
Time-series Interface
Bucketing RPC Compression
read
ingd
b
SQL
Storage mapper
MySQL
• Low-latency, high speed access to archived readings• 5 billion readings, 3.5 bytes/reading on disk
24
Archiver Query Language
• Column-like store with SQL-like interfaceselect distinct Metadata/SourceName
January 9, 2012 Local Winter Retreat 2012
Tag names look like columns
Arc
hive
r Q
uery
Lan
guag
e Q
uery
Sec
urity
che
ck a
dditi
on
Tra
nsfo
rmat
ion
to s
et o
pera
tions
Exe
cute
on
post
gres
25
Archiver Query Language Examples
query > select distinct Metadata/SourceNameSELECT DISTINCT m.tagval FROM metadata2 m, stream s, subscription sub WHERE s.id = m.stream_id AND s.subscription_id = sub.id AND tagname = 'Metadata/SourceName' AND (sub.public OR sub.key = 'jNiUiSNvb2A4ZCWrbqJMcMCblvcwosStiV71’)
query > select distinct Metadata/SourceName where Metadata/Instrument/Manufacturer ~ '^Dent'SELECT DISTINCT m.tagval FROM metadata2 m, stream s WHERE stream_id IN ( (SELECT stream_id AS cnt FROM metadata2 mi, subscription sub, stream si WHERE ((tagname = 'Metadata/Instrument/Manufacturer' AND tagval ~ E'^Dent')) AND (sub.public OR sub.key = 'jNiUiSNvb2A4ZCWrbqJMcMCblvcwosStiV71') AND mi.stream_id = si.id AND si.subscription_id = sub.id)) AND tagname = 'Metadata/SourceName' AND s.id = m.stream_id
January 9, 2012 Local Winter Retreat 2012 26
powerdb2: Plotting and Exploration
January 9, 2012 Local Winter Retreat 2012 27
slicr: Tags Generate Multiple Views
January 9, 2012 Local Winter Retreat 2012
[ { tag : "Metadata/SourceName", restrict: "has Metadata/Extra/EndUse"},
{ tag: "Metadata/Extra/EndUse"},
{ tag: "Metadata/Extra/Category", defaultSubStream: "Properties/UnitofMeasure = 'mW'", seriesLabel:["Metadata/Location/Room", "Metadata/Extra/Load"]}, { tag: "Metadata/Extra/ProductType", defaultSubStream: "Properties/UnitofMeasure = 'mW'", seriesLabel:["Metadata/Location/Room", "Metadata/Extra/Load"]}, { tag: "Metadata/Instrument/PartNumber", defaultSubStream: "Properties/UnitofMeasure = 'mW'", seriesLabel:["Metadata/Instrument/PartNumber", "Metadata/Location/Room", "Metadata/Extra/Load”]}, "Properties/UnitofMeasure”]
28
Status
• Nearly all sMAP 1 sources moved– Dent + Veris modbus electric– Siemens + Johnson BACnet – pQube, campus obvius electric + steam– Vaisala weather, wunderground, NOAA, NWS– Steam, condensate– BPA Hydro, CAISO price + consumption– ACme, Dust, Labjack, TED
• Sweet new sources– Real-time resampling, campus baselining
• Applications, too:– Data closet PUE– Lighting– Dashboards galore– Lots of undergrad involvement, projects in the works
• Entire system released: readingdb, smap-data, powerdb2
January 9, 2012 Local Winter Retreat 2012 29
Adoption & The Future
• CBE site– Shared backend– Store and organize BMS data from study participants– Mix of some realtime with mostly loaded data
• LBNL site– Full separate install
• CIEE Energy and Climate Research Innovation Seed Fund Grant
• Energy data application contest (thanks, Quanta!)
January 9, 2012 Local Winter Retreat 2012 30
DEMO & QUESTIONS
January 9, 2012 Local Winter Retreat 2012
http://new.openbms.org
http://code.google.com/p/smap-data/
31