advanced analytics with tibco enterprise runtime for rfiles.meetup.com/1731383/moc houston r july...

Advanced Analytics with TIBCO Enterprise Runtime for R Houston R User Group July 2014

Troy Ruths, PhD Matthew Slocum, PhD Ruths.AI Michael O’Connell, PhD TIBCO Data Science

© Copyright 2000-2014 TIBCO Software Inc.

2!© Copyright 2000-2014 TIBCO Software Inc.

•  Introduc)on / Context •  Analy&cs for Energy

•  Ma1hew Slocum •  Infill Analysis •  Structural Equa&on Modeling •  Voronoi Diagrams

•  Michael O’Connell

•  TERR with SpoAire: Custom Expressions & Data Func&ons •  Produc&on Op&miza&on

•  Wrap-‐up •  Insight to Ac&on

Increase Productivity

Grow Revenue

Value

Reduce Risk

ROI

Overview

Immediate Long-‐Term Compe55ve Advantage Value to the Organiza)on

Spo@ire is the only analy)cs pla@orm that can provide value to the organiza)on across the full spectrum of use cases

Self-‐service Dashboards Event Analy)cs

Predic)ve Analy)c Apps

Measure Diagnose Predict & Op&mize Opera&onalize Automate Transform

Analy&cs Maturity

Analy&cs Maturity Model

Immediate Long-‐Term Compe55ve Advantage Value to the Organiza)on

Spo@ire is the only analy)cs pla@orm that can provide value to the organiza)on across the full spectrum of use cases

Self-‐service Dashboards Event Analy)cs

Predic)ve Analy)c Apps

Measure Diagnose Predict & Op)mize Opera)onalize Automate Transform

Analy&cs Maturity

Analy&cs Maturity Model


TERR and Spotfire in the Energy Community


•  Ma1hew Slocum •  Infill Analysis •  Structural Equa&on Modeling •  Voronoi Diagrams


Grow Revenue

Value

Reduce Risk

ROI

Ruths.AI


•  Forecas)ng Y •  Forecasts of (financial) performance

•  Machine Learning & Data Mining Y = f (X, b) •  Comple&ons, Produc&on Op&miza&on

•  Segmenta)on and Clustering X •  G&G Analysis

•  Op)miza)on •  PorAolio, Resource Alloca&on

•  Simula)on •  Confidence Analyses e.g. P10, P50, P90; spa&al predic&on

Predictive Analytics


Software for Predictive Analytics

Open Source •  R •  Python (required packages) •  others...

Commercial •  SAS •  MATLAB •  SPSS •  others…

TIBCO’s focus and investment has been on R TIBCO Enterprise Run5me for R

R is the lingua franca of Statistical Computing

Date

R P a

c k a g e s

1/1/2002 1/1/2003 1/1/2004 1/1/2005 1/1/2006 1/1/2007 1/1/2008 1/1/2009 1/1/2010 1/1/2011 1/1/2012 1/1/2013

5000 4500 4000 3500 3000 2500 2000 1500 1000 500

0

Number of R-‐ or SAS-‐related posts to Stack Overflow by week. (copyright by r4stats.com)

Number of contributed packages on CRAN (h3p://cran.r-‐project.org/)

> 5,000 Packages !

10!

TIBCO Enterprise Runtime for R

•  TIBCO Enterprise Run)me for R (TERR) •  Latest sta&s&cs scrip&ng engine: S a S-‐PLUS® a R a TERR •  Developer Edi&on: www.TIBCOmmunity.com

•  Engine internals rebuilt from scratch at low-‐level •  Redesigned data objects, memory management •  Addresses long-‐standing issues with S (R) language

•  TERR addresses deployment issues with R •  Performance •  Big data, fast data

•  TERR is commercially licensed from TIBCO •  TERR Installs (free) with SpoAire Analyst / Desktop and other TIBCO products (CEP, Stats) •  SpoAire Server can manage all TERR / R scripts, ar&facts for reuse


11!11!

TERR Licensing & Deployment

•  TERR is commercially licensed from TIBCO •  TERR Installs (free) with SpoAire Analyst / Desktop and other TIBCO products (CEP, Stats) •  TERR can be embedded anywhere, under license

•  Open Source R is GPL-‐licensed •  Other socware vendors are forbidden to embed or redistribute open source R in

commercial socware (GPL license is viral) •  Convoluted integra&ons for all vendors (except TIBCO)

•  Centralized Analy)cs Management •  SpoAire library – TERR, R, (and SAS, Matlab) data func&ons •  TERR can use SpoAire Informa&on Links directly

•  Deploy through Big Data and Fast Data Environments •  Hadoop Streaming MapReduce •  Complex Event Processing / JMS


12!

Model Fitting: 5 Million Rows Model Scoring: 20 Million Rows

TERR 7X faster 84X © Copyright 2000-2014 TIBCO Software Inc.

TERR Performance


TERR in Spotfire

What does TERR do in Spo@ire? •  Runs TERR Data Func&ons in SpoAire analyses •  Powers the Predic&ve Modeling Tools •  Powers the Forecast tool •  Can be used directly in Expressions (new in Spo<ire 6.5)

TERR is embedded in Spo@ire Analyst/Desktop •  No other socware required, no connec&on to server required to get started

Ways of Using Spotfire and TERR


1.  No R code required •  Forecas&ng (Demo: natural gas consump&on)

2.  SpoAire analyst writes the R code (expression) (Demo: Western Canadian Sedimentary Basin geology)

3.  Sta&s&cal programmer writes the R code, SpoAire analyst uses it (expression func&on, data func&on) (Demo: Western Canadian Sedimentary Basin geology)

4.  Sta&s&cal programmer writes the R code, SpoAire analyst uses it (data func&on) (Demo: Bakken comple&ons)


Completions Demonstration

A1ributes, Loca)on Comple)on Geology Result En&ty ID Total Lateral Length Total Organic Content (TOC) EUR API # Hor or Vert % Clay 3 mo Cum Gas (cu c) Comple&on Date # of laterals % Quartz 6 mo Cum Gas (cu c) Forma&on # Stages % Pyrite 9 mo Cum Gas (cu c) County # Clusters % Carbonate 12 mo Cum Gas (cu c) Well Name # Shots per Stage Poisson's Ra&on 3 mo Cum Oil (bbl) LATITUDE # Clusters per Stage Young's Modulus 6 mo Cum Oil (bbl) LONGITUDE Average Rate Briqleness 9 mo Cum Oil (bbl) Reservoir Average Pressure Porosity 12 mo Cum Oil (bbl) Operator Proppant Type Reservoir pressure 3 mo Cum Water (bbl) Spud Date Total Proppant (lbs) Reservoir temperature 6 mo Cum Water (bbl) First Produc&on Date Proppant Size Reservoir thickness 9 mo Cum Water (bbl) Total Depth (TD) Proppant Concentra&on (lbs/gal) 12 mo Cum Water (bbl) True Ver&cal Depth (TVD) Proppant per Stage (lbs/stage) 3 mo Cum BOE (bbl) Re Frac (Y/N) Proppant per Foot (lbs/c) 6 mo Cum BOE (bbl) Hole Conddi&on Total Frac Fluid pumped (gal) 9 mo Cum BOE (bbl) Annular Isola&on Fluid per Stage (gal/stage) 12 mo Cum BOE (bbl) Internal/Mechanical Isola&on Frac Fluid per foot (gal/c) Screen Out (Y/N) Slickwater pumped (gal)

Slickwater per stage (gal/stage) Linear Gel Pumped (gal) Linear Gel per Stage (gal/stage) X-‐Link Pumped (gal) X-‐Link per Stage (gal/stage)


Analytic Components at Fingertips


Completions Analysis

•  Business Opportuni)es •  Comple&ons op&miza&on by well •  Produc&on predic&on for new wells •  Sand-‐out op&miza&on

•  Analysis and Data •  Subsurface (e.g. Spectra) •  Loca&on •  Comple&ons •  Produc&on

•  Value and Financial Impact •  Op&mal comple&ons •  Asset valua&on

Op5mize Comple5ons – Loca5on, Subsurface


Completions Analysis

•  Business Opportuni)es •  Comple&ons op&miza&on by well •  Produc&on predic&on for new wells •  Iden&fy factors driving produc&on vs expected produc&on e.g. operator

•  Analysis and Data •  Subsurface •  Loca&on •  Comple&ons •  Produc&on

•  Value and Financial Impact •  Op&mal comple&ons •  Opera&ons management •  Asset valua&on & “where to drill”

Op5mize Comple5ons – Loca5on, Subsurface

20!

Data Mashup, EDA and Features

•  Handling Missing Data •  G&G: Interpola&on from Spa&al Smoothing

•  Model interpola&on – predict flow from pressure and vice-‐versa

•  Sensors: interpolate across &me from Historian

•  Features (X variables) •  G&G •  Well aqributes

•  Response variables •  Produc&on: 30, 60, 90 day cum •  Produc&on: EUR (threshold via &me or produc&on)

•  Sensor readings


Spa5al Predic5on

21!

Production Forecasting

Forecast Produc5on – Set Expected Produc5on for Wells


•  Business Opportuni)es •  Asset valua&on •  Forecasts for surveillance benchmarks

•  Analysis and Data •  Produc&on analysis •  Well-‐level data

•  Value and Financial Impact •  EUR es&mates and Decline /Type Curves

•  Produc&on monitoring


Leveraging TERR

TIBCO Spo@ire Sta)s)cs Services •  Enables advanced analy&cs for SpoAire on web/cloud •  Managed pools of sta&s&cal engines -  TERR, R, S+, SAS®, MATLAB®

•  Load balancing, queuing, failover, paralleliza&on •  High-‐level APIs for loose integra&on •  Central management of analy&cs, packages

Embeddable TERR Engine •  Custom (&ght) integra&on, batch, exis&ng grids, etc. •  Faster than R, more robust, beqer memory management, fully

supported •  Low-‐level APIs for &ght integra&on •  Integrated into TIBCO products: CEP (Streambase), Cloud

Compute

S+

R

TERR

SAS®

MATLAB®


Supported Data Sources

In-‐Memory, In-‐Database and Data-‐On-‐Demand •  Apache Hadoop/Hive •  Cloudera Hive CDH4 •  Cloudera Impala CDH4, 0.6, 1.2.2, 1.2.3 •  Composite Informa&on Server 6.1, 6.2 •  Hortonworks Data PlaAorm 1.3, 2.0 •  HP Ver&ca 5.0, 6.0, 6.1 •  IBM DB2 LUW 8, 9, 9.5 •  IBM Informix 9.4 •  IBM Netezza 5, 6, 7 •  JDBC •  Microsoc SQL Server 2000, 2005, 2008, 2012 •  Oracle MySQL 4.1, 5.1, 5.5, 5.6 •  Oracle and Oracle Exadata (Oracle 9i, 10g, 11gR1 and R2, RAC) •  Pivotal Greenplum 3.3, 4.1, 4.2 •  Pivotal HAWQ •  PostgreSQL 8.4, 9.0, 9.1, 9.2 •  SAP HANA SPS5, SPS6 •  SAP Sybase 12.5, 15, 15.5 •  SAP Sybase IQ 15 •  Teradata 12.00.12, 13.00, 13.10, 14.00, 14.10 •  Teradata Aster 5.0, 5.11

In-‐Database only •  Microsoc SQL Server Analysis Services 2008, 2012 •  Oracle Essbase 9.3, 11.1 •  SAP NetWeaver Business Warehouse 7.0.1 SP10, 7.3

In-‐Memory and Data-‐On-‐Demand •  Aurea Sonic 7.5 •  Oracle E-‐Business Suite 11.5.8, 11.5.10 •  Oracle Siebel 7.7, 7.8, 8.0 •  Salesforce.com •  SAP R/3 4.7, mySAP 5.0, 6.0 •  TIBCO Ac&veMatrix BusinessWorks™ •  TIBCO Ac&veSpaces •  TIBCO StreamBase LiveView

In-‐Memory Only •  ADO.NET •  Comma-‐Separated Values (.csv) •  ESRI Shape Files (.shp) •  Microsoc Access Databases (.mdb, .mde) •  Microsoc Excel Workbooks (.xls, .xlsx, .xlsm) •  ODBC •  SAS Data Files (.sas7bdat, .sd2) •  SpoAire DecisionSite Files (.sfs) •  SpoAire Text Data Format (.stdf) •  SpoAire Binary Data Format (.sbdf) •  Text (.txt) •  TIBCO Formvine •  Universal Data Link (.udl)


TIBCO StreamBase and TERR

•  Event analy)cs •  Produc&on: Pump and

Compressor Monitoring •  Drilling: kick detec&on •  Produc&on: Flow

monitoring

•  Apply predic)ve models in real-‐)me decision making

•  CEP with seamless advanced analy&cs

•  Rapidly develop and iterate models in produc)on

25!

Streaming Data through TERR Rules/Models

1. Rules & Models pushed from SpoAire/TERR

2. Data streams into Streambase

3. Streambase TERR calcula&ons on data

4. SpoAire RCA on trigger Other no&fica&ons available



Wrap Up h

•  Spo@ire and R / TERR •  Rich Integra&on : Expressions, Data Func&ons, Menus •  Central Management of Ar&facts : SpoAire Library •  Model Export •  Enterprise Deployment : SpoAire Business Author, Consumer (Web) •  Enterprise Deployment : Sta&s&cs Services, CEP (Streambase)

•  Energy Analy)cs Case Studies

•  Produc&on Forecas&ng •  Produc&on Op&miza&on

Increase

Productivity

Grow Revenue

Value

Reduce Risk

ROI


TIBCO Spotfire Analytics


Documentation


Spotfire Resources

Training site: learn.spoAire.&bco.com Demo gallery: spoAire.&bco.com/demos


Spotfire Resources


Thank you! Michael O’Connell, PhD Chief Data Scien&st TIBCO moconnell@&bco.com hqp://about.me/moconnell +1-‐919-‐7401560

First to Insight, First to Action

© Copyright 2000-2013 TIBCO Software Inc. 31!

Data Mashups

Exploratory Data Analysis Features SpoQire

Dashboards Model & Predict

Deploy Champion Model

Test & Learn

SAP, IBM

Real-time Historian

Production

Well

Filter

Map

Merge

Shape

Geology

Geoformation

Improve Guided -‐-‐-‐-‐-‐-‐-‐-‐ Deploy -‐-‐-‐-‐-‐-‐-‐-‐ In-‐Line Explore Data

Aggregate

Prepare Data Business Case


Grow Revenue

Ensemble Forest

Regression Additive Models Completions

Visualize

Geo Location

Production

Challenger Models

At Rest

In Motion

Value Theses

Reduce Risk

ROI

G&G

Dashboard Updates

Data a Insight a Action

© Copyright 2000-2014 TIBCO Software Inc. 32!

advanced analytics with tibco enterprise runtime for rfiles.meetup.com/1731383/moc houston r july...

Documents