advanced analytics with tibco enterprise runtime for rfiles.meetup.com/1731383/moc houston r july...
Post on 03-Mar-2020
4 Views
Preview:
TRANSCRIPT
Advanced Analytics with TIBCO Enterprise Runtime for R Houston R User Group July 2014
Troy Ruths, PhD Matthew Slocum, PhD Ruths.AI Michael O’Connell, PhD TIBCO Data Science
© Copyright 2000-2014 TIBCO Software Inc.
2!© Copyright 2000-2014 TIBCO Software Inc.
• Introduc)on / Context • Analy&cs for Energy
• Ma1hew Slocum • Infill Analysis • Structural Equa&on Modeling • Voronoi Diagrams
• Michael O’Connell
• TERR with SpoAire: Custom Expressions & Data Func&ons • Produc&on Op&miza&on
• Wrap-‐up • Insight to Ac&on
Increase Productivity
Grow Revenue
Value
Reduce Risk
ROI
Overview
Immediate Long-‐Term Compe55ve Advantage Value to the Organiza)on
Spo@ire is the only analy)cs pla@orm that can provide value to the organiza)on across the full spectrum of use cases
Self-‐service Dashboards Event Analy)cs
Predic)ve Analy)c Apps
Measure Diagnose Predict & Op&mize Opera&onalize Automate Transform
Analy&cs Maturity
Analy&cs Maturity Model
Immediate Long-‐Term Compe55ve Advantage Value to the Organiza)on
Spo@ire is the only analy)cs pla@orm that can provide value to the organiza)on across the full spectrum of use cases
Self-‐service Dashboards Event Analy)cs
Predic)ve Analy)c Apps
Measure Diagnose Predict & Op)mize Opera)onalize Automate Transform
Analy&cs Maturity
Analy&cs Maturity Model
5!© Copyright 2000-2013 TIBCO Software Inc.
TERR and Spotfire in the Energy Community
6!© Copyright 2000-2014 TIBCO Software Inc.
• Ma1hew Slocum • Infill Analysis • Structural Equa&on Modeling • Voronoi Diagrams
Increase Productivity
Grow Revenue
Value
Reduce Risk
ROI
Ruths.AI
7!© Copyright 2000-2014 TIBCO Software Inc.
• Forecas)ng Y • Forecasts of (financial) performance
• Machine Learning & Data Mining Y = f (X, b) • Comple&ons, Produc&on Op&miza&on
• Segmenta)on and Clustering X • G&G Analysis
• Op)miza)on • PorAolio, Resource Alloca&on
• Simula)on • Confidence Analyses e.g. P10, P50, P90; spa&al predic&on
Predictive Analytics
8!© Copyright 2000-2014 TIBCO Software Inc.
Software for Predictive Analytics
Open Source • R • Python (required packages) • others...
Commercial • SAS • MATLAB • SPSS • others…
TIBCO’s focus and investment has been on R TIBCO Enterprise Run5me for R
R is the lingua franca of Statistical Computing
Date
R P a
c k a g e s
1/1/2002 1/1/2003 1/1/2004 1/1/2005 1/1/2006 1/1/2007 1/1/2008 1/1/2009 1/1/2010 1/1/2011 1/1/2012 1/1/2013
5000 4500 4000 3500 3000 2500 2000 1500 1000 500
0
Number of R-‐ or SAS-‐related posts to Stack Overflow by week. (copyright by r4stats.com)
Number of contributed packages on CRAN (h3p://cran.r-‐project.org/)
> 5,000 Packages !
10!
TIBCO Enterprise Runtime for R
• TIBCO Enterprise Run)me for R (TERR) • Latest sta&s&cs scrip&ng engine: S a S-‐PLUS® a R a TERR • Developer Edi&on: www.TIBCOmmunity.com
• Engine internals rebuilt from scratch at low-‐level • Redesigned data objects, memory management • Addresses long-‐standing issues with S (R) language
• TERR addresses deployment issues with R • Performance • Big data, fast data
• TERR is commercially licensed from TIBCO • TERR Installs (free) with SpoAire Analyst / Desktop and other TIBCO products (CEP, Stats) • SpoAire Server can manage all TERR / R scripts, ar&facts for reuse
© Copyright 2000-2014 TIBCO Software Inc.
11!11!
TERR Licensing & Deployment
• TERR is commercially licensed from TIBCO • TERR Installs (free) with SpoAire Analyst / Desktop and other TIBCO products (CEP, Stats) • TERR can be embedded anywhere, under license
• Open Source R is GPL-‐licensed • Other socware vendors are forbidden to embed or redistribute open source R in
commercial socware (GPL license is viral) • Convoluted integra&ons for all vendors (except TIBCO)
• Centralized Analy)cs Management • SpoAire library – TERR, R, (and SAS, Matlab) data func&ons • TERR can use SpoAire Informa&on Links directly
• Deploy through Big Data and Fast Data Environments • Hadoop Streaming MapReduce • Complex Event Processing / JMS
© Copyright 2000-2014 TIBCO Software Inc.
12!
Model Fitting: 5 Million Rows Model Scoring: 20 Million Rows
TERR 7X faster 84X © Copyright 2000-2014 TIBCO Software Inc.
TERR Performance
13!© Copyright 2000-2014 TIBCO Software Inc.
TERR in Spotfire
What does TERR do in Spo@ire? • Runs TERR Data Func&ons in SpoAire analyses • Powers the Predic&ve Modeling Tools • Powers the Forecast tool • Can be used directly in Expressions (new in Spo<ire 6.5)
TERR is embedded in Spo@ire Analyst/Desktop • No other socware required, no connec&on to server required to get started
Ways of Using Spotfire and TERR
© Copyright 2000-2013 TIBCO Software Inc.
1. No R code required • Forecas&ng (Demo: natural gas consump&on)
2. SpoAire analyst writes the R code (expression) (Demo: Western Canadian Sedimentary Basin geology)
3. Sta&s&cal programmer writes the R code, SpoAire analyst uses it (expression func&on, data func&on) (Demo: Western Canadian Sedimentary Basin geology)
4. Sta&s&cal programmer writes the R code, SpoAire analyst uses it (data func&on) (Demo: Bakken comple&ons)
15!© Copyright 2000-2014 TIBCO Software Inc.
Completions Demonstration
A1ributes, Loca)on Comple)on Geology Result En&ty ID Total Lateral Length Total Organic Content (TOC) EUR API # Hor or Vert % Clay 3 mo Cum Gas (cu c) Comple&on Date # of laterals % Quartz 6 mo Cum Gas (cu c) Forma&on # Stages % Pyrite 9 mo Cum Gas (cu c) County # Clusters % Carbonate 12 mo Cum Gas (cu c) Well Name # Shots per Stage Poisson's Ra&on 3 mo Cum Oil (bbl) LATITUDE # Clusters per Stage Young's Modulus 6 mo Cum Oil (bbl) LONGITUDE Average Rate Briqleness 9 mo Cum Oil (bbl) Reservoir Average Pressure Porosity 12 mo Cum Oil (bbl) Operator Proppant Type Reservoir pressure 3 mo Cum Water (bbl) Spud Date Total Proppant (lbs) Reservoir temperature 6 mo Cum Water (bbl) First Produc&on Date Proppant Size Reservoir thickness 9 mo Cum Water (bbl) Total Depth (TD) Proppant Concentra&on (lbs/gal) 12 mo Cum Water (bbl) True Ver&cal Depth (TVD) Proppant per Stage (lbs/stage) 3 mo Cum BOE (bbl) Re Frac (Y/N) Proppant per Foot (lbs/c) 6 mo Cum BOE (bbl) Hole Conddi&on Total Frac Fluid pumped (gal) 9 mo Cum BOE (bbl) Annular Isola&on Fluid per Stage (gal/stage) 12 mo Cum BOE (bbl) Internal/Mechanical Isola&on Frac Fluid per foot (gal/c) Screen Out (Y/N) Slickwater pumped (gal)
Slickwater per stage (gal/stage) Linear Gel Pumped (gal) Linear Gel per Stage (gal/stage) X-‐Link Pumped (gal) X-‐Link per Stage (gal/stage)
16!© Copyright 2000-2013 TIBCO Software Inc.
Analytic Components at Fingertips
17!© Copyright 2000-2013 TIBCO Software Inc.
Analytic Components at Fingertips
18!© Copyright 2000-2014 TIBCO Software Inc.
Completions Analysis
• Business Opportuni)es • Comple&ons op&miza&on by well • Produc&on predic&on for new wells • Sand-‐out op&miza&on
• Analysis and Data • Subsurface (e.g. Spectra) • Loca&on • Comple&ons • Produc&on
• Value and Financial Impact • Op&mal comple&ons • Asset valua&on
Op5mize Comple5ons – Loca5on, Subsurface
19!© Copyright 2000-2014 TIBCO Software Inc.
Completions Analysis
• Business Opportuni)es • Comple&ons op&miza&on by well • Produc&on predic&on for new wells • Iden&fy factors driving produc&on vs expected produc&on e.g. operator
• Analysis and Data • Subsurface • Loca&on • Comple&ons • Produc&on
• Value and Financial Impact • Op&mal comple&ons • Opera&ons management • Asset valua&on & “where to drill”
Op5mize Comple5ons – Loca5on, Subsurface
20!
Data Mashup, EDA and Features
• Handling Missing Data • G&G: Interpola&on from Spa&al Smoothing
• Model interpola&on – predict flow from pressure and vice-‐versa
• Sensors: interpolate across &me from Historian
• Features (X variables) • G&G • Well aqributes
• Response variables • Produc&on: 30, 60, 90 day cum • Produc&on: EUR (threshold via &me or produc&on)
• Sensor readings
© Copyright 2000-2014 TIBCO Software Inc.
Spa5al Predic5on
21!
Production Forecasting
Forecast Produc5on – Set Expected Produc5on for Wells
© Copyright 2000-2014 TIBCO Software Inc.
• Business Opportuni)es • Asset valua&on • Forecasts for surveillance benchmarks
• Analysis and Data • Produc&on analysis • Well-‐level data
• Value and Financial Impact • EUR es&mates and Decline /Type Curves
• Produc&on monitoring
22!© Copyright 2000-2014 TIBCO Software Inc.
Leveraging TERR
TIBCO Spo@ire Sta)s)cs Services • Enables advanced analy&cs for SpoAire on web/cloud • Managed pools of sta&s&cal engines - TERR, R, S+, SAS®, MATLAB®
• Load balancing, queuing, failover, paralleliza&on • High-‐level APIs for loose integra&on • Central management of analy&cs, packages
Embeddable TERR Engine • Custom (&ght) integra&on, batch, exis&ng grids, etc. • Faster than R, more robust, beqer memory management, fully
supported • Low-‐level APIs for &ght integra&on • Integrated into TIBCO products: CEP (Streambase), Cloud
Compute
S+
R
TERR
SAS®
MATLAB®
23!© Copyright 2000-2014 TIBCO Software Inc.
Supported Data Sources
In-‐Memory, In-‐Database and Data-‐On-‐Demand • Apache Hadoop/Hive • Cloudera Hive CDH4 • Cloudera Impala CDH4, 0.6, 1.2.2, 1.2.3 • Composite Informa&on Server 6.1, 6.2 • Hortonworks Data PlaAorm 1.3, 2.0 • HP Ver&ca 5.0, 6.0, 6.1 • IBM DB2 LUW 8, 9, 9.5 • IBM Informix 9.4 • IBM Netezza 5, 6, 7 • JDBC • Microsoc SQL Server 2000, 2005, 2008, 2012 • Oracle MySQL 4.1, 5.1, 5.5, 5.6 • Oracle and Oracle Exadata (Oracle 9i, 10g, 11gR1 and R2, RAC) • Pivotal Greenplum 3.3, 4.1, 4.2 • Pivotal HAWQ • PostgreSQL 8.4, 9.0, 9.1, 9.2 • SAP HANA SPS5, SPS6 • SAP Sybase 12.5, 15, 15.5 • SAP Sybase IQ 15 • Teradata 12.00.12, 13.00, 13.10, 14.00, 14.10 • Teradata Aster 5.0, 5.11
In-‐Database only • Microsoc SQL Server Analysis Services 2008, 2012 • Oracle Essbase 9.3, 11.1 • SAP NetWeaver Business Warehouse 7.0.1 SP10, 7.3
In-‐Memory and Data-‐On-‐Demand • Aurea Sonic 7.5 • Oracle E-‐Business Suite 11.5.8, 11.5.10 • Oracle Siebel 7.7, 7.8, 8.0 • Salesforce.com • SAP R/3 4.7, mySAP 5.0, 6.0 • TIBCO Ac&veMatrix BusinessWorks™ • TIBCO Ac&veSpaces • TIBCO StreamBase LiveView
In-‐Memory Only • ADO.NET • Comma-‐Separated Values (.csv) • ESRI Shape Files (.shp) • Microsoc Access Databases (.mdb, .mde) • Microsoc Excel Workbooks (.xls, .xlsx, .xlsm) • ODBC • SAS Data Files (.sas7bdat, .sd2) • SpoAire DecisionSite Files (.sfs) • SpoAire Text Data Format (.stdf) • SpoAire Binary Data Format (.sbdf) • Text (.txt) • TIBCO Formvine • Universal Data Link (.udl)
24!© Copyright 2000-2014 TIBCO Software Inc.
TIBCO StreamBase and TERR
• Event analy)cs • Produc&on: Pump and
Compressor Monitoring • Drilling: kick detec&on • Produc&on: Flow
monitoring
• Apply predic)ve models in real-‐)me decision making
• CEP with seamless advanced analy&cs
• Rapidly develop and iterate models in produc)on
25!
Streaming Data through TERR Rules/Models
1. Rules & Models pushed from SpoAire/TERR
2. Data streams into Streambase
3. Streambase TERR calcula&ons on data
4. SpoAire RCA on trigger Other no&fica&ons available
© Copyright 2000-2014 TIBCO Software Inc.
26!© Copyright 2000-2013 TIBCO Software Inc.
Wrap Up h
• Spo@ire and R / TERR • Rich Integra&on : Expressions, Data Func&ons, Menus • Central Management of Ar&facts : SpoAire Library • Model Export • Enterprise Deployment : SpoAire Business Author, Consumer (Web) • Enterprise Deployment : Sta&s&cs Services, CEP (Streambase)
• Energy Analy)cs Case Studies
• Produc&on Forecas&ng • Produc&on Op&miza&on
Increase
Productivity
Grow Revenue
Value
Reduce Risk
ROI
27!© Copyright 2000-2013 TIBCO Software Inc.
TIBCO Spotfire Analytics
28!© Copyright 2000-2013 TIBCO Software Inc.
Documentation
29!© Copyright 2000-2013 TIBCO Software Inc.
Spotfire Resources
Training site: learn.spoAire.&bco.com Demo gallery: spoAire.&bco.com/demos
30!© Copyright 2000-2013 TIBCO Software Inc.
Spotfire Resources
© Copyright 2000-2013 TIBCO Software Inc.
Thank you! Michael O’Connell, PhD Chief Data Scien&st TIBCO moconnell@&bco.com hqp://about.me/moconnell +1-‐919-‐7401560
First to Insight, First to Action
© Copyright 2000-2013 TIBCO Software Inc. 31!
Data Mashups
Exploratory Data Analysis Features SpoQire
Dashboards Model & Predict
Deploy Champion Model
Test & Learn
SAP, IBM
Real-time Historian
Production
Well
Filter
Map
Merge
Shape
Geology
Geoformation
Improve Guided -‐-‐-‐-‐-‐-‐-‐-‐ Deploy -‐-‐-‐-‐-‐-‐-‐-‐ In-‐Line Explore Data
Aggregate
Prepare Data Business Case
Increase Productivity
Grow Revenue
Ensemble Forest
Regression Additive Models Completions
Visualize
Geo Location
Production
Challenger Models
At Rest
In Motion
Value Theses
Reduce Risk
ROI
G&G
Dashboard Updates
Data a Insight a Action
© Copyright 2000-2014 TIBCO Software Inc. 32!
top related