a stock prediction system using open source software · 2017-12-14 · a stock prediction system...

40

Upload: others

Post on 22-May-2020

3 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito
Page 2: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito

A Stock Prediction System using open-source software

Fred Melo

[email protected]

@fredmelo_br

William Markito [email protected]

@william_markito

Page 3: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito
Page 4: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito

It's all about DATA

Data Sources Look for patterns

Prediction

Page 5: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito
Page 6: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito
Page 7: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito

Machine Learning is the answer

Neural Networks

Clustering Genetic Algorithms

Page 8: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito

Train with historical dataset

Apply model to the new input

Applying Machine Learning

Page 9: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito

Hard to add new data sources

Why?

Hard to scale

Why so hard?

Hard to make it real-time

Page 10: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito

HDFS

Data Lake

Store Analytics

Hard to change Labor intensive

Inefficient

No real-time information ETL based Data-source specific

Traditional models are reactive and static

Page 11: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito

HDFS Data Lake Expert System /

Machine Learning

In-Memory Real-Time Data

Continuous Learning Continuous Improvement

Continuous Adapting

Data Stream Pipeline

Multiple Data Sources Real-Time Processing Store Everything

Stream-based, real-time closed-loop analytics are needed

Page 12: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito

Info

Analysis

Look at past trends (for similar input)

Evaluate current input

Score / Predict

Neural Network

How can it be addressed?

Page 13: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito

Info

Analysis

Filter

[ json ]

Neural Network

How can it be addressed?

Page 14: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito

Info

Analysis

Filter Enrich Neural Network

How can it be addressed?

Page 15: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito

Info

Analysis

Neural Network Filter Enrich Transform

How can it be addressed?

Page 16: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito

Info

Analysis

Filter Enrich Transform

Neural Network

How can it be addressed?

Page 17: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito

Info

Analysis

Filter Enrich Transform

Transform

Neural Network

How can it be addressed?

Page 18: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito

Neural Network

In-Memory Data Grid Real-time scoring

How can it be addressed?

Train

Page 19: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito

Neural Network

In-Memory Data Grid

Front-end

Update Push

How can it be addressed?

Page 20: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito

Ingest Transform Sink SpringXD

Store / Analyze

Fast Data

Distributed Computing

Predict / Machine Learning

Other Sources and Destinations

JMS

Streaming real-time analytics architecture

Page 21: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito

Transform Sink

SpringXD

Extensible Open-Source Fault-Tolerant Horizontally Scalable

HTTP

Machine Learning

Fast Data

Filter

Predict Sink HTTP

Split

Dashboard

Push

Demo Architecture

Page 22: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito

SpringXD

shell - R

Transformer

geode-json client

geode-json client

http-client

http-server

obj-to-json

splitter

splitter

Simulator

tap

Page 23: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito

SpringXD

INGEST / SINK PROCESS ANALYZE

•  Little or no coding required

•  Dozens of built-in connectors

•  Seamless integration with Kafka, Sqoop

•  Create new connectors easily using Spring

•  Call Spark, Reactor or RxJava

•  Built-in configurable filtering, splitting and transformation

•  Out-of-box configurable jobs for batch processing

•  Import and invoke PMML jobs easily

•  Call Python, R, Madlib and other tools

•  Built-in configurable counters and gauges

Data Stream Pipelining

Page 24: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito

SpringXD

XD Nodes XD Nodes XD Nodes XD Nodes

Ingest

SpringXD

Split Filter Transform Sink

XD admin

XD Nodes

Ingest Split Filter Transform Sink

Stream Deployment

Messaging

Scale-Out and HA Architecture

Page 25: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito

Transform Sink

SpringXD

Extensible Open-Source Fault-Tolerant Horizontally Scalable

HTTP

Machine Learning

Fast Data

Filter

Predict Sink HTTP

Split

Dashboard

Push

Demo Architecture

Page 26: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito

Geode client-server architecture

Page 27: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito

Partitioned Regions

Page 28: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito

Event handling

Page 29: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito

Transform Sink

SpringXD

Extensible Open-Source Fault-Tolerant Horizontally Scalable

HTTP

Machine Learning

Fast Data

Filter

Predict Sink HTTP

Split

Dashboard

Push

Demo Architecture

Page 30: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito

Neural Networks

Page 31: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito

Neural Networks

Page 32: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito

medium avg (x+1)

relative strength (x)

medium avg (x)

price(x)

Neural Network

Page 33: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito

Neural Network

Page 34: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito

Transform Sink

SpringXD

Extensible Open-Source Fault-Tolerant Horizontally Scalable

HTTP

Machine Learning

Fast Data

Filter

Predict Sink HTTP

Split

Dashboard

Push

Demo Architecture

Page 35: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito
Page 36: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito

Demo Time

Page 37: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito

SpringXD

shell - R

Transformer

geode-json client

geode-json client

http-client

http-server

obj-to-json

splitter

splitter

Simulator

tap

Page 38: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito

SpringXD

http://projectgeode.org http://projects.spring.io/spring-xd http://www.r-project.org

Page 39: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito
Page 40: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito

Follow-up: In-Memory Unconference

"A place for all things in-memory: projects, people, ideas, roadmaps, discussions."Location: Hill Country A/B”

Weds 4:15pm - 6pm. (after this talk)

The demo code is on GitHub! @fredmelo_br @william_markito