paypal real time analytics

Open Source Real Time BI using Storm, Hadoop, Titan, Druid & D3

Anil MadanSr. Director Engineering, PayPal

$1 in every $6Spent on e-commerce is

spent through PayPal.*

*Source: Morgan Stanley, “eCommerce Disruption: A Global Theme,” January 6, 2013, p.21.

Creating Tomorrow’s

Mobile PaymentExperiences

25 countries with live PayPal fingerprint authenticationon Samsung devices.

Helping DevelopersInnovate & Monetize

New Mobile Apps

Braintree launches its new API, including Pay with PayPal.

PayPal Now Available in 203 Markets10 new markets added in the second quarter,

making PayPal available to 80 million new internet users.

Paraguay

Côte d’Ivoire

Nigeria

Monaco

Belarus

Moldova

Cameroon

Zimbabwe

Montenegro

Macedonia

We need to better understand our customers…

Where do prospects sign up for accounts?

How do prospective customers learn about

PayPal?

Acquisition Activation AdoptionAwareness

How can we help them

use PayPal even more?

How can we help them to

complete their 1st

payment?

Business Problem

How we solved it…

Direct/Home Page

ProductExperiences

Search EngineMarketing

TransactionEmails

Tracking MetadataTool

Taxonomy

Tracking Event Service

Tracking Servers

Tag Catalog

Tracking Validation Service

Marketing

Segmentation

Real Time Systems

Experimentation

Metadata

AttributionExploratory Analytics Predictive Analytics

Big Data

Mobile

Reporting & Visualization

Pathing Store

Logical View

Client Side Events

Page Performance

Events

Server Side Events

Collection Service

Sessionization

Behavioral Metrics

Marketing Metrics

Metadata Instrumentation Collection Processing Analytics

Performance Metrics

Operational Metrics (OpenTSDB)

DRUIDMetrics Store

Real Time Event

Metrics

Metadata –Logical Entity Model

COMPONENTS

PAGETEMPLATE

Metadata – Logical Event Model

ImpressionEvent

TrackingEvent

ReactionEvent

ComponentImpression

AdImpression

ClickEvent

Click-ThroughEvent

Mouse-overEvent

EntryEvent

ExitEvent

OutcomeEvent

PageImpression

Client PageImpression

Server PageImpression

Metadata - Self-Service Management Workflow…

DATA PIPELINEProcessing Analysis &

VisualizationClientSide

Metadata

Performance

Collection

Metrics

RESTSpout

Bot flagging

AggregationSessionization

RESTProxy

ServerSide

Geo Enrichment

Bolt Reporting

Data Stores

Apache Titan

DevelopersProduct Owners

Customers

Meta data

Reporting Consumers

Metadata Service

Druid Architecture

• Open-source• Distributed • Real-time • Highly-Available Data store• Column-oriented• Approximate or Exact

• Ingest data and buffer events in memory

• Incremental indexing• Query data as soon as it is

ingested• Periodically persist collected

events to disk • Combine multiple disk indexes

to create immutable ‘segments’• Log-structured merge-tree

Real Time Nodes

Druid Architecture

Historical Nodes

• Load immutable read-optimized data from deep storage

• Memory mapped storage engine• Caches segments • Supports tiered storage

Druid Architecture

Druid Systems Overview

"type": "doubleSum", "name": "pageviews", "fieldName": "PV" }, { "type": "doubleSum", "name": "bounces", "fieldName": "bnc" },.... { "type": "hyperUnique", "name": "unique_visits", "fieldName": "user_session_guid" }, { "type": "hyperUnique", "name": "unique_visitors", "fieldName": "user_guid" }

2014/06/11/10", "filter": "part-", "parser": { "type": "string", "timestampSpec": { "column": "timestamp", "format": "auto" }, "data": { "format": "json", "dimensions": [ "timestamp", "USER_GUID", "USER_SESSION_GUID", "PAGE_GROUP", "PAGE_NAME", "PAGEGROUP_LINK_NAME", "PAGE_LINK_NAME",

Metrics & Dimensions

Standard

Metrics

Estimated

Metrics

HyperLogLog

Dimensions

Sessionization

Visitor ID

SessionID

Timestamp EventPayload

V1 S1 2014-10-16 05:12

V2 S2 2014-10-16 05:14

V1 S1 2014-10-16 05:15

V1 S1 2014-10-16 05:20

V2 S2 2014-10-16 05:21

V1 S3 2014-10-16 05:25

… … … …

Visitor ID

SessionID

Payload

V1 S1 sf, mac, {flash, quicktime}, {ca, usa}, 480 secs,….

E4V2 S2 ff, win, {acrobat, mediaplayer}.

{wb, in}, 420 secs…..E2

V1 S3 sf, mac, {quicktime, java}, {on, ca}, 60 secs

Events VisitContainer

Druid Storage – Columns & Dictionaries Timestamp (Hr) Sessi

Country OS UserAgent

Page Name

2014-10-16 05 S1 US MAC SF LoginAccountOverview

2014-10-16 05 S2 DE WIN IE LoginPaymentReviewAccountHistory

2014-10-16 05 S3 US LNX FF LoginPaymentReview

Checkout

2014-10-16 05 S4 UK LNX FF LoginProfile

Checkout

2014-10-16 05 S5 DE WIN CR LoginProfile

2014-10-16 05 S6 UK MAC SF LoginAccountOverview

Checkout

Page Name

Dictionary

Login 0

AccountOverview

PaymentReview 2

AccountHistory 3

Checkout 4

Profile 5LZF

Druid Data Structure - Bitmap Indices

Herald – Self Service Analytics

Druid Metrics

Pathing

Fallout Reports

Visitor ID Current Page Next Page 1 Next Page 2 Prev Page 1 Prev Page 2

S1 A B C null null

S1 B C D A null

S1 C D X B A

S1 D X A C B

S1 X A M D C

S1 A M null X D

S1 M Null null A X

S2 A B C null Null

S2 B C D null A

S2 C D E B A

S2 D E Null C B

S2 E Null null D C

A->B->C->D->X->A->M and A->B->C->D->E Pathing

Next Page{ “queryType” : “groupBy” “dimensions” : (“current_page”, “dimensions like country, segmentation etc”} “aggregations” : [ { “type”: “count”, “name”: “next_page_count”, “fieldname” : “next_page, next_page2” }] “filter”: { “type”: “selector”, “dimension”: “current_page”, “value”: “C” }}

Previous Page{ “queryType” : “groupBy” “dimensions” : {“current_page”, “dimensions like country, segmentations etc”} “aggregations” : [ { “type”: “count”, “name”: “prev_page_count”, “fieldname” : “prev_page1, prev_page2” }] “filter”: { “type”: “selector”, “dimension”: “current_page”, “value”: “C” }}

Pathing

Fallout

• Apply them to the dictionary• Figure out the values that match• Take those bitmap indices• OR the bitmap indices together• Use the output bitmap as the filter

A->D-> X->M

“queryType” : “search” “dimensions” : { “current_page_path_count”, “dimensions like country, segmentation etc”} “filter”: { “type”: “regex”, “dimension”: “next_page_path”, “pattern”: “^A*D*X*M$” }}

A->B->C->D->X->A->M

Model View

Controller

NVD3Directives

RHerald Architecture

Herald Deployment

Login_2014101611

Country: USCount: 15

AccountOverview_2014101611

PaymentReview_

2014101611

Checkout_2014101611

Count: 8

Country: USCount: 5

Count: 7

Country: USCount: 5

Count: 5

Adhoc Graph Analytics

Name: Login_2014101611

Name: AccountOverview_2014101611

Name: PaymentReview_2014101611

Name: Checkout_2014101611

Count: 8

Country: USCount: 5

Count: 7

Country: USCount: 5

Count: 5

gremlin> g.v(‘Name’, ‘Login_2014101611').as('x’).

outE.inV.loop('x'){it.loops < 4}

{it.object.getProperty('name') ==

'Checkout_2014101611'}.path

Summary

• Problem• Understand our customer behavior• Across disparate channels & experiences

• Solution• Democratize data• Consistent standardized metadata• Disciplined instrumentation• Distributed scalable backend for adhoc & interactive analytics• Self-service BI through modern visualization tools

Questions ?

paypal real time analytics

paypal available

druid architecture opensource

14e2v1 s1

15e3v1 s1

druid d3anil madansr

druid systems overview

tiered storage

12e1v2 s2

Data & Analytics

real-time analytics with hbase

pygmalios analytics - cisco 11.1 · real-time analytics &...

real time analytics case study

real-time analytics with spark -...

real-time in-store analytics

real-time identification and analytics

real time analytics with cassandra

real-time analytics

real world analytics

demonstrating near real-time analytics with ibm db2...

accelerated real-time analytics

ibm: real-time transactional analytics

paypal phishing example. can you tell which is real? 1. 2

analytics: innovation in real-time

location analytics for real estate

backoffice - real green analytics

real-time analytics in financial

enabling real-time operational analytics

real time security analytics

streetlife's real time analytics stack