atlhug 20150625

Cardlytics & Drill Use Case: Matching Big Data

David Kim

Principal Engineer

2015.06.25

About Cardlytics

© 2013 Cardlytics. Proprietary and Confidential. 2

•  Privately held company leveraging proprietary purchase-‐driven intelligence pla6orm to provide ac7onable insights into consumer behavior to numerous organiza7ons using consumer purchase data that we have exclusive rights to

•  Founded in 2008 by ScoA Grimes (CEO) and Lynne Laube (COO) both former execu7ves at Capital One

•  Headquartered in Atlanta, we have 320 employees with offices in NY, Chicago, San Francisco & London

•  Owns mul7ple patents and nearly 700 banking rela7onships in the US and the UK represen7ng over 100 million households and $1 trillion in yearly spend

Problem Statement

A customer (advertiser) requested analysis to provide insight into their own business and customer base in order to better understand and make better business decisions. •  Must match advertiser customers to Cardlytics customers

•  Matches must be highly confident and unique


Our Approach: Pattern Matching

time


Challenges

•  Matches must be unique

•  Matches must be highly confident

•  Limited information available to match data points (no PII)

•  Missing data points

•  Scale (Drill) »  Depending on the advertiser, data points are sparse or densely packed


Scale Issues with Dense Data Points


Scale Issues

•  60M x 40M = 2.4T potential matches evaluated




Our Environment…

SQL Server: 64 cores (32 physical), 256GB RAM, direct-attached storage w/enterprise disks

Hadoop Cluster: 10 nodes, 32 cores/node, 128GB RAM, 12 x 4TB consumer grade disks


Actual Results…

•  POC 1: 60M customer data points x 40M Cardlytics data points collected over 2 years

»  SQL Server : ~20 hours

•  POC 2:120M x 120M over 6 months

»  SQL Server: 1~2 months »  Hive: Killed after several days (estimated to take about a week) »  Drill: 17-18 hours yielding 91+B matching data points

•  POC3: 590M x 130M over 1 year »  Drill: ~17 hours to yield 1.3T matches and 72TB »  Required some tweaking and turning some secret knobs


…PROBABLY

…from the MapR Drill team

Compliments of Jacques Nadeau/Aman Sinha

•  store.format

•  store.parquet.block-size

•  planner.broadcast_threshold

•  planner.broadcast_factor

•  planner.join.row_count_estimate_factor

•  planner.enable_multiphase_agg

•  planner.enable_mux_exchange

•  exec.min_hash_table_size

•  planner.enable_hashjoin

•  select * from sys.options;


Other Nuggets

•  Drill is memory intensive

•  You will always know more about your data than Drill

•  Hadoop and Drill are great tools but doesn’t solve stupidity

•  Some of the basic principles of querying a dataset still apply

»  Intelligent batching

»  Applying filters early to work with smaller datasets »  Bringing back only the data that you need

»  Partitioning

»  Understanding the configurations and internals of your tools


"Louis, I think this is the beginning of a beautiful friendship."

Our close partnership with MapR includes… •  Semi-weekly check-ins with Drill dev team

•  Weekly check-ins with MapR product managers

•  Improving Drill with real world applications, tests, and data

•  Input to future roadmap »  Large IN-clause

»  DST support

»  Auto-partitioning

»  Windowing functions

»  Support for inserts


Grab a seat at the cool kids’ table!!

Careers @Cardlytics

http://cardlytics.com/cardlytics/?s=career

Apache Drill

https://drill.apache.org/

https://drill.apache.org/docs/

MapR

https://www.mapr.com/products/product-overview/apache-drill


Michael Fabacher, VP of Data Development [email protected] David Kim, Principal Engineer [email protected]

atlhug 20150625

Documents