rdbms to graphs

RDBMS to Graphs Harnessing the Power of the Graph

September 2015

Ryan Boyd @ryguyrg

Agenda

•  Origins of Neo4j •  Benefits of Graphs •  Designing your Graph Model •  Query <me! •  Fi@ng Neo4j into your Enterprise Architecture •  Q&A

Neo Technology Overview

Product • Neo4j -‐ World’s leading graph database

• 150+ enterprise subscrip<on customers including over 50 of the Global 2000

Company • Neo Technology, Creator of Neo4j • 100 employees with HQ in Silicon Valley, London, Munich, Paris and Malmö

• $45M in funding

Neo4j AdopDon by Selected VerDcals FinancialServices Communications Health &

Life Sciences HR &

Recruiting Media &

Publishing SocialWeb

Industry & Logistics

Entertainment Consumer Retail Information Services Business Services

How Customers Use Neo4j Network &

Data Center Master DataManagement Social Recom–

mendations Identity

& Access Search &Discovery GEO

“Forrester es<mates that over 25% of enterprises will be using graph databases by 2017”

Neo4j Leads the Graph Database RevoluDon

“Neo4j is the current market leader in graph databases.”

“Graph analysis is possibly the single most effecDve compeDDve differenDator for organiza<ons pursuing data-‐driven opera<ons and decisions aaer the design of data capture.”

IT Market Clock for Database Management Systems, 2014 hbps://www.gartner.com/doc/2852717/it-‐market-‐clock-‐database-‐management TechRadar™: Enterprise DBMS, Q1 2014 hbp://www.forrester.com/TechRadar+Enterprise+DBMS+Q1+2014/fulltext/-‐/E-‐RES106801 Graph Databases – and Their PotenDal to Transform How We Capture Interdependencies (Enterprise Management Associates) hbp://blogs.enterprisemanagement.com/dennisdrogseth/2013/11/06/graph-‐databasesand-‐poten<al-‐transform-‐capture-‐interdependencies/

High Business Value in Data RelaDonships

Data is increasing in volume… •  New digital processes •  More online transac<ons •  New social networks •  More devices

Using Data RelaDonships unlocks value •  Real-‐<me recommenda<ons •  Fraud detec<on •  Master data management •  Network and IT opera<ons •  Iden<ty and access management •  Graph-‐based search … and is ge[ng more connected

Customers, products, processes, devices interact and relate to each other

Early adopters became industry leaders

RelaDonal DBs Can’t Handle RelaDonships Well

•  Cannot model or store data and rela>onships without complexity

•  Performance degrades with number and levels of rela<onships, and database size

•  Query complexity grows with need for JOINs •  Adding new types of data and rela>onships requires schema redesign, increasing <me to market

… making tradi<onal databases inappropriate when data rela<onships are valuable in real-‐Dme

Slow development Poor performance Low scalability Hard to maintain

Modeling as a Graph

The Whiteboard Model Is the Physical Model

CAR

name: “Dan” born: May 29, 1970 twiber: “@dan”

name: “Ann” born: Dec 5, 1975

since: Jan 10, 2011

brand: “Volvo” model: “V70”

Property Graph Model Components

Nodes •  The objects in the graph •  Can have name-‐value proper&es •  Can be labeled RelaDonships •  Relate nodes by type and direc<on •  Can have name-‐value proper&es

LOVES

LOVES

LIVES WITH PERSON PERSON

RelaDonal Versus Graph Models

RelaDonal Model Graph Model

KNOWS ANDREAS

TOBIAS

MICA

DELIA

Person Friend Person-‐Friend

ANDREAS DELIA

TOBIAS

MICA

Let’s Model!

Customer, Supplier, and Product (Master Data) Orders (AcDvity)

The Domain Model

Except…

Northwind Example!

The QuintessenDal Northwind Example!

NOT JUST ANY

(Northwind)-‐[:TO]-‐>(Graph) Building the Graph Model

Building RelaDonships in Graphs

SOLD

Employee Order Order

Locate Foreign Keys

(FKs)-‐[:BECOME]-‐>(RelaDonships) Correct DirecDons

Simple Join Tables Becomes RelaDonships

Afributed Join Tables Become RelaDonships with ProperDes

Working Subset (Today’s Exercise)

Northwind Graph Model

Querying Your Data

Basic Query: Who do people report to?

MATCH (:Employee{ firstName:“Steven”} ) -‐[:REPORTS_TO]-‐> (:Employee{ firstName:“Andrew”} )

REPORTS_TO Steven Andrew

LABEL PROPERTY

NODE NODE

LABEL PROPERTY


MATCH ! (e:Employee)<-[:REPORTS_TO]-(sub:Employee)!RETURN ! *!

Real Query from a Customer

Find all direct reports and how many people they manage,

each up to 3 levels down

(SELECT T.directReportees AS directReportees, sum(T.count) AS count FROM ( SELECT manager.pid AS directReportees, 0 AS count FROM person_reportee manager WHERE manager.pid = (SELECT id FROM person WHERE name = "fName lName") UNION SELECT manager.pid AS directReportees, count(manager.directly_manages) AS count FROM person_reportee manager WHERE manager.pid = (SELECT id FROM person WHERE name = "fName lName") GROUP BY directReportees UNION SELECT manager.pid AS directReportees, count(reportee.directly_manages) AS count FROM person_reportee manager JOIN person_reportee reportee ON manager.directly_manages = reportee.pid WHERE manager.pid = (SELECT id FROM person WHERE name = "fName lName") GROUP BY directReportees UNION SELECT manager.pid AS directReportees, count(L2Reportees.directly_manages) AS count FROM person_reportee manager JOIN person_reportee L1Reportees ON manager.directly_manages = L1Reportees.pid JOIN person_reportee L2Reportees ON L1Reportees.directly_manages = L2Reportees.pid WHERE manager.pid = (SELECT id FROM person WHERE name = "fName lName") GROUP BY directReportees ) AS T GROUP BY directReportees) UNION (SELECT T.directReportees AS directReportees, sum(T.count) AS count FROM ( SELECT manager.directly_manages AS directReportees, 0 AS count FROM person_reportee manager WHERE manager.pid = (SELECT id FROM person WHERE name = "fName lName") UNION SELECT reportee.pid AS directReportees, count(reportee.directly_manages) AS count FROM person_reportee manager

JOIN person_reportee reportee ON manager.directly_manages = reportee.pid WHERE manager.pid = (SELECT id FROM person WHERE name = "fName lName") GROUP BY directReportees UNION SELECT depth1Reportees.pid AS directReportees, count(depth2Reportees.directly_manages) AS count FROM person_reportee manager JOIN person_reportee L1Reportees ON manager.directly_manages = L1Reportees.pid JOIN person_reportee L2Reportees ON L1Reportees.directly_manages = L2Reportees.pid WHERE manager.pid = (SELECT id FROM person WHERE name = "fName lName") GROUP BY directReportees ) AS T GROUP BY directReportees) UNION (SELECT T.directReportees AS directReportees, sum(T.count) AS count OUTER UNIONS FROM( SELECT reportee.directly_manages AS directReportees, 0 AS count FROM person_reportee manager JOIN person_reportee reportee ON manager.directly_manages = reportee.pid WHERE manager.pid = (SELECT id FROM person WHERE name = "fName lName") GROUP BY directReportees count FROM person_reportee manager JOIN person_reportee L1Reportees ON manager.directly_manages = L1Reportees.pid JOIN person_reportee L2Reportees ON L1Reportees.directly_manages = L2Reportees.pid WHERE manager.pid = (SELECT id FROM person WHERE name = "fName lName") GROUP BY directReportees ) AS T GROUP BY directReportees) UNION (SELECT L2Reportees.directly_manages AS directReportees, 0 AS count FROM person_reportee manager JOIN person_reportee L1Reportees ON manager.directly_manages = L1Reportees.pid

JOIN person_reportee L2Reportees ON L1Reportees.directly_manages = L2Reportees.pid

WHERE manager.pid = (SELECT id FROM person WHERE name = "fName lName") )


MATCH (manager)-‐[:REPORTS_TO*0..3]-‐>(boss), (report)-‐[:REPORTS_TO*1..3]-‐>(manager) WHERE boss.name = “John Doe” RETURN manager.name AS Manager, count(report) AS TotalReports

Find all direct reports and how many people they manage, up to 3 levels down

Cypher Query



Cypher Query

SQL Query

MATCH (manager)-‐[:REPORTS_TO*0..3]-‐>(boss), (report)-‐[:REPORTS_TO*1..3]-‐>(manager) WHERE boss.name = “John Doe” RETURN manager.name AS Manager, count(report) AS TotalReports

MATCH (sub)-‐[:REPORTS_TO*0..3]-‐>(boss), (report)-‐[:REPORTS_TO*1..3]-‐>(sub) WHERE boss.name = “John Doe” RETURN sub.name AS Subordinate, count(report) AS Total

Express Complex Queries Easily with Cypher


Cypher Query

SQL Query

“We found Neo4j to be literally thousands of Dmes faster than our prior MySQL solu<on, with queries that require 10 to 100 Dmes less code. Today, Neo4j provides eBay with func<onality that was previously impossible.” Volker Pacher Senior Developer

Who is in Robert’s (direct, upwards) reporDng chain?

MATCH ! p=(e:Employee)<-[:REPORTS_TO*]-(sub:Employee)!WHERE! sub.firstName = ‘Robert’!RETURN ! p!

Who is in Robert’s (direct, upwards) reporDng chain?

Who’s the Big Boss?

MATCH ! p=(e:Employee)!WHERE! NOT (e)-[:REPORTS_TO]->()!RETURN ! e.firstName as bigBoss!

Who’s the Big Boss?

Product Cross-‐Sell MATCH ! (choc:Product {productName: 'Chocolade'})! <-[:PRODUCT]-(:Order)<-[:SOLD]-(employee),! (employee)-[:SOLD]->(o2)-[:PRODUCT]->(other:Product)!RETURN ! employee.firstName, other.productName, count(distinct o2) as count!ORDER BY ! count DESC!LIMIT 5;!

Product Cross-‐Sell

High Performance

Cypher vs SQL -‐ Paths

MATCH (u:User)-[:KNOWS*5..5]->(f5) WHERE u.name = 'John' RETURN count(f5) as size;

Cypher Find Size of John’s 5th degree Network

●  100k Users ●  5M

Rela<onships ●  Query took 5

min, 30s ●  Returns count of

312M Neo4j config: page-‐cache = 512m heap = 4G


SELECT count(*) FROM user, user_friend as uf1, user_friend as uf2, user_friend as uf3, user_friend as uf4, user_friend as uf5 user as f5 WHERE user.name='John' AND user.id = uf1.user_1 AND uf1.user_2 = uf2.user_1 AND uf2.user_2 = uf3.user_1 AND uf3.user_2 = uf4.user_1 AND uf4.user_2 = uf5.user_1 AND uf5.user_2 = f5.id;

SQL Find Size of John’s 5th degree Network

●  100k Users ●  5M Connec<ons ●  Query took 1hr 55 mins ●  Returns 312M

MySQL config: key_buffer = 2G join_buffer_size = 2G


SELECT count(*) FROM user, user_friend as uf1,

user_friend as uf2, user_friend as uf3, user_friend as uf4, user_friend as uf5

WHERE user.name='John' AND user.id = uf1.user_1 AND uf1.user_2 = uf2.user_1 AND uf2.user_2 = uf3.user_1 AND uf3.user_2 = uf4.user_1 AND uf4.user_2 = uf5.user_1;

SQL Op>mize: Only count on JOIN table

●  100k Users ●  5M Connec<ons ●  Query took 2 min, 30s ●  Returns count of 312M

MySQL config: key_buffer = 2G join_buffer_size = 2G


MATCH (u:User)-[:KNOWS*4..4]->(f4) WHERE u.name = 'John' RETURN sum(size((f4)-[:KNOWS]->()))

Cypher Op>mize: Only sum degree of last step

●  100k Users ●  5M

Rela<onships ●  Query takes 12

sec ●  Returns count of

312M Neo4j config: page-‐cache = 512m heap = 4G

Neo4j Clustering Architecture OpDmized for Speed & Availability at Scale

50

Performance Benefits •  No network hops within queries •  Real-‐>me opera>ons with fast and consistent response <mes

•  Cache sharding spreads cache across cluster for very large graphs

Clustering Features •  Master-‐slave replica<on with master re-‐elec>on and failover

•  Each instance has its own local cache •  Horizontal scaling & disaster recovery

Load Balancer

Neo4j Neo4j Neo4j

Ge[ng Data into Neo4j

Cypher-‐Based “LOAD CSV” Capability •  Transac<onal (ACID) writes •  Ini<al and incremental loads of up to 10 million nodes and rela<onships

Command-‐Line Bulk Loader neo4j-‐import •  For ini<al database popula<on •  For loads with 10B+ records •  Up to 1M records per second

4.58 million things and their rela<onships…

Loads in 100 seconds!

MIGRATE ALL DATA

MIGRATE GRAPH DATA

DUPLICATE GRAPH DATA

Non-‐graph data Graph data

Graph data All data

All data

RelaDonal Database

Graph Database

Applica<on

Applica<on

Applica<on

Three Ways to Load Data into Neo4j

Polyglot Persistence

Data Storage and Business Rules Execu<on

Data Mining and Aggrega<on

Neo4j Fits into Your Enterprise Environment

ApplicaDon

Graph Database Cluster

Neo4j Neo4j Neo4j

Ad Hoc Analysis

Bulk AnalyDc Infrastructure

Graph Compute Engine EDW …

Data ScienDst

End User

Databases Rela<onal NoSQL Hadoop

Neo4j + Mongo!

Users Love Neo4j

Learn the Way of the Graph Quickly and Easily

Quick Start in 1 minute

Quick Start: Plan Your Project

1

2

3

4

5

6

7

8

Learn Neo4j

Decide on Architecture

Import and Model Data

Build ApplicaDon

Test ApplicaDon

Deploy your app in as lible as 8 weeks

PROFESSIONAL SERVICES PLAN

There Are Lots of Ways to Easily Learn Neo4j

Huge Ecosystem of Graph Enthusiasts

•  1,000,000+ downloads •  20,000+ educa<on registrants •  18,000+ Meetup members •  100+ technology and service partners •  150+ enterprise subscrip<on customers including 50+ Global 2000 companies

Get Started Now

Summary of the Power of the Graph

•  Take rela<onships and connected data seriously •  Seriously easy to model •  Serious performance

•  Fits in with your Enterprise Architecture •  Easy to get started •  Fast to reap the benefits

RDBMS to Graphs Harnessing the Power of the Graph

Start of Q&A

Ryan Boyd @ryguyrg