rdbms to graphs

(RDBMS) -[:TO]->(Graph)An Introduction To Graphs and Neo4j

William Lyon @lyonwj

William LyonDeveloper Relations Engineer @neo4j

[email protected] @lyonwj lyonwj.com

mailto:[email protected]?subject=

http://lyonwj.com

AGENDA• Use Cases • SQL Pains • Building a Neo4j Application • Moving from RDBMS -> Graph Models

• Walk through an Example • Creating Data in Graphs • Querying Data

The Traditional Approach Towards Data

SYSTEMS OF RECORD

RDBMS

SYSTEMS OF RECORD

HR-tools Supply Payments Logistics CRM Support

TRADITIONAL DATA STRUCTURE

RDBMS RDBMS RDBMSRDBMS RDBMS RDBMS

SHIFT TOWARDS SYSTEMS OF ENGAGEMENT

Users Engaging With DevicesUsers Engaging With Users Devices Engaging With Devices

SYSTEMS OF ENGAGEMENT

SHIFT TOWARDS SYSTEMS OF ENGAGEMENT

You are here!

Data volume

SYSTEMS OF RECORDRelational Database Model

StructuredPre-computed

Based on rigid rules

SYSTEMS OF ENGAGEMENTNoSQL Database Model

Highly FlexibleReal-Time QueriesHighly Contextual

SYSTEMS OF RECORD

SYSTEMS OF ENGAGEMENT

This is data modeled as a graph!

WHAT IS A GRAPH?

A Graph Is

NODE

NODE

NODE

RELATIONSHIP

RELATIONSHIP

RELATIONSHIP

A Graph Is Connected Data

ROAD

TRAFFIC

LIGHTS

HA

S

AVAILABLE

HOTEL

ROOMS

AVAILABLE

CREATING COMPETITIVE ADVANTAGE FROM THE USE OF DATA

ACCOUNT HOLDER 2

ACCOUNT HOLDER 1

ACCOUNT HOLDER 3

CREDIT CARD

BANKACCOUNT

BANKACCOUNT

BANKACCOUNT

ADDRESS

PHONE NUMBER

PHONE NUMBER

SSN 2

UNSECURE LOAN

SSN 2

UNSECURE LOAN

CREDIT CARD

IntuitivnessSpeedAgility

IntuitivenessSpeedAgility

Intuitiveness

Speed

“We found Neo4j to be literally thousands of times faster than our prior MySQL solution, with queries that require

10-100 times less code. Today, Neo4j provides eBay with functionality that was previously impossible.”

- Volker Pacher, Senior Developer

“Minutes to milliseconds” performance Queries up to 1000x faster than RDBMS or other NoSQL

A Naturally Adaptive Model

A Query Language Designed for Connectedness

+

=Agility

CypherTypical Complex SQL Join The Same Query using Cypher

MATCH (boss)-[:MANAGES*0..3]->(sub), (sub)-[:MANAGES*1..3]->(report)WHERE boss.name = “John Doe”RETURN sub.name AS Subordinate, count(report) AS Total

Project ImpactLess time writing queries

Less time debugging queries

Code that’s easier to read

NEO4j USE CASESReal Time Recommendations

Master Data Management

Fraud Detection

Identity & Access Management

Graph Based Search

Network & IT-Operations



Fraud Detection


Graph Based Search


GRAPH THINKING: Real Time Recommendations

VIEW

ED

VIEWED

BOUG

HT

VIEWED BOUGHT

BOUGHT

BO

UG

HT

BOUG

HT

“As the current market leader in graph databases, and with enterprise features for scalability and availability, Neo4j is the right choice to meet our demands.” Marcos Wada

Software Developer, Walmart



Fraud Detection


Graph Based Search




Fraud Detection


Graph Based Search


GRAPH THINKING: Master Data Management

MANAGES

MANAGES

LEADS

REGION

MANAGES

MANAGES

REGION

LEADS

LEADS

COLL

ABO

RATE

S

Neo4j is the heart of Cisco HMP: used for governance and single source of truth and a one-stop shop for all of Cisco’s hierarchies.



Fraud Detection


Graph Based Search




Fraud Detection


Graph Based Search


GRAPH THINKING: Master Data Management

Solu%onSupportCase

SupportCase

KnowledgeBaseAr%cle

Message

KnowledgeBaseAr%cle

KnowledgeBaseAr%cle

Neo4j is the heart of Cisco’s Helpdesk Solution too.



Fraud Detection


Graph Based Search


GRAPH THINKING: Fraud Detection

OPENED_ACCOUNT

HAS IS_ISSUED

HAS

LIVES LIVES

IS_ISSUED

OPE

NED_

ACCO

UNT

“Graph databases offer new methods of uncovering fraud rings and other sophisticated scams with a high-level of accuracy, and are capable of stopping advanced fraud scenarios in real-time.”

Gorka SadowskiCyber Security Expert



Fraud Detection


Graph Based Search


GRAPH THINKING: Graph Based Search



Fraud Detection


Graph Based Search


PUBLISH

INCLUDE

INCLUDE

CREATE

CAPT

URE

IN

INSO

URCE

USES

USES

IN

IN

USES

SOURCE SOURCE

Uses Neo4j to manage the digital assets inside of its next generation in-flight entertainment system.



Fraud Detection


Graph Based Search




Fraud Detection


Graph Based Search


BROWSESCO

NNEC

TS

BRIDGES

ROUTES

POW

ERSROUTES

POWERSPOWERS

HOSTS

QUERIES

GRAPH THINKING: Network & IT-Operations

Uses Neo4j for network topology analysis for big telco service providers



Fraud Detection


Graph Based Search


GRAPH THINKING: Identity And Access Management



Fraud Detection


Graph Based Search


TRUSTS

TRUSTS

ID

ID

AUTHENTICATES

AUTH

ENTI

CATE

S

OWNS

OWNSC

AN

_REA

D

UBS was the recipient of the 2014 Graphie Award for “Best Identify And Access Management App”



Fraud Detection


Graph Based Search


Neo4j Adoption by Selected VerticalsSOFTWARE FINANCIAL

SERVICES RETAIL MEDIA & BROADCASTING

SOCIAL NETWORKS TELECOM HEALTHCARE

SQL

Day in the Life of a RDBMS Developer

SELECT p.name, c.country, c.leader, p.hair, u.name, u.pres, u.stateFROM people p LEFT JOIN country c ON c.ID=p.country LEFT JOIN uni u ON p.uni=u.idWHERE u.state=‘CT’

JOIN

JOIN

JOIN

JOIN

JOIN

JOIN

JOIN

JOIN

JOIN

JOIN

JOIN

JOIN

JOIN

JOIN

JOIN

JOIN

JOIN

JOIN

JOIN

JOIN

JOIN

JOIN

• Complex to model and store relationships • Performance degrades with increases in data • Queries get long and complex • Maintenance is painful

SQL Pains

• Easy to model and store relationships • Performance of relationship traversal remains constant with

growth in data size • Queries are shortened and more readable • Adding additional properties and relationships can be done on

the fly - no migrations

Graph Gains

Relational Versus Graph Models

Relational Model Graph Model

KNOWS

KNOWS

KNOWS

ANDREAS

TOBIAS

MICA

DELIA

Person FriendPerson-Friend

ANDREASDELIA

TOBIAS

MICA

What does this Graph look like?

CYPHER

Ann DanLoves

Property Graph Model

CREATE (:Person { name:“Dan”} ) - [:LOVES]-> (:Person { name:“Ann”} )

LOVES

LABEL PROPERTY

NODE NODE

LABEL PROPERTY

MATCH (p:Person)-[:WENT_TO]->(u:Uni), (p)-[:LIVES_IN]->(c:Country), (u)-[:LED_BY]->(l:Leader), (u)-[:LOCATED_IN]->(s:State)WHERE s.abbr = ‘CT’RETURN p.name, c.country, c.leader, p.hair, u.name, l.name, s.abbr

How do you use Neo4j?

CREATE MODEL

+

LOAD DATA QUERY DATA

How do you use Neo4j?

Language Drivers

User Defined Procedures

Architectural Options

DataStorageandBusinessRulesExecu5on

DataMiningandAggrega5on

Applica'on

GraphDatabaseCluster

Neo4j Neo4j Neo4j

AdHocAnalysis

BulkAnaly'cInfrastructureHadoop,EDW…

DataScien'st

EndUser

DatabasesRela5onalNoSQLHadoop

RDBMS to Graph Options

MIGRATEALLDATA

MIGRATESUBSET

DUPLICATESUBSET

Non-GraphQueries GraphQueries

GraphQueriesNon-GraphQueries

AllQueries

Rela3onalDatabase

GraphDatabase

Application

Application

Application

NonGraphData

AllData

FROM RDBMS TO GRAPHS

Northwind

Northwind - the canonical RDBMS Example

( )-[:TO]->(Graph)

( )-[:IS_BETTER_AS]->(Graph)

Starting with the ER Diagram

Locate the Foreign Keys

Drop the Foreign Keys

Find the JOIN Tables

(Simple) JOIN Tables Become Relationships

Attributed JOIN Tables -> Relationships with Properties

Querying a Subset Today

As a Graph

QUERYING THE GRAPH

using openCypher

Property Graph Model

CREATE(:Employee{firstName:“Steven”})-[:REPORTS_TO]->(:Employee{firstName:“Andrew”})

REPORTS_TO Steven Andrew

LABEL PROPERTY

NODE NODE

LABEL PROPERTY

Who do people report to?MATCH (e:Employee)<-[:REPORTS_TO]-(sub:Employee)RETURN *

Who do people report to?

Who do people report to?MATCH (e:Employee)<-[:REPORTS_TO]-(sub:Employee)RETURN e.employeeID AS managerID, e.firstName AS managerName, sub.employeeID AS employeeID, sub.firstName AS employeeName;

Who do people report to?

Who does Robert report to?

MATCH p=(e:Employee)<-[:REPORTS_TO]-(sub:Employee)WHERE sub.firstName = ‘Robert’RETURN p

Who does Robert report to?

What is Robert’s reporting chain?

MATCH p=(e:Employee)<-[:REPORTS_TO*]-(sub:Employee)WHERE sub.firstName = ‘Robert’RETURN p

What is Robert’s reporting chain?

Who’s the Big Boss?MATCH (e:Employee)WHERE NOT (e)-[:REPORTS_TO]->()RETURN e.firstName as bigBoss

Who’s the Big Boss?

Product Cross-SellingMATCH (choc:Product {productName: 'Chocolade'}) <-[:INCLUDES]-(:Order)<-[:SOLD]-(employee), (employee)-[:SOLD]->(o2)-[:INCLUDES]->(other:Product)RETURN employee.firstName, other.productName, COUNT(DISTINCT o2) as countORDER BY count DESCLIMIT 5;

Product Cross-Selling

https://neo4j.com/graphacademy/online-training/

https://neo4j.com/graphacademy/online-training/

POWERING AN APP

http://network.graphdemos.com/

Simple App

Simple Python Code

LOADING OUR DATA

CSV files for Northwind

3 Steps to Creating the Graph

IMPORT NODES CREATE INDEXES IMPORT RELATIONSHIPS

Importing Nodes// Create customersUSING PERIODIC COMMITLOAD CSV WITH HEADERS FROM "https://raw.githubusercontent.com/neo4j-contrib/developer-resources/gh-pages/data/northwind/customers.csv" AS rowCREATE (:Customer {companyName: row.CompanyName, customerID: row.CustomerID, fax: row.Fax, phone: row.Phone});

// Create productsUSING PERIODIC COMMITLOAD CSV WITH HEADERS FROM "https://raw.githubusercontent.com/neo4j-contrib/developer-resources/gh-pages/data/northwind/products.csv" AS rowCREATE (:Product {productName: row.ProductName, productID: row.ProductID, unitPrice: toFloat(row.UnitPrice)});

Importing Nodes// Create suppliersUSING PERIODIC COMMITLOAD CSV WITH HEADERS FROM "https://raw.githubusercontent.com/neo4j-contrib/developer-resources/gh-pages/data/northwind/suppliers.csv" AS rowCREATE (:Supplier {companyName: row.CompanyName, supplierID: row.SupplierID});

// Create employeesUSING PERIODIC COMMITLOAD CSV WITH HEADERS FROM "https://raw.githubusercontent.com/neo4j-contrib/developer-resources/gh-pages/data/northwind/employees.csv" AS rowCREATE (:Employee {employeeID:row.EmployeeID, firstName: row.FirstName, lastName: row.LastName, title: row.Title});

Importing Nodes// Create categoriesUSING PERIODIC COMMITLOAD CSV WITH HEADERS FROM "https://raw.githubusercontent.com/neo4j-contrib/developer-resources/gh-pages/data/northwind/categories.csv" AS rowCREATE (:Category {categoryID: row.CategoryID, categoryName: row.CategoryName, description: row.Description});

// Create ordersUSING PERIODIC COMMITLOAD CSV WITH HEADERS FROM "https://raw.githubusercontent.com/neo4j-contrib/developer-resources/gh-pages/data/northwind/orders.csv" AS rowMERGE (order:Order {orderID: row.OrderID}) ON CREATE SET order.shipName = row.ShipName;

Creating IndexesCREATE INDEX ON :Product(productID);CREATE INDEX ON :Product(productName);CREATE INDEX ON :Category(categoryID);CREATE INDEX ON :Employee(employeeID);CREATE INDEX ON :Supplier(supplierID);CREATE INDEX ON :Customer(customerID);CREATE INDEX ON :Customer(customerName);

Creating RelationshipsUSING PERIODIC COMMITLOAD CSV WITH HEADERS FROM "https://raw.githubusercontent.com/neo4j-contrib/developer-resources/gh-pages/data/northwind/orders.csv" AS rowMATCH (order:Order {orderID: row.OrderID})MATCH (customer:Customer {customerID: row.CustomerID})MERGE (customer)-[:PURCHASED]->(order);

USING PERIODIC COMMITLOAD CSV WITH HEADERS FROM "https://raw.githubusercontent.com/neo4j-contrib/developer-resources/gh-pages/data/northwind/products.csv" AS rowMATCH (product:Product {productID: row.ProductID})MATCH (supplier:Supplier {supplierID: row.SupplierID})MERGE (supplier)-[:SUPPLIES]->(product);

Creating RelationshipsUSING PERIODIC COMMITLOAD CSV WITH HEADERS FROM "https://raw.githubusercontent.com/neo4j-contrib/developer-resources/gh-pages/data/northwind/orders.csv" AS rowMATCH (order:Order {orderID: row.OrderID})MATCH (product:Product {productID: row.ProductID})MERGE (order)-[pu:INCLUDES]->(product)ON CREATE SET pu.unitPrice = toFloat(row.UnitPrice), pu.quantity = toFloat(row.Quantity);

USING PERIODIC COMMITLOAD CSV WITH HEADERS FROM "https://raw.githubusercontent.com/neo4j-contrib/developer-resources/gh-pages/data/northwind/orders.csv" AS rowMATCH (order:Order {orderID: row.OrderID})MATCH (employee:Employee {employeeID: row.EmployeeID})MERGE (employee)-[:SOLD]->(order);

Creating RelationshipsUSING PERIODIC COMMITLOAD CSV WITH HEADERS FROM "https://raw.githubusercontent.com/neo4j-contrib/developer-resources/gh-pages/data/northwind/products.csv" AS rowMATCH (product:Product {productID: row.ProductID})MATCH (category:Category {categoryID: row.CategoryID})MERGE (product)-[:PART_OF]->(category);

USING PERIODIC COMMITLOAD CSV WITH HEADERS FROM "https://raw.githubusercontent.com/neo4j-contrib/developer-resources/gh-pages/data/northwind/employees.csv" AS rowMATCH (employee:Employee {employeeID: row.EmployeeID})MATCH (manager:Employee {employeeID: row.ReportsTo})MERGE (employee)-[:REPORTS_TO]->(manager);

High Performance LOADingneo4j-import

4.58 million thingsand their relationships…

Loads in 100 seconds!

JDBCapoc.load.jdbc

https://github.com/neo4j-contrib/neo4j-apoc-procedures


User Defined Procedures

User-defined procedures are written in Java, deployed into the database, and called from Cypher.

http://neo4j.com/docs/developer-manual/current/#procedures

http://neo4j.com/docs/developer-manual/current/#procedures

Built-in Procedures

User-defined Procedures

https://github.com/neo4j-examples/neo4j-procedure-template

https://github.com/neo4j-examples/neo4j-procedure-template

Apoc Procedures



THERE’S A PROCEDURE FOR THAT



https://neo4j-contrib.github.io/neo4j-apoc-procedures/#_loading_data_from_web_apis_json_xml_csv

https://neo4j-contrib.github.io/neo4j-apoc-procedures/#_loading_data_from_web_apis_json_xml_csv

https://github.com/neo4j-contrib/neo4j-apoc-procedures/blob/master/readme.adoc

https://github.com/neo4j-contrib/neo4j-apoc-procedures/blob/master/readme.adoc

Northwind

Graph Based Recommendations

JDBCNeo4j-JDBC driver

https://neo4j.com/blog/icij-neo4j-unravel-panama-papers/

https://neo4j.com/blog/official-neo4j-jdbc-driver-3-0/

https://neo4j.com/blog/official-neo4j-jdbc-driver-3-0/

WRAPPING UP

Resources

http://neo4j.com/developer/

Neo4j Developer Resources

http://neo4j.com/developer/

http://graphdatabases.com/

http://graphdatabases.com/

http://neo4j.com/download

http://neo4j.com/download

:play northwind graph

THANK YOU!

rdbms to graphs

Technology