graphconnect europe 2016 - enterprise data integration with a new jdbc driver for neo4j 3.0 -...
TRANSCRIPT
/ 50
LONDON, 26 April 2016
1
LORENZO SPERANZONI | @inserpio CEO @ LARUS Business Automation - GCE2016 BRONZE SPONSOR
GraphConnect Europe 2016
Explore the new COUCHBASE and JDBC
integrations for Neo4j 3.0
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
AGENDA
✦ Let’s say Hi ( “Ciao” !! )
✦ Neo4j - Couchbase Connector
✦ JDBC Driver with BOLT for Neo4j 3.0
2
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
AGENDA
✦ Let’s say Hi ( “Ciao” !! )
✦ Neo4j - Couchbase Connector
✦ JDBC Driver with BOLT for Neo4j 3.0
3
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
WELCOME AND
THANKS FOR COMING TO THIS LIGHTNING TALK
4
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
5
[:WORKS_FOR { role: ‘CEO’ }]
Venice
[:BRONZE_SPONSOR]
Lorenzo Speranzoni
LARUS Business Automation
[:BASED_IN][:CERTIFIED_ON]
[:PARTNER_OF]
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
6
… this is a typical day in Venice George Clooney Wedding
BY THE WAY … … NOT THIS ONE DURING …
#*@!#!!
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
7
2016
Neo4j Contributors
20152011
First Spikes ( with neo4j 1.x !! )
2013
LARUS’ TRIP WITH NEO4J (quite a long and cool story)
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
8
CONSULTING
[:DELIVERS]
TRAININGS
SUPPORT
[:DELIVERS]
[:DELIVERS] [:FOR]
[:ON]
[:ON]
[:ADO
PTED
]
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
AGENDA
✦ Let’s say Hi ( “Ciao” !! )
✦ Neo4j - Couchbase Connector
✦ JDBC Driver with BOLT for Neo4j 3.0
9
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
The new Neo4j - Couchbase Connector
10
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
KEY ASPECTS
✦ BI-DIRECTIONAL
✦ CONFIGURABLE
11
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
https://developer.spotify.com/web-api/console/get-album
12
CA
SE S
TUDY
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
https://developer.spotify.com/web-api/console/get-album [ RESPONSE ]
13
CA
SE S
TUDY
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
BASIC JSON 2 GRAPH CONVERSION RULES
PLEASE NOTE.
✦ Most of the following approaches come from William Lion’s (@lionwj)MongoDB - Neo4j Connector (http://neo4j.com/blog/neo4j-doc-manager-polyglot-persistence-mongodb/)
✦ Don’t miss William’s session “NoSQL Polyglot Persistence: Tools and Integrations” here at GCE2016 (4:30PM), for many more examples about managing multiple databases.
14
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
PRIMITIVE ATTRIBUTES …
15
album_type: “U2” href: “https://api.spotify.com/v1/albums/586Z…” id: “586ZRfgsIckfcKvHVcGM4V” name: “The Joshua Tree” popularity: 53 release_date: “1987-03-10” release_date_precision: “day” type: “album” uri: “spotify:album:586ZRfgsIckfcKvHVcGM4V”
ALBUM
BECOME NODE’S ATTRIBUTES
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
ARRAY OF PRIMITIVE ATTRIBUTES …
16
album_type: “U2” available_markets: [ “MX”, “US” ] href: “https://api.spotify.com/v1/albums/586Z…” id: “586ZRfgsIckfcKvHVcGM4V” name: “The Joshua Tree” popularity: 53 release_date: “1987-03-10” release_date_precision: “day” type: “album” uri: “spotify:album:586ZRfgsIckfcKvHVcGM4V”
ALBUM
STILL BECOME NODE’S ATTRIBUTES
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
OBJECT ATTRIBUTES . . .
17
href: “https://api.spotify.com/v1/artists/51B…” id: “51Blml2LZPmy7TTiAg47vQ” name: “U2” type: “artist” uri: “spotify:album:51Blml2LZPmy7TTiAg47vQ”
ARTIST
BECOME CONNECTED NODES
name: “The Joshua Tree”
ALBUM
[ :ALBUM_ARTIST ]
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
ARRAY OF OBJECTS ATTRIBUTES …
18
ARTIST
AGAIN BECOME CONNECTED NODES
ALBUM disc_number: 1 duration_ms: 337973 href: “https://api.spotify.com/v1/tracks/3zq…” id: “3zqTlJtp7KTPrNUKNahvAo” name: “Where The Streets Have No Name” uri: “spotify:album:3zqTlJtp7KTPrNUKNahvAo” track_number: 1 type: “track”
ITEM
TRACKS
ITEM
2ITEM
3ITEM
11
ITEM
4
[ :TRA
CKS_IT
EMS ]
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
Neo4j - Couchbase Connector
COMPONENTS (and a bit of noisy code)
19
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
Neo4j - Couchbase Connector Components
20
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
1. Couchbase Mutation Listener public class CouchbaseEvent {
/** * Wrapped {@link CouchbaseMessage}. */
private CouchbaseMessage message;}
public class CouchbaseEventFilter {
/** * Returns true if event is a {@link MutationMessage}. */
public static boolean accept(final CouchbaseEvent couchbaseEvent) { return couchbaseEvent.getMessage() instanceof MutationMessage; }}
21
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
1. Couchbase Mutation Listener public class CouchbaseEventHandler implements EventHandler<CouchbaseEvent> {
/** * Handles {@link CouchbaseEvent}s that come into the response RingBuffer. */ @Override
public void onEvent(final CouchbaseEvent event, final long sequence, final boolean endOfBatch) throws Exception { if (CouchbaseEventFilter.accept(event)) { MutationMessage mutationMessage = (MutationMessage) event.getMessage(); ByteBuf jsonDocument = mutationMessage.content(); DocumentTransformer<String> remoteNeo4jTransformer = new RemoteNeo4jTransformer();
remoteNeo4jTransformer.transform(mutationMessage.key(), “COUCHBASE_DOCUMENT", asString(jsonDocument)); } } }
22
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
1. Couchbase Mutation Listener public class CouchbaseMutationListener extends Thread {
/** * Builds and starts the event consumer. */
public void buildDisruptor() { Disruptor<CouchbaseEvent> disruptor = new Disruptor<CouchbaseEvent>(new CouchbaseEventFactory(), 32, Executors.newCachedThreadPool()); disruptor.handleExceptionsWith(new CouchbaseExceptionLogger()); disruptor.handleEventsWith(new CouchbaseEventHandler()); this.couchbaseEventBuffer = disruptor.start(); }
/** * Connect this dispatcher to Couchbase. */
public void connect() { this.couchbaseClusterFacade.send(new SeedNodesRequest(couchbaseNodes)).timeout(2, TimeUnit.SECONDS).toBlocking().single(); this.couchbaseClusterFacade.send(new OpenBucketRequest(this.couchbaseBucket, couchbasePassword)).timeout(2, TimeUnit.SECONDS).toBlocking().single(); }
}
23
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
2. Neo4j JSON Loader public class JsonDocument {
private String id; private String type; private String content; private JsonMappingStrategy mappingStrategy; private List<JsonObjectDescriptor> objectDescriptors;}
public enum JsonMappingStrategy {
ATTRIBUTE_BASED(“attribute”), DOMAIN_DRIVEN(“domain”), STORED_PROCEDURE(“stored procedure”),}
24
public class JsonObjectDescriptor { private String entityName; private List<String> uniqueKeyAttributes; private String typeAttribute;}
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
2. Neo4j JSON Loader @Path("/")public class JsonLoaderRestController {
private final GraphDatabaseService graphDatabaseService;
public JsonLoaderRestController(@Context GraphDatabaseService graphDatabaseService) { this.graphDatabaseService = graphDatabaseService; }
@POST @Produces({ MediaType.APPLICATION_JSON, MediaType.APPLICATION_XML })
public Response loadJSON(JAXBElement<JsonDocument> jsonDocumentWrapper) {
JsonDocument jsonDocument = jsonDocumentWrapper.getValue(); JsonLoaderService loaderService = new DefaultJsonLoaderService(this.graphDatabaseService); QueryStatistics queryStatistics = loaderService.save(jsonDocument); return Response.ok(queryStatistics).build(); }}
neo4j-server.properties org.neo4j.server.thirdparty_jaxrs_classes=it.larusba.integration.neo4j.jsonloader.rest=/jsonloader
25
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
2. Neo4j JSON Loader
public class DefaultJsonLoaderService implements JsonLoaderService {
@Override public JsonLoaderStatistics save(JsonDocument jsonDocument) throws JsonParseException, JsonMappingException, IOException {
JsonTransformer<String> jsonTransformer = JsonTransformerFactory.getInstance(jsonDocument.getMappingStrategy()); String cypher = jsonTransformer.transform(jsonDocument);
try (Transaction tx = this.graphDatabaseService.beginTx()) {
Result result = this.graphDatabaseService.execute(cypher); jsonLoaderStatistics = new JsonLoaderStatistics(result.getQueryStatistics()); tx.success(); }
return jsonLoaderStatistics; }
26
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
3. Neo4j Mutation Listener public class Neo4jEventListener implements TransactionEventHandler<Void> {
private GraphDatabaseService db;
public Neo4jEventListener(GraphDatabaseService db) { this.db = db; }
public void afterCommit(TransactionData data, Void state) { List<JsonDocument> jsonDocuments = buildJSONDocument(data); if (CollectionUtils.isNotEmpty(jsonDocuments)) { CouchbaseJSONLoader.upsert(jsonDocuments); } }}
27
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
4. Couchbase JSON Loader public class CouchbaseJSONLoader
public static void upsert(List<JsonDocument> jsonDocuments) {
Cluster cluster = CouchbaseCluster.create(); Bucket defaultBucket = cluster.openBucket();
for (JsonDocument jsonDocument : jsonDocuments) { defaultBucket.upsert(jsonDocument); }
cluster.disconnect(); }}
28
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
Neo4j - Couchbase Connector
HOW ABOUT THE DOMAIN?
WOULDN’T BE COOL TO BE ABLE TO TRAIN THE “JSON LOADER” WITH THE BUSINESS DOMAIN
YOUR WORKING ON?
29
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
HOW ABOUT THESE INTERESTING ATTRIBUTES ?
30
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
31
WHY DON’T WE USE “ID” ATTRIBUTES TO UNIQUELY IDENTIFY NODES?
We could use them on MERGE statements: MERGE (album:ALBUM {id: ‘586ZRfgsIckfcKvHVcGM4V’}) MERGE (track:TRACK {id: ‘3zqTlJtp7KTPrNUKNahvAo’})
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
32
AND WHY DON’T WE USE “TYPE” ATTRIBUTES AS NODES’ LABELS?
MERGE (album:ALBUM {id: ‘586ZRfgsIckfcKvHVcGM4V’}) MERGE (track:TRACK {id: ‘3zqTlJtp7KTPrNUKNahvAo’})
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
DESCRIBE YOUR DOMAIN i.e. instruct how the JSON 2 GRAPH transformer should behave List<JsonObjectDescriptor> jsonObjectDescriptors = new ArrayList<JsonObjectDescriptor>();
jsonObjectDescriptors.add(new JsonObjectDescriptor("Album", Arrays.asList("id"), “type”));jsonObjectDescriptors.add(new JsonObjectDescriptor("Artists", Arrays.asList("id"), “type"));jsonObjectDescriptors.add(new JsonObjectDescriptor("Images", Arrays.asList("url"), null));jsonObjectDescriptors.add(new JsonObjectDescriptor("Tracks", Arrays.asList("href"), null));jsonObjectDescriptors.add(new JsonObjectDescriptor("Items", Arrays.asList("id"), “type”));
JsonDocument jsonDocument = new JsonDocument("U2TheJoshuaTreeAlbum", "Album", spotifyTheJoshuaTreeAlbumDocument, JsonMappingStrategy.DOMAIN_DRIVEN, jsonObjectDescriptors);
33
key type json strategy descriptors
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
DOMAIN DRIVER JSON TRANSFORMER public class DomainDrivenJsonTransformer implements JsonTransformer<String> {
@Override public String transform(JsonDocument jsonDocument) throws JsonParseException, JsonMappingException, IOException {
34
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
35
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
36
“DOMAIN DRIVEN” TRANSFORMERALLOWS YOU TO RE-USE“U2” and all other NODES
… SO THAT YOU GET
A REAL BENEFIT FROM RELATIONSHIPS
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
AGENDA
✦ Let’s say Hi ( “Ciao” !! )
✦ Neo4j - Couchbase Connector
✦ JDBC Driver with BOLT for Neo4j 3.0
37
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
38
The new JDBC Driver with BOLT
Neo4j 3.0
APPLICATION
JDBC
EMIL’S OPENING KEYNOTE
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
Neo4j JDBC Driver: A-TEAM
39
Project Leader
HTTP Protocol
Tester
BENOIT SIMARD
STEFAN ARMBRUSTER
MICHAEL HUNGER
Dev & Project LeaderRALF BECHER
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
KEY ASPECTS
✦ MAKE USE OF THE NEW B.O.L.T. PROTOCOL
✦ MODULAR DESING (bolt, http, in-memory, file)
✦ HEAVILY TESTED
✦ FOCUS ON PERFORMANCE
40
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
41
1. Make use of the new BOLT protocol
Neo4j 3.0
NEO4J-JDBC
BOLT module
neo4j-java-driver
APPLICATION
https://github.com/neo4j/neo4j-java-driver
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
42
2. Modular design
Neo4j 3.0
NEO4J-JDBC BOLT
HTTP
FILE
IN-MEMORY
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
3. Heavily tested
✦ ~300 unit tests
✦ ~50 integration tests
43
for ~60 classes
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
4. Focus on performance ✦ OLD: 30’20’’
✦ NEW: 17’11’’
44
30.000 CSV rows
WOW! ALMOST HALF TIME
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
45
INTEGRATE NEO4J WITH ALL YOUR FAVOURITE TOOLS AND PRODUCTS
Neo4j 3.0
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
Minimum Viable Snippet // Make sure Neo4j Driver is registeredClass.forName(“it.larusba.neo4j.jdbc.bolt.BoltDriver”);
// ConnectConnection con = DriverManager.getConnection(“jdbc:bolt://localhost”);
// Querytry (Statement stmt = con.createStatement()) { ResultSet rs = stmt.executeQuery(“MATCH (n:User) RETURN n.name”); while (rs.next()) { System.out.println(rs.getString(“n.name”)); }}
con.close();
46
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
47
RC1
Neo4j Contrib
M5M3 M4
Neo4j 3.0 JDBC Driver RELEASE PLAN
HTTP PROTOCOL java.sql.CallableStatement
FULLY RETRO-COMPATIBILITY
BOLT PROTOCOL
early autumn
PRODUCTION READY
TODAY AT GCE2016
early summer
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
HAVE FUN WITH US
✦ https://github.com/larusba/neo4j-jdbc
✦ https://github.com/larusba/neo4j-couchbase-connector
✦ https://github.com/larusba/neo4j-json-loader
48
/ 50LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0
COME AND TALK WITH THESE GUYS!!! THEY DID ALL THIS STUFF & THEY’RE IN THE ROOM :-)
✦ Mauro Roiter (@MauroRoiter) ✦ Marco Falcier (@mfalcier) ✦ Alberto D’Este (@ziotobia) ✦ Gianmarco Laggia (@glaggia)
49
/ 50
LONDON, 26 April 2016
50
LORENZO SPERANZONI | @inserpio CEO @ LARUS Business Automation - GCE2016 BRONZE SPONSOR
GraphConnect Europe 2016
THANK YOU