Fun Teaching MongoDB Fun Teaching MongoDB New TricksNew Tricks

Robert J. MooreRobert J. MoorePresidentPresidentAllanbank Consulting, Inc.Allanbank Consulting, Inc.

Robert.J.Moore@allanbank.comRobert.J.Moore@allanbank.com

AgendaAgenda

InspirationInspiration

A Little Fun: MongoDB asA Little Fun: MongoDB as ... a Coordination Service?... a Coordination Service? ... a Message Broker?... a Message Broker?

https://github.com/allanbank/mongodb-trickshttps://github.com/allanbank/mongodb-tricks

DisclaimerDisclaimer

Don't try these at homeDon't try these at home They are hacks in the best senseThey are hacks in the best sense

Professional developer on an air gap Professional developer on an air gap laptoplaptop

Enough! On with the funEnough! On with the fun

Inspiration - TopicsInspiration - Topics

Publish / SubscribePublish / Subscribe One writer, many readersOne writer, many readers

MongoDB Capped CollectionMongoDB Capped Collection Tailable CursorTailable Cursor Can also have selectorsCan also have selectors

Inspiration - TopicsInspiration - Topics

...

Cursor

Document

Create a Capped CollectionCreate a Capped Collection Acts like a ring bufferActs like a ring buffer

Tailable Cursor returns documents as Tailable Cursor returns documents as they are added in the futurethey are added in the future

Asynchronous Java DriverAsynchronous Java Driver

WebsiteWebsite http://www.allanbank.com/mongodb-async-driverhttp://www.allanbank.com/mongodb-async-driver

Focus on performance and usabilityFocus on performance and usability Lower latency, higher throughput even under Lower latency, higher throughput even under

heavy thread contentionheavy thread contention Benefit when using the synchronous Benefit when using the synchronous

interfaceinterface Greater benefit using the asynchronous Greater benefit using the asynchronous

interfaceinterface

Asynchronous Java DriverAsynchronous Java DriverPerformancePerformance

http://www.allanbank.com/mongodb-async-driver/performance/ycsb.htmlhttp://www.allanbank.com/mongodb-async-driver/performance/ycsb.html

Coordination ServiceCoordination Service

ZooKeeperZooKeeper Notification that something of interest Notification that something of interest

changedchanged WatchesWatches

Track entities in a groupTrack entities in a group Group ManagementGroup Management

WatchesWatches

Replica Set OplogReplica Set Oplog All Insert, Update, Delete operationsAll Insert, Update, Delete operations Capped – Tailable cursors againCapped – Tailable cursors again

Want to convert from the OplogWant to convert from the Oplog Different fields contain the _id of the Different fields contain the _id of the

documentdocument

Oplog DocumentsOplog Documents InsertInsert

{ "ts" : { "t" : 1362958492000, "i" : 1 }, { "ts" : { "t" : 1362958492000, "i" : 1 },

"h" : NumberLong("5915409566571821368"), "v" : 2, "h" : NumberLong("5915409566571821368"), "v" : 2,

"op" : "i", "op" : "i",

"ns" : "test.test", "ns" : "test.test",

"o" : { "_id" : "513d189c8544eb2b5e000001" } }"o" : { "_id" : "513d189c8544eb2b5e000001" } }

DeleteDelete{ ... "op" : "d", ..., "b" : true, { ... "op" : "d", ..., "b" : true,

"o" : { "_id" : "513d189c8544eb2b5e000001" } }"o" : { "_id" : "513d189c8544eb2b5e000001" } }

UpdateUpdate{ ... "op" : "u", ..., { ... "op" : "u", ...,

"o2" : { "_id" : "513d189c8544eb2b5e000001" }, "o2" : { "_id" : "513d189c8544eb2b5e000001" },

"o" : { "$set" : { "i" : 1 } } }"o" : { "$set" : { "i" : 1 } } }

Initialize a Watcher - CodeInitialize a Watcher - Codefinal DocumentAssignable wantQuery = or( where("ns").equals(ns) .and("op").in(constant("i"), constant("d")) .and("o._id").matches(myContext), where("ns").equals(ns) .and("op").equals("u") .and("o2._id").matches(myContext));

final Find.Builder builder = new Find.Builder();if (myLastTs != null) { final DocumentBuilder tsQuery = BuilderFactory.start(); tsQuery.push(myLastTs.getName()) .add(myLastTs.withName(ComparisonOperator.GT.getToken()));

builder.setQuery(and(tsQuery, wantQuery));}else { builder.setQuery(wantQuery);}

builder.tailable();

MongoDatabase localDb = myMongoClient.getDatabase("local");MongoCollection oplog = localDb.getCollection("oplog.rs");myControls = oplog.streamingFind(new OpLogNotification(), builder.build());

Watcher - CodeWatcher - Code

Watcher watcher = new Watcher(client, collection, Pattern.compile(".*"), new WatchListener() { @Override public void changed(Operation op, String context, Document document) { if (op == Operation.DELETE) { System.out.println(context + ": " + op); } else { System.out.println(context + ": " + op + ": " + document); } } });watcher.start();

Watcher DemoWatcher Demo

Group ManagementGroup Management

Notification of:Notification of: Adding a Group MemberAdding a Group Member Removing a Group MemberRemoving a Group Member

Or Group Member DisappearsOr Group Member Disappears

Combination of:Combination of: WatcherWatcher Scheduled Task for HeartbeatScheduled Task for Heartbeat Expire old membersExpire old members

Group Manager Client - CodeGroup Manager Client - CodeGroupManager manager = new GroupManager(executor, client, collection, rootContext);manager.addListener(new GroupListener() { @Override public void memberRemoved(String context) { System.out.println(context + " - Removed"); }

@Override public void memberAdded(String context) { System.out.println(context + " - Added"); }});manager.start();

final GroupMember member = manager.addMember();

// Faster cleanup, if we can.Runtime.getRuntime().addShutdownHook(new Thread() { @Override public void run() { member.remove(); }});

Group DemoGroup Demo

QueuesQueues

Divide and ConquerDivide and Conquer One writer, one readerOne writer, one reader

MongoDB Capped CollectionMongoDB Capped Collection Want to “share” a CursorWant to “share” a Cursor

Restartable CursorsRestartable Cursors

What if we had multiple applications passing the What if we had multiple applications passing the same “getmore” requests?same “getmore” requests?

““getmore” message references a cursorgetmore” message references a cursor

We need a way to save and restore the cursorWe need a way to save and restore the cursor Lets call them Restartable Cursors, but they don't Lets call them Restartable Cursors, but they don't

exist, yetexist, yet

struct { MsgHeader header; // standard message header int32 ZERO; // 0 - reserved for future use cstring fullCollectionName; // "dbname.collectionname" int32 numberToReturn; // number of documents to return int64 cursorID; // cursorID from the OP_REPLY}http://docs.mongodb.org/meta-driver/latest/legacy/mongodb-wire-protocol/#op-get-more

QueuesQueues

...

Cursor

Document

Create a Capped CollectionCreate a Capped Collection Initialize a Tailable CursorInitialize a Tailable Cursor Share with multiple processesShare with multiple processes

Application 1 Application 3Application 2

Restartable CursorsRestartable Cursors

Added the ability to:Added the ability to: Persist an iterator or stream as a doumentPersist an iterator or stream as a doument Halt or stop the iterator stream (after Halt or stop the iterator stream (after

exhausting the already requested exhausting the already requested documents).documents).

For restart add the ability to restart the For restart add the ability to restart the iterator or streamiterator or stream

Allows conversion from an iterator to a Allows conversion from an iterator to a stream and back.stream and back.

Queues – Initialization CodeQueues – Initialization Code Initialize a Tailable CursorInitialize a Tailable Cursor Share with multiple processesShare with multiple processes

Find.Builder builder = new Find.Builder(BuilderFactory.start()); builder.setTailable(true); builder.setAwaitData(false); // Sigh. MongoIterator<Document> cursor = collection.find(builder.build());

// Graceful shutdown of the iterator locally but not on the server. cursor.stop(); while (cursor.hasNext()) { System.out.println(cursor.next()); }

collection = db.getCollection("lookup"); collection.delete(BuilderFactory.start().add("_id", collectionName)); collection.insert(BuilderFactory.start().add("_id", collectionName) .add("cursor", cursor.asDocument()));

Queues – Consumer CodeQueues – Consumer Code

MongoCollection index = client.getDatabase(args[1]).getCollection( "lookup"); Document queueLookupDoc = index.findOne(BuilderFactory.start().add( "_id", args[2])); DocumentElement cursorElement = queueLookupDoc.get( DocumentElement.class, "cursor");

MongoIterator<Document> iter = client.restart(cursorElement .getDocument());

long lastCount = 0; while (iter.hasNext()) { Document doc = iter.next(); // Do stuff }

Queues DemoQueues Demo

Restartable Cursors - GotchaRestartable Cursors - Gotcha Remember this: Remember this: builder.setAwaitData(false); // Sigh.builder.setAwaitData(false); // Sigh. That masks a small issue...That masks a small issue...

Closes the cursor on the clientCloses the cursor on the client https://jira.mongodb.org/browse/SERVER-8602https://jira.mongodb.org/browse/SERVER-8602 https://github.com/allanbank/mongo/tree/concurrent_cursor_supporthttps://github.com/allanbank/mongo/tree/concurrent_cursor_support

Pin( long long cursorid ) : _cursorid( INVALID_CURSOR_ID ) { recursive_scoped_lock lock( ccmutex ); ClientCursor *cursor = ClientCursor::find_inlock( cursorid, true ); if ( cursor ) { uassert( 12051, "clientcursor already in use? driver problem?", cursor->_pinValue < 100 ); cursor->_pinValue += 100; _cursorid = cursorid; }}

Final ThoughtsFinal Thoughts

Coordination Service Coordination Service WatchersWatchers Group ManagementGroup Management

Message BrokerMessage Broker TopicsTopics QueuesQueues

Remember the DisclaimerRemember the Disclaimer

Questions?Questions?

Contact Information:Contact Information:Robert.J.Moore@allanbank.comRobert.J.Moore@allanbank.com