lessons from 4 years of driver develoment
DESCRIPTION
Lessons from 4 years of developing the mongodb driver for node.jsTRANSCRIPT
4 years of Node.js and Mongodb
Christian Amor KvalheimNow at 10gen
Friday, August 16, 13
How did I get into Node.js
Friday, August 16, 13
Thu Jan 7 13:53:14 2010
First Commit
Friday, August 16, 13
Growth
Friday, August 16, 13
IT was crap
• It had a “Hash” and “Ordered Hash” class
• Complete mess with horrible structure
• Used Hacked JSpec for testing at the start
Friday, August 16, 13
Node was liquid
• Constant changes in Node.js
• Constant changes in driver to work with Node.js
• OH yeah no NPM yet
Friday, August 16, 13
Challenges
Friday, August 16, 13
Types Oh Types
• No int64 support in Javascript
• Oh yeah Numbers is a Double (53 bit int max)
• Decided to use Google Long closure library class
Friday, August 16, 13
Features
• Started adding features like GridFS in late jan 2010
• Replicaset and Mongos support in 2010/2011
• Stream support for Cursor and GridFS
• High Availability
Friday, August 16, 13
C++
• Profiled and found BSON parser to be main bottleneck
• Wrote horrible C++ parser
• Later cleaned up by a real C++ developer from Lucasarts (RIP)
Friday, August 16, 13
C++
• Initial version had own implementation of all BSON types
• C++ Long, Binary etc.
• Hard as hell to maintain
• Bought me nothing in the end as V8 changed
Friday, August 16, 13
Making JS BSON fasterBuffers
• Another major refactoring to support properly
• Calculate size of Document
• Single allocation instead of multiple small ones
• Using Buffers (optimized by v8)
• Indexed entry a[0] direct memory access
Friday, August 16, 13
Pool handling
• No traditional Connection Pool
• No checking OUT and IN Connections
• Connection pinning
• The “Spice” must flow
Friday, August 16, 13
Write concerns
• Mongodb wire protocol
• Insert/Update/Remove + GetLastError
• Operations must be pipelined
• Write the op + GetLastError in one operation
Friday, August 16, 13
Reconnecting
• No Blocking == difficult reconnect
• Buffer operations until reconnect
• Still a possible problem waiting for a solution
Friday, August 16, 13
Custom Serialization off Object to BSON
• Added toBSON support
• Specify on your objects and return the true representation of the object
• Modeled after toJSON
Friday, August 16, 13
Replicaset
• Problem
• Servers come and leave the cluster
• Need to update status of servers
• Need to pick the right server for the readPreference
Friday, August 16, 13
Replicaset
• Single thread SUCKS
• Have to use setTimeout/setInterval
• Has tons of subtle state changes
• Never going to get Webworkers in Node.js
• Processes are to granular
Friday, August 16, 13
Multi Platform
Friday, August 16, 13
Windows
• Hello Windows Users
• Node.js goes Windows
• What to do About C++ ?
• Build DLL’s for win32 and win64 in Windows 7 and package with driver
Friday, August 16, 13
Windows Build Hell
• Oh yeah Node extensions only build with Visual Studio 2010 + Windows 7 64 bit SDK
• Maybe better now but I doubt it
• Node-gyp will fail to build extensions on Win causes confusion
Friday, August 16, 13
Windows Build Hell
• Confusion about build process on Windows
• More “Cannot” install module questions on Win than you can swing a cat at
• Mostly PBKC
• Who the Hell still uses XP (SERIOUSLY)
Friday, August 16, 13
ARM
• Build C++ extension on ARM
• Learned about memory alignment difference between ARM and X86
• Runs on Raspberry PI (HUZZA)
Friday, August 16, 13
Regret
Friday, August 16, 13
Regrets
var A = function() { this.something = "a"}
A.prototype.some = function() { return this.something}
Friday, August 16, 13
Should have
var B = function() { var something = "a"
this.some = function() { return something }}
Friday, August 16, 13
Lesson
• If it’s possible to access internal state they will access internal state
• Only expose state and functions you wish end user should be able to use
• Examples of My BAD
• Collection class
• Cursor class
Friday, August 16, 13
Testing It
Friday, August 16, 13
Testing it ALL
• In the beginning
• Simple integration testing with JSpec
• Later moved to JUnit
Friday, August 16, 13
3D Reality hitsMongo 2.0 Mongo 2.2 Mongo 2.4
Node 0.8
Node 0.10
Node 0.X
with authreplicasetsharding
+2.2 features
+2.4 features
etc etc etc
etc etc etc
Friday, August 16, 13
A New Beginning
• New test framework
• Introduce configurations (replset/etc)
• Introduce runners (in auth mode, etc)
• Run tests against configurations
• Test does not manage configurations
• Still not there
• Need to work on the concept more
Friday, August 16, 13
Documenting It
Friday, August 16, 13
Documentation
• Had to provide tons of usage examples
• Realized I had them in code already
• Built some tools to help me generate most of the docs
• Spaghetti JS + Python Spinx
Friday, August 16, 13
Documentation/** * Example of inserting a document containing functions * * @_class collection * @_function insert * @ignore */exports.test = function(configuration, test) { var db = configuration.db(); // DOC_LINE var db = new Db('test', new Server('locahost', 27017)); // DOC_START // Fetch a collection to insert document into var collection = db.collection("simple_document_insert_with_function_safe"); // Insert a single document collection.insert({hello:'world' , func:function() {}}, {w:1, serializeFunctions:true}, function(err, result) { test.equal(null, err);
// Fetch the document collection.findOne({hello:'world'}, function(err, item) { test.equal(null, err); test.ok("function() {}", item.code); test.done(); }) }); // DOC_END}
Friday, August 16, 13
Result
Friday, August 16, 13
Open Source
Friday, August 16, 13
On Backwards Compatibility
• Drivers should be boring
• Change is bad
• Yeah some of the API sucks but to many depends on it
• This week CoffeeScript next week LiveScript (REALLY?)
Friday, August 16, 13
On Managing The Project
• Challenge people to help
• Be specific about what you need
• Give credit for pull requests
• Ignore flames or criticism
• Just ignore bad faith issues
• Your code is always going to suck, roll with it
Friday, August 16, 13
Wrapping Up
Friday, August 16, 13
Take aways
• No Environment is Static
• Testing Drivers is Very Hard
• Don’t take criticism personally
• Challenge people to help you
• Most people are nice and will help
• Credit people who help you
Friday, August 16, 13
In Conclusion
• It’s been A crazy 3 1/2 years of serving the Node community
• I’ve made tons of mistakes a long the way
• I’m sure to make a ton of mistakes in the future
• I would do it all again in a second
Friday, August 16, 13
Feel free to Ask Questions
Friday, August 16, 13