node in production at aviary

Click here to load reader

Post on 18-Jan-2015




2 download

Embed Size (px)


Aviary's customizable SDK powers cross-platform photo editing for over 6,500 partners and over 70 million monthly active users across the globe. Some of our notable partners include Walgreens, Squarespace, Yahoo Mail, Flickr, Photobucket, and Wix. At Aviary, we use node.js for several mission-critical projects in production and have seen extremely positive results. In this talk, we will discuss how we approach some common situations that developers deploying node.js projects will likely need to tackle. We will walk you through our routing mechanism, our automated deployment system, some of our custom middleware, and our testing philosophy.


  • 1. Node.js in Production at Aviary NYC Node.js Meetup March 5, 2014

2. Aviary Photo-Editing SDK & Apps Fully-Baked UI Configurable, High-Quality Tools Over 6,500 Partners Over 70 Million Monthly Users Over 6 Billion Photos Edited iOS, Android, Web, Server J 3. Who Are We? NirJackLead ServersideDirector ofEngineerEngineeringLikes: Automated deployment Big-O notation BrainteasersHates: CilantroLikes: Parallelizing processes DRY code SeltzerHates: Food after the sell-by date 4. Who Are We? JeffAriServersideDeveloperEngineerEvangelistLikes: Performance Profiling Spaces, not tabs BikesHates: His PhotoLikes: Empowering Developers Refactoring/Patterns DancingHates: Forrest Gump 5. How Do We Use Node? In Production: Future:Analytics Dashboard Server-side Rendering API Content Delivery (CDS) Automated billing Public Website Receipts Collection Monitoring Tools 6. Why Do We Use Node? Extremely fast and lightweight Easy to iterate on Common language for client and server JSON Cross Platform npm express module Active Community 7. Setting Up Your Server 8. Request Routing Our API servers all require(routes.json) { home: { getVersion: { verb: get, path: /version } }, queue: { updateContent: { verb: put, path: /content/:id, permissions: [content] } } }for (var controllerName in routes) { var controller = require(ctrlrDir + controllerName); for (var endpointName in routes[controllerName]) { var endpoint = routes[controllerName][endpointName]; var callback = controller[endpointName]; app[endpoint.verb]( endpoint.path, ensurePermissions(endpoint.permissions), callback ); } } 9. Authentication: Overview RequestServer listensMiddlewareLogged in?No YesResponseRequest handler takes overDoes user have permission?Redirected to login pageAuthenticated user saved in cookie 10. Authentication: Login passport.use(new GoogleStrategy({ returnURL: DOMAIN + '/auth/return' }, function (identifier, profile, done) { var userInfo = { name:, fullName: }; userRepository.findUserByName(, function (findErr, foundUser) { // ... if (foundUser.length === 0) { done('Invalid user', null); return; } userInfo.userId = foundUser.user_id; userInfo.permissions = foundUser.permissions; done(null, userInfo); }); } )); 11. Working with JSON 12. Validation - JSON Schema SCHEMA JSON-basedJSON{{ type: object additionalProperties: false properties: { status: { type: string, enum: [ok, error], required: true }, data: { type: object, required: false } } Like XML Schema Validation modules Used throughout Aviarys systems}status: ok } { status: error, data: { reason: hoisting } } { status: gladys, node: meetup } 13. Advanced JSON - Content EffectsFramesStickersMessages 14. The One-To-Many Problem Android expects responses to look like this: iOS 1.0 expects responses to look like this: iOS 2.0 expects responses to look like this: 15. Response Formatting - The Model Content EntryResponse FormatsResponsesJSON document describing content itemJSON documents defining mappings from entry to responsesActual JSON responses delivered to devices 16. Response Formatting - Details The Single Content Entry "identifier": "com.aviary.stickers.234fe"The Response Format "id": {"dataKey": "identifier""icon": { "path": "cds/hats/icon.png" "path-100": "cds/hats/icon100.png"}, "isPaid": {"isPaid": true, "iconImagePath": "cds/hats/icon100.png" "stickers": [ {"type": "boolean",},"dataKey": "isFree",{"identifier": "1""transformations": ["negateBool"]"items": ["imageUrl": "cds/hats/1.png"},"identifier": "1"}"iconImagePath": {"imageUrl": "cds/hats/1.png""type": "string",} ]"id": "com.aviary.stickers.234fe", "type": "string","isFree": false,The Response"dataKey": "icon.path-100" }, "stickers": { "type": "array", "dataKey": "items"] 17. Code Sample (Dumbed Down) var formattedResponse = {};for (var propName in responseFormat) { var val = contentEntry[responseFormat[propName].dataKey]; for (var transformation in responseFormat[propName].transformations) { val = transformationModule[transformation](val); } formattedResponse[propName] = val; }return formattedResponse; 18. Interacting with External Processes 19. Image Rendering Challenge: Use our existing image rendering .NET/C++ process from node server Solution: require(child_process).spawn(renderer.exe)Benefits: Easy IPC, asynchronous workflow 20. Code Sample var spawn = require(child_process).spawn;var renderer = spawn(renderer.exe, [-i, inputImage.jpg, ]); // read text renderer.stderr.setEncoding(utf8); renderer.stderr.on(data, function (data) { json += data; }); // or binary data renderer.stdout.on(data, function (data) { buffers.push(data); });renderer.on(close, function (code, signal) { // respond to exit code, signal (e.g. SIGTERM), process output var diagnostics = JSON.parse(json); var img = Buffer.concat(buffers); }); 21. Going Live 22. Testing Philosophy Unit tests (sparingly) End-to-end integration tests Mocha Enforced before push (master / development) 23. Example Integration Test #!/bin/bash Bash scriptmocha scopecreation && mocha cmsformatcreation && mocha crfcreation && Independent filesmocha mrfcreation && mocha rflcreation && mocha appcreation && Shared configurationmocha contentcreation && mocha manifestcreation && mocha push && Single failure stops processmocha cmsformatupdate && mocha crfaddition && mocha rfladdition && mocha contentupdate && mocha manifestupdate 24. Automated Deployment: Overview GitS3 2) Jenkins polls for repo changes1) Code is pushed to master3) Code is zipped and uploaded to S3Jenkins 4) Get a list of live servers in this groupAWS API5) SSH into each server and run the bootstrap script 25. Automated Deployment: Bootstrap 5) SSH into each server and run the bootstrap script#!/bin/bash ZIP_LOCATION="s3://aviary/projectX/"; cd ~/projectX; sudo apt-get -y -q install nodejs@0.10.0; sudo apt-get -y -q install s3cmd; sudo npm install -g forever@0.10.8;Goals of the 1. Ensure all dependencies are installed 2. Download and extract project 3. Ensure HTTP traffic is routed to the proper port 4. Keep the old version of the project live until the moment the new one is ready to go live# Missing step: create s3 configuration file s3cmd -c /usr/local/s3cfg.config get "$ZIP_LOCATION"; unzip -q -o iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-ports 8142; forever stopall; forever start server.js; 26. Summary 27. Lessons Learned (1) Integration tests! Watch out for node and npm updates Hardcode the node version youre using If youre using package.json, version everything Node.js + MongoDb are a great couple Make sure you understand hoisting 28. Lessons Learned (2) Always callback in async functions Always return after a callback Node doesnt always run the same on all platforms Use middleware only when necessary Always store dates as Unix Timestamps Timezones are a pain in your future Throwing unhandled errors will crash your process 29. Conclusion Today, our production node servers: serve dynamic content to 20MM people (soon 70MM) power our website: log real-time receipt data for every in-app purchase allow us to analyze hundreds of millions of events daily power quick scripts and one-off internal tools 30. Questions? Comments also welcome - - - and by the way, WERE HIRING!

View more