mongodb and content delivery at aviary by nir zicherman and jack sisson

Content Delivery At Aviary

MongoDb User Group 11/19/13

Aviary

Fully-Baked UI

Configurable, High-Quality Tools

Over 5,000 Partners

Over 50 Million Monthly Users

Over 4 Billion Photos Edited

iOS, Android, Web, Windows, Server

Photo-Editing SDK & Apps

J

Who Are We?

JackDirector of Engineering

Nir Lead Serverside Engineer

● Automated deployment● Big-O notation● Brainteasers

Likes:

● Cilantro

Hates:

● Parallelizing processes● DRY code● Seltzer

Likes:

● Food after the sell-by date

Hates:

ContentEffects Frames Stickers Messages

J

The “Fat Tiny” Situation

We want to dynamically deliver the Fat Tiny

stickers to our users

How can we do that?

J

The CDSAviary’s Content Delivery System

Version 1

● Static files on S3

● Excess data sent

● Data stored in MySQL

● New features meant new code

N

The “Fat Tiny” Situation

We want to display the Fat Tiny sticker pack only

to Picstitch users in Japan who use iOS 7

How can we do that?

N

CDS V2Version 2 Overview

Stack

Load Balancer

APIServers

DatabaseCluster

Content Delivery Network

Management System

GlobalUsers

Akamai

AWS Cloud FormationAWS Elastic LoadBalancer (ELB)

Node.jsAWS Elastic Cloud Computing (EC2)UbuntuBash

MongoDbJSONJSON Schema

N

Why MongoDB?

● Great with Node (JSON-based)

● Schema-less is easy to change/query

● Read-heavy

● Relatively small data set

N

How It WorksA Behind-the-Scenes Look

Delivered Types

● Manifest JSON○ Content Set○ Content Versions

● Content JSON○ Content Metadata○ All Assets

N

The “Fat Tiny” Situation

We want to manage the Fat Tiny sticker pack

as a single entity, but we want to deliver it to

each device in its own

optimal format.

How can we do that?N

Response Formatting ModelContent Entry Response Formats Responses

JSON document describing content item

JSON documents defining mappings from entry to responses

Actual JSON responses delivered to devices J

Content Formats

● JSON schema

● Added properties

● Custom types

● Validates entries

{

"type":"object",

"properties":{

"metadata": {

"type": "object",

"properties":{

"displayName": {

"type": "string",

}

}

},

"icon": {

"type": "object",

"customType": "image"

"properties":{

"path": {

"type": "string",

"required": true, J

Response Formats

● JSON schema

● dataKey property

● Defines response

structure

● Maps content

{

"type":"object",

"properties":{

"identifier": {

"type": "string",

"dataKey": "identifier"

},

"name": {

"type": "string",

"dataKey": "metadata.displayName"

},

"iconImagePath": {

"type": "string",

"dataKey": "icon.path-100"

},

"items": {

"type": "array",

"dataKey": "items" J

Content Deployment

"type":"object",

"properties":{

"id": {

"type": "string",

"dataKey": "identifier"

},

"name": {

"type": "string",

"dataKey": "metadata.displayName"

},

"iconImagePath": {

"type": "string",

"dataKey": "icon.path-100"

},

"stickers": {

"type": "array",

"dataKey": "items"

"identifier": "com.aviary.stickers.234fe"

"metadata": {

"displayName": "Hats"

},

"icon": {

"path": "cds/hats/icon.png"

"path-100": "cds/hats/icon100.png"

},

"items": [

{

"identifier": "1"

"imageUrl": "cds/hats/1.png"

}

]

"id": "com.aviary.stickers.234fe",

"name": "Hats",

"iconImagePath": "cds/hats/icon100.png"

"stickers": [

{

"identifier": "1"

"imageUrl": "cds/hats/1.png"

}

],

"versionKey": "e4532fd342"

1. Insert/Update CMS Entry 2. Find Response Formats 3. Generate+Insert Responses

J

Manifests "stickers": [

{

"id": "com.aviary.stickers.234fe",

"versionKey": "e4532fd342"

},

{

"id": "com.aviary.stickers.fed34",

"versionKey": "c54532343d"

}

],

"frames": [

{

"id": "com.aviary.frames.25435",

"versionKey": "fd4324323"

}

] J

Manifest Deployment

{

$match:{

formatId:{$in:formatIds},

identifier:{$in:identifiers}

}

},

{

$sort: { _id: -1 }

},

{

$group: {

_id: "$identifier",

versionKey:{$first:"$versionKey"}

}

}

Using Aggregate to Find the Newest Versions Manifest with Correct Version Keys

"stickers": [

{

"id": "com.aviary.stickers.234fe",

"versionKey": "e4532fd342"

}

],

"effects": [

{

"id": "com.aviary.effects.25435",

"versionKey": "fd4324323"

}

]

J

Scopes and Targeting

Manifest 1

"targetingScope": {

"apiKey": "abc",

"country": ["JP"]

},

"formattingScope": {

"platform": "ios",

"minOsVersion": "7.0.0"

}

Manifest 2

"targetingScope": {

"apiKey": "def",

},

"formattingScope": {

"platform": "android",

"minOsVersion": "6.0.0"

}

Deployed Manifests Have Scopes

/manifest?

apiKey=abc&

country=JP&

language=ja&

platform=ios&

osVersion=7.2.0

End Users Have Scope Parameters

N

API ServersScope parameters are converted into queries

/manifest?

apiKey=abc&

country=JP&

language=ja&

platform=ios&

osVersion=7.2.0

db.manifest.find({

"apiKey": {$in: ["abc", null]},

"country": {$in: ["JP", null]},

"language": {$in: ["ja", null]},

"platform": {$in: ["ios", null]},

"minOsVersion": {$lte: 7002000}

}).sort({

"apiKey": -1,

"language": -1,

"country": -1,

"minOsVersion": -1,

"platform": -1,

"_id": -1

}).limit(1)

7002000

Manifest 1

"targetingScope": {

"apiKey": "abc",

"country": ["JP"]

},

"formattingScope": {

"platform": "ios",

"minOsVersion": "7.0.0"

}

Manifest 2

"targetingScope": {

"apiKey": "abc",

},

"formattingScope": {

"platform": "ios",

"minOsVersion": "6.0.0"

}N

Versioned Content

"stickers": [

{

"id": "com.aviary.stickers.234fe",

"versionKey": "e4532fd342"

},

{

"id": "com.aviary.stickers.fed34",

"versionKey": "c54532343d"

}

],

"frames": [

{

"id": "com.aviary.frames.25435",

"versionKey": "fd4324323"

}

]

Received Manifests Contain VersionKeys

/content?

versionKey=e4532fd342

db.content.findOne({

"versionKey": “e4532fd342”

});

N

Response Caching/manifest?

apiKey=abc&

country=JP&

language=ja&

platform=ios&

osVersion=7.2.0

db.manifests.find({

"apiKey": {$in: ["abc", null]},

"country": {$in: ["JP", null]},

"language": {$in: ["ja", null]},

"platform": {$in: ["ios", null]},

"minOsVersion": {$gte: 7002000}

}).sort({

…,

"_id": -1

}).limit(1)

db.cachedManifests.findOne({

"url": "/manifest?apiKey=abc&country=JP&language=ja&osVersion=7.2.0&platform=ios"

})

N

PAULAThe CDS Management Console

Auto-generated UI

J

Other Mongo Usage

● PAULA permissions in user objects

● Integration tests interact with schemaless db willy nilly

users collection

{

"name": "nir",

"email": "nir@aviary.com",

"permissions": [

"content",

"dev",

"admin",

"partying"

]

}

N

ConclusionThe Takeaway

The Facts

● Built and deployed in 3 months

● Very few struggles with MongoDB

● Seamless management

● Graceful scaling from 0 to over 20M MAUs

● Happy serverside engineersJ

The Future

● Targeted Translations

● Granular User Targeting

● PAULA for the masses

N

Questions?Comments also welcome

nir@aviary.com jack@aviary.com

…and by the way, WE’RE HIRING!