mongodb and content delivery at aviary by nir zicherman and jack sisson
DESCRIPTION
Aviary's customizable SDK powers cross-platform photo editing for over 4,500 partners and over 50 million monthly active users across the globe. Some of our notable partners include Walgreens, Squarespace, Yahoo Mail, Flickr, Photobucket, and Wix. Aviary's network has grown to thousands of partners and over 50 million active users since the fall of 2011. To optimize the photo editing experience, we recently built a content delivery system that targets users with customized effects, stickers, frames, etc. Today, we can distribute targeted content based on a seamlessly extendable set of parameters, including a user's location, language, app, and device.TRANSCRIPT
Content Delivery At Aviary
MongoDb User Group 11/19/13
Aviary
Fully-Baked UI
Configurable, High-Quality Tools
Over 5,000 Partners
Over 50 Million Monthly Users
Over 4 Billion Photos Edited
iOS, Android, Web, Windows, Server
Photo-Editing SDK & Apps
J
Who Are We?
JackDirector of Engineering
Nir Lead Serverside Engineer
● Automated deployment● Big-O notation● Brainteasers
Likes:
● Cilantro
Hates:
● Parallelizing processes● DRY code● Seltzer
Likes:
● Food after the sell-by date
Hates:
ContentEffects Frames Stickers Messages
J
The “Fat Tiny” Situation
We want to dynamically deliver the Fat Tiny
stickers to our users
How can we do that?
J
The CDSAviary’s Content Delivery System
Version 1
● Static files on S3
● Excess data sent
● Data stored in MySQL
● New features meant new code
N
The “Fat Tiny” Situation
We want to display the Fat Tiny sticker pack only
to Picstitch users in Japan who use iOS 7
How can we do that?
N
CDS V2Version 2 Overview
Stack
Load Balancer
APIServers
DatabaseCluster
Content Delivery Network
Management System
GlobalUsers
Akamai
AWS Cloud FormationAWS Elastic LoadBalancer (ELB)
Node.jsAWS Elastic Cloud Computing (EC2)UbuntuBash
MongoDbJSONJSON Schema
N
Why MongoDB?
● Great with Node (JSON-based)
● Schema-less is easy to change/query
● Read-heavy
● Relatively small data set
N
How It WorksA Behind-the-Scenes Look
Delivered Types
● Manifest JSON○ Content Set○ Content Versions
● Content JSON○ Content Metadata○ All Assets
N
The “Fat Tiny” Situation
We want to manage the Fat Tiny sticker pack
as a single entity, but we want to deliver it to
each device in its own
optimal format.
How can we do that?N
Response Formatting ModelContent Entry Response Formats Responses
JSON document describing content item
JSON documents defining mappings from entry to responses
Actual JSON responses delivered to devices J
Content Formats
● JSON schema
● Added properties
● Custom types
● Validates entries
{
"type":"object",
"properties":{
"metadata": {
"type": "object",
"properties":{
"displayName": {
"type": "string",
}
}
},
"icon": {
"type": "object",
"customType": "image"
"properties":{
"path": {
"type": "string",
"required": true, J
Response Formats
● JSON schema
● dataKey property
● Defines response
structure
● Maps content
{
"type":"object",
"properties":{
"identifier": {
"type": "string",
"dataKey": "identifier"
},
"name": {
"type": "string",
"dataKey": "metadata.displayName"
},
"iconImagePath": {
"type": "string",
"dataKey": "icon.path-100"
},
"items": {
"type": "array",
"dataKey": "items" J
Content Deployment
"type":"object",
"properties":{
"id": {
"type": "string",
"dataKey": "identifier"
},
"name": {
"type": "string",
"dataKey": "metadata.displayName"
},
"iconImagePath": {
"type": "string",
"dataKey": "icon.path-100"
},
"stickers": {
"type": "array",
"dataKey": "items"
"identifier": "com.aviary.stickers.234fe"
"metadata": {
"displayName": "Hats"
},
"icon": {
"path": "cds/hats/icon.png"
"path-100": "cds/hats/icon100.png"
},
"items": [
{
"identifier": "1"
"imageUrl": "cds/hats/1.png"
}
]
"id": "com.aviary.stickers.234fe",
"name": "Hats",
"iconImagePath": "cds/hats/icon100.png"
"stickers": [
{
"identifier": "1"
"imageUrl": "cds/hats/1.png"
}
],
"versionKey": "e4532fd342"
1. Insert/Update CMS Entry 2. Find Response Formats 3. Generate+Insert Responses
J
Manifests "stickers": [
{
"id": "com.aviary.stickers.234fe",
"versionKey": "e4532fd342"
},
{
"id": "com.aviary.stickers.fed34",
"versionKey": "c54532343d"
}
],
"frames": [
{
"id": "com.aviary.frames.25435",
"versionKey": "fd4324323"
}
] J
Manifest Deployment
{
$match:{
formatId:{$in:formatIds},
identifier:{$in:identifiers}
}
},
{
$sort: { _id: -1 }
},
{
$group: {
_id: "$identifier",
versionKey:{$first:"$versionKey"}
}
}
Using Aggregate to Find the Newest Versions Manifest with Correct Version Keys
"stickers": [
{
"id": "com.aviary.stickers.234fe",
"versionKey": "e4532fd342"
}
],
"effects": [
{
"id": "com.aviary.effects.25435",
"versionKey": "fd4324323"
}
]
J
Scopes and Targeting
Manifest 1
"targetingScope": {
"apiKey": "abc",
"country": ["JP"]
},
"formattingScope": {
"platform": "ios",
"minOsVersion": "7.0.0"
}
Manifest 2
"targetingScope": {
"apiKey": "def",
},
"formattingScope": {
"platform": "android",
"minOsVersion": "6.0.0"
}
Deployed Manifests Have Scopes
/manifest?
apiKey=abc&
country=JP&
language=ja&
platform=ios&
osVersion=7.2.0
End Users Have Scope Parameters
N
API ServersScope parameters are converted into queries
/manifest?
apiKey=abc&
country=JP&
language=ja&
platform=ios&
osVersion=7.2.0
db.manifest.find({
"apiKey": {$in: ["abc", null]},
"country": {$in: ["JP", null]},
"language": {$in: ["ja", null]},
"platform": {$in: ["ios", null]},
"minOsVersion": {$lte: 7002000}
}).sort({
"apiKey": -1,
"language": -1,
"country": -1,
"minOsVersion": -1,
"platform": -1,
"_id": -1
}).limit(1)
7002000
Manifest 1
"targetingScope": {
"apiKey": "abc",
"country": ["JP"]
},
"formattingScope": {
"platform": "ios",
"minOsVersion": "7.0.0"
}
Manifest 2
"targetingScope": {
"apiKey": "abc",
},
"formattingScope": {
"platform": "ios",
"minOsVersion": "6.0.0"
}N
Versioned Content
"stickers": [
{
"id": "com.aviary.stickers.234fe",
"versionKey": "e4532fd342"
},
{
"id": "com.aviary.stickers.fed34",
"versionKey": "c54532343d"
}
],
"frames": [
{
"id": "com.aviary.frames.25435",
"versionKey": "fd4324323"
}
]
Received Manifests Contain VersionKeys
/content?
versionKey=e4532fd342
db.content.findOne({
"versionKey": “e4532fd342”
});
N
Response Caching/manifest?
apiKey=abc&
country=JP&
language=ja&
platform=ios&
osVersion=7.2.0
db.manifests.find({
"apiKey": {$in: ["abc", null]},
"country": {$in: ["JP", null]},
"language": {$in: ["ja", null]},
"platform": {$in: ["ios", null]},
"minOsVersion": {$gte: 7002000}
}).sort({
…,
"_id": -1
}).limit(1)
db.cachedManifests.findOne({
"url": "/manifest?apiKey=abc&country=JP&language=ja&osVersion=7.2.0&platform=ios"
})
N
PAULAThe CDS Management Console
Auto-generated UI
J
Other Mongo Usage
● PAULA permissions in user objects
● Integration tests interact with schemaless db willy nilly
users collection
{
"name": "nir",
"email": "[email protected]",
"permissions": [
"content",
"dev",
"admin",
"partying"
]
}
N
ConclusionThe Takeaway
The Facts
● Built and deployed in 3 months
● Very few struggles with MongoDB
● Seamless management
● Graceful scaling from 0 to over 20M MAUs
● Happy serverside engineersJ
The Future
● Targeted Translations
● Granular User Targeting
● PAULA for the masses
N
