mongodb at-gilt-groupe-seattle-2012-09-14-final
TRANSCRIPT
![Page 1: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/1.jpg)
September 2012Sean Sullivan
@
![Page 2: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/2.jpg)
• software engineer
• ~18 months at Gilt
• work in Gilt’s Portland office
• back office applications
About me
![Page 3: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/3.jpg)
• Gilt Groupe
• Gilt’s technology stack
• MongoDB at Gilt
• Q&A
Agenda
![Page 4: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/4.jpg)
Gilt Groupehttp://www.gilt.com
![Page 5: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/5.jpg)
flash sales
![Page 6: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/6.jpg)
Everyday at 12 noon
![Page 7: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/7.jpg)
what does Gilt sell?
![Page 8: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/8.jpg)
Apparel
![Page 9: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/9.jpg)
![Page 10: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/10.jpg)
![Page 11: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/11.jpg)
Kids merchandise
![Page 12: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/12.jpg)
![Page 13: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/13.jpg)
Home furnishings
![Page 14: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/14.jpg)
![Page 15: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/15.jpg)
Food
![Page 16: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/16.jpg)
![Page 17: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/17.jpg)
Local deals
![Page 18: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/18.jpg)
![Page 19: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/19.jpg)
Travel
![Page 20: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/20.jpg)
![Page 21: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/21.jpg)
![Page 22: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/22.jpg)
Gilt technology stack
![Page 23: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/23.jpg)
![Page 24: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/24.jpg)
Gilt architecture
![Page 25: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/25.jpg)
monolithicapplication
service-oriented approach
2007 2012
![Page 26: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/26.jpg)
2007
![Page 27: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/27.jpg)
2012
service A
service B
service C
service D
service E
legacyweb app
gilt.com
![Page 28: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/28.jpg)
Data storage @ Gilt
![Page 29: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/29.jpg)
@
![Page 30: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/30.jpg)
Why MongoDB?
• Ease of use
• Horizontal scaling
• High availability
• Automatic failover
![Page 31: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/31.jpg)
Why MongoDB?
• Stability
• Support
• Drivers
![Page 32: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/32.jpg)
• MongoDB 2.0
• sharded and non-sharded data
• Solid State Drives
• MMS for monitoring
![Page 33: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/33.jpg)
Use case #1: user profiles
![Page 34: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/34.jpg)
• user data in Postgres
• legacy Rails app expects to find user data in Postgres
• we want Gilt’s customer facing application code to retrieve user data from MongoDB
Challenges
![Page 35: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/35.jpg)
• keep user data in both MongoDB and Postgres
• replicate table changes from Postgres to MongoDB
Solution
![Page 36: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/36.jpg)
Replicating data
user service
legacyweb app
replication
![Page 37: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/37.jpg)
Use case #2: feature configuration
Goal
manage the release of new features on gilt.com
How
feature configuration persisted in MongoDB
![Page 38: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/38.jpg)
Rolling out a new feature
1. deploy new application code to production
2. enable feature for Gilt Tech employees
3. ... then enable for all Gilt employees
4. ... then enable for a subset of users
5. gradually ramp up to 100% of users
![Page 39: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/39.jpg)
Feature configservice
Feature configuration
gilt.com
![Page 40: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/40.jpg)
Use case #3: favorite brands
![Page 41: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/41.jpg)
userpreference
service
AJAX
Favorite brands
![Page 42: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/42.jpg)
Application development
![Page 43: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/43.jpg)
mongo-java-driver
Morphia
Casbah
![Page 44: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/44.jpg)
MongoDB Java driver
• main class: com.mongodb.Mongo
• Mongo object maintains a pool of connections
• Latest version: 2.9.1
![Page 45: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/45.jpg)
MongoDB Java driver connection pool tuning
MongoOptions opts = new MongoOptions();
// example values
opts.connectionsPerHost = 50;
opts.threadsAllowedToBlockForConnectionMultiplier = 5;
![Page 46: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/46.jpg)
com.mongodb.WriteConcern
• NORMAL
• SAFE
• REPLICAS_SAFE
• MAJORITY
• FSYNC_SAFE
• JOURNAL_SAFE
![Page 47: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/47.jpg)
Morphia library
![Page 48: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/48.jpg)
Morphia
• object-document mapper for Java
• built on top of mongo-java-driver
• map fields using Java annotations
![Page 49: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/49.jpg)
// Morphia example
import com.google.code.morphia.annotations.*;
@Entity(value="features", noClassnameStored = true)
public class Feature {
@Id
ObjectId featureId;
@Property("feature_key") @Indexed(unique=true)
String featureKey;
@Property("release_to_percentage")
int releaseToPercentage;
}
![Page 50: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/50.jpg)
http://code.google.com/p/morphia/
![Page 51: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/51.jpg)
Casbah library
![Page 52: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/52.jpg)
Casbah
• Scala toolkit for MongoDB
• built on top of the mongo-java-driver
![Page 53: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/53.jpg)
Casbah
• Scala idioms
• Scala collections
• fluid query syntax
![Page 54: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/54.jpg)
https://twitter.com/max4f/status/230503836958199808
![Page 55: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/55.jpg)
Gilt Scala code
![Page 56: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/56.jpg)
trait MongoWriteConcern {
def withSession[T](f: => T)(implicit mongoDb: MongoDB) {
mongoDb.requestStart()
mongoDb.requestEnsureConnection()
try {
f
} finally {
mongoDb.requestDone()
}
}
}
![Page 57: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/57.jpg)
Best Practices
![Page 58: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/58.jpg)
Best practices
Connection tuning
explicitly configure Mongo Java Driver connection pool size
![Page 59: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/59.jpg)
Best practices
Use caution when creating new indexes
“creating a new index on a production mongo server can basically cause it to stop working while it builds the index”
(source: Gilt production incident report)
![Page 60: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/60.jpg)
http://docs.mongodb.org/manual/administration/indexes/#index-building-replica-sets
Minimizing impact of building a new index
• remove one secondary from replica set
• create/rebuild index on this instance
• rejoin replica set
• repeat on all remaining secondaries
• run rs.stepDown() on primary member
![Page 61: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/61.jpg)
Best practicesuse short names for fields to avoid wasted space
http://christophermaier.name/blog/2011/05/22/MongoDB-key-names
{
city: “Portland”,
state: “Oregon”,
country: “US”
}
{
ci: “Portland”,
st: “Oregon”,
co: “US”
}
vs
![Page 62: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/62.jpg)
Best practices
use explain() during development
db.collection.find(query).explain()
![Page 63: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/63.jpg)
Best practices
use caution when choosing a shard key
“It is generally not a good idea to use the default ObjectId as the shard key”
source: http://stackoverflow.com/questions/9164356/sharding-by-objectid-is-it-the-right-way
![Page 64: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/64.jpg)
Future
• standardized read preferences
• tag aware sharding
MongoDB 2.2
Mongo Java driver 2.9.x
• enhanced read preference support
(see: com.mongodb.ReadPreference)
![Page 65: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/65.jpg)
Gilt Groupe is hiring!http://techjobs.gilt.com
![Page 66: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/66.jpg)
Questions?
![Page 67: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/67.jpg)
[email protected]@tinyrobots
![Page 68: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/68.jpg)
The end
![Page 69: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/69.jpg)
Bonus slides
![Page 70: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/70.jpg)
Gilt tech talks
• Apache Camel and MongoDB @ Gilt http://bit.ly/OYO37K
• Deploying new features @ Gilt http://slidesha.re/OoCYfd
• Voldemort @ Gilt
http://bit.ly/b9Qhib
![Page 71: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/71.jpg)
![Page 72: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/72.jpg)
![Page 73: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/73.jpg)
![Page 74: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/74.jpg)
![Page 75: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/75.jpg)
![Page 76: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/76.jpg)
![Page 77: Mongodb at-gilt-groupe-seattle-2012-09-14-final](https://reader034.vdocuments.mx/reader034/viewer/2022052310/554f7be1b4c9052a518b4aa7/html5/thumbnails/77.jpg)