sharded by business line: migrating to a core database using mongodb and solr
DESCRIPTION
TRANSCRIPT
![Page 1: Sharded By Business Line: Migrating to a Core Database using MongoDB and Solr](https://reader035.vdocuments.mx/reader035/viewer/2022081414/54dc82124a7959b6618b48c2/html5/thumbnails/1.jpg)
Click to edit Master title styleThe Knot Search Platform MongoLA 2013
![Page 2: Sharded By Business Line: Migrating to a Core Database using MongoDB and Solr](https://reader035.vdocuments.mx/reader035/viewer/2022081414/54dc82124a7959b6618b48c2/html5/thumbnails/2.jpg)
About Us• Weddings, Newlyweds, Babies, Style• NYSE-traded under "XOXO"• Founded in 1996 as AOL Channel• 11 million uniques / month
• Articles / Blogs CMS• Photo Galleries• Membership / Favorites• Community Forums• Planning Tools• Local Directory• Gift Registry• Ecommerce
If you haven't heard of us… watch out, your girlfriend probably has!
![Page 3: Sharded By Business Line: Migrating to a Core Database using MongoDB and Solr](https://reader035.vdocuments.mx/reader035/viewer/2022081414/54dc82124a7959b6618b48c2/html5/thumbnails/3.jpg)
About Me
[email protected]://jasonsirota.com/
@jasonsirota
Jason SirotaDirector of Software Architecture
XO Group Inc. (The Knot)
![Page 4: Sharded By Business Line: Migrating to a Core Database using MongoDB and Solr](https://reader035.vdocuments.mx/reader035/viewer/2022081414/54dc82124a7959b6618b48c2/html5/thumbnails/4.jpg)
Current Arch: Sharded By Business LineD
ata
ba
ses
En
terp
ris
eC
on
su
me
r
GP GP ManCRM
eCommUX
My Account
UX
LocalUX / API
NationalUX
RegistryUX / API
FatTail
ContentUX
ToolsUX / API
Comm.UX
Business Intelligence
Memb.UX / API
Responsys
eCommBusiness Logic
My Acct.Business Logic
LocalBusiness Logic
NationalBusiness Logic
RegistryBusiness Logic
ContentBusiness Logic
ToolsBusiness Logic
Comm.Business Logic
Memb.Business Logic
UW(SQL)
Registry(Oracle)
Local Profiles(SQL)
Member-ship(SQL)
Ecom(SQL)
ODB(SQL)
Pluck
Comm-unity(SQL)
GR360(MySQL)
CES/ATS(SQL)
National(SQL)
Tools(SQL)
Sitecore(SQL)
UGC Photos(SQL)
![Page 5: Sharded By Business Line: Migrating to a Core Database using MongoDB and Solr](https://reader035.vdocuments.mx/reader035/viewer/2022081414/54dc82124a7959b6618b48c2/html5/thumbnails/5.jpg)
Migrating to OSS and Cloud
Co
ns
um
er
So
luti
on
s
Enterprise Service BusMSMQ (On-Premise)SQS (Cloud)
Web APIs AWS Elastic BeanstalkFederated via Apigee Services
Relational DataSQL Server
Document DataMongoDB
RelationshipsNeo4j
Free-Text SearchSolr
Data AnalysisHadoop
Key/Value DataDynamoDB
CachingCouchbase
![Page 6: Sharded By Business Line: Migrating to a Core Database using MongoDB and Solr](https://reader035.vdocuments.mx/reader035/viewer/2022081414/54dc82124a7959b6618b48c2/html5/thumbnails/6.jpg)
Why MongoDB?
• RavenDB (C#)– Worked well with C# LINQ – Cross-collection Joins (but slow..)– Very new to NoSQL Landscape– Limited to C# and REST interface
• Couchbase 2.0– Already familiar with Ops from Caching– Masterless horizontal scaling– Still in Beta during choice– Map/Reduce-based queries only
• MongoDB– Mature Document Data Store– Enterprise-level Support– High user-base– LINQ and JSON-based querying– Locking issues resolved– GeoSearching– AWS I/O issues N/A
Document DB Decided First: Schemaless Design
![Page 7: Sharded By Business Line: Migrating to a Core Database using MongoDB and Solr](https://reader035.vdocuments.mx/reader035/viewer/2022081414/54dc82124a7959b6618b48c2/html5/thumbnails/7.jpg)
Started this Migration with ourSearch Application
![Page 8: Sharded By Business Line: Migrating to a Core Database using MongoDB and Solr](https://reader035.vdocuments.mx/reader035/viewer/2022081414/54dc82124a7959b6618b48c2/html5/thumbnails/8.jpg)
![Page 9: Sharded By Business Line: Migrating to a Core Database using MongoDB and Solr](https://reader035.vdocuments.mx/reader035/viewer/2022081414/54dc82124a7959b6618b48c2/html5/thumbnails/9.jpg)
![Page 10: Sharded By Business Line: Migrating to a Core Database using MongoDB and Solr](https://reader035.vdocuments.mx/reader035/viewer/2022081414/54dc82124a7959b6618b48c2/html5/thumbnails/10.jpg)
Many Data Stores
Gowns Products
"Gown"
Photos Local Directory Articles/Blogs ECommerce User Photos
![Page 11: Sharded By Business Line: Migrating to a Core Database using MongoDB and Solr](https://reader035.vdocuments.mx/reader035/viewer/2022081414/54dc82124a7959b6618b48c2/html5/thumbnails/11.jpg)
Message-oriented Realtime Publishing
![Page 12: Sharded By Business Line: Migrating to a Core Database using MongoDB and Solr](https://reader035.vdocuments.mx/reader035/viewer/2022081414/54dc82124a7959b6618b48c2/html5/thumbnails/12.jpg)
I hope you guys can read JSON
![Page 13: Sharded By Business Line: Migrating to a Core Database using MongoDB and Solr](https://reader035.vdocuments.mx/reader035/viewer/2022081414/54dc82124a7959b6618b48c2/html5/thumbnails/13.jpg)
Message Format
Message: {EntityId: "6765aec7-370d-4f1d-82d2-97647ccea94e",SearchType: "Product",Title: "Sloan by Sottero and Midgley",Url:"http://www.theknot.com/wedding-dress/sottero-midgley/
sloansottero",
//Images appear in the Image SearchImages:[{
Id: "04ed3a07-fcb5-41da-aa74-11214dcc8e27",Url:
"http://xoedge.com/objects/0031/0107148/main_image.jpg",}],
//Used for Solr IndexingCategories:["Gowns","Fashion"],Tags: ["modern","romantic"],Facets:[
"Color":["White","Ivory"]],Attributes: {
"FeaturedVendor": true},
}
![Page 14: Sharded By Business Line: Migrating to a Core Database using MongoDB and Solr](https://reader035.vdocuments.mx/reader035/viewer/2022081414/54dc82124a7959b6618b48c2/html5/thumbnails/14.jpg)
Message-oriented Realtime Publishing
![Page 15: Sharded By Business Line: Migrating to a Core Database using MongoDB and Solr](https://reader035.vdocuments.mx/reader035/viewer/2022081414/54dc82124a7959b6618b48c2/html5/thumbnails/15.jpg)
Persister
var server = MongoServer.Create(ConfigurationManager.ConnectionStrings["MongoDB"].ToString());
var db = server.GetDatabase(config.MongoDatabase);
var collection = db.GetCollection(config.MongoCollection);
collection.Save(entity);
"EntityMappings": [{"SearchType": "LocalProfile","PersistenceType": "XO.Vendors.Core.Domain.Profile, XO.Vendors.Core","MongoDatabase": "search","MongoCollection": "profiles"
}
namespace XO.Vendors.Core.Domain{
public class Profile : Entity, IReviewAggreate {
public Address Address { get; set; }
public string Headline { get; set; }
public List<Guid> ImageIds { get; set; }
public string ImageId { get; set; }
Configure…
…Define…
…Save.
![Page 16: Sharded By Business Line: Migrating to a Core Database using MongoDB and Solr](https://reader035.vdocuments.mx/reader035/viewer/2022081414/54dc82124a7959b6618b48c2/html5/thumbnails/16.jpg)
Document Structure
![Page 17: Sharded By Business Line: Migrating to a Core Database using MongoDB and Solr](https://reader035.vdocuments.mx/reader035/viewer/2022081414/54dc82124a7959b6618b48c2/html5/thumbnails/17.jpg)
Message-oriented Realtime Publishing
![Page 18: Sharded By Business Line: Migrating to a Core Database using MongoDB and Solr](https://reader035.vdocuments.mx/reader035/viewer/2022081414/54dc82124a7959b6618b48c2/html5/thumbnails/18.jpg)
Search API
![Page 19: Sharded By Business Line: Migrating to a Core Database using MongoDB and Solr](https://reader035.vdocuments.mx/reader035/viewer/2022081414/54dc82124a7959b6618b48c2/html5/thumbnails/19.jpg)
MongoDB Challenges
• UUID Endianness– Write C# GUID to Mongo– Retrieve UUID from Python, reverses
Endianness, different value out
def upendUUID(orig):return uuid.UUID(bytes=orig.bytes_le)
• C# Driver logged phantom errors at first:– "Could not Find MongoDB"– No other indicators of outage
![Page 20: Sharded By Business Line: Migrating to a Core Database using MongoDB and Solr](https://reader035.vdocuments.mx/reader035/viewer/2022081414/54dc82124a7959b6618b48c2/html5/thumbnails/20.jpg)
Demo?
![Page 21: Sharded By Business Line: Migrating to a Core Database using MongoDB and Solr](https://reader035.vdocuments.mx/reader035/viewer/2022081414/54dc82124a7959b6618b48c2/html5/thumbnails/21.jpg)
MongoDB Instances
• 1 Replica Set
• 5 MongoDB Instances
• 3 Availability Zones
• 20 EBS Volumes (R10)
• 250 IOPS per Volume
• EBS Snapshot Backups
• S3 Data Dump
![Page 22: Sharded By Business Line: Migrating to a Core Database using MongoDB and Solr](https://reader035.vdocuments.mx/reader035/viewer/2022081414/54dc82124a7959b6618b48c2/html5/thumbnails/22.jpg)
Tested Traffic: Queries per Minute
![Page 23: Sharded By Business Line: Migrating to a Core Database using MongoDB and Solr](https://reader035.vdocuments.mx/reader035/viewer/2022081414/54dc82124a7959b6618b48c2/html5/thumbnails/23.jpg)
Actual Traffic: Queries per Minute
![Page 24: Sharded By Business Line: Migrating to a Core Database using MongoDB and Solr](https://reader035.vdocuments.mx/reader035/viewer/2022081414/54dc82124a7959b6618b48c2/html5/thumbnails/24.jpg)
How to lie with statistics…
what we tested…
…what we got
![Page 25: Sharded By Business Line: Migrating to a Core Database using MongoDB and Solr](https://reader035.vdocuments.mx/reader035/viewer/2022081414/54dc82124a7959b6618b48c2/html5/thumbnails/25.jpg)
The Knot
[email protected]://jasonsirota.com/
@jasonsirota
Jason SirotaDirector of Software Architecture
XO Group Inc. (The Knot)