advanced administration, monitoring and backup
TRANSCRIPT
![Page 1: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/1.jpg)
Advanced AdministrationMonitoring and Backup at Scale
Dr. Jeffrey BergerLead Database Engineer - Sailthru
![Page 2: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/2.jpg)
Scale The Universe!
![Page 3: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/3.jpg)
Scale The Universe!
Flat FRW Metric for isotropic cosmological geometry
![Page 4: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/4.jpg)
Scale The Universe!
Flat FRW Metric for isotropic cosmological geometry
Scale Factor
![Page 5: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/5.jpg)
Scale The Universe!
Flat FRW Metric for isotropic cosmological geometry
Scale Factor
Related to the hubble constant for an expanding universe, this does a great job of actually scaling our universe.
In fact the rate of expansion is continuing to grow and accelerate!
![Page 6: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/6.jpg)
Scale The Universe!
![Page 7: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/7.jpg)
Uhh maybe just the galaxy
![Page 8: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/8.jpg)
The world is a lot still
![Page 9: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/9.jpg)
Keep zooming in...
![Page 10: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/10.jpg)
‘Big Data’...?
![Page 11: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/11.jpg)
Sailthru
![Page 12: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/12.jpg)
Sailthru
● Extremely early adopter of MongoDB ~2009
● 4 Clusters and 9 Stand-Alone RS
● Largest is 32 shards and 5.5TB with ~1.5 billion profiles
● All production systems are housed in a colo data center
on hardware owned and operated by Sailthru
![Page 13: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/13.jpg)
Sailthru
● 4 DB Team Members
○ Me
○ Dr. Joshua Wickman
○ Chandrakant Gopalan
○ Tim Burrington
![Page 14: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/14.jpg)
Sailthru
● Our systems are composed of replica sets of 2 live nodes and 1 arbiter
● Many of our systems are ‘microsharded’
PRIMARY
ARBITER
SECONDARY
PRIMARY
ARBITER
SECONDARY
![Page 15: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/15.jpg)
Two tales of DBA struggle
No DBEs to DB team Mass Migration
What do you do if you have to move data from one data-center to another, while moving 17 replica sets into a single sharded cluster
with no (minimal) downtime?
What do you do when you join an organization which has been using
MongoDB without any DBA oversight?
![Page 16: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/16.jpg)
Welcome to the DB team
What are the most important things for a DB to set up?
MONITORING BACKUPS
![Page 17: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/17.jpg)
Monitoring
Microsharded systems are not easy to monitor!
● Multiple replica sets on a single machine
● Primaries and Secondaries often sharing hardware
● Monitoring systems for Mongo are at a instance level not server level
SHARD 1 PRIMARY
SHARD 2 SECONDARY
MEMORY
DISK IO
NETWORK IO
![Page 18: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/18.jpg)
Monitoring - MMS
MMS is a great tool for all Mongo deployments
● Built in user level permissions● Automatic topology discovery● Graphs and time series data● Breakdown by replica set for
clusters● Pulls a wealth of data
![Page 19: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/19.jpg)
Monitoring - MMS
● Built in alerting
● Many variable alerting criteria
● Integration with email, SMS, Pagerduty and more
![Page 20: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/20.jpg)
Monitoring - MMS
MMS is our backup monitoring system
● Alerting time sometimes lags behind issue time
● Organizational decision not to host MMS and that we need an internal monitoring system as our main monitor
![Page 21: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/21.jpg)
Monitoring - MMS
What we are looking forward to:● Proactive Support has some great features coming
through MMS● Enhanced monitoring and alerting options● Logging long queries? Non-indexed queries?● Perhaps we can run custom scripts and checks against
the system eventually!
![Page 22: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/22.jpg)
Monitoring - Zabbix
“Quis custodiet ipsos custodes?” - ZABBIX
![Page 23: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/23.jpg)
Monitoring - Zabbix
Monitoring mongo with Zabbixhttps://github.com/sailthru/mongodb-zabbix
● Number of voting members● Long query logging● Chunk distribution in a sharded cluster● Fsync lock status● Failover notification
![Page 24: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/24.jpg)
Monitoring - Zabbix
Custom checks and graphs - cluster monitoring
![Page 25: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/25.jpg)
Monitoring - Zabbix
Long Query Logging
![Page 26: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/26.jpg)
Monitoring - Zabbix
Zabbix does not have any automated topology discovery!
Sailthru has created its own MongoDB topological discovery tool : DB Map
● Python Process● Automatically discovers nodes or config changes● Outputs all servers and information to a Mongo collection
![Page 27: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/27.jpg)
Admin Tool - DB Map
Useful for many processes in our system
● Management scripts● Execute aggregation queries to pull specific systems● Keep Zabbix in sync using it as a source of truth● Exportable for Ansible inventory files or other
management software● Soon to be Open Sourced
Built By : Dr. Joshua Wickman
![Page 28: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/28.jpg)
Backups
Many ways to skin a… cluster....?
● Volume snapshots (within our Datacenter)
● Snapshots of cloud secondaries (Hybrid Cloud)
● MMS Backups
![Page 29: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/29.jpg)
Backups - Hybrid Cloud
SECONARY(HIDDEN)
SECONDARY
PRIMARY
DATACENTER
CLOUD
Sailthru had a hybrid cloud-physical topology.
![Page 30: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/30.jpg)
Backups - Hybrid Cloud
● Disaster recovery is immediate● Backups can be taken care of by EC2
snapshotting
There are benefits to a hybrid setup
![Page 31: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/31.jpg)
Backups - Hybrid Cloud
PRIMARY PRIMARY PRIMARY PRIMARY
SECONDARY SECONDARY SECONDARY SECONDARY
SECONDARY(hidden)
SECONDARY(hidden)
SECONDARY(hidden)
SECONDARY(hidden)
DC
Cloud
![Page 32: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/32.jpg)
Backups - Hybrid Cloud
PRIMARY PRIMARY
SECONDARY SECONDARY
SECONDARY(hidden)
SECONDARY(hidden)
● Are these secondaries on hardware provisioned equally to the others?
● Is there enough bandwidth?● Can the disks keep up with
bursts of write activity?● Are the oplogs on these
secondaries long enough?● Is the connection to the
cloud secure and stable?
![Page 33: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/33.jpg)
Backups - Hybrid Cloud
DO YOU HAVE THE TIME AND RESOURCES TO DO ALL OF THAT WORK??
We all just want backups that are fire-and-forget it!
![Page 34: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/34.jpg)
Backups - MMS
● Save on your team’s time● Save on the provisioned hardware● Much cheaper than hybrid cloud solution
Sailthru has saved almost 1 million dollars year over year
![Page 35: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/35.jpg)
Backups - MMS
● UI is easy to use and great for small/individual sets
● Need automation in order to bring up a cluster of any reasonable size○ Automation tools not yet
available out of the box● Pulls your data across the
internet - make sure you allocate this time!
![Page 36: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/36.jpg)
The Power is Turning Off...
During 2014 Sailthru was forced to move Data Centers
Additionally we made the infrastructure decision to move from 17+ separate replica sets to a sharded cluster.
![Page 37: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/37.jpg)
Data Migrations
DC1 DC2
CLOUD
With limited bandwidth and servers this becomes some interview’s brain teaser
![Page 38: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/38.jpg)
Data Migrations - Dumps
DC1 DC2
MongodumpNetcat
Write to file then Mongorestore
● Lots of combinations, none ended up being fast enough.● Hampered by disk writes and reads.● If you touch disk you lose! The floor is lava!
![Page 39: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/39.jpg)
Data Migrations - Mongopipe
Custom multiprocessing python process to insert without hitting disk
● Using python, multiprocessing, ZMQ, and some custom C objects
● Got around 2.4 bulk insert issue by sorting on shard key● Never touches disk, all processing is done in memory● Directly insert into many local mongos instances● Open source coming soon!
![Page 40: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/40.jpg)
Data Migrations - Mongopipe
Cursor
Cursor
Cursor
Writer
Writer
Writer
Mongos
Mongos
Mongos
Target Cluster
ZMQ Batch Inserts
Sort on Shard Key
DC1 DC2
![Page 41: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/41.jpg)
Data Migrations - Mongopipe
insert query update delete getmore command
64982 25 *0 *0 0 45|0
62484 23 *0 *0 0 50|0
37490 15 *0 *0 0 25|0
-1073585030 -4978381 *0 *0 -163 -5042014|0
197448 70 *0 *0 0 144|0
227440 105 *0 *0 0 181|0
49986 45 *0 *0 0 59|0
![Page 42: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/42.jpg)
Data Migrations - Mongo Connector
● Mongoconnector is a way to mirror mongodb operations, creating almost a virtual secondary without adding it to a replica set
● Great for data migrations without downtime
https://github.com/10gen-labs/mongo-connector
![Page 43: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/43.jpg)
Data Migrations - Mongo Connector
MONGO
OP LOG1….2….3….
TARGETDATASTORE
Elasticsearch..Solr...
Mongodb...
MONGO CONNECTOR
OPLOGMNGR.
DOC MNGR.
DOC MNGR.
DOC MNGR.
![Page 44: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/44.jpg)
Access Patterns - Keystore
● What if I want to do a lot of findOnes on a cluster?● On many unique fields?● Am I doomed to many scatter gathers?
SHARD SHARD SHARD SHARD
MONGOSAssume sharded on _id: hashed
findOne({“ssn”: X}) findOne({“cell_phone”: X}) findOne({“_id”: X})
Created by : Ian White
![Page 45: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/45.jpg)
Access Patterns - Keystore
Find by SSN
SHARDED COLLSharded on:{_id: hashed}
Doc:{ _id: SSN sid: ObjectId()}
Query on _id (shard key)Return an ObjectId
Main Sharded CollectionSharded on :{_id: hashed}
Use sid that was found to query the _id in the main collection
![Page 46: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/46.jpg)
Access Patterns - Keystore
2 queries rather than n where n is your number of shards** Not useful unless you are sharded out very far **
● Time averaged by keystore : ~30 seconds● Time averaged by direct lookup: ~170 seconds
** tests done on a 32 shard cluster
![Page 47: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/47.jpg)
Other Tools - Mongoexup
● Cron jobs are unreliable● Any ‘prototype’ inevitably becomes production● Constructed a python scheduler daemon to execute
these tasks● Looking to open source in the future
Business need to regularly execute mongoexport and uploads
Built By : Chandrakant Gopalan
![Page 48: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/48.jpg)
Other Tools - Mongoexup
Mongo MongoExUp S3Greenlets Greenlets
Job Status Information
![Page 49: Advanced Administration, Monitoring and Backup](https://reader030.vdocuments.mx/reader030/viewer/2022013108/55c9a73ebb61eb141c8b456d/html5/thumbnails/49.jpg)
What are we doing next?
● Open source even more of our tools● Ansible Automation● Building API layers around all our DBs
○ Tornado - ASYNC RULES● MongoDB + Other Data Stores
○ Enhancing the Keystore concept● Upgrading
○ WT○ RocksDB