lightning talk: mongodb sharding

9
I have a good shard key now what? David Murphy , Mongo Master Lead DBA, ObjectRocket @dmurphy_data @objectrocket

Upload: mongodb

Post on 01-Nov-2014

488 views

Category:

Technology


11 download

DESCRIPTION

 

TRANSCRIPT

  • 1. I have a good shard key now what?David Murphy , Mongo MasterLead DBA, ObjectRocket@dmurphy_data @objectrocket

2. Background 16 yrs in databases, development, & system engineering MySQL and NoSQL Architect for largest game company Lead DBA @ ObjectRocket Mongo Master with a focus on sharding, chunks, andscaling mongo beyond normal means. 3. The balancer is great but you can also perform some proactive checks to predict issues.Understanding the reasons why not all chunks are equalCounting Chunks per shard without sh.status()Finding out how many chunkMoves or splits occur at a timeGetting the size of all chunks for a collection 4. Sharding, the balancer, and chunks 5. Ways to Count ChunksMany functions for this: sh.status() db.collection.stats()Simple query for counting chunks: db.chunks.count({ns:"database.collection"});Why I prefer using Aggregation: Faster as you only look at one type of data Provides chunk count per shard Can programmatically use the output 6. Tracking moveChunks and SplitsThe config.changelog tracks chunk changes.The types of changes to chunks are split moveChunk moveChunk.start moveChunk.from moveChunk.to moveChunk.commit*Bold means I track with aggregations by hour/day 7. Sizing ChunksdataSize command - Very expensiveScans all documents to be 100% accuratefind().count() * avgObjSize on chunk rangesRough size estimate based on statsChunkHunter.pyNew script from ObjectRocket to:a) copy chunks meta data to new collectionb) scan each chunk there with dataSize or countc) output summary 8. Anatomy of a ChunkHunter 9. Contact@dmurphy_data@[email protected]://www.objectrocket.comWE ARE HIRING! (DBA,DEVOPS, and more)https://www.objectrocket.com/careersGitHub for ChunkHunter (later today):https://github.com/objectrocket/Utils/blob/master/DBA/ChunkHunter.py