Cassandra Community Webinar: MySQL to Cassandra - What I Wish I'd Known

Download Cassandra Community Webinar: MySQL to Cassandra - What I Wish I'd Known

Post on 26-Jan-2015




1 download

Embed Size (px)


A brief intro to how Barracuda Networks uses Cassandra and the ways in which they are replacing their MySQL infrastructure, with Cassandra. This presentation will include the lessons they've learned along the way during this migration. Speaker: Michael Kjellman, Software Engineer at Barracuda Networks Michael Kjellman is a Software Engineer, from San Francisco, working at Barracuda Networks. Michael works across multiple products, technologies, and languages. He primarily works on Barracuda's spam infrastructure and web filter classification data.


<ul><li> 1. Hindsight is 20/20:MySQL to CassandraMichael Kjellman (@mkjellman)Barracuda Networks</li></ul> <p> 2. What I Do Build and maintain real-time Spamdetection and Web Filter classification Java/Perl/C (and bits of everything else) Author perlcassa (Perl C* client) Frontend? Backend? Customer? Internal?Broken RAID Card? Bad Disk? I touch it all. 3. Our C* Cluster In production for ~2 years since 0.8 Running 1.2.5 + minor patches 24 nodes in 2 datacenters (2) 2TB Hard Drives (no RAID) (1) Small SSD for small hot CFs 64GB of RAM Puppet for management Cobbler for deployment Target max load at 600GB per node 4. What is real-time exactly? 5. Our Rewrite by the NumbersCassandraBasedMySQLBasedAverage ApplicationLatency2.41ms 5.0msElements in Database 32,836,767 3,946,713Elements ApplicationHandles32,836,767 314,974Element Seen Prior toTracking1st request VariousThresholdsDatacenters 2 1Average Latency ofAutomatedClassification3 seconds 8 minutes 6. Should you Rewrite? How To Survive a Ground-Up Rewrite Without LosingYour Sanity[1] Joel Spolsky Past engineering decisions preventingimplementation of new business requirements New threats smarter and more targeted[1] 7. Evolving Legacy Systems Even good developers can write sloppy code Too much duct tape Most layers applied around the database 8. Hitting the Reset Button Plan for continuous failure Easily Scalable No Single Point of Failure that you know of Many smaller boxes vs. one monolithic box 9. Whiteboard to Reality Get technical buy-in from all parties Migrate and rewrite in stages Business requirements forced hybrid period withthe old and new systems operated in parallel 10. Cassandra is Not1. Direct MySQL replacement2. Magic bullet to solve everything 11. Migrating Painful Painful Painful Tons of rewriting Tons of regressions Did I say painful? 12. So Why Migrate? C* is the best option for persistence tier Business success motivation Dont let your database hold you back 13. Lessons Learned (the good) Carefully defining data model up front Creating a flexible systems architecture thatadapts well to changes during implementation Seriously Measure twice, cut once. 14. Lessons Learned (the bad) Consider migration and delivery requirementsfrom the very beginning Adjust expectations didnt expect relying onlegacy systems for so long Make syncing data between systems a priority 15. Tips1. Start with the queries2. Think differently regarding reads3. Syncing and migrating data4. Dont use C* as a queue5. Estimate capacity6. Automate, Automate, Automate7. Some maintenance required 16. 1. Start with the Queries C* != #dontneedtothinkaboutmyschema Counters and Composites Optimize for use case Dont be afraid of writes. Storage is cheap. Optimize to reduce the number of tombstones 17. 2. Think Differently Regarding Reads Do you really need all that data at once? mysql&gt; SELECT * FROM mysupercooltableWHERE foo = bar; Slow, but eventually will work cqlsh&gt; SELECT * FROM myreallybigcfWHERE foo = bar; Wont work. Expect RPC timeout exceptions on readsgenerally after ~10,000 rows even with paging Our solutions: ElasticSearch Hadoop/Pig 18. 3. Syncing and Migrating Data Sync and migration scripts take moreseriously than production code Design sync to be continuous with bothsystems running in parallel during migration Prioritize the sync 19. 4. Dont use C* as a Queue Cassandra anti-patterns: Queues and queue-like datasets[2] Aleksey Yeschenko Tombstones + read performance Our solution: Kafka (multiple publisher, multiple consumerdurable queue)[2] 20. 5. Estimate Capacity Dont forget the Java heap (8GB Max) Plan capacity today and future Stress Tool profile node and multiply MySQL hardware != Cassandra hardware New bottlenecks thanks to C* being soawesome? I/O still an important concern with C* 21. 6. Automate, Automate, Automate Love your inner Ops self. Distributed systemsmove complexity to operations. Puppet or something similar (really) Learn CCM earlier rather than later 22. 7. Some Maintenance Required Repairs &amp; Cleanup ops automate and run frequently Rolling restart meet rollingrepair Learn jconsole Solution: Jolokia (JMX via HTTP) 23. Where is Barracuda Today? 2 years in production with Cassandra Definitely the right choice for our persistencetier 2 product lines on C* based system andanother major product in beta Achieved real-time response 24. 2.0 and Beyond Thrift -&gt; CQL CQL helps the MySQL to C* migration Easier to comprehend / grasp Everyone understands SELECT * FROM cf WHEREkey = foo; CAS and other 2.0 features make C* an evenbetter replacement option for MySQL 25. C* Community Supercalifragilisticexpialidocious community! Riak, HBase, Oracle are other options. How istheir dev community? Great client support. Great people. Greatmotivated developers. IRC: #cassandra on freenode Mailing List: </p>