queue in the cloud with mongo db
DESCRIPTION
Leveraging MongoDB as a queue infrastructure. This talk describes a low-overhead approach for building a queue as a service built on top of MongoDB, covering concerns of durability, long running processes and distributed processing.TRANSCRIPT
![Page 1: Queue in the cloud with mongo db](https://reader033.vdocuments.mx/reader033/viewer/2022042714/55515adab4c9059f768b4aa1/html5/thumbnails/1.jpg)
QUEUE IN THE CLOUD WITH MONGODBMONGODB LA 2013
NURI HALPERIN
![Page 2: Queue in the cloud with mongo db](https://reader033.vdocuments.mx/reader033/viewer/2022042714/55515adab4c9059f768b4aa1/html5/thumbnails/2.jpg)
QUEUE
![Page 3: Queue in the cloud with mongo db](https://reader033.vdocuments.mx/reader033/viewer/2022042714/55515adab4c9059f768b4aa1/html5/thumbnails/3.jpg)
USAGE
Ordered execution
Buffering consumer/producer
Work distribution
![Page 4: Queue in the cloud with mongo db](https://reader033.vdocuments.mx/reader033/viewer/2022042714/55515adab4c9059f768b4aa1/html5/thumbnails/4.jpg)
GOALS OF PROJECT
Leverage Mongo
• Reduce ops overhead by reusing infrastructure• Map queue semantics to Mongo’s strengths
Reliable
• Durable - support long running process• Resilient to machine failure• Narrow down window of failure/ data loss.
Centralized, distributed:
• Multiple producers• Multiple consumers
![Page 5: Queue in the cloud with mongo db](https://reader033.vdocuments.mx/reader033/viewer/2022042714/55515adab4c9059f768b4aa1/html5/thumbnails/5.jpg)
ITERATION 0
Capped collection – not the perfect choice
• Tailing queue seems attractive, but…• Need external sync to avoid double-consume• Secondary indexes and updating are anti-pattern
Relaxing FIFO is OK
• No guarantee that first-popped is first done• Multi-client is negated if they have to sync on execution order• Race condition for queue insertion has same effect
Conclusion: Project doesn’t use capped collection and relaxes FIFO.
![Page 6: Queue in the cloud with mongo db](https://reader033.vdocuments.mx/reader033/viewer/2022042714/55515adab4c9059f768b4aa1/html5/thumbnails/6.jpg)
PARANOID BY DESIGN
Network diesProcess dies
DB dies
Machine dies Poison letter Dead letter
![Page 7: Queue in the cloud with mongo db](https://reader033.vdocuments.mx/reader033/viewer/2022042714/55515adab4c9059f768b4aa1/html5/thumbnails/7.jpg)
ITERATION 1
db.q4foo.save({v:{f:1}})
db.q4foo.findAndModify({query: {}, sort: {_id:1}, remove: true})
Hot: quick and simple
Not: dead client, dead in transit, no trace
![Page 8: Queue in the cloud with mongo db](https://reader033.vdocuments.mx/reader033/viewer/2022042714/55515adab4c9059f768b4aa1/html5/thumbnails/8.jpg)
ARE WE THERE YET?
Network diesProcess dies
DB dies
Machine dies Poison letter Dead letter
![Page 9: Queue in the cloud with mongo db](https://reader033.vdocuments.mx/reader033/viewer/2022042714/55515adab4c9059f768b4aa1/html5/thumbnails/9.jpg)
QUEUE SEMANTICSLocal / Memory Distributed
Push Put
Pop Get << visibility >>
<< exception >> Release << retry >>
Delete
<< exception >>
![Page 10: Queue in the cloud with mongo db](https://reader033.vdocuments.mx/reader033/viewer/2022042714/55515adab4c9059f768b4aa1/html5/thumbnails/10.jpg)
ITERATION 2db.q4foo.save({v:{f:1}, dq: null})
db.q4foo.findAndModify( { query: { dq: null}, sort: {_id:1}, update:{ $set: { dq: later(60)}}})
… If processing was success => delete..
Hot: If client dies, item remains in queue. Data not lost.
Not: index on _id less useful in high volume.
![Page 11: Queue in the cloud with mongo db](https://reader033.vdocuments.mx/reader033/viewer/2022042714/55515adab4c9059f768b4aa1/html5/thumbnails/11.jpg)
ARE WE THERE YET?
Network diesProcess dies
DB dies
Machine dies Poison letter Dead letter
![Page 12: Queue in the cloud with mongo db](https://reader033.vdocuments.mx/reader033/viewer/2022042714/55515adab4c9059f768b4aa1/html5/thumbnails/12.jpg)
ITERATION 3db.q4foo.save({v:{f:1}, dq: null, pc: 0})
db.q4foo.findAndModify({query: { dq: null, pc:{$lt:3}}, sort: {_id:1}, update:{$set:{dq:later(60)},$inc:{pc:1}}}) // consume
db.q4foo.findAndModify({query: {_id:"..."}, update:{$set:{dq: null}}}) // release
Hot: An item can be retried automatically (pc) after released. Exhausted item remains in queue.
Not: Not strict FIFO.
![Page 13: Queue in the cloud with mongo db](https://reader033.vdocuments.mx/reader033/viewer/2022042714/55515adab4c9059f768b4aa1/html5/thumbnails/13.jpg)
ARE WE THERE? YES.
Network diesProcess dies
DB dies
Machine dies Poison letter Dead letter
![Page 14: Queue in the cloud with mongo db](https://reader033.vdocuments.mx/reader033/viewer/2022042714/55515adab4c9059f768b4aa1/html5/thumbnails/14.jpg)
ITERATION 4
Ensure your queue writes use applicable durability
• db.q4foo.save() + getLastError(…)• db.q4foo.findAndModify () + getLastError(…)
Replica sets for durability only. No capacity or speed gain.
![Page 15: Queue in the cloud with mongo db](https://reader033.vdocuments.mx/reader033/viewer/2022042714/55515adab4c9059f768b4aa1/html5/thumbnails/15.jpg)
OTHER THOUGHTSCreate admin jobs to monitor queues:
• Growth• Retries exhausted
Consider TTL risks (ex: client failure before calling Release())
Consider idempotent operations when possible
Design clients to back off polling
Separate queue vs. extra “topic” field
Consider dedicated DB for write-lock scope
Capped vs. regular collection – capped now can have _id, in-place update.