On MongoDB Backup

2014.05.02.葉秉哲

William Yeh

Why backup?

Availability vs. recovery

Cost is impacted by these decisions… How much data can you afford to lose? How long can you afford to be off-line?

Ref: “MongoDB Backups & Disaster Recovery”http://www.mongodb.com/presentations/webinar-backups-and-disaster-

recovery

2.

Tasks

3.

Backup Restore

One-time snapshot

Incremental

Two kinds of backup

One-time snapshot Capture a consistent snapshot at a specific

point in timeNot easy in a distributed zero-downtime

production system.

Incremental backup Capture differences since last one-time

snapshotOplog is N/A in MongoDB config servers (to be

discussed later)

4.

Cluster architecture in production

5.

How about MMS?MongoDB Management Service http://mms.mongodb.com

6.

$$

7.

8.

Backup Restore

One-time snapshot

Incremental

One-time snapshot steps

1. Stop balancer (mongos + config)

2. Take snapshot of config data

3. For each shard, take snapshot of one secondary

4. Re-start the balancer

5. Copy snapshots in steps 2 & 3

Ref: “Backup a Sharded Cluster with Filesystem Snapshots”http://bit.ly/1hqO1kj

9.

10.10.

1

2

3

4

5 Copy outCopy out

Take snapshotTake snapshot

#1#1Stop BalancerStop Balancer

11.

Stop balancer… Why?

“If MongoDB migrates a chunk during a backup, you can end with an inconsistent snapshot of your sharded cluster. Never run a backup while the balancer is active.”

Ref: “Disable Balancing During Backups”http://bit.ly/1rnRMfV

12.

#1: Stop balancer

var sleep_time_in_ms = 10000;

sh.setBalancerState(false);

while (sh.getBalancerState()

|| sh.isBalancerRunning()) {

sleep(sleep_time_in_ms);

}

// now the balancer is stopped…

Ref: “Disable Balancing During Backups”http://bit.ly/1rnRMfV

13.

#2#2Snapshot -Snapshot -config dataconfig data

14.

#3#3Snapshot -Snapshot -secondarysecondary

of each shardof each shard

15.

Snapshot the “dbpath”

Snapshoting step, almost the same: #2: Config server #3: Secondary

… but just one difference: “Never use db.fsyncLock() on config

databases.” Ref: “Backup a Sharded Cluster with Filesystem Snapshots”

http://bit.ly/1hqO1kj

16.

Snapshot approaches

“True” snapshot LVM, ZFS, Btrfs, etc. Cloud snapshot (EC2, GCE, etc.)

“Fake-and-slowwwww” dump mongodump fsyncLock() + tar + pigz Wordnik tool: https://github.com/wordnik/wordnik-oss

17.

Considerations for snapshot

Timing “Freeze” time period Archive (including compression &

transmission)

Storage efficiency Copy-on-write snapshot is better

18.

Snapshot: copy on write

Ref: “ 磁碟配額 (Quota) 與進階檔案系統管理”http://bit.ly/R67Yrd

19.

#4#4Re-start BalancerRe-start Balancer

20.

#4: Re-start balancer

sh.setBalancerState(true);

// now the balancer is re-started…

Ref: “Enable the Balancer”http://bit.ly/1hfKFl2

21.

#5#5Copy SnapshotCopy Snapshot

22.

Considerations for snapshot

Timing “Freeze” time Archive (including compression &

transmission)

Storage efficiency Copy-on-write snapshot is better

23.

24.

Backup Restore

One-time snapshot

Incremental

Restore approaches

“True” snapshot LVM, ZFS, Btrfs, etc. Cloud snapshot (EC2, GCE, etc.)

“Fake-and-slowwwww” dump mongorestore Daemon stop + untar Wordnik tool: https://github.com/wordnik/wordnik-oss

25.

26.

Backup Restore

One-time snapshot

Incremental

Oplog

Ordered list of write operations

27.

Oplog tools

Ready-to-use command line tools mongodump + mongorestore Wordnik tool: https://github.com/wordnik/wordnik-oss

Tayra tool:http://www.jroller.com/DhavalDalal/entry/tayra_an_incremental_backup_tool

mongosync:http://nosqldb.org/topic/5173d275cbce24580a033bd8

Still many others: https://github.com/search?q=mongo+oplog&type=Repositories

28.

Oplog in real-life

Replication lag Inspect it in real production environment!

Oplog is N/A in MongoDB config servers … Still need to deal with the balancer … Still need to use snapshot techniques

29.

Replication lag in real-life

30.

References

MongoDB: The Definitive Guide, 2/e MongoDB Manual

http://docs.mongodb.org/manual/administration/backup/

MongoDB Backups & Disaster Recovery http://www.mongodb.com/presentations/webinar-backups-and-disaster-recovery

Backup Strategies: Keeping Your Data Safe

http://www.mongodb.com/presentations/backup-strategies-keeping-your-data-safe

http://www.slideshare.net/fehguy/keeping-mongodb-data-safe

31.