open source backup conference 2014: bareos in scientific environments, by jan behrend
TRANSCRIPT
für Radioastronomie
Max-Planck-Institut
Bareos in Radio Astronomy - Scaling up using Virtual Full Backups
Bareos in Radio Astronomy –Scaling up using Virtual Full Backups
Jan Behrend
Max Planck Institute for Radio Astronomy
Open Source Backup ConferenceSeptember 23rd 2014
für Radioastronomie
Max-Planck-Institut
Bareos in Radio Astronomy - Scaling up using Virtual Full Backups
OverviewAbout the InstituteBackup Goals and LimitationsThe ChallengeImplementationConfiguraton StrategyVirtual Full BackupsIntegration with DRBDIntegration with REARWishlist
Bareos in Radio Astronomy - Scaling up using Virtual Full Backups
Max Planck Institute for Radio Astronomy
– Founded in 1966– Main Building Bonn / 100m Radio Telescope Effelsberg– Apex Telescope in the Atacama Desert, Chile
– IT staff of 11 people– 204 servers (132 virtualized, 110 Bareos clients)– 64 node cluster– . . .
http://mpifr.de
für Radioastronomie
Max-Planck-Institut
Bareos in Radio Astronomy - Scaling up using Virtual Full Backups
Lofar Antenna Field
“Software” Telescope44 Stations
für Radioastronomie
Max-Planck-Institut
Bareos in Radio Astronomy - Scaling up using Virtual Full Backups
Scientific Raw Data
Project Output per Observation RunK-Band receiver: 2 TiBH1-Survey: 3 TiBPulsar search: 4 TiBLeap: 7 TiBLofar: 50 TiBGlow: 120 TiB. . .
∑186 TiB
Forget it!
für Radioastronomie
Max-Planck-Institut
Bareos in Radio Astronomy - Scaling up using Virtual Full Backups
Scientific Raw Data
Project Output per Observation RunK-Band receiver: 2 TiBH1-Survey: 3 TiBPulsar search: 4 TiBLeap: 7 TiBLofar: 50 TiBGlow: 120 TiB. . .
∑186 TiB
Forget it!
für Radioastronomie
Max-Planck-Institut
Bareos in Radio Astronomy - Scaling up using Virtual Full Backups
GoalsFast recovery of:
specific files / directories(many) complete systems
LimitationsTime
Backup window (scientists are night birds)Network bandwidth (usually 1GBit/s)Resources on backup clients (I/O, RAM, CPU)
Storage space (disk and tape)
What can we do?parallel jobs, Virtual Full Backupsvolume retention
für Radioastronomie
Max-Planck-Institut
Bareos in Radio Astronomy - Scaling up using Virtual Full Backups
GoalsFast recovery of:
specific files / directories(many) complete systems
LimitationsTime
Backup window (scientists are night birds)Network bandwidth (usually 1GBit/s)Resources on backup clients (I/O, RAM, CPU)
Storage space (disk and tape)
What can we do?parallel jobs, Virtual Full Backupsvolume retention
für Radioastronomie
Max-Planck-Institut
Bareos in Radio Astronomy - Scaling up using Virtual Full Backups
GoalsFast recovery of:
specific files / directories(many) complete systems
LimitationsTime
Backup window (scientists are night birds)Network bandwidth (usually 1GBit/s)Resources on backup clients (I/O, RAM, CPU)
Storage space (disk and tape)
What can we do?parallel jobs, Virtual Full Backupsvolume retention
für Radioastronomie
Max-Planck-Institut
Bareos in Radio Astronomy - Scaling up using Virtual Full Backups
“The Remains of the Day” without Raw Data
Full Backup Volume: 55 TiB (1.5M files / TiB)Differential Backup Volume: 566 GiB (4K files / GiB)Incremental Backup Volume: 102 GiBTwo backup copies
Time for a complete Full Backup and its 2nd copy
55 TiB130 MiB
s
· 2 · 2 · 1 d60 · 60 · 24 s
= 20.54 days
High Volume “Longterm” ClientsVirtual Fulls every half year, Incrementals every SaturdayLongterm Full Backup Volume: - 50 TiB (1M files / TiB)⇒ Full Volume every 8 weeks: 5 TiB (6M files / TiB)
für Radioastronomie
Max-Planck-Institut
Bareos in Radio Astronomy - Scaling up using Virtual Full Backups
“The Remains of the Day” without Raw Data
Full Backup Volume: 55 TiB (1.5M files / TiB)Differential Backup Volume: 566 GiB (4K files / GiB)Incremental Backup Volume: 102 GiBTwo backup copies
Time for a complete Full Backup and its 2nd copy
55 TiB130 MiB
s
· 2 · 2 · 1 d60 · 60 · 24 s
= 20.54 days
High Volume “Longterm” ClientsVirtual Fulls every half year, Incrementals every SaturdayLongterm Full Backup Volume: - 50 TiB (1M files / TiB)⇒ Full Volume every 8 weeks: 5 TiB (6M files / TiB)
für Radioastronomie
Max-Planck-Institut
Bareos in Radio Astronomy - Scaling up using Virtual Full Backups
“The Remains of the Day” without Raw Data
Full Backup Volume: 55 TiB (1.5M files / TiB)Differential Backup Volume: 566 GiB (4K files / GiB)Incremental Backup Volume: 102 GiBTwo backup copies
Time for a complete Full Backup and its 2nd copy
55 TiB130 MiB
s
· 2
· 2 · 1 d60 · 60 · 24 s
= 20.54 days
High Volume “Longterm” ClientsVirtual Fulls every half year, Incrementals every SaturdayLongterm Full Backup Volume: - 50 TiB (1M files / TiB)⇒ Full Volume every 8 weeks: 5 TiB (6M files / TiB)
für Radioastronomie
Max-Planck-Institut
Bareos in Radio Astronomy - Scaling up using Virtual Full Backups
“The Remains of the Day” without Raw Data
Full Backup Volume: 55 TiB (1.5M files / TiB)Differential Backup Volume: 566 GiB (4K files / GiB)Incremental Backup Volume: 102 GiBTwo backup copies
Time for a complete Full Backup and its 2nd copy
55 TiB130 MiB
s
· 2 · 2
· 1 d60 · 60 · 24 s
= 20.54 days
High Volume “Longterm” ClientsVirtual Fulls every half year, Incrementals every SaturdayLongterm Full Backup Volume: - 50 TiB (1M files / TiB)⇒ Full Volume every 8 weeks: 5 TiB (6M files / TiB)
für Radioastronomie
Max-Planck-Institut
Bareos in Radio Astronomy - Scaling up using Virtual Full Backups
“The Remains of the Day” without Raw Data
Full Backup Volume: 55 TiB (1.5M files / TiB)Differential Backup Volume: 566 GiB (4K files / GiB)Incremental Backup Volume: 102 GiBTwo backup copies
Time for a complete Full Backup and its 2nd copy
55 TiB130 MiB
s
· 2 · 2 · 1 d60 · 60 · 24 s
= 20.54 days
High Volume “Longterm” ClientsVirtual Fulls every half year, Incrementals every SaturdayLongterm Full Backup Volume: - 50 TiB (1M files / TiB)⇒ Full Volume every 8 weeks: 5 TiB (6M files / TiB)
für Radioastronomie
Max-Planck-Institut
Bareos in Radio Astronomy - Scaling up using Virtual Full Backups
“The Remains of the Day” without Raw Data
Full Backup Volume: 55 TiB (1.5M files / TiB)Differential Backup Volume: 566 GiB (4K files / GiB)Incremental Backup Volume: 102 GiBTwo backup copies
Time for a complete Full Backup and its 2nd copy
55 TiB130 MiB
s
· 2 · 2 · 1 d60 · 60 · 24 s
= 20.54 days
High Volume “Longterm” ClientsVirtual Fulls every half year, Incrementals every SaturdayLongterm Full Backup Volume: - 50 TiB (1M files / TiB)⇒ Full Volume every 8 weeks: 5 TiB (6M files / TiB)
für Radioastronomie
Max-Planck-Institut
Bareos in Radio Astronomy - Scaling up using Virtual Full Backups
Backup Hardware2 Tape Libraries: Spectra Logic T950
2 x LTO5 drives80 Slots a 1,5 TiB = 160 TiB (too small by now)Dedicated cleaning partition
2 physical servers (Fujitsu RX300S6)2x 30 TiB RAID storage (Nexan Satabeast2)(Mostly) dedicated 1GbE/10GbE backup network
für Radioastronomie
Max-Planck-Institut
Bareos in Radio Astronomy - Scaling up using Virtual Full Backups
Hardware Infrastructure
FC SAN FC SAN
BACKUPSRV2cold standby
BACKUPSRV1
BACKUP LAN BACKUP LAN
INTRANET BONNINTRANET EFFELSBERG
TAPE LIBRARY 1TAPE LIBRARY 2
8GBit
1GBit
3x10GBit
10GBit 10GBit
SATABEAST (30 TiB)
40km
SATABEAST (30 TiB)
für Radioastronomie
Max-Planck-Institut
Bareos in Radio Astronomy - Scaling up using Virtual Full Backups
Configuration StrategyIncrementals to disk for better restore performanceFulls and Differentials to cheaper tape storagePer client config (DIR and SD) for easier configmanagement:
compression, quota, reservation, encryption ...easy templating for new backup clients
Focus on schedules / retentionDaily backup summary (3rd party tools)Dedicated DB partition· · ·
für Radioastronomie
Max-Planck-Institut
Bareos in Radio Astronomy - Scaling up using Virtual Full Backups
Virtual Full Backups
Reorganizes existing backups to create a new Full
Last Full (2 months ago)FullPool
Last Differential (1 week ago)DiffPool
Incrementals (x days ago)IncrementalPool
New Virtual Full (today)VirtualPool
Subsequent Migration
für Radioastronomie
Max-Planck-Institut
Bareos in Radio Astronomy - Scaling up using Virtual Full Backups
Virtual Full Backups
+++ No backup client interaction!(basically) no backup time limit
– Needs designated pool to avoid lockups– No “MaxFullInterval” config option
für Radioastronomie
Max-Planck-Institut
Bareos in Radio Astronomy - Scaling up using Virtual Full Backups
What is DRBD?
Taken from http://www.drbd.org/home/what-is-drbd
für Radioastronomie
Max-Planck-Institut
Bareos in Radio Astronomy - Scaling up using Virtual Full Backups
Integrating Bareos with HA / DRBD
DRBD volume backed up by HA controlled BAREOS instance
IMAP FailoverIP: 134.104.18.110BAREOS on /REAR enabled
IMAP FailoverIP: 134.104.18.111BAREOS on /REAR enabled
/var/mail on DRBDBAREOS on /var/mailIP: 134.104.18.77...
Client {Address = 134.104.18.110...FileSet {File = /
Client {Address = 134.104.18.77...FileSet {File = /var/mail
Client {Address = 134.104.18.111...FileSet {File = /
für Radioastronomie
Max-Planck-Institut
Bareos in Radio Astronomy - Scaling up using Virtual Full Backups
REAR: Relax and Recover
Linux bare metal disaster recovery solutionCreates bootable mediaRecovery takes care of all disk configurationCan use Bareos to restore content
für Radioastronomie
Max-Planck-Institut
Bareos in Radio Astronomy - Scaling up using Virtual Full Backups
Bareos Client config
/etc/rear/local.confGRUB_RESCUE=OUTPUT=ISOBACKUP=BAREOSBACKUP_URL="rsync://backupsrv/srv/rear/"
/etc/bareos/bconsole.confConsole {
Name = client-restorePassword = "password"
}
für Radioastronomie
Max-Planck-Institut
Bareos in Radio Astronomy - Scaling up using Virtual Full Backups
Bareos Client config
/etc/rear/local.confGRUB_RESCUE=OUTPUT=ISOBACKUP=BAREOSBACKUP_URL="rsync://backupsrv/srv/rear/"
/etc/bareos/bconsole.confConsole {
Name = client-restorePassword = "password"
}
für Radioastronomie
Max-Planck-Institut
Bareos in Radio Astronomy - Scaling up using Virtual Full Backups
Bareos Server config
/etc/bareos/conf.d/client.dir.confConsole {Name = client-restorePassword = "password"ClientACL = client-fdJobACL = Restore, clientFileSetACL = clientCommandACL = status, restore[...]
}
http://relax-and-recover.org
für Radioastronomie
Max-Planck-Institut
Bareos in Radio Astronomy - Scaling up using Virtual Full Backups
WishlistIntegration of virtualization solutions on hypervisor levelWrite to more than one destination at onceRolling Spool/DespoolAutomatic spooling when writing to tapeNo copy of jobs using a base job?”MaxFullInterval“ for Virtual Full Backups
für Radioastronomie
Max-Planck-Institut
Bareos in Radio Astronomy - Scaling up using Virtual Full Backups
Want configs? Send me a mail to [email protected]
Any questions?
Notable reference:http://myunix.dk/category/bacula
für Radioastronomie
Max-Planck-Institut
Bareos in Radio Astronomy - Scaling up using Virtual Full Backups
Want configs? Send me a mail to [email protected]
Any questions?
Notable reference:http://myunix.dk/category/bacula