the modern way postgresql backups - hagander backups... · 3/ 25/ 2019 p o stgre sql ba ck ups the...

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 1/50

PostgreSQL BackupsPostgreSQL Backupsthe Modern Waythe Modern Way

Nordic PGDay 2019 Copenhagen, Denmark

Magnus Hagander magnus@hagander.net

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 2/50

Magnus HaganderMagnus HaganderRedpill Linpro

Infrastructure servicesPrincipal database consultant

PostgreSQLCore Team memberCommitterPostgreSQL Europe

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 3/50

So, backups...So, backups...Do you make them?

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 4/50

BackupsBackupsAre not superseded by replicationOr cloudOr containers..

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 5/50

BackupsBackupsAre boringBut I'm glad you have them

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 6/50

BackupsBackupsWhen did you last restore?

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 7/50

PostgreSQL backupsPostgreSQL backupsOk, enough genericWhat about backups in PostgreSQL?

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 8/50

Seen this before?Seen this before?pg_dump options:pg_dump options:

-Fc = custom format -Z = compression -j = parallel -a = data only, -s = schema only -n = schema, -t = table ...

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 9/50

pg_dumppg_dumpDon't use for backups

Has other good usecasesToo slow to restoreToo much overheadNo PITRExceptions, of course

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 10/50

Physical backupsPhysical backupsBase backupsWith or without log archiveFast restoreFull cluster onlyPlatform specific

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 11/50

Base backupsBase backups #!/bin/bash set -e psql -U postgres -q "SELECT pg_start_backup('foo')" tar cfz /backup/$(date +%Y%m%d).tar.gz /var/lib/pgsql/data psql -U postgres -q "SELECT pg_stop_backup()"

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 12/50

Base backupsBase backupsSo many ways to get that wrong

Spot one?

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 13/50

Base backupsBase backupsThis used to be the only wayMany scripts around that does itMost of those are broken...

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 14/50

pg_basebackuppg_basebackupBase backup over replication protocolSafeError handling and recoveryFor most cases

(we'll cover other options later)

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 15/50

pg_basebackuppg_basebackup #!/bin/bash set -e pg_basebackup -D /backup/$(date +%Y%m%d) -Ft

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 16/50

Needs replicationNeeds replicationEnabled by default in 10!Older versions:

wal_level = hot_standby max_wal_senders = 10

local replication postgres peer

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 17/50

Backup formatsBackup formatsplain

Safe copy of data directoryNot good with multiple tablespaces

tarDestination still a directoryEach tablespace gets one file

base.tar

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 18/50

Transaction logTransaction logWAL required to restore backupFrom beginning of backup to endIn the log archive, right?

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 19/50

Including WALIncluding WALAlways use -x or -X to include WAL

Default in 10Makes backup independently consistent

With or without log archiveMay back up WAL twice

Use even with log archive!

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 20/50

Including WALIncluding WAL-X fetch

Fetches WAL at end of backupCan fail if WAL rotated

-X streamReplicates WAL over secondary connectionFewer failure scenariosDoes not work with tar prior to version 10

-X noneTurn off (10+ only)

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 21/50

Replication slotsReplication slotspg_basebackup can fall behind on < 10Use replication slot

Don't forget to drop!PostgreSQL 10 uses ephemeral slot

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 22/50

Backup compressionBackup compressionpg_basebackup -Z

Compression happens in pg_basebackupTar format onlyCPU usageRemote server?

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 23/50

Transfer compressionTransfer compressionSSL compression

Much harder these daysssh tunneling

Does not work with WAL ssh mydbserver -c "pg_basebackup -Ft -D- -Z9 -Xnone" > backup.tgz

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 24/50

That's it!That's it!With that, you have backupsThat workAnd are (reasonably) safe

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 25/50

PITRPITRPoint in time recoveryYou all want itA bit more setting up

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 26/50

archive_commandarchive_commandTo use PITR, we use log archivinglike this?

archive_command = 'test ! -f /mnt/archivedir/%f && cp %p /mnt/archivedir/%f'

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 27/50

Don't do that!Don't do that!

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 28/50

pg_receivewalpg_receivewalRuns on archive serverUses streaming replicationGenerates log archive

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 29/50

pg_receivewalpg_receivewalMore granular recoverySafe against server restartsCan follow timeline switches on master

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 30/50

pg_receivewalpg_receivewalAlways use with replication slot

As of 9.4But we said modern..

Backups should block

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 31/50

pg_receivewalpg_receivewal pg_receivewal -D /log/archive -h master -S backup

Ensure it's restarted!

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 32/50

Backup retentionBackup retentionHow long to keep around?What granularity?...

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 33/50

Backup retentionBackup retentionRecovery needs:

Base backupAll WAL from start to endAll WAL from end to pitr

(that's why we use -x!)

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 34/50

Backup retentionBackup retentionfind is o�en enoughDelete logs older than X, base older than Y

Safe if -x was used! #!/bin/bash find /var/backups/basebackup -type f -mtime +30 -print0 | xargs -0 -r /bin/rm find /var/backups/wal -type f -mtime +7 -print0 | xargs -0 -r /bin/rm

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 35/50

Not enough?Not enough?Handles the simple casesBut has limitationsParticularly in management

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 36/50

Other toolsOther toolspgBackRestBarman

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 37/50

pgBackRestpgBackRestBackup schedulingLog archivingRetention managementMulti-serverRestore shortcutsObsessive validation

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 38/50

pgBackRestpgBackRestDeveloped by CrunchyDataPerl

Moving to CMIT licensessh but not rsync

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 39/50

pgBackRestpgBackRestCustom protocolParallel backups (compression)Full/Differential/Incremental

Segment basedDelta restore

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 40/50

pgBackRestpgBackRestValidates checksumsChecksums backups

Every time

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 41/50

pgBackRestpgBackRestNo pg_receivewal support

YetNo Windows support

Yet

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 42/50

BarmanBarmanBackup schedulingLog archivingRetention managementMulti-serverRestore shortcuts

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 43/50

BarmanBarmanDeveloped by 2ndQuadrantPythonGPLv3Primarily ssh+rsync

1.6 learned about pg_receivewal2.0 learned about pg_basebackup

Before that, no (safe) concurrent backup support

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 44/50

What aboutWhat about

Enterprise product XEnterprise product X??

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 45/50

Enterprise product XEnterprise product Xpg_basebackup

Run as pre-backup commandOptionally clean up in post-backup

pgbackrest/barmanTo diskLet backup so�ware take it form there

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 46/50

SummarySummary

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 47/50

Don't roll your own!Don't roll your own!

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 48/50

Don't roll your ownDon't roll your ownToo many pitfallsBoth base backups and archivingBackups are too important!

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 49/50

Don't roll your ownDon't roll your ownPrimary choice

Built-in (pg_basebackup)If it's enough

Secondary choicepgBackRestBarman

Tertiary choiceRestart from top of slide

3/25/2019 PostgreSQL Backups the Modern Way

localhost:9999/?print-pdf/#/ 50/50

Thank you!Thank you!Magnus Hagander

magnus@hagander.net @magnushagander

https://www.hagander.net/talks/

This material is licensed