richard yen pg conf silicon valley 2019 - postgres conf · 2019-11-07 · next, try pglogical or...

33
Choosing from the Many Flavors of PostgreSQL Replication Richard Yen PG Conf Silicon Valley 2019

Upload: others

Post on 25-Jun-2020

9 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Richard Yen PG Conf Silicon Valley 2019 - Postgres Conf · 2019-11-07 · Next, try pglogical or EPRS • Replication engines supporting Logical Decoding • pglogical • EnterpriseDB

Choosing from the Many Flavors of PostgreSQL ReplicationRichard Yen

PG Conf Silicon Valley 2019

Page 2: Richard Yen PG Conf Silicon Valley 2019 - Postgres Conf · 2019-11-07 · Next, try pglogical or EPRS • Replication engines supporting Logical Decoding • pglogical • EnterpriseDB

DisclaimerThis is not an EDB talk!

Page 3: Richard Yen PG Conf Silicon Valley 2019 - Postgres Conf · 2019-11-07 · Next, try pglogical or EPRS • Replication engines supporting Logical Decoding • pglogical • EnterpriseDB

Why Replication?

• Because one copy is not enough!

• Fault-Tolerance

• Scalability

• Recoverability

• People want “real” data to work with

Page 4: Richard Yen PG Conf Silicon Valley 2019 - Postgres Conf · 2019-11-07 · Next, try pglogical or EPRS • Replication engines supporting Logical Decoding • pglogical • EnterpriseDB

Which one is for me?

• Streaming replication

• Logical replication

• Slony

• Bucardo

• Londiste

• DRBD

• Corosync

• 2ndQ BDR

• pg_shard

• pglogical

• Daffodil

• Mammoth

• Pgpool

• EDB Replication Server

• Spectral Core Replicator Pro

• And More!

Page 5: Richard Yen PG Conf Silicon Valley 2019 - Postgres Conf · 2019-11-07 · Next, try pglogical or EPRS • Replication engines supporting Logical Decoding • pglogical • EnterpriseDB

It all boils down to…Trigger-based | WAL-based

Page 6: Richard Yen PG Conf Silicon Valley 2019 - Postgres Conf · 2019-11-07 · Next, try pglogical or EPRS • Replication engines supporting Logical Decoding • pglogical • EnterpriseDB

Trigger-based Replication

• Relies on PostgreSQL’s trigger mechanism to log all DML into event/shadow tables

• Data from shadow tables ingested by another program and replayed to other nodes

• Very reliable: replicating production environments since 2003

Page 7: Richard Yen PG Conf Silicon Valley 2019 - Postgres Conf · 2019-11-07 · Next, try pglogical or EPRS • Replication engines supporting Logical Decoding • pglogical • EnterpriseDB

Trigger-based Replication

• Main players

• Slony — written in C

• Londiste — written in Python

• Bucardo — written in Perl

Page 8: Richard Yen PG Conf Silicon Valley 2019 - Postgres Conf · 2019-11-07 · Next, try pglogical or EPRS • Replication engines supporting Logical Decoding • pglogical • EnterpriseDB

Trigger-based Pros & Cons

• Select which tables to replicate

• Enables row-based/column-based replication

• Major-version upgrades

• Master-master replication (Bucardo only)

Page 9: Richard Yen PG Conf Silicon Valley 2019 - Postgres Conf · 2019-11-07 · Next, try pglogical or EPRS • Replication engines supporting Logical Decoding • pglogical • EnterpriseDB

Trigger-based Pros & Cons• Primary keys required

• Limited ability to replicate DDL

• Large object replication not available

• Trigger activity generates additional I/O

• Event tracking consumes network bandwidth

• Hard to scale up

• Infrastructure is brittle

Page 10: Richard Yen PG Conf Silicon Valley 2019 - Postgres Conf · 2019-11-07 · Next, try pglogical or EPRS • Replication engines supporting Logical Decoding • pglogical • EnterpriseDB

WAL-based Replication

• WAL files are the mechanism to recover from crashes

• WAL files can be used to replay DML on remote servers

• Streaming WAL activity effectively creates an up-to-date clone

Page 11: Richard Yen PG Conf Silicon Valley 2019 - Postgres Conf · 2019-11-07 · Next, try pglogical or EPRS • Replication engines supporting Logical Decoding • pglogical • EnterpriseDB

WAL-based Pros & Cons• It's fast

• Easy to set up

• Requires no additional software, I/O, bandwidth

• Exact replica provides assurance for backups

• Exact replica means all data types are supported

• Synchronous replication is possible

Page 12: Richard Yen PG Conf Silicon Valley 2019 - Postgres Conf · 2019-11-07 · Next, try pglogical or EPRS • Replication engines supporting Logical Decoding • pglogical • EnterpriseDB

WAL-based Pros & Cons• All or nothing

• No major version upgrades

• Query cancellations

• Storage impact on primary

• wal_keep_segments

• replication slots

Page 13: Richard Yen PG Conf Silicon Valley 2019 - Postgres Conf · 2019-11-07 · Next, try pglogical or EPRS • Replication engines supporting Logical Decoding • pglogical • EnterpriseDB

Middle Ground: Logical Replication

• Introduced in v. 10

• Based off Logical Decoding framework introduced in v. 9.4

• WAL files are reconstructed into SQL statements

• Paves the way to more granular replication filters

• Happy-medium

Page 14: Richard Yen PG Conf Silicon Valley 2019 - Postgres Conf · 2019-11-07 · Next, try pglogical or EPRS • Replication engines supporting Logical Decoding • pglogical • EnterpriseDB

Best of Both Worlds?• Like Trigger-based Replication:

• Still need uniqueness/PK

• DDL is not replicated

• Sequences are not replicated

• Like WAL-based Replication

• Still need to monitor storage impact

• Can’t create column-level filters (yet)

Page 15: Richard Yen PG Conf Silicon Valley 2019 - Postgres Conf · 2019-11-07 · Next, try pglogical or EPRS • Replication engines supporting Logical Decoding • pglogical • EnterpriseDB

Additional Gotchas

• WAL events are unrolled into individual SQL queries

• COPY statements are translated into INSERTs

• UPDATE on 1000 rows are translated into 1000 individual UPDATEs

• TRUNCATEs are replicated, but not cascaded

• Partitioned tables cannot be easily replicated

Page 16: Richard Yen PG Conf Silicon Valley 2019 - Postgres Conf · 2019-11-07 · Next, try pglogical or EPRS • Replication engines supporting Logical Decoding • pglogical • EnterpriseDB

Other options

• DRBD, Corosync, Windows Cluster

• Basically, mirrors a disk or other block-level device

• Doesn't support read-only standby

• Use cases for these are somewhat rare

• Use at your own discretion

Page 17: Richard Yen PG Conf Silicon Valley 2019 - Postgres Conf · 2019-11-07 · Next, try pglogical or EPRS • Replication engines supporting Logical Decoding • pglogical • EnterpriseDB

How do you choose?

Page 18: Richard Yen PG Conf Silicon Valley 2019 - Postgres Conf · 2019-11-07 · Next, try pglogical or EPRS • Replication engines supporting Logical Decoding • pglogical • EnterpriseDB

It all depends on your requirementsTolerances, privacy, etc.

Page 19: Richard Yen PG Conf Silicon Valley 2019 - Postgres Conf · 2019-11-07 · Next, try pglogical or EPRS • Replication engines supporting Logical Decoding • pglogical • EnterpriseDB

Process of Elimination

Page 20: Richard Yen PG Conf Silicon Valley 2019 - Postgres Conf · 2019-11-07 · Next, try pglogical or EPRS • Replication engines supporting Logical Decoding • pglogical • EnterpriseDB

Start with built-in Streaming Replication

• Super-simple to set up

• No additional software

• Best performance, least impact

• Good for most use-cases

Page 21: Richard Yen PG Conf Silicon Valley 2019 - Postgres Conf · 2019-11-07 · Next, try pglogical or EPRS • Replication engines supporting Logical Decoding • pglogical • EnterpriseDB

Streaming ReplicationX

Page 22: Richard Yen PG Conf Silicon Valley 2019 - Postgres Conf · 2019-11-07 · Next, try pglogical or EPRS • Replication engines supporting Logical Decoding • pglogical • EnterpriseDB

Next, try Logical Replication

• Pick and choose which tables to replicate

• WAL-based replication performance

• Zero-downtime major-version upgrades

• Only available in v. 10 and later

Page 23: Richard Yen PG Conf Silicon Valley 2019 - Postgres Conf · 2019-11-07 · Next, try pglogical or EPRS • Replication engines supporting Logical Decoding • pglogical • EnterpriseDB

Streaming Replication

Logical Replication

XX

Page 24: Richard Yen PG Conf Silicon Valley 2019 - Postgres Conf · 2019-11-07 · Next, try pglogical or EPRS • Replication engines supporting Logical Decoding • pglogical • EnterpriseDB

Next, try pglogical or EPRS

• Replication engines supporting Logical Decoding

• pglogical

• EnterpriseDB Postgres Replication Server 7 (EPRS7 — Limited Availability)

• Row-level and column-level filters

• Still WAL-based

Page 25: Richard Yen PG Conf Silicon Valley 2019 - Postgres Conf · 2019-11-07 · Next, try pglogical or EPRS • Replication engines supporting Logical Decoding • pglogical • EnterpriseDB

Streaming Replication

Logical Replication

pglogical | EPRS7

XXX

Page 26: Richard Yen PG Conf Silicon Valley 2019 - Postgres Conf · 2019-11-07 · Next, try pglogical or EPRS • Replication engines supporting Logical Decoding • pglogical • EnterpriseDB

Use a Trigger-based Solution

Page 27: Richard Yen PG Conf Silicon Valley 2019 - Postgres Conf · 2019-11-07 · Next, try pglogical or EPRS • Replication engines supporting Logical Decoding • pglogical • EnterpriseDB

Use a Trigger-based Solution

• But please upgrade soon!

• Impacts performance

• Often difficult to administration

• Logical Replication is the future

Page 28: Richard Yen PG Conf Silicon Valley 2019 - Postgres Conf · 2019-11-07 · Next, try pglogical or EPRS • Replication engines supporting Logical Decoding • pglogical • EnterpriseDB

RDS?Try Bucardo

Page 29: Richard Yen PG Conf Silicon Valley 2019 - Postgres Conf · 2019-11-07 · Next, try pglogical or EPRS • Replication engines supporting Logical Decoding • pglogical • EnterpriseDB

Multi-Master Replication?

Page 30: Richard Yen PG Conf Silicon Valley 2019 - Postgres Conf · 2019-11-07 · Next, try pglogical or EPRS • Replication engines supporting Logical Decoding • pglogical • EnterpriseDB

Multi-Master Replication?

• Holy Grail of replication solutions?

• First, ask yourself if you REALLY need it

• Consistency

• Performance

• Conflict Resolution

• Often, the other options are good enough?

Page 31: Richard Yen PG Conf Silicon Valley 2019 - Postgres Conf · 2019-11-07 · Next, try pglogical or EPRS • Replication engines supporting Logical Decoding • pglogical • EnterpriseDB

Multi-Master Replication• Trigger-based

• Bucardo

• XDB by EnterpriseDB

• WAL-based

• Bi-Directional Replication (BDR) by 2nd Quadrant

• requires a custom compile/installation of PG

• XDB/EPRS7 by EnterpriseDB

Page 32: Richard Yen PG Conf Silicon Valley 2019 - Postgres Conf · 2019-11-07 · Next, try pglogical or EPRS • Replication engines supporting Logical Decoding • pglogical • EnterpriseDB

Q & A

Page 33: Richard Yen PG Conf Silicon Valley 2019 - Postgres Conf · 2019-11-07 · Next, try pglogical or EPRS • Replication engines supporting Logical Decoding • pglogical • EnterpriseDB

Thank You!Richard Yen

Principal Support Engineer, [email protected]

richyen.com