the future of postgres sharding - bruce momjian · the future of postgres sharding bruce momjian...

The Future of Postgres Sharding

BRUCE MOMJIAN

This presentation will cover the advantages of sharding andfuture Postgres sharding implementation requirements.Creative Commons Attribution License http://momjian.us/presentations

Last updated: September, 2018

1 / 22

Outline

1. Scaling

2. Vertical scaling options

3. Non-sharding horizontal scaling

4. Existing sharding options

5. Future sharding

2 / 22

Scaling

Database scaling is the ability to increase database throughput byutilizing additional resources such as I/O, memory, CPU, oradditional computers.However, the high concurrency and write requirements ofdatabase servers make scaling a challenge. Sometimes scaling isonly possible with multiple sessions, while other options requiredata model adjustments or server configuration changes.Postgres Scaling Opportunities http://momjian.us/main/presentations/overview.html#scaling

3 / 22

http://momjian.us/main/presentations/overview.html#scaling

http://momjian.us/main/presentations/overview.html#scaling

Vertical Scaling

Vertical scaling can improve performance on a single server by:

◮ Increasing I/O with

◮ faster storage◮ tablespaces on storage devices◮ striping (RAID 0) across storage devices◮ Moving WAL to separate storage

◮ Adding memory to reduce read I/O requirements

◮ Adding more and faster CPUs

4 / 22

Non-sharding Horizontal Scaling

Non-sharding horizontal scaling options include:

◮ Read scaling using Pgpool and streaming replication

◮ CPU/memory scaling with asynchronous multi-master

The entire data set is stored on each server.

5 / 22

Why Use Sharding?

◮ Only sharding can reduce I/O, by splitting data across servers

◮ Sharding benefits are only possible with a shardableworkload

◮ The shard key should be one that evenly spreads the data

◮ Changing the sharding layout can cause downtime

◮ Additional hosts reduce reliability; additional standbyservers might be required

6 / 22

Typical Sharding Criteria

◮ List

◮ Range

◮ Hash

7 / 22

Existing Sharding Solutions

◮ Application-based sharding

◮ PL/Proxy

◮ Postgres-XC/XL

◮ pg_shard

◮ Hadoop

The data set is sharded (striped) across servers.

8 / 22

Sharding Using Foreign Data Wrappers (FDW)

��

��

��

��

��

��

SQL Queries

SQL Queries

PG FDW

Foreign Server Foreign Server Foreign Server

https://wiki.postgresql.org/wiki/Built-in_Sharding

9 / 22

https://wiki.postgresql.org/wiki/Built-in_Sharding

FDW Sort/Join/Aggregate Pushdown

��

��

��

��

��

��

SQL Queries

PG FDW


aggregates (10)joins (9.6)

sorts (9.6)aggs on partitions (11)

10 / 22

Advantages of FDW Sort/Join/Aggregate Pushdown

◮ Sort pushdown reduces CPU and memory overhead on thecoordinator

◮ Join pushdown reduces coordinator join overhead, andreduces the number of rows transferred

◮ Aggregate pushdown causes summarized values to be passedback from the shards

◮ WHERE clause restrictions are also pushed down

11 / 22

FDW DML Pushdown in Postgres 9.6

��

��

��

��

��

��

SQL Queries

PG FDW


UPDATE, DELETE

12 / 22

Pushdown of Static Tables

��

��

��

��

��

��

SQL Queries

SQL Queries

with joins to static data

PG FDW

and static data restrictions

Foreign S. Foreign S. Foreign S.static static static

13 / 22

Implementing Pushdown of Static Tables

Static table pushdown allows join pushdown where the queryrestriction is on the static table and not on the sharded column.Static tables can be pushed to shards usinglogical replication, implemented in Postgres 10. The optimizermust also know which tables are duplicated on the shards for joinpushdown.

14 / 22

Shard Management

��

��

��

��

��

��

SQL Queries

DDL Queries

PG FDW


Shard management will be added to the partitioning syntax,which was added in Postgres 10.

15 / 22

Parallel Shard Access

��

��

��

��

��

��

SQL Queries

PG FDW


Shard Parallel Access

16 / 22

Advantages of Parallel Shard Access

◮ Can use libpq’s asynchronous API to issue multiple pendingqueries

◮ Ideal for queries that must run on every shard, e.g.,

◮ restrictions on static tables◮ queries with no sharded-key reference◮ queries with multiple shared-key references

◮ Aggregate across shards

17 / 22

Global Snapshot Manager

��

��

��

��

��

��

SQL Queries

SQL Queries

ManagerGlobal Snapshot

PG FDW


18 / 22

Implementing a Global Snapshot Manager

◮ We already support sharing snapshots among clients withpg_export_snapshot()

◮ We already support exporting snapshots to other serverswith the GUC hot_standby_feedback

19 / 22

Global Transaction Manager

��

��

��

��

��

��

SQL Queries

SQL Queries

Foreign Server

ManagerGlobal Transaction

PG FDW

Foreign Server Foreign Server

20 / 22

Implementing a Global Transaction Manager

◮ Can use prepared transactions (two-phase commit)

◮ Transaction manager can be internal or external

◮ Can use an industry-standard protocol like XA

21 / 22

Conclusion

http://momjian.us/presentations https://www.flickr.com/photos/anotherpintplease/

22 / 22

the future of postgres sharding - bruce momjian · the future of postgres sharding bruce momjian...

Documents