best practices for a complete postgres enterprise architecture setup

© 2015 EnterpriseDB Corporation. All rights reserved. 1

Best Practices for a Complete Postgres Enterprise Architecture Setup

To watch to the presentation visit - www.EnterpriseDB.com > Resources > On-Demand Webcasts


•  Introduction to EnterpriseDB •  Enterprise Data Management Architecture

−  OLTP Infrastructure −  High Availability −  Disaster Recovery −  Data Integration −  Monitoring and Management −  Security

•  Summary and Resources •  Q&A

Agenda

We will take two quick polls during the webinar to gather group data on best practices of interest


POSTGRES innovation

ENTERPRISE reliability

24/7 support

Services & training

Enterprise-class features, tools &

compatibility

Indemnification

Product road-map

Control

Thousands of developers

Fast development

cycles

Low cost

No vendor lock-in

Advanced features

Enabling commercial adoption of Postgres


EDB Customers EDB currently has over 2,500 total customers including 50 of the Fortune 500 and 98 of the Forbes Global 2000


EDB is a Industry Leader Magic Quadrant for Operational DBMS, Q4 ‘14

• Gartner Comments − “EnterpriseDB is responsible

for many features of PostgreSQL, contributing to JSON, materialized views and partitioning.” − “Clients report that the

functionality of EnterpriseDB's Postgres Plus Oracle Compatibility Feature is now more than sufficient to run both mission-critical and non- mission-critical applications.” − “Customers commend the

compatibility with Oracle, the stability of the DBMS and the product support.”


EDB is an Open Source Community Leader

Amit Kapila

Ashesh Vashi

Bruce Momjian

Dave Page

Devrim Gunduz

Jan Wieck

Kevin Grittner

Korry Douglas

Muhammad Usama

Robert M Haas

Thom Brown


Where Do Users Need Help # Tickets Phase Category POC Dev Deployment Maintenance Grand Total Connectors 0% 2% 1% 0% 3%

Bug 0% 0% 0% 0% 1% How to 0% 1% 0% 0% 1% Product Awareness 0% 1% 0% 0% 1%

Database 10% 9% 17% 37% 73% Bug 1% 2% 1% 2% 6% Corruption 0% 0% 0% 2% 2% Enhancements 0% 0% 0% 0% 1%

How to 4% 3% 6% 10% 23% Product Awareness 4% 3% 8% 15% 31% Tuning 0% 0% 1% 7% 9%

Replication 1% 4% 4% 2% 12% Bug 0% 1% 0% 0% 2% How to 0% 0% 3% 1% 4% Product Awareness 1% 2% 1% 0% 4% Tuning 0% 0% 0% 1% 1%

Utilities 1% 3% 6% 3% 12% Bug 0% 1% 1% 0% 2% Enhancements 0% 0% 0% 0% 0% How to 1% 1% 3% 1% 6% Product Awareness 0% 1% 3% 1% 5%

Grand Total 12% 17% 28% 42% 100%

Questions focus on deploying, supporting

and maintaining Postgres in the

Enterprise

© 2013 EDB All rights reserved 8.1. 8

Enterprise Architecture with Postgres


•  Effective data center implementations use a reference architecture to: −  Accelerate the implementation of a solution −  Lower operational costs −  Build-in flexibility −  Enhance performance −  Lower complexity −  Reduce risk −  Avoid the ‘OMG – MTP’ moment!

•  EnterpriseDB Postgres reference architecture −  modular and standards-based −  supports a wide range of enterprise deployments

Why?

OMG! Tomorrow is

MTP!


•  OLTP Infrastructure −  ACID Compliant −  High Performance −  Scalable −  Reliable

•  High Availability −  Protect against hardware

failure −  Protect against software failure

•  Disaster Recovery −  Protect against site failure −  Protect against operator error

•  Data Integration −  Integration with other Line of

Business systems −  Heterogeneous integration with

Microsoft, Oracle, etc

•  Monitoring and Management −  Capacity Planning −  Event management and

alerting

•  Security −  Authentication and

Authorization −  Encryption and data protection

Key Components of a Enterprise Data Management Architecture


? Tool and Infrastructure Choices

Hig

h Av

aila

bilit

y Monitoring and Management

Disaster R

ecovery

Data Integration

Open PostgreSQL Monitoring

pgAdmin phpPgAdmin

repmgr HandyRep

PgHA

Slony Bucardo

Londiste BDR

BARMan pg_rman

TeamPostgreSQL OmniPITR

Circonus pgBadger

pgFouine

powa

pitrery wal-e

•  Multiple license models (PostgreSQL, GPL, AGPL) •  Multiple vendors with different SLAs •  Overlap and gaps

check_postgres

PITRTools

Nagios XI wizard


EDB Integrated Postgres Portfolio

•  OLTP Infrastructure −  Postgres Plus Advanced

Server PPAS −  Oracle Compliant −  Enhanced partitioning, security

and tooling (Profiler, Index Advisor, Runtime Statistics)

•  High Availability −  EDB Failover Manager −  Robust, cost effective,

commodity hardware

•  Disaster Recovery −  EDB BART (Backup and

Recovery Tool) −  Native Streaming Replication

•  Data Integration −  xDB Replication (Single

Master and Multi Master) −  Heterogeneous integration with

Oracle, MS SQL Server and Postgres

•  Monitoring and Management −  Postgres Enterprise Manager −  Enterprise level management,

monitoring and alerting for database and infrastructure

•  Security −  PPAS Enhancements −  Virtual Private Database (VPD) −  Code protection −  SQL Injection Attack Guard


PEM Client

Oracle/SQL Server/PG

Rep

licat

ion

DB Master Witness Offsite Replica

Virtual IP

DB Client

Backup & Recovery

PEM Server High Availability

Disaster Recovery HA Replica

Read/HA Replica

Robust, Resilient, Scalable Postgres

Enterprise Architecture

Mon

itorin

g

Dat

a In

tegr

atio

n

Simple, development

focused setup


OLTP Infrastructure


Security

Tools

Performance

Compatibility

ADVANCED SERVER

POSTGRES

PLUS


Postgres Plus Advanced

Server

Resource Manager

(CPU & I/O)

Reporting

Transactions

80%

20%

PPAS 9.4 Resource Manager

•  DBA assigns CPU & I/O to job groups

•  Allocates and prioritizes consumption of resources

•  Low priority jobs don’t hurt high priority jobs


SQL Performance Management

−  SQL Profiler captures a SQL workload and locates the worst running SQL

−  Both ad-hoc and scheduled operations supported

−  Provides a number of SQL-based performance metrics along with EXPLAIN analysis

−  Integrates with Index Advisor, which analyzes SQL statements and recommends new indexes to improve performance

−  EDB SQL Profiler & Index advisor are components of EDB Postgres Enterprise Manager (PEM)

•  Poorly optimized SQL code is the NUMBER ONE cause of database problems—EDB SQL Profiler & Index Advisor tools can help:


•  Types of storage systems −  Direct Attached Storage (DAS) −  Storage Area Network (SAN) w. Fiber Channel −  Storage Area Network (SAN) w. iSCSI −  Network File System (NFS)

•  DAS: fast and low latency •  SAN: fast, expensive (w. Fiber Channel), scalable, can

include redundancy and smart file system operations •  NFS: not an optimal solution for PostgreSQL data files

Caveat Emptor: File System and Storage Subsystem


High Availability System Design with Streaming Replication


•  Shared Disk −  Red Hat Cluster Server, Veritas Cluster, … −  Mostly used for local failover −  High $$$

•  Native streaming replication-based −  Available with Postgres 9.X −  Log (WAL)-based, fast and reliable −  Replicate the database locally or remote −  Synchronous or asynchronous −  Hub & spoke or cascading

•  EDB Failover Manager (EFM) −  Leverages native streaming replication −  Integrates ‘Witness’ concept to create quorum and avoid false

alarms

High Availability Options for Postgres


EDB

EDB Failover Manager creates fault tolerant database clusters to minimize downtime when a master database fails by keeping data online in high availability configurations

FAILOVER MANAGER Master

Streaming Replica

Clients

Witness

Network / Internet

The MASTER is down!

I agree! Take control now!


Disaster Recovery


Backup and Standby – Reality Check

hot 47%

warm 23%

None 18%

cold 12%

Standby Approach Physical

Only 36%

Logical Only 29%

Physical and

Logical 14%

FS Snapshot

7%

None 14%

Backup Approach 43% of customers could not execute PITR (prior to EDB Architecture Engagement) – some had NO backup in place

30% of customers would have been slow or unable to recover from failure


•  Why do you need backup and recovery? Backup and recovery strategies protect you in case of: −  Catastrophic device failure −  Site failure −  Maintenance −  Operator error −  Compliance −  Data corruption

•  Multiple components: −  Logical backup provides granularity in objects (tables, table

spaces, databases) −  Physical backups provide granularity in time for PITR −  Cold/off-line backups −  Hot/on-line backup, with WAL archiving

Backup and Recovery Strategies


•  Backup Strategy Framework −  Logical backup after structural changes or

major updates of reference/meta data −  Physical backup (daily…) −  WAL file archiving −  Combine backup/recovery and streaming replication −  Uses replica to offload hot backup and logical backup

•  Consider −  Allowable PITR timeframe (MTR – Mean Time to Recover) and

allowable data loss (RPO – Recovery Point Objective) −  Data retention policy −  Test, test, test −  Periodic backup validation

Backup and Recovery Best Practices


Disaster Recovery for Postgres

•  Two components −  Offsite replication

−  Streaming replication to Disaster Recovery site −  Backup

−  Logical backup −  Physical backup −  WAL archiving for PITR


•  Simplifies and reduces errors with a system-wide catalog and command line tool

•  Online backup and recovery across local and remote servers

•  Local and remote online physical backups

•  Auto-compression and MD5 checksum verification

•  Continuous WAL Archiving and PITR

•  Support for tablespace restoration on different paths

BART Architecture


Database Infrastructure Management and Monitoring


•  Monitor the health of your server and databases for proactive management and issue prevention

Monitoring and Management

• What should I monitor? • How do I reduce downtime? • How do I know if I need more space? • How to find bloat in tables & indexes?


Maintenance Highlights from 20+ AHCs

Unused Indexes

39%

Bloated Indexes

30%

Bloated Tables 13%

Missing Indexes

9% Missing

PK 9%

Bloat management (43%) is a key

production problem

specific to Postgres


•  Utilize alerting and event management across statistics −  Storage, user activity, connections −  I/O, memory analysis −  Database size, tablespace size −  Session activity - workload, locks, waits

•  Perform regular capacity planning −  Analyze historical usage statistics of objects −  Project the anticipated usage statistics for an object −  Collect and analyze metrics for specific:

−  Host/operating system −  Postgres server −  Database −  Database object (table, index, function etc.)

Effective Monitoring and Capacity Planning


General Monitoring Recommendations Description Frequency Alerting Criteria / Parameters monitored Postgres instance is running and connection count

Every minute (i) When postgres instance is not running, servers down (ii) When number of connections reaches percentage of max_connections (iii) When number of connections crosses threshold

Load average, disk space

Every 15 minutes

(i) When load average is above threshold (ii) When disk consumption percentage above threshold (iii) When disk space available is below threshold

Long running and waiting queries in database

Every 10 minutes

(i) When any query is running for longer than threshold (ii) When any query is in state of "idle in transaction" for more than threshold (iii) When any query is waiting for more than threshold

Bloats for tables and indexes

Once a day (i)  Total table bloat, highest table bloat (ii) When database tables and/or indexes individually

bloated more than critical threshold

Vacuum maintenance

Once a day

(i)  Last vacuum (ii) Last auto-vacuum

Streaming replication sync with primary

Every 5 minutes

(i) When lag detected on streaming cluster exceeds threshold

Standby server lag behind the master by WAL segments Standby server lag behind the master by WAL pages


•  Consumption projection: −  At current consumption rates when will I drop below 100MB of disk? −  At the consumption rate of the past 3 days when will I drop below 1 GB of disk

space?

•  Point in Time activity −  Users were complaining of performance problems from 3 to 4 PM today. What

was going on in the system, the database, how many users were connected? −  We get a load spike every day at 11. What was happing at the time of the load

spike?

•  Root cause for slower query performance −  Queries that access the employ table seem to be taking longer and longer.

View the growth of the number of dead tuples in the table, see what kind of scans have been executing against the table (index or sequential)

•  Activity monitoring −  Which database on the server is the most active? What kind of growth is it

seeing?

Capacity Planning and Management – Use Cases


•  Single management console allows easy visual control •  Works with both PostgreSQL and Postgres Plus •  Start/stop, configure, define and manage storage, security and database objects via

single graphical console

MONITOR MANAGE TUNE

EDB Postgres Enterprise Manager (PEM)

Mission critical OS and database statistics collection

Monitor real-time alerts

Predefined (200+) and custom alerts via SMTP or SNMP

Predefined & custom at-a-glance global dashboards

Replication monitoring

CRUD operations on all database objects

Bulk operations across multiple servers

Historical reports

Capacity Manager for planning & forecasting

Customizable GUI charts, tables & graphs

Identify poorly running SQL statements

SQL/Profiler to speed up large workloads

Index Advisor to suggest and create indexes

Postgres Expert for best practice enforcement

Tuning Wizard for machine utilization and load profiles


Unmanaged Hosts

An efficient distributed architecture perfectly suited for managing, monitoring and tuning large numbers of Postgres servers in multiple locations

Monitoring Data

Enterprise Management Connections

PEM Clients or Web Clients

Managed Hosts

with PEM Agents

Enterprise Manager Agent Installed on each managed machine; collects data on OS and database health and operations PEM

Server

Enterprise Manager Server Centralized storage for agent-collected data and client dashboards

Host Database Management

Connection

Enterprise Manager Clients User GUI console with global at-a-glance monitoring dashboards; used to carry out centralized database administration and tuning

Platform-specific and web clients

POSTGRES ENTERPRISE MANAGER (PEM) Architecture

EDB


Monitor All Your Postgres Databases From One Screen

•  Customized global dashboard

•  View up/down status of all agents

•  Monitor alerts from many servers in one place

•  Navigate to Dashboards for further analysis


Data Integration


•  Postgres solutions exist in context with other Line of Business solutions

•  Options −  Physical data replication – xDB Replication Engine −  Virtual data replication – Foreign Data Wrappers −  ETL (Extract, Transfer, Load)

Data Integration with Postgres


Oracle SQL Server PostgreSQL Advanced Server

READ/WRITE

PostgreSQL Advanced Server

Data filtering

Scheduling

READ

MASTER

REPLICA

Flexible §  Heterogeneous §  Filter only data you need replicated §  Continuous or scheduled §  Cascading

Multiple Applications §  Performance—shift read operations

from master to hot replica Backup §  Testing/new development copies §  Migrate data to new systems

EDB xDB REPLICATION SINGLE MASTER

Near real-time replication


xDB Graphical UI: Making Replication Easy

Browsing publications and subscriptions

Setting up new publications

Configuring new publications

Adding criteria

Adding subscriptions

Reviewing results


Security


Security

•  Conceptual Layers −  Authentication −  Authorization −  Auditing −  Data Security −  SQL Injection Attacks

•  Approach −  Secure physical access −  Network access limitation −  Host access limitation −  Database access

limitation −  Data access limitation


“By default, PostgreSQL is probably the most security-aware database available ...”

--Database Hacker's Handbook

Postgres Plus Advanced Server Security •  Object level privileges assigned to roles and users •  Row Level Security (Virtual Private Database) •  EAL2 Certification (augmented with ALC_FLR.2), NIPRNet, SIPRNet, JWICS •  Stored procedure obfuscation •  Protection against SQL injection attacks •  Kerberos and LDAP authentication •  SQL USAGE privilege and VIEW Security Barriers •  SSL communication •  Data Level Encryption (AES, 3DES, etc.) •  Ability to utilize 3rd party Key Stores in a full PKI Infrastructure •  Foundation for full compliance with the strictest of security standards (PCI Data

Security Standard)


Security Best Practices for PostgreSQL Security Best Practices for PostgreSQL and Postgres Plus Advanced Server

Executive Summary ................................................................3

Introduction ….........................................................................4

Postgres Security Features within the Above Framework ......6

Authentication .........................................................6

Authorization ….......................................................7

Accounting/Auditing ................................................8

Data Security ….......................................................9

SQL Injection Attacks ............................................10

Postgres Plus Advanced Server Security Features ..............11

Further Reading and Useful Links ........................................12

About EnterpriseDB ..............................................................12

© 2013 EnterpriseDB Corporation. All rights reserved. EnterpriseDB and Postgres Plus are trademarks of EnterpriseDB Corporation. Other names may be trademarks of their respective owners. http://www.enterprisedb.com

Page: 2

Table of Contents

Disclaimer The following is intended as an outline of EnterpriseDB’s general product direction. It is intended for informational purposes only, and it should not be relied upon in making purchasing decisions. This information may not be incorporated into any contract. It is not a commitment or obligation on the part of EnterpriseDB to release, launch or deliver any updates, modifications, material, code or functional improvements and may change at EnterpriseDB’s sole discretion.

http://info.enterprisedb.com/rs/enterprisedb/images/Whitepapers_Security_BP_PostgreSQL_and_Postgres_Plus_AS.pdf


•  A successful Postgres implementation is about more than using Postgres

•  It includes −  High Availability −  Disaster Recovery −  Backup Strategy −  Integration Strategy −  Monitoring and Management −  Security

Summary


PEM Client


Rep

licat

ion


Virtual IP

DB Client

Backup & Recovery



Read/HA Replica

Mon

itorin

g

Dat

a In

tegr

atio

n

EDB Failover Manager

EDB Backup

and Recovery

xDB Replication

Postgres Enterprise Manager

Postgres Streaming Replication

best practices for a complete postgres enterprise architecture setup

Technology

enterprisedb corporation

product awareness

gartner comments enterprisedb

edb customers edb

product support

total customers

group data

advanced features