enterprise grade deployment and security with postgresql
TRANSCRIPT
Enterprise grade deployment and security with PostgreSQL
PGConf US 2015
March 25 - 27, 2015
Himanchali
www.inmobi.com
Independent mobile ad network
Customer engagement platform through a suit of product ranging from app distribution & monetization to brand advertisement
Serves billion of ads
Hosting the first official PGDay India event in April http://www.meetup.com/India-PUG/events/220553997/
What kind of application do we have..
OLTP
OLAP
What exactly am I talking today ??
Production Deployment
All main transactional and CI reporting DBs
Replication across globe
Load balancing and connection pooling through PGBouncer
Automation through puppet
Version control for DB level changes
Data security
Data security
User access control
OS level security
Data growth
0
10
20
30
40
50
60
70
80
90
2010 2011 2012 2013 2014 2015
Main Transactional DBs(GB)
Reporting DB (TB)
Others (GB)
Current stats of Postgres DB Servers
About 60 hosts spread across 4 regions
DB size ranges from 10 GB – 20 TB
The critical transactional DBs :
Average size 80GB
Average 18K QPS with peak of 54K QPS
Reporting DB
Average 200 GB data load everyday
Simple expectation of management
99.99% Uptime for DB servers
No data loss
Good performance
Achieving 99.99% Uptime
A proper DB Architecture
Multiple disaster recovery Plan
A time to time disaster drill
Proper data backup
PITR setup
A proper DB Architecture
The architecture..
Proper data modeling
Use of constraints carefully
Minimize the dependency between DBs
Use of proper data types
Proper retention policies
Disaster Recovery plan
Question is when to touch the trigger ??
What if standby doesn’t come up ?? (rare condition)
Are my backups up to date ??
Do I have an option to revert some drop of a table?
Data availability in different regions
Cascaded SR setup
A boon for cross region replication
One node as entry point to the region
Cascade slaves within LAN from above node
A little precaution needed with this setup
When the master is down and its standby is being triggered as master
Proper monitoring that no slave is out of sync
Read distribution across multiple slaves
Load balancer
Pgbouncer
Connection pooling
Connection mode control , session or transaction
Maintenance without downtime
What about performance ??
Proper index creation
Query optimization
Table partitioning
Proper vacuum
What else …
How can I forget those many stored procedures in my DB .
User defined functions for complex business logic
Saving time between application and DB
Get rid of dependency of language used by application
Practice adopted for DB changes
Version control of schema changes
Version control of data changes
Minimize the deployment effort
Using puppet and chef modules
No direct change on the host manually
Configuration changes pushed via puppet
Packages installation using chef-solo
Adding spice of monitoring…
Use of enriched statistics collector
User stats
DB stats
Table stats
Index stats
And many more…
pg_log monitoring
Log everything if you want to monitor everything
Security aspects
Control user access
Limited access to application users
Give only what they need
Control at DB, schema or table level
Use user groups to manage if many users
Manage and standardize DBA access
A common user
Maintain default privilege
Some exampleREVOKE CREATE ON SCHEMA public FROM PUBLIC;
REVOKE ALL on schema public from write_user,read_user;
GRANT USAGE on schema public to write_user,read_user ;
GRANT all on all tables in schema public to write_user;
GRANT all on all sequences in schema public to write_user;
REVOKE ALL on all functions in schema public from write_user,read_user;
GRANT EXECUTE on all functions in schema public to write_user,read_user;
GRANT SELECT on all sequences in schema public to read_user;
GRANT SELECT on all tables in schema public to read_user;
ALTER DEFAULT PRIVILEGES FOR ROLE enterprisedb,postgres IN SCHEMA public GRANT all on tables to write_user;
ALTER DEFAULT PRIVILEGES FOR ROLE enterprisedb,postgres IN SCHEMA public GRANT EXECUTE on functions to write_user,read_user;
ALTER DEFAULT PRIVILEGES FOR ROLE enterprisedb,postgres IN SCHEMA public GRANT all on sequences to write_user;
ALTER DEFAULT PRIVILEGES FOR ROLE enterprisedb,postgres IN SCHEMA public GRANT SELECT on sequences to read_user;
ALTER DEFAULT PRIVILEGES FOR ROLE enterprisedb,postgres IN SCHEMA public GRANT SELECT on tables to read_user;
Access restriction to DB
Use pg_hba.conf and pg_ident.conf
Avoid using trust
Use MD5 authentication
IP / Subnet access allocation for DB or user level
Restrict the connection for any particular subnet for any user/DB
Use pg_ident.conf if using ident authentication in pg_hba.conf
If that’s not enough…
Use secure TCP/IP Connections with SSL
IPTables
Control the traffic by making some rules
Allow or reject
SE-Linux - control access at process level
setenforce=1
Myth broken !!
Thank You !!
QUESTIONS??