practical postgresql - drexel ccitwc24/cs500_su11/slides/michaelgleasman.pdf · practical...

Practical PostgreSQLat [email protected]

CS 500 Database TheoryDepartment of Computer Science

Drexel University2010–08–05

Thursday, August 5, 2010

mailto:[email protected]

mailto:[email protected]

· myYearboook.com

· mind/body

· failure

· myb topology

· myb ecology

· measurement

· admin/maintenance


ecology: proxy, pool, cache, queuequestions any time

myYearbook.com


by genderby gender

52% female

48% male

by ageby ageby age

37% 13–17

27% 18–24

13% 25–34

12% 35–44

11% 45+

myYearbook.comcasual social network

founded in 2006

Google Analytics


myYearbook Teens and Twitter Survey (August 2009, ~%)myYearbook Teens and Twitter Survey (August 2009, ~%)myYearbook Teens and Twitter Survey (August 2009, ~%)myYearbook Teens and Twitter Survey (August 2009, ~%)

myYearbookmyYearbook FacebookFacebook

70% meet people 80% keep in touch with friends

50% play games 35% meet people

40% keep in touch with friends 30% share photos

35% flirt/date 25% play games


comScore Teens Category (April 2009)comScore Teens Category (April 2009)comScore Teens Category (April 2009)comScore Teens Category (April 2009)comScore Teens Category (April 2009)comScore Teens Category (April 2009)

rank site visits (K) uniques (K) minutes (M) page views (M)

1 myYearbook 55,808 4,604 851 1,630

2 MEEZ 8,629 1,407 226 469

3 Zwinky 11,558 3,691 153 108

4Hearst Teen Network

6,940 2,314 53 117

5 Quizilla 7,924 2,058 75 75


comScore page viewsJuly 2009



rank site views (M)

20 GaiaOnline 1,105

21 Chase 1,056

22 ESPN 984

23 myYearbook 953

24 Wikipedia 903

25 Onemanga 840

26 Mapquest 833

27 Foxsports 815

comScore time spentJuly 2009



rank site minutes (M)

20 Amazon 806

21 CNN 744

22 GaiaOnline 709

23 Bing 707

24 MSNBC 691

25 myYearbook 678

26 Iwin 670

27 NickJr 665


mind/body


relational theoryvalues & operators


normalizationprevent update anomalies


logical


I do declare!


logicalversus

physicalThursday, August 5, 2010

Hive

MonetDB

Truviso


Hive: SQL over HadoopMonetDB: column storeTruviso: streaming data

plannercompiler


thoughtworddeed


EXPLAIN


denormalization/

materialization


naturalversus

surrogate


+ size on disk, fast comparison, don’t need ON UPDATE CASCADE- additional lookups

Relative data access latencies

1 CPU Register

1–2 L1 cache

6–10 L2 cache

25–50 main memory (RAM)

10,000,000 hard disk

100,000,000 LAN

1,000,000,000 WAN

Thursday, August 5, 2010hard disk 1E7, LAN 1E7-1E8, WAN 1E9-2E9source: http://www.slideshare.net/guest22d4179/latency-trumps-all

main memory

disk

kernelbuffer

postgres

postgresbuffers

tempdataWAL


persistence


Fusion-iosolid state cards


http://www.fusionio.com/Steve Wozniak is Chief Scientist

Fusion-io

Postgres buffer

kernel buffer

CPU

Fusion-io card

CPU

application buffer

Fusion-io card

CPU

direct IO buffered IO


SSDsolid state drives

look promising


DASdirect attached storage

SANstorage area network


fsync


checkpoint


MVCC


rollback optimized

VACUUM


VLAMF


vacuum like a mo-fono autovacuum: scheduled for predictabilityautovacuum is good for general case, more difficult at scale

reindexing


failure


failure happens

plan for it


backup

warm standby


pg_dumpdisk IO

network IO


activeactive recovery

postgres

WAL

1

data

2

postgres

WAL

recovery

data


recoveryactive recovery

postgres

WAL

1

data

2

postgres

WAL

recovery

data


warm standbyactive

standby

postgres

WAL

1

data

2

WAL

copy

postgres

recovery

data


log shipping

SANfor warm standby


warm standby

SAN

VM

data 1 data 2 data 3

warm standby 1 warm standby 2 warm standby 3

postgres 1

log shipping

postgres 2

log shipping

postgres 3

log shipping

postgres spare


SAN

VM

data 1 data 2 data 3

warm standby 1 warm standby 2 warm standby 3

postgres 1

log shipping

postgres 2

log shipping

postgres 3

postgres spare

failover


Postgres performs single IO writes

kernel IO scheduler can’t schedule writes

writes queued in kernel


HAhigh availability

automate failoverequivalent standby hardware

Red Hat Cluster, Steeleye, Veritas


financial companies

If it’s not automated, it’s wrong.


myb topology


app-server 1

postgres 1 postgres 2

app-server 2 app-server 3


more activity? views/month 2007 100M ➙ 2009 1.5G

more TPSmore serversmore connectionsmore configurationmore pain!


SkypePL/Proxy

PgBouncerThursday, August 5, 2010

PL/Proxy


app-server 1

postgres 1


postgres 2

PL/Proxy

less configuration

fewer connections


CREATE FUNCTION pref.set_member_preference(in_member_id BIGINT, in_preference_name TEXT, in_preference_value_name TEXT) RETURNS BOOLEANLANGUAGE plpgsql STRICT VOLATILE AS $$DECLARE v_preference_id pref.preferences.preference_id%TYPE := pref.preference_id(in_preference_name); v_preference_value_id pref.preference_values.preference_value_id%TYPE := pref.preference_value_id(v_preference_id, in_preference_value); v_did_loop BOOLEAN DEFAULT FALSE;BEGIN << upsert >> LOOP UPDATE pref.member_preferences SET preference_value_id = v_preference_value_id WHERE (member_id, preference_id) = (in_member_id, v_preference_id); v_did_update := FOUND; EXIT upsert WHEN v_did_update OR v_did_loop; BEGIN INSERT INTO pref.member_preferences (member_id, preference_id, preference_value_id) VALUES (in_member_id, v_preference_id, v_preference_value_id); EXIT upsert WHEN FOUND; EXCEPTION WHEN unique_violation THEN v_did_loop := TRUE; -- loop to update END; RETURN v_did_update;END;$$;


CREATE FUNCTION pref.member_preference(in_member_id BIGINT, in_preference_name TEXT, OUT preference_value_name TEXT) RETURNS BOOLEANLANGUAGE plpgsql STRICT STABLE AS $$DECLARE v_preference_id pref.preferences.preference_id%TYPE := pref.preference_id(in_preference_name);BEGIN SELECT INTO preference_value_name pref.preference_value_name(v_preference_id, mp.preference_value_id) FROM pref.member_preferences AS mp WHERE (mp.member_id, mp.preference_id) = (in_member_id, v_preference_id);

IF NOT FOUND THEN preference_value_name := pref.default_preference_value(v_preference_id, in_preference_name); END IF;

RETURN;END;$$;


CREATE FUNCTION pref.member_preference(in_member_id BIGINT, in_preference_name TEXT) RETURNS TEXTLANGUAGE plproxy STRICT VOLATILE AS $$ CLUSTER 'pref'; RUN ON pref.partition_by_member_id(in_member_id);$$;

CREATE FUNCTION pref.set_member_preference(in_member_id BIGINT, in_preference_name TEXT, in_preference_value_name TEXT) RETURNS BOOLEANLANGUAGE plproxy STRICT VOLATILE AS $$ CLUSTER 'pref'; RUN ON pref.partition_by_member_id(in_member_id);$$;

SELECT get_cluster_partitions FROM plproxy.get_cluster_partitions('pref'); get_cluster_partitions --------------------------------------------------- host=10.10.10.10 port=6543 dbname=pref01 user=web host=10.10.10.10 port=6543 dbname=pref02 user=web host=10.10.10.10 port=6543 dbname=pref03 user=web host=10.10.10.10 port=6543 dbname=pref04 user=web


app-server 1

postgres 1


postgres 2 postgres 3 postgres 4

fewer connections!


connection overhead

1 process per backend

lock management


app server 1

postgres

app server 2 app server 3 app server 4 app server n


app server 1

pgbouncer

app server 2 app server 3 app server 4 app server n

postgres


app-server 1

pgbouncer 1


postgres 1



app-server 1

pgbouncer 1


postgres 1

internal pgbouncer



app-server 1

pgbouncer 1


postgres 1 postgres 2 postgres 3 postgres 4

pgproxy

internal pgbouncer

reduce TPS/server

less configuration


app-server 1 app-server 2 app-server 2 app-server n

external pgbouncer tier

app-server 1

app-pgbouncer 1

load balancer

app-server 2

app-pgbouncer 2

app-server 3

app-pgbouncer 3

app-server n

app-pgbouncer n

pgbouncer 3pgbouncer 2pgbouncer 1

pgproxy

internal pgbouncer

postgres 1 postgres 2 postgres 3 postgres 4 postgres n




internal pgbouncer tier

app-server 1

app-pgbouncer 1

load balancer

app-server 2

app-pgbouncer 2

app-server 3

app-pgbouncer 3

app-server n

app-pgbouncer n

pgbouncer 3pgbouncer 2pgbouncer 1

pgproxy

load balancer

pgbouncer 4

postgres 1 postgres 2 postgres 3 postgres n

pgbouncer 5




pgproxy 2


app-server 1

app-pgbouncer 1

load balancer

app-server 2

app-pgbouncer 2

app-server 3

app-pgbouncer 3

app-server n

app-pgbouncer n

load balancer

pgbouncer 3 pgbouncer 2pgbouncer 1 pgbouncer

pgbouncer 4pgbouncer 5

pgproxy pgproxy





pgproxy 3pgproxy 2pgproxy 1


app-server 1

app-pgbouncer 1

load balancer

app-server 2

app-pgbouncer 2

app-server 3

app-pgbouncer 3

app-server n

app-pgbouncer n

pgbouncerpgbouncerpgbouncer

pgproxy

load balancer

pgproxypgproxy

pgbouncer 4pgbouncer 5



28 servers avg 90% idle

464 cores

3.3 TB memory

3.8 TB on disk

35 TB total disk

15K avg TPS (> 27K)Fall 2009


shardinginter and intra server


roll off old datalog.events_201005log.events_201006log.events_201007log.events_201008log.events_201009


truncate, drop is better than delete

connection pooling

➙ pgBouncer


simplify interface

➙ PL/Proxy

function API


myb ecology


reduce TPS!

➙ memcached

1 TB

get 140K/s, set 15K/s

PgFouine


memcached

set

get

clear

memcached

application

1 get 3 set

postgres

2 get


asynchronous

➙ message queues


message queues

consumer count

queue depth

async

queue broker

application

postgres

sync

queue 1 queue n

consumer 1 consumer n


measurement


measure


tuning(troubleshooting)one variable at a time, people


benchmarksrepeatability

statistics

practical usage


OLTPversus

OLAPThursday, August 5, 2010

online transaction processingonline analytical processing

pgreplayreplay log files


7analyze

host

bloat

DTrace/SystemTap

logs

contrib

statistics collector


cpumemory

ioThursday, August 5, 2010

SNMP

RRDtool

Staplr


Simple Network Management Protocol

overview


http://localhost/~grzm/stats/db/?interval=h

http://localhost/~grzm/stats/db/?interval=h

logs


log_min_duration_statement

log_duration

log_lock_waits

deadlock_timeout

log_temp_files

log_connections

log_disconnections


LOG: EXECUTOR STATISTICSDETAIL: ! system usage stats: ! 0.017621 elapsed 0.004762 user 0.000816 system sec ! [6.012501 user 0.336354 sys total] ! 0/0 [0/0] filesystem blocks in/out ! 0/0 [0/0] page faults/reclaims, 0 [0] swaps ! 0 [1] signals rcvd, 0/10 [4/14944] messages rcvd/sent ! 2/0 [210/0] voluntary/involuntary context switches ! buffer usage stats: ! Shared blocks: 9 read, 0 written, buffer hit rate = 0.00% ! Local blocks: 0 read, 0 written, buffer hit rate = 0.00% ! Direct blocks: 0 read, 0 writtenSTATEMENT: select * from posuta.index_statistics limit 1000;LOG: duration: 42.422 ms

log_statement_statslog_parser_statslog_planner_statslog_executor_stats


18.7. Error Reporting and Logging18.8. Run-Time Statistics

csv2009-05-19 10:25:35.470 EDT,"grzm","posuta_production",99595,"[local]",4a12c078.1850b,28,"SELECT",2009-05-19 10:21:44 EDT,2/30525,0,LOG,00000,"EXECUTOR STATISTICS","! system usage stats:! 1.786288 elapsed 0.065964 user 0.074493 system sec! [6.079580 user 0.412469 sys total]! 2/0 [2/0] filesystem blocks in/out! 0/0 [0/0] page faults/reclaims, 0 [0] swaps! 0 [1] signals rcvd, 0/13 [5/14960] messages rcvd/sent! 1008/0 [1230/0] voluntary/involuntary context switches! buffer usage stats:! Shared blocks: 1073 read, 0 written, buffer hit rate = 0.00%! Local blocks: 0 read, 0 written, buffer hit rate = 0.00%! Direct blocks: 0 read, 0 written",,,,,"select * from posuta.index_statistics where index_id = 265 limit 1000;",,


Splunk


http://www.splunk.com/

admin/maintenance


bloatpg_database_size

pg_relation_size

pg_total_relation_size

pg_column_size

pg_size_pretty


bloat report


http://localhost/~grzm/bloat/

http://localhost/~grzm/bloat/

key indexeskey constraints

➙ unique indexes

cached plans (≤8.2)


reindex


snapshot


Chapter 26. Monitoring Database Activity

pgStatIO Heavy Index Hitters - 10/20/09 20:02:06 - Interval: 5s----------------------------------------------------------------------------------Table Last Difference Total Hits----------------------------------------------------------------------------------1. schema_a.table_a 146502 7100138408792. schema_b.table_b 92171 554732596073. schema_a.table_c 38684 1069502426904. schema_a.table_d 32797 1105412280955. schema_a.table_e 25096 439398036636. schema_a.table_f 20940 755012502347. schema_a.table_g 10982 1265582078968. schema_a.table_h 9337 147866373045

topHeapHitters

topIndexHitters


0300


schema management

transactional DDL


upgradeslony

pg_upgrade


configurationversion control


look around you


shoulders of giants


Oracle · IBM DB2

InnoDB · BerkleyDB

Hadoop · Cassandra

BOOM · MongoDB


StonebrakerIngres

Postgres, Illustra

c-store, Vertica

h-store, VoltDBThursday, August 5, 2010

ActiveMQ http://activemq.apache.org/

Fusion-io http://www.fusionio.com/

MonetDB http://monetdb.cwi.nl/

myYearbook http://myyearbook.com/

memcached http://memcached.org/

nagios http://www.nagios.org/

PgBouncer http://wiki.postgresql.org/wiki/PgBouncer

pgFouine http://pgfouine.projects.postgresql.org/

pgreplay http://pgreplay.projects.postgresql.org/

PL/Proxy http://pgfoundry.org/projects/plproxy/

RabbitMQ http://www.rabbitmq.com/

Reconnoiter https://labs.omniti.com/labs/reconnoiter

RRDtool http://oss.oetiker.ch/rrdtool/

Slony http://www.slony.info/

Splunk http://www.splunk.com/

Truviso http://www.truviso.com/


http://www.fusionio.com






http://pgreplay.projects.postgresql.org






http://wiki.postgresql.org/wiki/PgBouncer


http://pgfouine.projects.postgresql.org

http://pgfouine.projects.postgresql.org





http://www.splunk.com












practical postgresql - drexel ccitwc24/cs500_su11/slides/michaelgleasman.pdf · practical...

Documents