on beyond (postgresql) data types
TRANSCRIPT
On Beyond Data TypesJonathan S. Katz
PostgreSQL España February 16, 2015
About• CTO, VenueBook
• Co-Organizer, NYC PostgreSQL User Group (NYCPUG)
• Director, United States PostgreSQL Association
• ¡Primera vez en España!
• @jkatz05
2
A Brief Note on NYCPUG• Active since 2010
• Over 1,300 members
• Monthly Meetups
• PGConf NYC 2014
• 259 attendees
• PGConf US 2015:
• Mar 25 - 27 @ New York Marriott Downtown
• Already 160+ registrations
3
http://www.pgconf.us
Community Updates
Community Updates
8
Data Types• Fundamental
• 0 => 1
• 00001111
• Building Blocks
• 0x41424344
• Accessibility
• 1094861636
• 'ABCD'
9
C• char
• int
• float, double
• bool (C99)
• (short, long, signed, unsigned)
10
PostgreSQL
• char, varchar, text
• smallint, int, bigint
• real, double
• bool
11
I kid you not, I can spend close to an hour
on just those data types
12
PostgreSQL Primitives Oversimplified Summary
• Strings
• Use "text" unless you need actual limit on strings, o/w use "varchar"
• Don't use "char"
• Integers
• Use "int"
• If you seriously have big numbers, use "bigint"
• Numerical types
• Use "numeric" almost always
• If have IEEE 754 data source you need to record, use "float"
13
And If We Had More Time• (argh no pun intended)
• timestamp with time zone, timestamp without time zone
• date
• time with time zone, time without time zone
• interval
14
Summary of PostgreSQL Date/Time Types
• They are AWESOME
• Flexible input that you can customize
• Can perform mathematical operations in native format
• Thank you intervals!
• IMO better support than most programming languages have, let alone databases
15
16
PostgreSQL is a ORDBMS
• Designed to support more complex data types
• Complex data types => additional functionality
• Data Integrity
• Performance
17
Let's Start Easy: Geometry
18
PostgreSQL Geometric Types
19
Name Size Representation Format
point 16 bytes point on a plane (x,y)
lseg 32 bytes finite line segment ((x1, y1), (x2, y2))
box 32 bytes rectangular box ((x1, y1), (x2, y2))
path 16 + 16n bytes
closed path (similar to polygon, n = total points
((x1, y1), (x2, y2), …, (xn, yn))
path 16 + 16n bytes
open path, n = total points
[(x1, y1), (x2, y2), …, (xn, yn)]
polygon 40 bytes + 16n
polygon ((x1, y1), (x2, y2), …, (xn, yn))
circle 24 bytes circle – center point and radius
<(x, y), r>
http://www.postgresql.org/docs/current/static/datatype-geometric.html
Geometric Operators• 31 different operators built into PostgreSQL
20
obdt=# SELECT point(1,1) + point(2,2);!----------! (3,3)
obdt=# SELECT point(1,1) ~= point(2,2);!----------! f!!obdt=# SELECT point(1,1) ~= point(1,1);!----------! t
obdt=# SELECT point(1,1) <-> point(4,4);!------------------! 4.24264068711929
Equivalence
Translation
Distance
http://www.postgresql.org/docs/current/static/functions-geometry.html
Geometric Operators
21
obdt=# SELECT '(0,0),5)'::circle && '((2,2),3)'::circle;!----------! t
obdt=# SELECT '(0,0),5)'::circle @> point(2,2);!----------! t
Overlapping
Containment
obdt=# SELECT '((0,0), (1,1))'::lseg ?|| '((1,-1), (2,0))'::lseg; !----------! t
Is Parallel?
http://www.postgresql.org/docs/current/static/functions-geometry.html
obdt=# SELECT '((0,0), (1,1))'::lseg ?# '((0,0), (5,5))'::box;!----------! t
Intersection
Geometric Functions• 13 non-type conversion functions built into PostgreSQL
22
obdt=# SELECT area('((0,0),5)'::circle);!------------------! 78.5398163397448
Area
obdt=# SELECT center('((0,0),(5,5))'::box);!-----------! (2.5,2.5)
Center
obdt=# SELECT length('((0,0),(5,5))'::lseg);!------------------! 7.07106781186548
Length
obdt=# SELECT width('((0,0),(3,2))'::box);!-------! 3
obdt=# SELECT height('((0,0),(3,2))'::box);!--------! 2
Width
Height
Geometric Performance
• Size on Disk
• Consider I/O on reads
• But indexing should help!!
23
Geometric Performance
24
CREATE TABLE houses (plot box);!!INSERT INTO houses!SELECT box(!! point((500 * random())::int, (500 * random())::int),! ! point((750 * random() + 500)::int, (750 * random() + 500)::int)! )!FROM generate_series(1, 1000000);
obdt=# CREATE INDEX houses_plot_idx ON houses (plot);!ERROR: data type box has no default operator class for access method "btree"!HINT: You must specify an operator class for the index or define a default operator class for the data type.
Solution #1: Expression Indexes
25
obdt=# EXPLAIN ANALYZE SELECT * FROM houses WHERE area(plot) BETWEEN 50000 AND 75000;!-------------! Seq Scan on houses (cost=0.00..27353.00 rows=5000 width=32) (actual time=0.077..214.431 rows=26272 loops=1)! Filter: ((area(plot) >= 50000::double precision) AND (area(plot) <= 75000::double precision))! Rows Removed by Filter: 973728! Total runtime: 215.965 ms
obdt=# CREATE INDEX houses_plot_area_idx ON houses (area(plot));!!obdt=# EXPLAIN ANALYZE SELECT * FROM houses WHERE area(plot) BETWEEN 50000 AND 75000;!------------! Bitmap Heap Scan on houses (cost=107.68..7159.38 rows=5000 width=32) (actual time=5.433..14.686 rows=26272 loops=1)! Recheck Cond: ((area(plot) >= 50000::double precision) AND (area(plot) <= 75000::double precision))! -> Bitmap Index Scan on houses_plot_area_idx (cost=0.00..106.43 rows=5000 width=0) (actual time=4.300..4.300 rows=26272 loops=1)! Index Cond: ((area(plot) >= 50000::double precision) AND (area(plot) <= 75000::double precision))! Total runtime: 16.025 ms
http://www.postgresql.org/docs/current/static/indexes-expressional.html
Solution #2: GiST Indexes
26
obdt=# EXPLAIN ANALYZE SELECT * FROM houses WHERE plot @> '((100,100),(300,300))'::box;!------------! Seq Scan on houses (cost=0.00..19853.00 rows=1000 width=32) (actual time=0.009..96.680 rows=40520 loops=1)! Filter: (plot @> '(300,300),(100,100)'::box)! Rows Removed by Filter: 959480! Total runtime: 98.662 ms
obdt=# CREATE INDEX houses_plot_gist_idx ON houses USING gist(plot);!!obdt=# EXPLAIN ANALYZE SELECT * FROM houses WHERE plot @> '((100,100),(300,300))'::box;!------------! Bitmap Heap Scan on houses (cost=56.16..2813.20 rows=1000 width=32) (actual time=12.053..24.468 rows=40520 loops=1)! Recheck Cond: (plot @> '(300,300),(100,100)'::box)! -> Bitmap Index Scan on houses_plot_gist_idx (cost=0.00..55.91 rows=1000 width=0) (actual time=10.700..10.700 rows=40520 loops=1)! Index Cond: (plot @> '(300,300),(100,100)'::box)! Total runtime: 26.451 ms
http://www.postgresql.org/docs/current/static/indexes-types.html
Solution #2+: KNN-Gist
27
obdt=# CREATE INDEX locations_geocode_gist_idx ON locations USING gist(geocode);!!obdt=# EXPLAIN ANALYZE SELECT * FROM locations ORDER BY geocode <-> point(41.88853,-87.628852) LIMIT 10;!------------!Limit (cost=0.29..1.06 rows=10 width=16) (actual time=0.098..0.235 rows=10 loops=1)! -> Index Scan using locations_geocode_gist_idx on locations (cost=0.29..77936.29 rows=1000000 width=16) (actual time=0.097..0.234 rows=10 loops=1)! Order By: (geocode <-> '(41.88853,-87.628852)'::point)!
Total runtime: 0.257 ms
obdt=# CREATE TABLE locations (geocode point);!!obdt=# INSERT INTO locations!SELECT point(90 * random(), 180 * random())!FROM generate_series(1, 1000000);
obdt=# EXPLAIN ANALYZE SELECT * FROM locations ORDER BY geocode <-> point(41.88853,-87.628852) LIMIT 10;!------------! Limit (cost=39519.39..39519.42 rows=10 width=16) (actual time=319.306..319.309 rows=10 loops=1)! -> Sort (cost=39519.39..42019.67 rows=1000110 width=16) (actual time=319.305..319.307 rows=10 loops=1)! Sort Key: ((geocode <-> '(41.88853,-87.628852)'::point))! Sort Method: top-N heapsort Memory: 25kB! -> Seq Scan on locations (cost=0.00..17907.38 rows=1000110 width=16) (actual time=0.019..189.687 rows=1000000 loops=1)! Total runtime: 319.332 ms
http://www.slideshare.net/jkatz05/knn-39127023
• For when you are doing real things with shapes
28• (and geographic information systems)
Solution #3: PostGIS
For more on PostGIS, please go back in time to yesterday
and see Regina & Leo's tutorial
29
Let's Take a Break With UUIDs
30
2024e06c-44ff-5047-b1ae-00def276d043
! Universally Unique Identifiers
! 16 bytes on disk
! Acceptable input formats include:
– A0EEBC99-9C0B-4EF8-BB6D-6BB9BD380A11
– {a0eebc99-9c0b-4ef8-bb6d-6bb9bd380a11}
– a0eebc999c0b4ef8bb6d6bb9bd380a11
– a0ee-bc99-9c0b-4ef8-bb6d-6bb9-bd38-0a11
– {a0eebc99-9c0b4ef8-bb6d6bb9-bd380a11}
UUID + PostgreSQL
31
http://www.postgresql.org/docs/current/static/datatype-uuid.html
UUID Functions
32http://www.postgresql.org/docs/current/static/uuid-ossp.html
obdt=# CREATE EXTENSION IF NOT EXISTS "uuid-ossp";!"obdt=# SELECT uuid_generate_v1();! uuid_generate_v1 !--------------------------------------! d2729728-3d50-11e4-b1af-005056b75e1e!"obdt=# SELECT uuid_generate_v1mc();! uuid_generate_v1mc !--------------------------------------! e04668a2-3d50-11e4-b1b0-1355d5584528!"obdt=# SELECT uuid_generate_v3(uuid_ns_url(), 'http://www.postgresopen.org');! uuid_generate_v3 !--------------------------------------! d0bc1ba2-bf07-312f-bf6a-436e18b5b046!"obdt=# SELECT uuid_generate_v4();! uuid_generate_v4 !--------------------------------------! 0809d8fe-512c-4f02-ba37-bc2e9865e884!"obdt=# SELECT uuid_generate_v5(uuid_ns_url(), 'http://www.postgresopen.org');! uuid_generate_v5 !--------------------------------------! d508c779-da5c-5998-bd88-8d76d446754e
Network Address Types• inet (IPv4 & IPv6)
– SELECT '192.168.1.1'::inet;
– SELECT '192.168.1.1/32'::inet;
– SELECT '192.168.1.1/24'::inet;
• cidr (IPv4 & IPv6)
– SELECT '192.168.1.1'::cidr;
– SELECT '192.168.1.1/32'::cidr;
– SELECT '192.168.1.1/24'::cidr;
• macaddr
– SELECT '08:00:2b:01:02:03'::macaddr;
33
http://www.postgresql.org/docs/current/static/datatype-net-types.html
Networks can do Math
34
http://www.postgresql.org/docs/current/static/functions-net.html
Postgres Can Help Manage Your Routing Tables
35
http://www.postgresql.org/docs/current/static/functions-net.html
...perhaps with a foreign data wrapper and a background worker, perhaps it can fully mange your routing tables?
Arrays
• ...because a database is an "array" of tuples
• ...and a "tuple" is kind of like an array
• ...can we have an array within a tuple?
36
37
Array Facts
obdt=# SELECT (ARRAY[1,2,3])[1];!-------! 1
38
obdt=# SELECT (ARRAY[1,2,3])[0];!-------!
Arrays are 1-indexed
obdt=# CREATE TABLE lotto (!! ! numbers int[3]! );!"obdt=# INSERT INTO lotto VALUES (!! ARRAY[1,2,3,4]! );!"obdt=# SELECT * FROM lotto;!-----------! {1,2,3,4}
Size constraints not enforced
Arrays Are Malleable
39
obdt=# UPDATE lotto SET numbers = ARRAY[1,2,3];!"obdt=# SELECT * FROM lotto;!---------! {1,2,3}!"obdt=# UPDATE lotto SET numbers[3] = '7';!"obdt=# SELECT * FROM lotto;!---------! {1,2,7}!"obdt=# UPDATE lotto SET numbers[1:2] = ARRAY[6,5];!"obdt=# SELECT * FROM lotto;!---------! {6,5,7}
Array Operations• <, <=, =, >= >, <>
– full array comparisons
– B-tree indexable
40
SELECT ARRAY[1,2,3] @> ARRAY[1,2];!SELECT ARRAY[1,2] <@ ARRAY[1,2,3];
SELECT ARRAY[1,2,3] || ARRAY[3,4,5];!SELECT ARRAY[ARRAY[1,2]] || ARRAY[3,4];!SELECT ARRAY[1,2,3] || 4;
SELECT ARRAY[1,2,3] && ARRAY[3,4,5]; Overlaps
Containment
Concatenation
Integer Arrays Use GIN
41
obdt=# CREATE INDEX int_arrays_data_gin_idx ON int_arrays USING GIN(data);!"obdt=# EXPLAIN ANALYZE SELECT *!FROM int_arrays!WHERE 5432 = ANY (data);!---------------!Seq Scan on int_arrays (cost=0.00..30834.00 rows=5000 width=33) (actual time=1.237..157.397 rows=3 loops=1)! Filter: (5432 = ANY (data))! Rows Removed by Filter: 999997! Total runtime: 157.419 ms!"obdt=# EXPLAIN ANALYZE SELECT * FROM int_arrays
WHERE ARRAY[5432] <@ data;!---------------! Bitmap Heap Scan on int_arrays (cost=70.75..7680.14 rows=5000 width=33) (actual time=0.023..0.024 rows=3 loops=1)! Recheck Cond: ('{5432}'::integer[] <@ data)! -> Bitmap Index Scan on int_arrays_data_gin_idx (cost=0.00..69.50 rows=5000 width=0) (actual time=0.019..0.019 rows=3 loops=1)! Index Cond: ('{5432}'::integer[] <@ data)!
Total runtime: 0.090 ms
Array Functions• modification
! SELECT array_append(ARRAY[1,2,3], 4);!
! SELECT array_prepend(1, ARRAY[2,3,4]);!
! SELECT array_cat(ARRAY[1,2], ARRAY[3,4]);!
! SELECT array_remove(ARRAY[1,2,1,3], 1);!
! SELECT array_replace(ARRAY[1,2,1,3], 1, -4);!
• size
! SELECT array_length(ARRAY[1,2,3,4], 1); -- 4!
! SELECT array_ndims(ARRAY[ARRAY[1,2], ARRAY[3,4]]);!
! -- 2!
! SELECT array_dims(ARRAY[ARRAY[1,2], ARRAY[3,4]]);!
! -- [1:2][1:2]
42
http://www.postgresql.org/docs/current/static/functions-array.html
Array Functions
43
obdt=# SELECT array_to_string(ARRAY[1,2,NULL,4], ',', '*');!-----------------! 1,2,*,4
obdt=# SELECT unnest(ARRAY[1,2,3]);! unnest !--------! 1! 2! 3
Array to String
Array to Set
http://www.postgresql.org/docs/current/static/functions-array.html
array_agg• useful for variable-length lists or "unknown # of columns"
obdt=# SELECT!! t.title!! array_agg(s.full_name)!FROM talk t!JOIN speakers_talks st ON st.talk_id = t.id!JOIN speaker s ON s.id = st.speaker_id!GROUP BY t.title;!
" title | array_agg !---------------------+-----------! Data Types | {Jonathan, Jim}! Administration | {Bruce}! User Groups | {Josh, Jonathan, Magnus}
44
http://www.postgresql.org/docs/current/static/functions-array.html
Ranges• Scheduling
• Probability
• Measurements
• Financial applications
• Clinical trial data
• Intersections of ordered data
45
Why Range Overlaps Are Difficult
46
Before Postgres 9.2• OVERLAPS
"
"
"
• Limitations:
• Only date/time
• Start <= x <= End
SELECT!! ('2013-01-08`::date, '2013-01-10'::date) OVERLAPS ('2013-01-09'::date, '2013-01-12'::date);
47
Postgres 9.2+• INT4RANGE (integer)!
• INT8RANGE (bigint)!
• NUMRANGE (numeric)!
• TSRANGE (timestamp without time zone)!
• TSTZRANGE (timestamp with time zone)!
• DATERANGE (date)
48
http://www.postgresql.org/docs/current/static/rangetypes.html
Range Type Size• Size on disk = 2 * (data type) + 1
• sometimes magic if bounds are equal
obdt=# SELECT pg_column_size(daterange(CURRENT_DATE, CURRENT_DATE));!----------------! 9!
"obdt=# SELECT pg_column_size(daterange(CURRENT_DATE,CURRENT_DATE + 1));!----------------! 17
49
Range Bounds• Ranges can be inclusive, exclusive or both
• [2,4] => 2 ≤ x ≤ 4
• [2,4) => 2 ≤ x < 4
• (2,4] => 2 < x ≤ 4
• (2,4) => 2 < x < 4
"
• Can also be empty
50
Infinite Ranges• Ranges can be infinite
– [2,) => 2 ≤ x < ∞
– (,2] => -∞ < x ≤ 2
• CAVEAT EMPTOR
– “infinity” has special meaning with timestamp ranges
– [CURRENT_TIMESTAMP,) = [CURRENT_TIMESTAMP,]
– [CURRENT_TIMESTAMP, 'infinity') <> [CURRENT_TIMEAMP, 'infinity']
51
Constructing Ranges
obdt=# SELECT '[1,10]'::int4range;!-----------! [1,11)
52
Constructing Ranges• Constructor defaults to '[)'
53
obdt=# SELECT numrange(9.0, 9.5); !------------! [9.0,9.5)
Finding Overlapping Ranges
obdt=# SELECT *!FROM cars!
WHERE cars.price_range && int4range(13000, 15000, '[]')!ORDER BY lower(cars.price_range);!
-----------! id | name | price_range !----+---------------------+---------------! 5 | Ford Mustang | [11000,15001)! 6 | Lincoln Continental | [12000,14001)
54
http://www.postgresql.org/docs/current/static/functions-range.html
Ranges + GiSTobdt=# CREATE INDEX ranges_bounds_gist_idx ON cars USING gist (bounds);!
"obdt=# EXPLAIN ANALYZE SELECT * FROM ranges WHERE int4range(500,1000) && bounds;!
------------!Bitmap Heap Scan on ranges !(actual time=0.283..0.370 rows=653 loops=1)! Recheck Cond: ('[500,1000)'::int4range && bounds)! -> Bitmap Index Scan on ranges_bounds_gist_idx (actual time=0.275..0.275 rows=653 loops=1)! Index Cond: ('[500,1000)'::int4range && bounds)! Total runtime: 0.435 ms
55
Large Search Range?test=# EXPLAIN ANALYZE SELECT * FROM ranges WHERE int4range(10000,1000000) && bounds;! QUERY PLAN !-------------! Bitmap Heap Scan on ranges! (actual time=184.028..270.323 rows=993068 loops=1)! Recheck Cond: ('[10000,1000000)'::int4range && bounds)! -> Bitmap Index Scan on ranges_bounds_gist_idx ! !(actual time=183.060..183.060 rows=993068 loops=1)! Index Cond: ('[10000,1000000)'::int4range && bounds)! Total runtime: 313.743 ms
56
SP-GiST• space-partitioned generalized search tree
• ideal for non-balanced data structures
– k-d trees, quad-trees, suffix trees
– divides search space into partitions of unequal size
• matching partitioning rule = fast search
• traditionally for "in-memory" transactions, converted to play nicely with I/O
57
http://www.postgresql.org/docs/9.3/static/spgist.html
GiST vs SP-‐GiST: Space
GiST Clustered SP-GiST Clustered GiST Sparse SP-GiST Sparse
100K Size 6MB 5MB 6MB 11MB
100K Time 0.5s .4s 2.5s 7.8s
250K Size 15MB 12MB 15MB 28MB
250K Time 1.5s 1.1s 6.3s 47.2s
500K Size 30MB 25MB 30MB 55MB
500K Time 3.1s 3.0s 13.9s 192s
1MM Size 59MB52MB
60MB 110MB
1MM Time 5.1s 5.7s 29.2 777s
58
Schedulingobdt=# CREATE TABLE travel_log (! id serial PRIMARY KEY,! name varchar(255),! travel_range daterange,! EXCLUDE USING gist (travel_range WITH &&)!);!"obdt=# INSERT INTO travel_log (name, trip_range) VALUES ('Chicago',
daterange('2012-03-12', '2012-03-17'));!"obdt=# INSERT INTO travel_log (name, trip_range) VALUES ('Austin',
daterange('2012-03-16', '2012-03-18'));!"ERROR: conflicting key value violates exclusion constraint
"travel_log_trip_range_excl"!DETAIL: Key (trip_range)=([2012-03-16,2012-03-18)) conflicts with
existing key (trip_range)=([2012-03-12,2012-03-17)).
59
Extending Ranges
obdt=# CREATE TYPE inetrange AS RANGE (!! SUBTYPE = inet!);!"obdt=# SELECT '192.168.1.8'::inet <@ inetrange('192.168.1.1', '192.168.1.10');!----------! t!"obdt=# SELECT '192.168.1.20'::inet <@ inetrange('192.168.1.1', '192.168.1.10');!----------! f
60
Now For Something Unrelated
Let's talk non-relational data in PostgreSQL
61
hstore• key-value store in PostgreSQL • binary storage • key / values represented as strings when
querying
CREATE EXTENSION hstore;
SELECT 'jk=>1, jm=>2'::hstore; !--------------------! "jk"=>"1", "jm"=>"2"
62
http://www.postgresql.org/docs/current/static/hstore.html
Making hstore objectsobdt=# SELECT hstore(ARRAY['jk', 'jm'], ARRAY['1', '2']);!---------------------! "jk"=>"1", "jm"=>"2"!"obdt=# SELECT hstore(ARRAY['jk', '1', 'jm', '2']);!---------------------! "jk"=>"1", "jm"=>"2"!"obdt=# SELECT hstore(ROW('jk', 'jm'));!---------------------! "f1"=>"jk", "f2"=>"jm"
63
Accessing hstoreobdt=# SELECT ('jk=>1, jm=>2'::hstore) -> 'jk';!----------! 1!"obdt=# SELECT ('jk=>1, jm=>2'::hstore) -> ARRAY['jk','jm'];!----------! {1,2}!"obdt=# SELECT delete('jk=>1, jm=>2'::hstore, 'jm');!-----------! "jk"=>"1"
64
hstore operatorsobdt=# SELECT ('jk=>1, jm=>2'::hstore) @> 'jk=>1'::hstore;!----------! t!
"obdt=# SELECT ('jk=>1, jm=>2'::hstore) ? 'sf';!----------!f!
"obdt=# SELECT ('jk=>1, jm=>2'::hstore) ?& ARRAY['jk', 'sf'];!----------!f!
"obdt=# SELECT ('jk=>1, jm=>2'::hstore) ?| ARRAY['jk', 'sf'];!----------!t
65
hstore Performance
66
obdt=# EXPLAIN ANALYZE SELECT * FROM keypairs WHERE data ? '3';!-----------------------! Seq Scan on keypairs (cost=0.00..19135.06 rows=950 width=32) (actual time=0.071..214.007 rows=1 loops=1)! Filter: (data ? '3'::text)! Rows Removed by Filter: 999999! Total runtime: 214.028 ms
obdt=# CREATE INDEX keypairs_data_gin_idx ON keypairs USING gin(data);!"obdt=# EXPLAIN ANALYZE SELECT * FROM keypairs WHERE data ? '3';!--------------! Bitmap Heap Scan on keypairs (cost=27.75..2775.66 rows=1000 width=24) (actual time=0.046..0.046 rows=1 loops=1)! Recheck Cond: (data ? '3'::text)! -> Bitmap Index Scan on keypairs_data_gin_idx (cost=0.00..27.50 rows=1000 width=0) (actual time=0.041..0.041 rows=1 loops=1)! Index Cond: (data ? '3'::text)! Total runtime: 0.073 ms
JSON and PostgreSQL• Started in 2010 as a Google Summer of Code Project
• https://wiki.postgresql.org/wiki/JSON_datatype_GSoC_2010
• Goal:
• be similar to XML data type functionality in Postgres
• be committed as an extension for PostgreSQL 9.1
67
What Happened?• Different proposals over how to finalize the
implementation
• binary vs. text
• Core vs Extension
• Discussions between “old” vs. “new” ways of packaging for extensions
68
Foreshadowing
69
Foreshadowing
70
PostgreSQL 9.2: JSON• JSON data type in core PostgreSQL
• based on RFC 4627
• only “strictly” follows if your database encoding is UTF-8
• text-based format
• checks for validity
71
PostgreSQL 9.2: JSON
obdt=# SELECT '[{"PUG": "NYC"}]'::json;!------------------! [{"PUG": "NYC"}]!
""obdt=# SELECT '[{"PUG": "NYC"]'::json;!ERROR: invalid input syntax for type json at character 8!DETAIL: Expected "," or "}", but found "]".!CONTEXT: JSON data, line 1: [{"PUG": "NYC"]
72
http://www.postgresql.org/docs/current/static/datatype-json.html
PostgreSQL 9.2: JSON• array_to_json
73
obdt=# SELECT array_to_json(ARRAY[1,2,3]);!---------------! [1,2,3]
PostgreSQL 9.2: JSON• row_to_json
74
obdt=# SELECT row_to_json(category) FROM category;!------------!{"cat_id":652,"cat_pages":35,"cat_subcats":17,"cat_files":0,"title":"Continents"}
PostgreSQL 9.2: JSON
• In summary, within core PostgreSQL, it was a starting point
75
PostgreSQL 9.3: JSON Ups its Game
• Added operators and functions to read / prepare JSON
• Added casts from hstore to JSON
76
PostgreSQL 9.3: JSONOperator Description Example
-> return JSON array element OR JSON object field
’[1,2,3]’::json -> 0; ’{"a": 1, "b": 2, "c": 3}’::json -> ’b’;
->> return JSON array element OR JSON object field AS text
[’1,2,3]’::json ->> 0; ’{"a": 1, "b": 2, "c": 3}’::json ->> ’b’;
#> return JSON object using path ’{"a": 1, "b": 2, "c": [1,2,3]}’::json #> ’{c, 0}’;
#>> return JSON object using path AS text
’{"a": 1, "b": 2, "c": [1,2,3]}’::json #> ’{c, 0}’;
77
http://www.postgresql.org/docs/current/static/functions-json.html
Operator GotchasSELECT * FROM category_documents!
WHERE data->’title’ = ’PostgreSQL’;!
ERROR: operator does not exist: json = unknown!
LINE 1: ...ECT * FROM category_documents WHERE data->’title’ = ’Postgre... ^HINT: No operator matches the given name and argument type(s). You might need to add explicit type casts.
78
Operator GotchasSELECT * FROM category_documents!
WHERE data->>’title’ = ’PostgreSQL’;!
-----------------------!
{"cat_id":252739,"cat_pages":14,"cat_subcats":0,"cat_files":0,"title":"PostgreSQL"}!
(1 row)
79
For the Upcoming Examples• Wikipedia English category titles – all 1,823,644 that I
downloaded"• Relation looks something like:
80
Column | Type | Modifiers !-------------+---------+--------------------! cat_id | integer | not null! cat_pages | integer | not null default 0! cat_subcats | integer | not null default 0! cat_files | integer | not null default 0! title | text |
Performance?EXPLAIN ANALYZE SELECT * FROM category_documents!
WHERE data->>’title’ = ’PostgreSQL’;!
---------------------!
Seq Scan on category_documents (cost=0.00..57894.18 rows=9160 width=32) (actual time=360.083..2712.094 rows=1 loops=1)!
Filter: ((data ->> ’title’::text) = ’PostgreSQL’::text)!
Rows Removed by Filter: 1823643!
Total runtime: 2712.127 ms
81
Performance?
CREATE INDEX category_documents_idx ON category_documents (data);!
ERROR: data type json has no default operator class for access method "btree"!
HINT: You must specify an operator class for the index or define a default operator class for the data type.
82
Let’s Be Clever• json_extract_path, json_extract_path_text
• LIKE (#>, #>>) but with list of args
83
SELECT json_extract_path(!! ’{"a": 1, "b": 2, "c": [1,2,3]}’::json,!! ’c’, ’0’);!--------!1
Performance RevisitedCREATE INDEX category_documents_data_idx!ON category_documents!! (json_extract_path_text(data, ’title’));!"obdt=# EXPLAIN ANALYZE!SELECT * FROM category_documents!WHERE json_extract_path_text(data, ’title’) = ’PostgreSQL’;!------------! Bitmap Heap Scan on category_documents (cost=303.09..20011.96 rows=9118 width=32) (actual time=0.090..0.091 rows=1 loops=1)! Recheck Cond: (json_extract_path_text(data, VARIADIC ’{title}’::text[]) = ’PostgreSQL’::text)! -> Bitmap Index Scan on category_documents_data_idx (cost=0.00..300.81 rows=9118 width=0) (actual time=0.086..0.086 rows=1 loops=1)! Index Cond: (json_extract_path_text(data, VARIADIC ’{title}’::text[]) = ’PostgreSQL’::text)!" Total runtime: 0.105 ms!
84
The Relation vs JSON• Size on Disk
• category (relation) - 136MB
• category_documents (JSON) - 238MB
• Index Size for “title”
• category - 89MB
• category_documents - 89MB
• Average Performance for looking up “PostgreSQL”
• category - 0.065ms
• category_documents - 0.070ms
85
JSON Aggregates• (this is pretty cool) • json_agg
86
http://www.postgresql.org/docs/current/static/functions-json.html
SELECT b, json_agg(stuff)!FROM stuff!GROUP BY b;!" b | json_agg !------+----------------------------------! neat | [{"a":4,"b":"neat","c":[4,5,6]}]! wow | [{"a":1,"b":"wow","c":[1,2,3]}, +! | {"a":3,"b":"wow","c":[7,8,9]}]! cool | [{"a":2,"b":"cool","c":[4,5,6]}]
hstore gets in the game• hstore_to_json
• converts hstore to json, treating all values as strings
• hstore_to_json_loose
• converts hstore to json, but also tries to distinguish between data types and “convert” them to proper JSON representations
SELECT hstore_to_json_loose(’"a key"=>1, b=>t, c=>null, d=>12345, e=>012345, f=>1.234, g=>2.345e+4’);
----------------
{"b": true, "c": null, "d": 12345, "e": "012345", "f": 1.234, "g": 2.345e+4, "a key": 1}
87
Next Steps?
• In PostgreSQL 9.3, JSON became much more useful, but…
• Difficult to search within JSON
• Difficult to build new JSON objects
88
89
“Nested hstore”• Proposed at PGCon 2013 by Oleg Bartunov and Teodor Sigaev
• Hierarchical key-value storage system that supports arrays too and stored in binary format
• Takes advantage of GIN indexing mechanism in PostgreSQL
• “Generalized Inverted Index”
• Built to search within composite objects
• Arrays, fulltext search, hstore
• …JSON?
90http://www.pgcon.org/2013/schedule/attachments/280_hstore-pgcon-2013.pdf
How JSONB Came to Be• JSON is the “lingua franca per trasmissione la data
nella web”
• The PostgreSQL JSON type was in a text format and preserved text exactly as input
• e.g. duplicate keys are preserved
• Create a new data type that merges the nested Hstore work to create a JSON type stored in a binary format: JSONB
91
JSONB ≠ BSON
BSON is a data type created by MongoDB as a “superset of JSON” "
JSONB lives in PostgreSQL and is just JSON that is stored in a binary format on disk
92
JSONB Gives Us More Operators
• a @> b - is b contained within a?
• { "a": 1, "b": 2 } @> { "a": 1} -- TRUE!
• a <@ b - is a contained within b?
• { "a": 1 } <@ { "a": 1, "b": 2 } -- TRUE!
• a ? b - does the key “b” exist in JSONB a?
• { "a": 1, "b": 2 } ? 'a' -- TRUE!
• a ?| b - does the array of keys in “b” exist in JSONB a?
• { "a": 1, "b": 2 } ?| ARRAY['b', 'c'] -- TRUE!
• a ?& b - does the array of keys in "b" exist in JSONB a?
• { "a": 1, "b": 2 } ?& ARRAY['a', 'b'] -- TRUE
93
JSONB Gives us GIN• Recall - GIN indexes are used to "look inside"
objects
• JSONB has two flavors of GIN:
• Standard - supports @>, ?, ?|, ?&
"
• "Path Ops" - supports only @>
94
CREATE INDEX category_documents_data_idx USING gin(data);
CREATE INDEX category_documents_path_data_idx USING gin(data jsonb_path_ops);
JSONB Gives Us Flexibilityobdt=# SELECT * FROM category_documents WHERE!! data @> '{"title": "PostgreSQL"}';!"----------------! {"title": "PostgreSQL", "cat_id": 252739, "cat_files": 0, "cat_pages": 14, "cat_subcats": 0}!""obdt=# SELECT * FROM category_documents WHERE!! data @> '{"cat_id": 5432 }';!"----------------! {"title": "1394 establishments", "cat_id": 5432, "cat_files": 0, "cat_pages": 4, "cat_subcats": 2}
95
JSONB Gives Us SpeedEXPLAIN ANALYZE SELECT * FROM category_documents!! WHERE data @> '{"title": "PostgreSQL"}';! !------------! Bitmap Heap Scan on category_documents (cost=38.13..6091.65 rows=1824 width=153) (actual time=0.021..0.022 rows=1 loops=1)! Recheck Cond: (data @> '{"title": "PostgreSQL"}'::jsonb)! Heap Blocks: exact=1! -> Bitmap Index Scan on category_documents_path_data_idx (cost=0.00..37.68 rows=1824 width=0) (actual time=0.012..0.012 rows=1 loops=1)! Index Cond: (data @> '{"title": "PostgreSQL"}'::jsonb)! Planning time: 0.070 ms! Execution time: 0.043 ms
96
JSONB + Wikipedia Categories: By the Numbers
• Size on Disk
• category (relation) - 136MB
• category_documents (JSON) - 238MB
• category_documents (JSONB) - 325MB
• Index Size for “title”
• category - 89MB
• category_documents (JSON with one key using an expression index) - 89MB
• category_documents (JSONB, all GIN ops) - 311MB
• category_documents (JSONB, just @>) - 203MB
• Average Performance for looking up “PostgreSQL”
• category - 0.065ms
• category_documents (JSON with one key using an expression index) - 0.070ms
• category_documents (JSONB, all GIN ops) - 0.115ms
• category_documents (JSONB, just @>) - 0.045ms
97
Wow
• That was a lot of material
98
In Summary• PostgreSQL has a lot of advanced data types
• They are easy to access
• They have a lot of functionality around them
• They are durable
• They perform well (but of course must be used correctly)
• Furthermore, you can extend PostgreSQL to:
• Better manipulate your favorite data type
• Create more data types
• ...well, do basically what you want it to do
99
And That's All
• Thank You!
• Questions?
• @jkatz05
100