pl/perl - new features in postgresql 9.0
DESCRIPTION
Slides from my talk at PGcon 2010 in Ottawa.TRANSCRIPT
![Page 1: PL/Perl - New Features in PostgreSQL 9.0](https://reader034.vdocuments.mx/reader034/viewer/2022051110/54c8dbaf4a7959e2498b461d/html5/thumbnails/1.jpg)
PL/Perl New Featuresin PostgreSQL 9.0
Tim Bunce - June 2010
Creative Commons BY-NC-SA 3.0
![Page 2: PL/Perl - New Features in PostgreSQL 9.0](https://reader034.vdocuments.mx/reader034/viewer/2022051110/54c8dbaf4a7959e2498b461d/html5/thumbnails/2.jpg)
PL/Perl Changes
‣ USER‣ INTERNAL‣ DBA‣ NYTPROF
![Page 3: PL/Perl - New Features in PostgreSQL 9.0](https://reader034.vdocuments.mx/reader034/viewer/2022051110/54c8dbaf4a7959e2498b461d/html5/thumbnails/3.jpg)
PL/Perl Changes
‣ USER‣ INTERNAL‣ DBA‣ NYTPROF
![Page 4: PL/Perl - New Features in PostgreSQL 9.0](https://reader034.vdocuments.mx/reader034/viewer/2022051110/54c8dbaf4a7959e2498b461d/html5/thumbnails/4.jpg)
New Builtins
![Page 5: PL/Perl - New Features in PostgreSQL 9.0](https://reader034.vdocuments.mx/reader034/viewer/2022051110/54c8dbaf4a7959e2498b461d/html5/thumbnails/5.jpg)
quote_...
quote_literal( "foo" ) ==> "'foo'" quote_literal( "don't \"carp\"" ) ==> "'don''t \"carp\"'" quote_literal( "" ) ==> "''"
quote_nullable( "foo" ) ==> "'foo'" quote_nullable( "don't \"carp\"" ) ==> "'don''t \"carp\"'" quote_nullable( "" ) ==> "''"
quote_ident( "foo" ) ==> "foo" quote_ident( "don't \"carp\"" ) ==> "\"don't \"\"carp\"\"\"" quote_ident( "" ) ==> "\"\""
![Page 6: PL/Perl - New Features in PostgreSQL 9.0](https://reader034.vdocuments.mx/reader034/viewer/2022051110/54c8dbaf4a7959e2498b461d/html5/thumbnails/6.jpg)
quote_...(undef)
quote_literal( undef ) ==> undef
quote_nullable( undef ) ==> "NULL"
quote_ident( undef ) ==> "\"\"" (warn)
![Page 7: PL/Perl - New Features in PostgreSQL 9.0](https://reader034.vdocuments.mx/reader034/viewer/2022051110/54c8dbaf4a7959e2498b461d/html5/thumbnails/7.jpg)
{en,de}code_bytea encode_bytea( "foo" ) ==> "\\x666f6f" decode_bytea( "\\x666f6f" ) ==> "foo" decode_bytea( "\\146\\157\\157") ==> "foo"
encode_bytea( "\x{263a}" ) ==> "\\xe298ba" UTF8 decode_bytea( "\\xe298ba" ) ==> "\342\230\272" Not UTF8
encode_bytea( "" ) ==> "\\x" decode_bytea( "\\x" ) ==> "" decode_bytea( "" ) ==> ""
encode_bytea( undef ) ==> "\\x" (warn) decode_bytea( undef ) ==> "" (warn)
![Page 8: PL/Perl - New Features in PostgreSQL 9.0](https://reader034.vdocuments.mx/reader034/viewer/2022051110/54c8dbaf4a7959e2498b461d/html5/thumbnails/8.jpg)
looks_like_number
looks_like_number( 1 ) ==> 1 looks_like_number( 0 ) ==> 1 looks_like_number( "+7.2e-9" ) ==> 1
looks_like_number( "foo" ) ==> 0 looks_like_number( "" ) ==> 0 looks_like_number( undef ) ==> undef
looks_like_number( " 4 " ) ==> 1
looks_like_number( "5plus" ) ==> 0 (but '5plus'+0=5)
![Page 9: PL/Perl - New Features in PostgreSQL 9.0](https://reader034.vdocuments.mx/reader034/viewer/2022051110/54c8dbaf4a7959e2498b461d/html5/thumbnails/9.jpg)
encode_array_*
encode_array_literal( ["foo","bar"] ) ==> "{\"foo\", \"bar\"}"
encode_array_constructor( ["foo","bar"] ) ==> "ARRAY['foo', 'bar']"
encode_array_literal( [1,[2,[undef]]] ) ==> "{\"1\", {\"2\", {NULL}}}"
encode_array_constructor( [1,[2,[undef]]] ) ==> "ARRAY['1', ARRAY['2', ARRAY[NULL]]]"
![Page 10: PL/Perl - New Features in PostgreSQL 9.0](https://reader034.vdocuments.mx/reader034/viewer/2022051110/54c8dbaf4a7959e2498b461d/html5/thumbnails/10.jpg)
encode_array_*
encode_array_literal( "foo" ) ==> "foo"encode_array_constructor( "foo" ) ==> "'foo'"
encode_array_literal( undef ) ==> undefencode_array_constructor( undef ) ==> "NULL"
![Page 11: PL/Perl - New Features in PostgreSQL 9.0](https://reader034.vdocuments.mx/reader034/viewer/2022051110/54c8dbaf4a7959e2498b461d/html5/thumbnails/11.jpg)
Trusted require/use
• require/use work for already loaded modules
use strict; # old way: BEGIN { strict->import(); }
• extra pre-loaded modules
use warnings;use Carp;use feature qw(say); # for perl 5.10 or lateruse utf8; # if server_encoding is utf8
![Page 12: PL/Perl - New Features in PostgreSQL 9.0](https://reader034.vdocuments.mx/reader034/viewer/2022051110/54c8dbaf4a7959e2498b461d/html5/thumbnails/12.jpg)
CONTEXT: ...
• PL/Perl tracks the context of log messages- before:
WARNING: ...some warning from perl code...
- now:WARNING: ...some warning from perl code...CONTEXT: PL/Perl function "..."
![Page 13: PL/Perl - New Features in PostgreSQL 9.0](https://reader034.vdocuments.mx/reader034/viewer/2022051110/54c8dbaf4a7959e2498b461d/html5/thumbnails/13.jpg)
DO '...' LANGUAGE plperl;
• Arbitrary chunks of perl code can be executed directly from psql, or client apps, via DO
• No need to create and run a stored procedure each time.
DO $$spi_exec("... $_ ...") for 'a'..'z';
$$ language plperl;
![Page 14: PL/Perl - New Features in PostgreSQL 9.0](https://reader034.vdocuments.mx/reader034/viewer/2022051110/54c8dbaf4a7959e2498b461d/html5/thumbnails/14.jpg)
Other Changes
• Using $a and $b in sort blocks now works!
• eval { ... } and eval "..."- can now be used in plperl
• END blocks are now run at end of session- they can't (currently) access the database
• Warnings from perl are now WARNINGs- they used to be NOTICE
![Page 15: PL/Perl - New Features in PostgreSQL 9.0](https://reader034.vdocuments.mx/reader034/viewer/2022051110/54c8dbaf4a7959e2498b461d/html5/thumbnails/15.jpg)
PL/Perl Changes
‣ USER‣ INTERNAL‣ DBA‣ NYTPROF
![Page 16: PL/Perl - New Features in PostgreSQL 9.0](https://reader034.vdocuments.mx/reader034/viewer/2022051110/54c8dbaf4a7959e2498b461d/html5/thumbnails/16.jpg)
INTERNAL
• The Safe module is no longer used for plperl- Improved security and reduced call overheads
- Upgrade to latest security patch!
• Validates return values are in server encoding- ERROR: invalid byte sequence for encoding
• Internal code refactoring and cleanup
![Page 17: PL/Perl - New Features in PostgreSQL 9.0](https://reader034.vdocuments.mx/reader034/viewer/2022051110/54c8dbaf4a7959e2498b461d/html5/thumbnails/17.jpg)
PL/Perl Changes
‣ USER‣ INTERNAL‣ DBA‣ NYTPROF
![Page 18: PL/Perl - New Features in PostgreSQL 9.0](https://reader034.vdocuments.mx/reader034/viewer/2022051110/54c8dbaf4a7959e2498b461d/html5/thumbnails/18.jpg)
New plperl.* Config
• Specify perl code to run during initialization:
plperl.on_init = '...perl...'plperl.on_plperlu_init = '...perl...'plperl.on_plperl_init = '...perl...'
• Can only be set by superuser or postgres.conf
• Code can't access the database
![Page 19: PL/Perl - New Features in PostgreSQL 9.0](https://reader034.vdocuments.mx/reader034/viewer/2022051110/54c8dbaf4a7959e2498b461d/html5/thumbnails/19.jpg)
~ Timeline ~
▾ Perl interpreter created on demand▿ Options from PERL5OPT env var are processed
▿ PL/Perl support code bootstraps
▿ plperl.on_init code runs
Above may happen in postmaster at startup if plperl is loaded via shared_preload_libraries
▾ Interpreter is specialised for plperl (or plperlu)▿ Modules loaded: strict, warnings, features, Carp
▿ Perl operators are restricted (require, open etc.)
▿ DynaLoader package is deleted
▿ plperl.on_plperl_init code runs
▿ Database access is enabled
![Page 20: PL/Perl - New Features in PostgreSQL 9.0](https://reader034.vdocuments.mx/reader034/viewer/2022051110/54c8dbaf4a7959e2498b461d/html5/thumbnails/20.jpg)
plperl.on_init
• Handy to set global perl configuationplperl.on_init='use lib qw(/myapp); use ...;'plperl.on_init='require "plperloninit.pl";'
• SECURITY RISK!Only load modules you're happy for plperl code to use!Also check any other modules loaded as dependencies!Use Devel::TraceLoad to see what's actually loaded:PERL5OPT='-MDevel::Trace=summary' pg_ctl ...
![Page 21: PL/Perl - New Features in PostgreSQL 9.0](https://reader034.vdocuments.mx/reader034/viewer/2022051110/54c8dbaf4a7959e2498b461d/html5/thumbnails/21.jpg)
PL/Perl Best Practice
• Include explicit use statements in functions
• Don't assume modules have been pre-loaded
• For plperlu that'll actually load if needed
• For plperl it'll check that module was loaded- so you'll get an immediate clear failure if not- for example on a replica with old postgres.conf file
![Page 22: PL/Perl - New Features in PostgreSQL 9.0](https://reader034.vdocuments.mx/reader034/viewer/2022051110/54c8dbaf4a7959e2498b461d/html5/thumbnails/22.jpg)
plperl.on_plperl_init
• Originally intended for things like- PGOPTIONS="-c plperl.on_plperl_init='...'"
- to enable debug or profiling for a session
• But...• Can only be set by superuser or postgres.conf- sadly, due to SECURITY DEFINER risks.- You shouldn't write SECURITY DEFINER
functions in plperl if untrusted users can use plperl!
![Page 23: PL/Perl - New Features in PostgreSQL 9.0](https://reader034.vdocuments.mx/reader034/viewer/2022051110/54c8dbaf4a7959e2498b461d/html5/thumbnails/23.jpg)
PL/Perl Changes
‣ USER‣ INTERNAL‣ DBA‣ NYTPROF
![Page 24: PL/Perl - New Features in PostgreSQL 9.0](https://reader034.vdocuments.mx/reader034/viewer/2022051110/54c8dbaf4a7959e2498b461d/html5/thumbnails/24.jpg)
Devel::NYTProfPerl Source Code Profiler
PostgreSQL::PLPerl::NYTProf
![Page 25: PL/Perl - New Features in PostgreSQL 9.0](https://reader034.vdocuments.mx/reader034/viewer/2022051110/54c8dbaf4a7959e2498b461d/html5/thumbnails/25.jpg)
Enabling NYTProf
• Via postgres.conf:plperl.on_init='use PostgreSQL::PLPerl::NYTProf'
• Via environment variable:PERL5OPT='-MPostgreSQL::PLPerl::NYTProf' pg_ctl ...
• Immediately active. To enable on demand:NYTPROF=start=no PERL5OPT=... pg_ctl ...DO 'DB::enable_profile' LANGUAGE plperl;
![Page 26: PL/Perl - New Features in PostgreSQL 9.0](https://reader034.vdocuments.mx/reader034/viewer/2022051110/54c8dbaf4a7959e2498b461d/html5/thumbnails/26.jpg)
Reporting from NYTProf
• Writes per-backend data files:
$PGDATA/nytprof.out.$pid
• To generate a report:
nytprofhtml --file=$PGDATA/nytprof.out.4321 --open
![Page 27: PL/Perl - New Features in PostgreSQL 9.0](https://reader034.vdocuments.mx/reader034/viewer/2022051110/54c8dbaf4a7959e2498b461d/html5/thumbnails/27.jpg)
~ Demo ~
Screencast: http://timbunce.blip.tvVideo: http://www.fosslc.org/drupal/content/plperl-new-features-90
![Page 29: PL/Perl - New Features in PostgreSQL 9.0](https://reader034.vdocuments.mx/reader034/viewer/2022051110/54c8dbaf4a7959e2498b461d/html5/thumbnails/29.jpg)
http://xkcd.com/519/