PL/Perl New Featuresin PostgreSQL 9.0

Tim Bunce - Dec 2010

Creative Commons BY-NC-SA 3.0

PL/Perl Changes

‣ USER‣ INTERNAL‣ DBA‣ NYTPROF‣ FUTURE

Some features are in recent 8.x.y due to back-porting of security changes

PL/Perl Changes

‣ USER‣ INTERNAL‣ DBA‣ NYTPROF‣ FUTURE

New Builtins

quote_...

quote_literal( "foo" ) ==> "'foo'" quote_literal( "don't \"carp\"" ) ==> "'don''t \"carp\"'" quote_literal( "" ) ==> "''"

quote_nullable( "foo" ) ==> "'foo'" quote_nullable( "don't \"carp\"" ) ==> "'don''t \"carp\"'" quote_nullable( "" ) ==> "''"

quote_ident( "foo" ) ==> "foo" quote_ident( "don't \"carp\"" ) ==> "\"don't \"\"carp\"\"\"" quote_ident( "" ) ==> "\"\""

quote_...(undef)

quote_literal( undef ) ==> undef

quote_nullable( undef ) ==> "NULL"

quote_ident( undef ) ==> "\"\"" (warn)

{en,de}code_bytea encode_bytea( "foo" ) ==> "\\x666f6f" decode_bytea( "\\x666f6f" ) ==> "foo" decode_bytea( "\\146\\157\\157") ==> "foo"

encode_bytea( "\x{263a}" ) ==> "\\xe298ba" UTF8 decode_bytea( "\\xe298ba" ) ==> "\342\230\272" Not UTF8

encode_bytea( "" ) ==> "\\x" decode_bytea( "\\x" ) ==> "" decode_bytea( "" ) ==> ""

encode_bytea( undef ) ==> "\\x" (warn) decode_bytea( undef ) ==> "" (warn)

looks_like_number

looks_like_number( 1 ) ==> 1 looks_like_number( 0 ) ==> 1 looks_like_number( "+7.2e-9" ) ==> 1

looks_like_number( "foo" ) ==> 0 looks_like_number( "" ) ==> 0 looks_like_number( undef ) ==> undef

looks_like_number( " 4 " ) ==> 1

looks_like_number( "5plus" ) ==> 0 (but '5plus'+0=5)

encode_array_*

encode_array_literal( ["foo","bar"] ) ==> "{\"foo\", \"bar\"}"

encode_array_constructor( ["foo","bar"] ) ==> "ARRAY['foo', 'bar']"

encode_array_literal( [1,[2,[undef]]] ) ==> "{\"1\", {\"2\", {NULL}}}"

encode_array_constructor( [1,[2,[undef]]] ) ==> "ARRAY['1', ARRAY['2', ARRAY[NULL]]]"

encode_array_*

encode_array_literal( "foo" ) ==> "foo"encode_array_constructor( "foo" ) ==> "'foo'"

encode_array_literal( undef ) ==> undefencode_array_constructor( undef ) ==> "NULL"

Trusted require/use

• require/use work for already loaded modules

use strict; # old way: BEGIN { strict->import(); }

• extra pre-loaded modules

use warnings;use Carp;use feature qw(say); # for perl 5.10 or lateruse utf8; # if server_encoding is utf8

CONTEXT: ...

• PL/Perl tracks the context of log messages- before:

WARNING: ...some warning from perl code...

- now:WARNING: ...some warning from perl code...CONTEXT: PL/Perl function "..."

- Thanks to Alexey Klyukin.

DO '...' LANGUAGE ...;

• Arbitrary chunks of code can be executed directly from psql, or client apps, via DO- Thanks to Petr Jelinek, Joshua Tolley, Hannu Valtonen

• No need to create and run a stored procedure each time:

DO $$spi_exec("... $_ ...") for 'a'..'z';

$$ language plperl;

Other Changes

• Using $a and $b in sort blocks now works!

• eval { ... } and eval "..." now work!

• END blocks are now run at end of session

- they can't (currently) access the database

• Warnings from perl are now WARNINGs- they used to be NOTICE

PL/Perl Changes

‣ USER‣ INTERNAL‣ DBA‣ NYTPROF‣ FUTURE

INTERNAL

• The Safe module is no longer used for plperl- Now faster, simpler, and more secure

• Validates return values are in server encoding- ERROR: invalid byte sequence for encoding

- Thanks to Andrew Dunstan

• Internal code refactoring and cleanup

PL/Perl Changes

‣ USER‣ INTERNAL‣ DBA‣ NYTPROF‣ FUTURE

New plperl.* Config

• Specify perl code to run during initialization:

plperl.on_init = '...perl code...'plperl.on_plperlu_init = '...perl code...'plperl.on_plperl_init = '...perl code...'

• Can only be set by superuser or postgres.conf

• Code can't access the database

Interpreter Lifecycle

1. Perl interpreter created

2. Options from PERL5OPT env var are processed

3. PL/Perl support bootstrap code is executed

4. plperl.on_init code runs (unrestricted)

Above steps may happen in postmaster process at startup, if plperl is loaded via shared_preload_libraries.

Otherwise they happen at first use.

No access to database.

Birth

6. Interpreter is specialised for plperl (if that’s used first)

• Modules loaded: strict, warnings, features, Carp

• Unsafe perl ops are restricted (require, open etc.)

• DynaLoader package is deleted

• plperl.on_plperl_init code runs (restricted)

7. Database access is enabled

8. Perl interpreter is made available for use

9. Executes whatever action called it into existence

Adolescence

• If plplerlu code is run later- then a new interpreter is created

- similarly if plperlu is run first and plperl run later

• If plplerl with a different security context is run- then a new interpreter is created for the ROLE

- That’s a recent security fix:http://wiki.postgresql.org/wiki/20101005securityrelease

Note impact on shared_preload_libraries in these cases

Siblings

• Finally, when the session ends:- Access to the database is disabled

- END blocks, if any, are run (if exiting cleanly)

Death

plperl.on_init

• Handy to set global perl configuationplperl.on_init='use lib qw(/myapp); use ...;'plperl.on_init='require "plperloninit.pl";'Effectively defines ‘approved’ modules for plperl

• SECURITY RISK!Only load modules you're happy for plperl code to use.Also check any other modules loaded as dependencies!Use Devel::TraceLoad to see what's actually loaded:PERL5OPT='-MDevel::Trace=summary' pg_ctl ...

PL/Perl Best Practice

• Include explicit use statements in functions

For plperlu that'll actually load if neededFor plperl it'll check that module is loaded- so you'll get an immediate clear failure if not- (e.g., on a replica with old postgres.conf file)

plperl.on_plperl_init

• Originally intended for things like- PGOPTIONS="-c plperl.on_plperl_init='...'"

- to enable debug or profiling for a session

• But...• Can only be set by superuser or postgres.conf- due to SECURITY DEFINER risk at the time- that’s now been patched (20101005, CVE-2010-3433)

- so this restriction may be removed in future

PL/Perl Changes

‣ USER‣ INTERNAL‣ DBA‣ NYTPROF‣FUTURE

Devel::NYTProfPerl Source Code Profiler

PostgreSQL::PLPerl::NYTProf

Enabling NYTProf• Via postgres.conf:

plperl.on_init='use PostgreSQL::PLPerl::NYTProf'

• Via environment variable:PERL5OPT='-MPostgreSQL::PLPerl::NYTProf' pg_ctl ...

• Is immediately active for all connections.• To enable on demand for one connection:

NYTPROF=start=no PERL5OPT=... pg_ctl ...DO 'DB::enable_profile' LANGUAGE plperl;

Reporting from NYTProf

• Writes per-backend data files:

$PGDATA/nytprof.out.$pid

• To generate a report:

nytprofhtml --file=$PGDATA/nytprof.out.4321 --open

~ Demo ~

Of plperl.on_init in postgresql.confAnd use of PostgreSQL::PLPerl::NYTProf

Screencast: http://timbunce.blip.tv/file/3691795/Video: http://www.fosslc.org/drupal/content/plperl-new-features-90

PL/Perl Changes

‣ USER‣ INTERNAL‣ DBA‣ NYTPROF‣ FUTURE

Future Possibilities

• Optimize trigger handling overheads

• Remove overheads of threaded perls

• $array_ref = decode_array_literal(‘{...}’)

• $hash_ref = decode_hstore_literal(‘x=>42’)

• Array params as array refs (with overloading)

• Rewrite encode_array_literal in C.

Questions?

Tim.Bunce@pobox.comhttp://blog.timbunce.org

@timbunce on twitter

http://xkcd.com/519/