Topic 8: DatabasesTopic 8: Databases
CSE2395/CSE3395Perl Programming
CSE2395/CSE3395Perl Programming
Llama3 chapter 16, pages 221-224
Camel3 pages 363-398
Programming the Perl DBI chapters 2-5, pages 7-135
perlfunc, perlmod, perlmodlib, Data::Dumper, perltie,
DB_File, DBI manpages
2Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University
In this topicIn this topic
Modules► use Module;► Perl standard library► CPAN► writing modules
Databases► persistent data► Data::Dumper► DBM
– databases in files► pack and unpack
– flat-file databases► DBI
– interfacing with SQL databases
Modules► use Module;► Perl standard library► CPAN► writing modules
Databases► persistent data► Data::Dumper► DBM
– databases in files► pack and unpack
– flat-file databases► DBI
– interfacing with SQL databases
3Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University
ModulesModules
Much of Perl’s functionality is contained in optional modules
A module is a file which contains re-usable Perl code for a particular task
Much of Perl’s functionality is contained in optional modules
A module is a file which contains re-usable Perl code for a particular task
Your Perl code
Module
Your code is importing; module is exporting
imported module
4Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University
ModulesModules
Modules can be► found in the Perl Standard Library
– supplied with Perl and already installed► downloaded from CPAN
– tar xzf Module.tar.gz– cd Module– perl Makefile.PL– make– make install
► written by you– using package keyword and saving file as Module.pm
Modules can be► found in the Perl Standard Library
– supplied with Perl and already installed► downloaded from CPAN
– tar xzf Module.tar.gz– cd Module– perl Makefile.PL– make– make install
► written by you– using package keyword and saving file as Module.pm
Camel3 pages 299-307, 831-915perlmod, perlmodlib, perlfunc manpages
5Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University
Using a moduleUsing a module
Import a module into your code with use keyword at top of your program
use Data::Dumper; use CGI ":all"; use Dog qw(bark bite); Read module documentation to see how to
access module’s functionality► perldoc Data::Dumper► perldoc CGI
Import a module into your code with use keyword at top of your program
use Data::Dumper; use CGI ":all"; use Dog qw(bark bite); Read module documentation to see how to
access module’s functionality► perldoc Data::Dumper► perldoc CGI
Llama3 pages 282-283Camel3 pages 299-300, 822-823perlfunc manpage
6Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University
Writing a moduleWriting a module
To create a module called Dog:► start file with package Dog;
– package keyword ensures that Dog’s variables and functions don’t interfere with anyone else’s
► use Exporter;► @ISA = qw(Exporter);► set @EXPORT_OK to a list of function names (bark, bite)
– these two lines allow Perl to put Dog’s functions into code that says use Dog ("bark", "bite");
► write the functions for Dog (bark, bite)► end file with a nonzero expression
– needed or else use Dog will fail► save file as Dog.pm in current directory
– or any directory named in special Perl @INC array
To create a module called Dog:► start file with package Dog;
– package keyword ensures that Dog’s variables and functions don’t interfere with anyone else’s
► use Exporter;► @ISA = qw(Exporter);► set @EXPORT_OK to a list of function names (bark, bite)
– these two lines allow Perl to put Dog’s functions into code that says use Dog ("bark", "bite");
► write the functions for Dog (bark, bite)► end file with a nonzero expression
– needed or else use Dog will fail► save file as Dog.pm in current directory
– or any directory named in special Perl @INC array
Camel3 pages 301-305perlmod, perlfunc manpages
7Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University
TimeoutTimeout
package DayOfWeek;use Exporter; @ISA = qw(Exporter);@EXPORT_OK = "weekday"; # Allow import of &weekday.
# weekday(y,m,d): Return weekday of date (0 = Sun).sub weekday{ my ($y, $m, $d) = @_; # Uses array references (topic 10). ( [[6,2,2,5,0,3,5,1,4,6,2,4], [6,2,3,6,1,4,6,2,5,0,3,5]] -> [ ((! ($y % 400) || ($y % 100)) && (! ($y % 4))) || 0 ][ $m-1 ] + $d + $y + int(($y-1)/4) - int(($y-1)/100) + int(($y-1)/400) ) % 7;}1; # Indicate successful definition of module.
package DayOfWeek;use Exporter; @ISA = qw(Exporter);@EXPORT_OK = "weekday"; # Allow import of &weekday.
# weekday(y,m,d): Return weekday of date (0 = Sun).sub weekday{ my ($y, $m, $d) = @_; # Uses array references (topic 10). ( [[6,2,2,5,0,3,5,1,4,6,2,4], [6,2,3,6,1,4,6,2,5,0,3,5]] -> [ ((! ($y % 400) || ($y % 100)) && (! ($y % 4))) || 0 ][ $m-1 ] + $d + $y + int(($y-1)/4) - int(($y-1)/100) + int(($y-1)/400) ) % 7;}1; # Indicate successful definition of module.
8Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University
TimeoutTimeout
# Import the weekday function from DayOfWeek.pmuse DayOfWeek qw(weekday);
# Names of days of week.@days = qw(Sunday Monday Tuesday Wednesday Thursday Friday Saturday);
# Check usage.die "Usage: $0 year month day\n" if @ARGV != 3;
$daynum = weekday(@ARGV[0..2]);
print "That day is $days[$daynum]\n";
# Import the weekday function from DayOfWeek.pmuse DayOfWeek qw(weekday);
# Names of days of week.@days = qw(Sunday Monday Tuesday Wednesday Thursday Friday Saturday);
# Check usage.die "Usage: $0 year month day\n" if @ARGV != 3;
$daynum = weekday(@ARGV[0..2]);
print "That day is $days[$daynum]\n";
9Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University
DatabasesDatabases
A database► contains organized data► contains relations between data► is persistent
A hash► contains organized data► has a key/value relationship► ceases to exist at end of scope
– never beyond end of program
A persistent hash would be great► but how?
A database► contains organized data► contains relations between data► is persistent
A hash► contains organized data► has a key/value relationship► ceases to exist at end of scope
– never beyond end of program
A persistent hash would be great► but how?
10Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University
Persistent dataPersistent data
To make a variable persistent, need a way to make it last between invocations of program
► by saving its state in a file
Write-back (“freeze/thaw”, serialization) strategy► dump the contents of variable to file at program exit (“freeze”)► initialize variable with dumped contents at program entry (“thaw”)► file is up-to-date only when program is not running► for example: Data::Dumper, Storable
Write-through (“tie”) strategy► update file every time the variable is changed► requires help from Perl to tie the variable to the file► file is always up-to-date► for example: DBM files
To make a variable persistent, need a way to make it last between invocations of program
► by saving its state in a file
Write-back (“freeze/thaw”, serialization) strategy► dump the contents of variable to file at program exit (“freeze”)► initialize variable with dumped contents at program entry (“thaw”)► file is up-to-date only when program is not running► for example: Data::Dumper, Storable
Write-through (“tie”) strategy► update file every time the variable is changed► requires help from Perl to tie the variable to the file► file is always up-to-date► for example: DBM files
11Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University
Data::DumperData::Dumper
Perl Standard Library module► use Data::Dumper;
Example of write-back (freeze/thaw) strategy Writes Perl code to file
► when later evaluated by Perl, data is reconstructed To dump (freeze) a variable %hash:
► open the persistent file for writing► Data::Dumper->Dump([\%hash], [qw(*hash)]);
– [...] is array reference, see topic 10.
To restore (thaw) from a file file:► do "file";► Perl evaluates the statements in file, reconstructing the
variable %hash
Perl Standard Library module► use Data::Dumper;
Example of write-back (freeze/thaw) strategy Writes Perl code to file
► when later evaluated by Perl, data is reconstructed To dump (freeze) a variable %hash:
► open the persistent file for writing► Data::Dumper->Dump([\%hash], [qw(*hash)]);
– [...] is array reference, see topic 10.
To restore (thaw) from a file file:► do "file";► Perl evaluates the statements in file, reconstructing the
variable %hash
Camel3 pages 702, 882Data::Dumper, perlfunc manpages
12Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University
TimeoutTimeout
# Freezing and thawing some variables.use Data::Dumper;
# Thaw saved values, if they exist.if (-f "storage") { do "storage"; # Restore previous values.} else { $important = 0; %valuable = (); # First run.}
# ... do stuff with $important and %valuable ...
# Freeze values for next time before program exit.open STORAGE, ">storage";print STORAGE Data::Dumper->Dump( [$important, \%valuable], [qw(important *valuable)]);
# Freezing and thawing some variables.use Data::Dumper;
# Thaw saved values, if they exist.if (-f "storage") { do "storage"; # Restore previous values.} else { $important = 0; %valuable = (); # First run.}
# ... do stuff with $important and %valuable ...
# Freeze values for next time before program exit.open STORAGE, ">storage";print STORAGE Data::Dumper->Dump( [$important, \%valuable], [qw(important *valuable)]);
13Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University
DBM filesDBM files
DBM files are a simple database format common to Unix► also available on other platforms
Data stored in a binary file► usually with a sophisticated data structure like a B-
tree
Roughly equivalent to a hash► can look up value given key► keys are strings, values are strings► may be arbitrary limits on data size
DBM files are a simple database format common to Unix► also available on other platforms
Data stored in a binary file► usually with a sophisticated data structure like a B-
tree
Roughly equivalent to a hash► can look up value given key► keys are strings, values are strings► may be arbitrary limits on data size
Llama3 pages 221-222Camel3 pages 696-697, 883-884, 985perlmod manpage
14Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University
DBM filesDBM files
Perl provides a simple interface to DBM files► using dbmopen or tie functions► connects a hash to the DBM file► accessing the hash transparently accesses the file
instead– all hash operations supported
► example of write-through (tie) strategy To tie a hash %hash to a file file:
► either dbmopen %hash, "file", 0600– mode (0600) only used if file needs to be created
► or tie %hash, "DB_file", "file"
Perl provides a simple interface to DBM files► using dbmopen or tie functions► connects a hash to the DBM file► accessing the hash transparently accesses the file
instead– all hash operations supported
► example of write-through (tie) strategy To tie a hash %hash to a file file:
► either dbmopen %hash, "file", 0600– mode (0600) only used if file needs to be created
► or tie %hash, "DB_file", "file"
Llama3 pages 222-224Camel3 pages 363-398, 697-698perltie, perlfunc manpages
15Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University
TimeoutTimeout
# Use DBM to track mail aliases.
# Use Berkeley-DB format for DBM file.# This is the most flexible one.use DB_File;dbmopen %alias, "aliases", 0600 or die "Can't tie to DBM file: $!\n";
# Define functions that modify file# through %alias hash.sub get_alias { return $alias{$_[0]}; }sub set_alias { $alias{$_[0]} = $_[1]; }sub delete_alias { delete $alias{$_[0]}; }sub exists_alias { return exists $alias{$_[0]}; }
# Use DBM to track mail aliases.
# Use Berkeley-DB format for DBM file.# This is the most flexible one.use DB_File;dbmopen %alias, "aliases", 0600 or die "Can't tie to DBM file: $!\n";
# Define functions that modify file# through %alias hash.sub get_alias { return $alias{$_[0]}; }sub set_alias { $alias{$_[0]} = $_[1]; }sub delete_alias { delete $alias{$_[0]}; }sub exists_alias { return exists $alias{$_[0]}; }
16Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University
pack and unpackpack and unpack
Files store only strings If data other than strings are to be stored, need to
encode into strings before storing► and decode from strings when fetching
To encode data into a string, use pack► $string = pack "a10 l", $word, $int;
To decode data from a string, use unpack► ($word, $int) = unpack "a10 l", $string;
Format string ("a10 l") specifies how to pack or unpack
► dozens of different specifiers
Files store only strings If data other than strings are to be stored, need to
encode into strings before storing► and decode from strings when fetching
To encode data into a string, use pack► $string = pack "a10 l", $word, $int;
To decode data from a string, use unpack► ($word, $int) = unpack "a10 l", $string;
Format string ("a10 l") specifies how to pack or unpack
► dozens of different specifiers
Llama3 page 224Camel3 pages 757-762, 819-821perlfunc manpage
17Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University
Random-access filesRandom-access files
Can store fixed-length records in a file► e.g., 30 bytes per record
– 20 for name, 8 for ID number, 1 for age, 1 for mark► bytes 0 to 29 for first record► bytes 30 to 59 for second record► etc.
Can store fixed-length records in a file► e.g., 30 bytes per record
– 20 for name, 8 for ID number, 1 for age, 1 for mark► bytes 0 to 29 for first record► bytes 30 to 59 for second record► etc.
record 1 record 2 record 3
18Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University
Random-access filesRandom-access files
Open file for both reading and writing To modify or create record
► create record with pack► seek to position in file with seek► write packed record to file with print
To read record► seek to position in file with seek► read appropriate number of bytes with read► extract fields from string with unpack
Open file for both reading and writing To modify or create record
► create record with pack► seek to position in file with seek► write packed record to file with print
To read record► seek to position in file with seek► read appropriate number of bytes with read► extract fields from string with unpack
Llama3 pages 225-227Camel3 pages 769, 780perlfunc manpage
19Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University
TimeoutTimeout
open FILE, "+< database"; # Open for read/write.
# write_record(recno, name, id, age, mark)sub write_record { my $recno = shift; my $string = pack "a20 a8 c c", @_; seek FILE, ($recno-1)*30, 0; print FILE $string;}
# (name, id, age, mark) = read_record(recno)sub read_record { my $recno = shift; seek FILE, ($recno-1)*30, 0; read FILE, $string, 30; return unpack "a20 a8 c c", $string;}
open FILE, "+< database"; # Open for read/write.
# write_record(recno, name, id, age, mark)sub write_record { my $recno = shift; my $string = pack "a20 a8 c c", @_; seek FILE, ($recno-1)*30, 0; print FILE $string;}
# (name, id, age, mark) = read_record(recno)sub read_record { my $recno = shift; seek FILE, ($recno-1)*30, 0; read FILE, $string, 30; return unpack "a20 a8 c c", $string;}
20Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University
Client/server databasesClient/server databases
Sometimes necessary to interact with an existing database► database is kept on a server
– e.g., Oracle, MySQL, Microsoft Access (ODBC), etc.– server handles mundane aspects like storage, access control,
file/record locking, etc.► your program is client
– makes requests to server on database’s behalf
Need to interact with server using its interface► usually requires making a network connection
Commonly done with SQL► Structured Query Language, a standard way of phrasing queries
and database transformations► SQL is not examinable in this subject!
But► SQL dialects vary slightly► network protocols and interfaces vary
Sometimes necessary to interact with an existing database► database is kept on a server
– e.g., Oracle, MySQL, Microsoft Access (ODBC), etc.– server handles mundane aspects like storage, access control,
file/record locking, etc.► your program is client
– makes requests to server on database’s behalf
Need to interact with server using its interface► usually requires making a network connection
Commonly done with SQL► Structured Query Language, a standard way of phrasing queries
and database transformations► SQL is not examinable in this subject!
But► SQL dialects vary slightly► network protocols and interfaces vary
21Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University
DBIDBI
Perl provides a unified interface for all database servers► called Database Interface, or DBI► use DBI;
Client uses DBI to access database► DBI speaks to database through a system-specific
database driver (DBD)► DBD translates client’s SQL request for specific
server dialects► DBD massages server responses into standard form
for client
Perl provides a unified interface for all database servers► called Database Interface, or DBI► use DBI;
Client uses DBI to access database► DBI speaks to database through a system-specific
database driver (DBD)► DBD translates client’s SQL request for specific
server dialects► DBD massages server responses into standard form
for client
22Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University
DBIDBI
Your Perl program
DBI (interface)
DBD::Oracle (driver)
DBD::CSV (driver)
DBI system
Oracle database
Oracle database
CSV database
Always one instance of DBI
One driver per type of database
23Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University
DBI handlesDBI handles
DBI system uses handles to manage connections to database system► database handle
– one per database– used to perform instantaneous actions and to initiate longer
statements► statement handle
– one per SQL statement– usually used to perform lengthy SELECT queries
DBI system uses handles to manage connections to database system► database handle
– one per database– used to perform instantaneous actions and to initiate longer
statements► statement handle
– one per SQL statement– usually used to perform lengthy SELECT queries
24Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University
Connecting to a databaseConnecting to a database
Need to know database’s data source name Need to know database’s data source name
DBI:mSQL:frotz.org:base:4910
Always DBI
Database driver
Database identifier; format differs for each driver (here: hostname,
database name, port number
25Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University
TimeoutTimeout
use DBI;
# Connect to the database.# OO: class->method() notation is how to call# a class method (in this case, a constructor).my $db_handle = DBI->connect( "DBI:Ingres:data", $username, $passwd, { PrintError => 1 } # Hash ref: topic 10 ) or die "Can't connect: $DBI::errstr\n";
# ... do stuff ...
# Now disconnect.# OO: $object->method() notation is how to call# an object method (here, a destructor).$db_handle->disconnect() or warn "Disconnection error: $DBI::errstr\n";
use DBI;
# Connect to the database.# OO: class->method() notation is how to call# a class method (in this case, a constructor).my $db_handle = DBI->connect( "DBI:Ingres:data", $username, $passwd, { PrintError => 1 } # Hash ref: topic 10 ) or die "Can't connect: $DBI::errstr\n";
# ... do stuff ...
# Now disconnect.# OO: $object->method() notation is how to call# an object method (here, a destructor).$db_handle->disconnect() or warn "Disconnection error: $DBI::errstr\n";
26Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University
Executing DBI statementsExecuting DBI statements
Actions on database are performed through database handle
Simple (atomic) statements can be performed using do method► $db_handle->do("DELETE FROM ...");► return value indicates success or failure
Complex statements, which may produce several results in sequence, require a statement handle► $st_handle = $db_handle->prepare("SELECT...");► statement handle used to iterate over results
– @fields = $st_handle->fetchrow_array()
Actions on database are performed through database handle
Simple (atomic) statements can be performed using do method► $db_handle->do("DELETE FROM ...");► return value indicates success or failure
Complex statements, which may produce several results in sequence, require a statement handle► $st_handle = $db_handle->prepare("SELECT...");► statement handle used to iterate over results
– @fields = $st_handle->fetchrow_array()
27Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University
TimeoutTimeout
# Insert a name into a phone database# usage: thisprogram name number
use DBI;
# ... connect to the database ...
# quote() method correctly escapes strings# with metacharacters in them.$db_handle->do(" INSERT INTO phonedir (name, phone_number) VALUES ( " . $db_handle->quote($ARGV[0]) . ", " . $db_handle->quote($ARGV[1]) . ")") or warn "Insert error: $DBI::errstr\n";
# Insert a name into a phone database# usage: thisprogram name number
use DBI;
# ... connect to the database ...
# quote() method correctly escapes strings# with metacharacters in them.$db_handle->do(" INSERT INTO phonedir (name, phone_number) VALUES ( " . $db_handle->quote($ARGV[0]) . ", " . $db_handle->quote($ARGV[1]) . ")") or warn "Insert error: $DBI::errstr\n";
28Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University
TimeoutTimeout
# Extract entries from the phone database# usage: thisprogram name
# ... connect to the database ...
$st_handle = $db_handle->prepare(" SELECT name, phone_number FROM phonedir WHERE name LIKE " . $db_handle->quote($ARGV[0]));$st_handle->execute();
# Iterate over the data returned by the query.while (($name, $num) = $st_handle->fetchrow_array()){ print "$name: $num\n";}
# Extract entries from the phone database# usage: thisprogram name
# ... connect to the database ...
$st_handle = $db_handle->prepare(" SELECT name, phone_number FROM phonedir WHERE name LIKE " . $db_handle->quote($ARGV[0]));$st_handle->execute();
# Iterate over the data returned by the query.while (($name, $num) = $st_handle->fetchrow_array()){ print "$name: $num\n";}
29Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University
Covered in this topicCovered in this topic
Modules► use Module;► Perl standard library► CPAN► writing modules
Databases► persistent data► Data::Dumper► DBM
– databases in files► pack and unpack
– flat-file databases► DBI
– interfacing with SQL databases
Modules► use Module;► Perl standard library► CPAN► writing modules
Databases► persistent data► Data::Dumper► DBM
– databases in files► pack and unpack
– flat-file databases► DBI
– interfacing with SQL databases
30Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University
Going furtherGoing further
Tying► adding new behaviour to built-in-types► perldoc -f tie
Transactions► protecting databases from corruption► supported by some DBD modules
Multi-level DBM files► storing nested data structures in a DBM file► MLDBM module
ODBC► Microsoft’s database standard► DBI::ODBC and Win32::ODBC modules
Tying► adding new behaviour to built-in-types► perldoc -f tie
Transactions► protecting databases from corruption► supported by some DBD modules
Multi-level DBM files► storing nested data structures in a DBM file► MLDBM module
ODBC► Microsoft’s database standard► DBI::ODBC and Win32::ODBC modules
31Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University
Next topicNext topic
The World Wide Web Writing a Perl web client
► LWP module
Dynamic web pages► Common Gateway Interface (CGI)
The World Wide Web Writing a Perl web client
► LWP module
Dynamic web pages► Common Gateway Interface (CGI)
LWP, CGI manpages