perl -- aa dbadba and ddevelopers best evelopers best ... · perl - a dba and developers best...

Perl - A DBA and Developers best (forgotten) friend

1 Session #376

PP ERL ERL -- AA DBADBA AND AND DD EVELOPERS BEST EVELOPERS BEST (( FORGOTTENFORGOTTEN )) FRIENDFRIEND AA BEGINNERS GUIDE TO BEGINNERS GUIDE TO PP ERLERL

Arjen Visser, Dbvisit Software Limited

INTRODUCTION This white paper is focused on the programming language Perl. Its intent is to reintroduce Perl to the Oracle community, especially the DBAs and Developers that use Oracle database technology on an everyday basis.

For many DBAs and Developers Perl needs no introduction. However there is also a large group that have heard about Perl, and would like to use Perl, but do now know how to begin. This paper guides Oracle DBAs and Developers on how to get started.

This paper introduces the basics of Perl, shows what is important, identifies some of the Perl gotchas and traps, and also demonstrates how to use Perl with the Oracle database. To conclude there is a programing project detailed to get you get started on actually using Perl.

The title of this white paper is “A DBA and Developers best (forgotten) friend”. The term forgotten is used, as Perl is no longer in the limelight as much as other languages such as Java, Python and Ruby. That is unfortunate and underserved as Perl is actually a truly great language. It has a huge community of users and is extremely useful in our daily working lives.

BRIEF HISTORY OF PERL Perl is one of the most portable languages around. Larry Wall created it in 1987 while working at Unisys. Larry Wall was asked to write a configuration management and control system with cross-reference reports between the 6 servers on the East and West coast of the US. The awk utility of the day could not handle the opening and closing of multiple files so Larry set out by creating his own language to enable this. He called it Pearl first, but found he was too lazy to type 5 letter words and to make it into a 4-letter word, he changed it to Perl. Some people say Perl stands for Practical Extraction Reporting Language.

Perl 5 was rewritten and released on October 17, 1994 and included:

• objects

• references

• modules

Perl is still in active development. On May 14, 2011, Perl 5.14 was released followed on September 26, 2011, by the latest version, 5.14.2.

WHAT MAKES PERL SO GREAT ADVANTAGES

• Interpreted language (not compiled). Perl is an interpreted or scripting language, so you don't have to compile it like you do Java, C, or C++. For fast development work, that is great because you can execute your program immediately, you do not have to wait until it is compiled. Perl is similar to Python and Ruby which are also interpreted languages.

• Concise language allowing you to write programs quickly.

• Allows handling of complex data structures.


2 Session #376

• You can get under the "hood" of Perl (like v$tables). This is something that Oracle DBAs and Developers will appreciate and is very similar to what is available in Oracle. Oracle keeps most of its internal structures and information in the database itself so that you can view what Oracle is doing internally. This is similar in Perl. Perl keeps its internal variables and data structures in Perl variables so that these can be accessed for greater transparency and instrumentation in your code.

• Very strong pattern matching with regular expressions. This is how Perl got started and is one of its strengths. It is quite often used in Data Warehouse projects to manipulate and cleanse data.

• Easy to get started.

• Inter Process Communication (IPC) using Sockets and pipes for building web servers, file transportation mechanisms, peer checking etc.

• Tied in close to OS to make it great for automating specific OS tasks such as log reading, log rotation, file checking, backups etc.

• CPAN – the true power of Perl. CPAN stands for Comprehensive Perl Archive Network with 23,500 modules by more than 7,000 authors that are free to download and use in your code. It is the “Apps” store for Perl.

DISADVANTAGES

• Can be cryptic to read especially pattern matching. Perl is not the easiest of languages to read and some Perl programmers make it a sport to be as cryptic as possible. But a good programmer knows how to write code that is easy to read and good to maintain.

• No standard named parameters with calling functions. Perl lacks the ability to call functions with named parameters, but there is a way around this that will be highlighted later in the white paper.

• Hard to master, but is true for most languages.

• Not as fast as natively compiled programs such as C. Perl will never be as fast as a fully compiled language such as C. However in Perl, you can include C code for specific functions that need fast performance such as IO related functions. This C code is natively complied for each platform and included as part of the Perl package or module.

• GUI applications written in Perl look dated (but you should not be writing GUI applications, they should all be web based).

• Must have Perl installed to run your program. Being an interpreted language it means that if you want to deploy your program to another environment, you have to have Perl installed on the target environment with all the related modules and libraries. However with Perl there are tools available that allow Perl programs to be “compiled” or “packaged” as standalone and self-contained executables that contain your program as well as the Perl interpreter and related modules and libraries. This is an all in one executable making it very easy to deploy your Perl code to a new target environment.

WHAT TO USE PERL FOR Perl is great for the following applications:

• Any shell or command line scripting or programs.

• Batch type programming / Backend processing.

• Data loading, manipulation (Data warehousing).

• Installation routines.

• Heavy OS interfacing.

• Backend for Web processing and validation.

• Building a web front end for a batch process application.


3 Session #376

MAYBE NOT USE PERL (EXCLUSIVELY) FOR Perl is perhaps not best suited to the following type of applications

• New Facebook/twitter web app, probably better languages such as PHP to do this. But Perl would be great for the backend.

• Big corporate systems such as a Billing or Financial systems

• Windows GUI applications (like Thunderbird)

• Building an Operating System

IS PERL STILL CURRENT? There is a perception that Perl is not so current as Python, Ruby, Java etc. This is only a perception because many companies continue to use Perl to write software.

The following are examples of why Perl is still current:

• Oracle uses it in the latest release of their database. Some of their tools are written in Perl. An example is the command line tool (asmcmd) to interface to the Oracle clustering file system ASM is written in Perl.

• Perl executable and libraries are standard installed with Oracle (also on Windows)

• VMware uses it for their installation routines

• Dbvisit Standby and Replicate uses Perl

• Many websites use it (Amazon.com, bbc.co.uk, Zappos.com)

Perl is available on all flavours of Linux, most Unix, Windows, Macs and most hardware like Intel, Itanium, PPC, SPARC and RISC.

CPAN CPAN stands for Comprehensive Perl Archive Network and this one of the reasons why Perl is such a great language. It is the Apps store for Perl and is a collection of “plug and play” modules and libraries that are free to use in your programs. Other languages have similar libraries but none are as comprehensive as CPAN.

To search for a particular module go to http://search.cpan.org/


4 Session #376

Program libraries are only useful if they are kept up to date. This is certainly true for CPAN. It is very much up to date. Everyday there are about roughly 50 updates. This can be verified by going to http://search.cpan.org/recent

HOW TO USE CPAN ON UNIX/LINUX WITH AN INTERNET CONNECTION Type cpan $ cpan

As an example we are going to install a CPAN module that will tell us what the Google Pagerank is of a particular website.

Use the search functionality to search for the right module. cpan> i /google::pagerank/ Module POE::Component::IRC::Plugin::Google::PageRank Module POE::Component::WWW::Google::PageRank Module = WWW::Google::PageRank

The module that we want is the last one. Type it exactly as written as it is case sensitive: cpan> install WWW::Google::PageRank

And now the module is ready to be used in your program.

ON UNIX/LINUX WITHOUT AN INTERNET CONNECTION 1. You manually download the module from CPAN and then install it your self.

2. Download the zipped tar file from CPAN (WWW-Google-PageRank-0.17.tar.gz)

3. Unzip and untar the file which will create a new temp directory.

4. cd into temp file directory: perl Makefile.PL make make test make install

5. Delete temp file directory.


5 Session #376

ON WINDOWS, USE PPM On Windows the ActiveState Perl Package Manager can be used. Start a command prompt and type ppm:

These modules are precompiled because compiling on Windows is not as straightforward as on Unix and Linux. The Perl Package Manager is managed by ActiveState: http://www.activestate.com/

If you do want to compile your own modules, or the module that you required is not present in ppm, you can use StrawberryPerl which comes with its own compilers for Windows http://strawberryperl.com/

FIRST LOOK AT A PERL PROGRAM Here is our first Perl program, which shows the power of CPAN. What we would like is to have a Perl program that can give us the Google pagerank of a particular website. The Google pagerank is a number between 0 and 10 which indicates how relevant or popular the website is. The best score is 10.

We are assuming that the WWW::Google::PageRank module has been installed as above. We will not go into the Perl syntax at this stage:

# First complete Perl program! use WWW::Google::PageRank; ##### # Set website and get pagerank ##### my $website = 'http://www.oracle.com'; my $pr = WWW::Google::PageRank->new(); my $rank = $pr->get($website); ##### # Print out pagerank ##### print "PageRank: " . $rank . " for website: $website";

Save this program in a file called pagerank.pl and then type: perl pagerank.pl

The output of the program is: PageRank: 8 for website: http://www.oracle.com


6 Session #376

Just 5 lines of code produced this result. The CPAN module does all the hard work. We do not need to know anything about web services, http headers, http protocols, GET request etc. We leave all that to the CPAN module. Very simple.

We make use of the module by the use command. Then we initiate the module with new, give it a URL and we print the output.

One of Perl’s motto’s is that there is always more than one way to do it. Here is the same program, but written in 3 lines: use WWW::Google::PageRank; my $pr = WWW::Google::PageRank->new; print scalar($pr->get('http://www.oracle.com/')), "\n";

PERL LANGUAGE OVERVIEW IN 10 STEPS STEP 1 There are 3 Variable types in Perl:

• Scalar: $ (any single value numeric or character)

• Array: @ (ordered list – has an index)

• Hash: % (unordered set of scalars – key value pairs)

Example of Scalar $answer = 42; $colour="red"; $cwd = `pwd`; # The output of running the pwd command in Linux/Unix

Notes:

• In PL/SQL the scalar equivalent would be varchar2, numeric, integer, number etc.

Example of Array @versions = ("7.3","8i","9i","10g","11g"); $versions[0] = "7.3"; $versions[1] = "8i"; @ordered_versions = sort @versions;

Notes:

• In Perl an array starts at position 0.

• There is an automatic scalar associated with each position in the array, eg: $versions[0]

Example of Hash %summer = ( "dec" => "December", "jan" => "January", "feb" => "February");

STEP 2 Declaring - local variables

The variables are local to your functions and do not exist outside of the scope in which they are declared. my $i; my @array_versions; my $i = 2;

Declaring - global variables

The variables are global to your program. our $Logfile; our @Datafiles; our %Ora_init;

Increment


7 Session #376

Increment is done in C style. $i++; # $i = $i + 1; $i--; # $i = $i – 1;

STEP 3 All lines end with ;

Comparisons

== Numeric comparison. eg test if ($pi == 3.14159)

>= Greater than

<= Smaller than

!= Not equal to

eq String comparison (equal) ($colour eq "red")

ne String comparison (not equal)

Assignment

= Assignment ($a = $b) Remember this is not comparing

STEP 4 Control structures:

while ( cond ) { ... } for ( init-expr ; cond-expr ; incr-expr ) { ... } foreach var ( list ) { ... } if ( cond ) { ... } if ( cond ) { ... } else { ... } if ( cond ) { ... } elsif ( cond ) { ... } else { ... } if ($version eq "8i" ) { print "Your Oracle version is old, but still good!\n" } if ($age == 40 ) { print "What a great age!\n" }

(\n is newline)

STEP 5 Internal build in variables

$_

In Perl the $_ is a very powerful default internal variable that Perl uses when you have not explicitly set a variable. For example when reading through an array and printing out each element in an array, this can be done with $_ or with an explicit variable:

With $_ foreach (@colours) { print "colour is: $_\n"; }

No explicit variable is set to store each element of the array as we are reading through, so $_ is used.

With explicit variable foreach $colour (@colours) print "colour is: $colour\n";


8 Session #376

}

An explicit scalar variable ($colour) is used to store each element of the array as we are processing through the array.

More internal build in variables

$@ Output of eval command

$? Return code of child program

$! OS error (example open file, or file delete error)

$^O OS name (Solaris, Linux, Windows etc)

@ARGV Array containing the arguments to the program

my ($db, $oracle_home) = @ARGV;

STEP 6 Pattern matching

Based on Unix/Linux and Awk style regular expression

=~ Is the main pattern matching (binding) operator.

Examples: $answer = "Y"; # Can be 'y', 'Y’, 'Yes', 'YES’, even 'Yellow'

if ($answer =~ /^y/i) { print "Yes\n" }

Metacharacters: if ($file =~ /\s+/) { print "File contains spaces\n" }

Substitution: $switch =~ s/on/off/; # Substitute on to off.

Advanced: $ora_error =~ /ORA-(01345|01110).+?['"](.+?)['"]/gms

$ora_data_file{$1} = $2; # $2 will contain the actual Oracle error message

STEP 7 Quotation marks – single and double quotes

" (double quotes) do variable interpolation (variable substitution)

' (single quotes) suppress variable interpolation (no variable substitution) $date = "11 October 2011";

$today = "Double quotes: Today is $date\n";

print $today;

Double quotes: Today is 11 October 2011

$today = 'Single quotes: Today is $date\n';

print $today;

Single quotes: Today is $date\n

Escape character \ $today = "\'Today is $date\'\n";

print $today;

'Today is 11 October 2011'

STEP 8 Reading files


9 Session #376

my $file = 'c:\temp\logfile.txt';

open(FILE, "<", $file) or die "Cannot open $file.\n$!\n";

while (<FILE>) {

chomp; # Gets rid of white spaces and line breaks

# do stuff on each line. $_ contains each line

print "$_\n";

}

close (FILE);

Note: The $_ internal variable is used to read through the file. Each line in the file is automatically assigned to $_.

Writing files open(FILE, ">", $file) # Create new file and write to it

open(FILE, ">>", $file) # Append to existing file or create it

FILE is filehandle and can be any name. Standard is to use uppercase names.

STEP 9 Functions / subroutines

sub set_oracle_home {

my $db = shift;

my $oracle_home = shift;

#####

# Function logic starts here

#####

print "db = $db\n";

print "oracle_home = $oracle_home\n";

}

Calling the function: set_oracle_home ("PROD1", '/oracle/product/11g/');

What if we want to call: set_oracle_home ('/oracle/product/11g/');

Just calling the function with the second parameter will not work because the function expects both parameters. Not having named parameters is one of the disadvantages of Perl, however because of the power of having access to the internal variables of Perl, it is possible to create our own named parameter calling which is detailed in step 10.

STEP 10 Functions / subroutines (parameter calling)

# Declare the function with parameter calling

sub set_oracle_home {

my %params = @_;

my $db = $params{db};

my $oracle_home = $params{oracle_home};

#####

# Function logic starts here

#####

print "db = $db\n";


10 Session #376

print "oracle_home = $oracle_home\n";

}

Call the function with parameter calling. set_oracle_home ( db => "PROD1", oracle_home => '/oracle/product/11g/');

Reverse the parameters still gives the same result. set_oracle_home (oracle_home => '/oracle/product/11g/', db => "PROD1");

Perl overview completed in 10 steps!

GOTCHAS IN PERL Every programming language has some gotchas. So has Perl.

GOTCHA - 1 if (!$sequence) { print "sequence is empty($sequence)\n" }

This applies when $sequence is empty AND $sequence == 0.

Better: if (!defined($sequence)) { print "sequence is empty($sequence)\n" }

Or if ($sequence eq "") { print "sequence is empty($sequence)\n" }

GOTCHA - 2 We define an array

@sequence = (2,3,4,5);

Now with two almost identical strings we get something from the array:

i) $first_one = @sequence;

ii) ($first_one) = @sequence;

Now we print the result print "First one: $first_one\n";

The two statements produce very different results. Be aware of this.

i) "First one: 4" <- Prints out the number of elements in the array!

ii) "First one: 2" <- Display the first element in the array.

MAKING PERL PORTABLE Perl is already a very portable language but there are still a few issues that need to be addressed when designing a program for seamless portability. One of these is file path components.

There are modules available on CPAN that help to addresses these. Even though you may not have the need for portability, it is a good idea to get into the habit of writing for portability because you never know when you have the requirement to use your Perl programs on multiple platforms.

File path components:

/ on Linux and Unix

\ on Windows

: Mac


11 Session #376

Use File::Spec CPAN module to address this and use catfile function:

Eg on Linux:

Example: # curdir = /home/users

$new_file = catfile( curdir(), "temp", "dbvisit.trc");

Result: $new_file = /home/users/temp/dbvisit.trc

Eg on Windows:

Example: # curdir = C:\Documents and Settings\All Users

$new_file = catfile( curdir(), "temp", "dbvisit.trc");

Result: $new_file = C:\Documents and Settings\All Users\temp\dbvisit.trc

PERL AND ORACLE: - SHELL How to connect to Oracle through a Perl script? For those familiar with shell scripting, the normal method is to connect directly to sqlplus using IO redirect and inline data. A typical shell program connecting to Oracle would look like:

sqlplus -s <<- EOF > /usr/tmp/sqlplus_tmp.log \/ as sysdba clear columns set linesize 10000 set pause off set verify off set trimspool on set pages 0 set feedback off select member from v\$logfile; exit EOF echo "Output =====>" cat /usr/tmp/sqlplus_tmp.log

Output: Output =====>

/oracle/oradata/dbvisitp/redo03.log



PERL AND ORACLE: - PERL In Perl we can do a similar connection to Oracle using sqlplus. However we first create our sqlplus command file and then we call sqlplus and run the command file.

#####

# Declaration

#####

my $connect = qq("\/ as sysdba");


12 Session #376

my $sql_extra1 = "clear columns

set linesize 10000

set pause off

set verify off

set trimspool on

set pages 0

set feedback off\n";

#####

# Create the SQL command file

#####

open (SQL,">","/usr/tmp/sqlplus_tmp.sql") or die "cannot open file\n";

print SQL $sql_extra1;

print SQL "select member from v\$logfile;\n";

print SQL "exit";

close (SQL);

#####

# Run the command and capture the output. qx() is equivalent to backticks: ``

#####

@sqlplus_output = qx(sqlplus -s $connect \@/usr/tmp/sqlplus_tmp.sql);

print "Output =====>\n";

foreach (@sqlplus_output){

chomp;

print "$_\n";

}

Output: Output =====>




Notes:

We are connecting to Oracle as sysdba. No listener is required as this is a direct connection to the database.

PERL AND ORACLE: - DBD::ORACLE It is also possible to use a CPAN module to make the connection to an Oracle database. In this case sqlplus is not required.

The DBD::Oracle module is very powerful module and is the recommended way to connect to an Oracle database. However, if you have many different systems and Operating Systems, then DBD::Oracle will have to be installed on each system. The installation of DBD::Oracle and its prerequisite DBI can be challenging on some systems.


13 Session #376

An example of using DBD::Oracle. use DBI;

$dbh = DBI->connect("dbi:Oracle:$dbname", $user, $passwd);

my $SEL = "select member from v\$logfile;";

my $sth = $db->prepare($SEL);

$sth->execute();

@data = $sth->fetchrow_array());

The advantage with this method is that you have the result set already in a Perl variable (@data). With previous sqlplus method you always have to parse the resulting set as the output is unformatted text.

POPULAR CPAN MODULES MIME::Lite Send email on all platforms.

Log::Log4perl Flexible logging for debugging and log files.

Number::Format Format numbers.

File::Basename Parse file paths into directory, filename and suffix.

File::Temp Automatically find the temp system dir.

Sys::Hostname Try every conceivable way to get hostname.

File::Spec Portably perform operations on file names.

Template::Toolkit HTML template Processing System.

DBD::Oracle Oracle database driver for the DBI module.

MooseX::Declare The postmodern object system for Perl 5. There is a startup and memory overhead.

BEST WAY TO LEARN PERL Best Perl book: Programming Perl

By Larry Wall, Tom Christiansen and Job Orwant


14 Session #376

The best way to learn Perl is so set a small programming project. Examples of projects can be:

• Check alert log for errors and email them

• Check rman backups and email if errors

• Show free space in Oracle tablespace and file systems (see example below)

PERL PROJECT A nice Perl project is to show the free space in an Oracle tablespace and file system for a platform independent system.

The Perl script is called FreeSpace.pl and takes two arguments:

1. Oracle SID

2. Oracle Home

Example of running the script against an XE database on Windows: perl FreeSpace.pl XE C:\oracle\xe\app\oracle\product\10.2.0\server ==>Database: XE Tablespace Used Mb Free Mb Total Mb Pct Free ------------- ----------- ----------- ----------- ---------- SYSTEM 447 3 450 1 SYSAUX 430 10 440 2 USERS 7 93 100 93 UNDO 5 205 210 98 OS : MSWin32 Hostname: laptop03 Filesystem C:\ Total Mb: 139,746.99 Used Mb : 117,393.2 Free Mb : 22,353.8 Pct Free: 84 Filesystem D:\ Total Mb: 10,240 Used Mb : 3,890.92 Free Mb : 6,349.08 Pct Free: 38

The code for this can be downloaded from http://www.dbvisit.com/content/freeTechGuides/FreeSpace

Add the following functionality to complete the project:

1. Add total Database size.


15 Session #376

2. Supply threshold in % for tablespace and filesystem and send emails if thresholds have exceeded.

3. Only require the Oracle SID for Linux/Unix. Work out the ORACLE_HOME from /etc/oratab or /var/opt/oracle/oratab

PERL LINKS

• www.perl.org

• search.cpan.org

• perl6.org – The future for Perl

• perlmonks.org

• strawberryperl.com – Perl for Windows

• activestate.com – Perl for Windows

• perldoc.perl.org/perlintro.html

• perl2exe.com – Convert scripts to standalone exe files

• perl-compiler.org – Compile Perl to C code (experimental)

• www.perl.org/books/beginning-perl/

• en.wikipedia.org/wiki/Perl_language_structure