cummingsdceluna2012

From Voyager to your website: Using Linux Shell scripts and Oracle SQL*Plus to generate web pages.

ExLibris ELUNA 2012 Salt Lake City, UT 1 WRLC & George Washington University

Michael CummingsThe George Washington University

James O. AustinWashington Research Library Consortium

From Voyager to your websiteUsing Linux Shell scripts and Oracle SQL*Plus to generate web pages

ELUNA 2012 Salt Lake City, UT



• About WRLC and GWU

• WRLC services / reports

• ExLibris MS Access reports

• Oracle SQL*Plus

• GWU New Books List

• Steps to creating the List• Setup at Your Library

Install SQL*Plus on Linux

Get the newbooks scripts

Code Walk Through

• Questions?

Agenda

Agenda



Georgetown

University

American

UniversityCatholic

University

Gallaudet

University

About WRLC and GWU

Voyager 8

9.6 million bibliographic records

10.5 million holdings

5.8 million items



“Canned Reports”

WRLC provides technical services, including reports



Microsoft Access reports from ExLibris are often utilized

Reports.mdb



Linux

Alternative to MS Access: Oracle SQL*Plus (for Linux)



George Washington University Libraries New Books List one example



George Washington University Libraries New Books List Updated weekly



George Washington University Libraries New Books List Call Number Guide



George Washington University Libraries New Books List RSS Menu



George Washington University Libraries New Books List RSS feed



George Washington University New Titles, Media another example



George Washington University New Titles, Media



Steps to creating the New Book Shelf web page Pick and Scan



<1> Connect to the web server

via SSH.

Oracle SQL*Plus

Shell / Perl

you could

use a cron

task instead

Shell scripts and SQL*Plus

<2> Run the newbookshelf shell script.

Steps to creating the New Book Shelf web page



Setup at your library




• These instructions have been validated using

Ubuntu 10.04 LTS in both i686 and x64 desktop

and server editions.

• In order to download files from Oracle, you will

need to register for a free account at

Oracle.com

• We presume in these instructions that you are

using the BASH shell.

• The person performing the setup needs root

user sudo rights, or root account.

Assumptions



Install SQL*Plus on Linux Get files from Oracle

• To obtain the required installable packages, navigate to

http://www.oracle.com/technetwork/indexes/downloads/index.html

Instant Client Package - Basic

Instant Client Package - SQL*Plus

• Scroll down to the Database category and choose Instant Client.

• Choose the appropriate Operating System.

• Check the box indicating you accept the licensing agreement.

• Download the two client packages listed below in .rpm format.

We recommend 10.2.0.4

• Transfer the two files to the Linux server if you downloaded them

somewhere else first.



Install SQL*Plus on Linux Install files from Oracle

The next steps are completed in a terminal window.

Convert the .rpm packages to .deb to install them in Ubuntu.

Do this by using an application called 'alien'.

If it is not already installed, you can do so by typing:

sudo apt-get install alien

Next, navigate to where you saved the .rpm packages and type:

sudo alien -k oracle-instantclient*.rpm

The .deb packages are ready to be installed. Do so by typing:

sudo dpkg -i oracle-instantclient*basic*.deb

sudo dpkg -i oracle-instantclient*sqlplus*.deb




The library files are installed but we must create an environment variables

called LD_LIBRARY_PATH and SQLPATH in order for SQL*Plus to locate

them. We'll do this by typing:

gedit ~/.bashrc and pasting into it:

export LD_LIBRARY_PATH=/usr/lib/oracle/10.2.0.4/client64/lib

export SQLPATH=/usr/lib/oracle/10.2.0.4/client64/lib

Note: The path to ../10.2.0.4/client64/lib will be different depending on which version of SQL*Plus you

installed and which architecture your platform runs on. Change '10.2.0.4' and 'client64' part of the

path accordingly. If you are installing on a server platform use vi or nano instead of gedit.

We need to fix one more dependency, so do: sudo apt-get install libaio1

You may now go ahead and delete the .rpm files downloaded and subsequent

.deb that were created.

Install files from Oracle



Oracle SQL*Plus is now installed and configured. Create a blank

tnsnames.ora file in your home directory by typing:

touch ~/tnsnames.ora

Create an environment variable so SQL*Plus can reference this tnsnames.ora

file we created.

Type: gedit ~/.bashrc and paste into it: export TNS_ADMIN=~

Logout of your account and then log back in to apply the changes to your

shell.

Note: use sqlplus or sqlplus64 as appropriate and change your USERNAME, PASSWORD, HOSTNAME, PORT and

SID as applicable.

You can test your installation by typing on one line:sqlplus

USERNAME/PASSWORD@'(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(

HOST=HOSTNAME)(PORT=1521)))(CONNECT_DATA=(SID=VGER)))‘




You can put the appropriate information in your tnsnames.ora file using

the following as a template:

IDENTIFIER =

(DESCRIPTION =

(ADDRESS = (PROTOCOL= TCP)(Host=HOSTNAME)(Port= 1521))

(CONNECT_DATA = (SID = VGER))

)

You will then be able to connect with the following command:

sqlplus USERNAME/PASSWORD@IDENTIFIER




Download the files from https://github.com/cummingsm/VoyagerELUNA2012

Get the newbooks scripts, install instructions, this slide file

https://github.com/cummingsm/VoyagerELUNA2012



pageHeader.template

lcguideHeaderpt1.out


lcguideFactory.out

authorsectionHeader.template

callnumbersectionFactory.out

authorsectionFactory.out

titlesectionHeader.template

titlesectionFactory.out

pagefooterFactory.out

Sections of newlibrarybooks.html

The completed web page is a

combination of several sections, some

are boilerplate templates, others are

rows of information output produced by

the script having parsed and

reformatted the SQL results.

• Page header

• The LC Call Number and RSS guide

• List sorted by Call Number

• List sorted by Author

• List sorted by Title

• Page footer

We also generate separate RSS pages.



newbookshelf.sh

This Shell script calls the other shell and Perl scripts,

and the SQLplus query. (Snippet shown on next slide)

sqlplusQuery.sh

Reads several parameters from the configuration file config.cnf

(Oracle username, password, port, host, etc) and generates

tempQuery.sh. It saves a SQL statement in newbook.sql.

Then it runs tempQuery.sh which saves output in

a tab-delimited file named tbl_newbooklocation.tab

config.cnf

This is the local configuration setting file.

Example: ORACLEPORT=1521

You must edit it.

Code walk-through pg 1 of 17



log=/tmp/newlibrarybooks.log

echo "newbookshelf.sh " > $log

date >> $log

# run the SQLPlus query

echo "calling sqlplusQuery.sh" >> $log

sh sqlplusQuery.sh;

echo -n ':::::::::: Processing the new libray books query output :';

#

echo 'Step 1: calling doPrelim.sh to save previous file' >> $log

sh doPrelim.sh

echo -n ':'

echo 'Step 2: calling sortCallNumbers.sh (and sortLC.pl) ' >> $log

sh sortCallNumbers.sh;

.

. s n i p

.

echo 'DONE!'

echo 'Copy newlibrarybooks.html file to web site directory; copy /rss

directory and files to a directory below the one containing

newlibrarybooks.html.'

newbookshelf.shCode walk-through pg 2 of 17

User runs

this script.

newbookshelf.sh calls the other scripts and creates the log file.



#ORACLEUSER =`grep ORACLEUSER config.cnf | cut -f2 -d'='`

ORACLEPWD =`grep ORACLEPWD config.cnf | cut -f2 -d'='`

ORACLEHOST =`grep ORACLEHOST config.cnf | cut -f2 -d'='`

ORACLEPORT =`grep ORACLEPORT config.cnf | cut -f2 -d'='`

ORACLESID =`grep ORACLESID config.cnf | cut -f2 -d'='`

ORACLEPATH =`grep ORACLEPATH config.cnf | cut -f2 -d'='`

myTEMPLOC =`grep TEMPLOC config.cnf | cut -f2 -d'='`

#LD_LIBRARY_PATH=$ORACLEPATH

SQLPATH=$ORACLEPATH

#

# Generate the complete sql file by merging templates with values from the config file

#cat sqlplus_newbooktop.template > newbook.sql

echo "item.temp_location='$myTEMPLOC'" >> newbook.sql

cat sqlplus_newbookbottom.template >> newbook.sql

sqlplusQuery.shCode walk-through pg 3 of 17

Read config

file. Make

SQL from

template.

sqlplusQuery.sh generates the SQL command and query statement



# Output commands,parameters to shell script

cmdstring_exec=$SQLPATH/sqlplus

cmdstring_acct=$ORACLEUSER/$ORACLEPWD@

cmdstring_desc="'(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)

(HOST=$ORACLEHOST)(PORT=$ORACLEPORT)))(CONNECT_DATA=(SID=$ORACLESID)))'"

cmdstring_file="< newbook.sql"

echo 'LD_LIBRARY_PATH='$ORACLEPATH > tempQuery.sh

echo 'SQL_PATH='$ORACLEPATH >> tempQuery.sh

echo 'export LD_LIBRARY_PATH' >> tempQuery.sh

echo 'export SQLPATH' >> tempQuery.sh

echo $cmdstring_exec $cmdstring_acct$cmdstring_desc $cmdstring_file >>

tempQuery.sh

#

# run the script

#

sh tempQuery.sh

sqlplusQuery.sh

Generate

a command

line; run it

sqlplusQuery.sh executes the query




set ECHO OFF

set COLSEP ' '

set LINESIZE 3000

set PAGESIZE 500

set WRAP OFF

set HEADING OFF

set FEEDBACK OFF

set SCAN OFF

set ESCAPE \

column callnum Format A30

column callnumgroup Format A1

column title Format A255

column bib Format 99999999

SPOOL tbl_newbooklocation.tab

http://ss64.com/ora/syntax-sqlplus-set.html

SQL*PLUS set statements

Optional

formatting

newbook.sql




select

bib_vw.call_no as callnum,

bib_vw.call_no as callnumgroup,

bib_item.bib_id as bib,

bib_text.title,

bib_text.author || ' ' as author_td,

publisher || ' ' || pub_place || ' ' || publisher_date as pub

from

bib_vw,

bib_item,

bib_text,

item

SQL Query

Author

always has

the string




SQL Query

where

item.temp_location='$myTEMPLOC'

and

bib_vw.bib_id=bib_text.bib_id

and

bib_text.bib_id=bib_item.bib_id

and

bib_item.item_id=item.item_id

and

bib_vw.call_no NOT LIKE 'GW%'

and

…etc

bib_vw.call_no NOT LIKE '9%' and bib_vw.call_no

is not NULL

order by bib_vw.call_no

/

SPOOL OFF

We filter

certain

callnums

Temp loc

from the

config file




sortCallNumbers.sh

Removes the query statement from the query output and then

passes the output to a Perl script (sortLC.pl) for better

sorting by LC Class call number. Result is sortedfile.tab

callNumberFactory.sh

This script and others with “Factory” in the name produce a

file with snippets of HTML code.

Using awk, read sortedfile.tab and generate the HTML section

listing the titles sorted by call number.

doPrelim.sh

Removes problem characters from tbl_newbooklocation.tab

Saves a backup copy of the previous html and sql output files




callNumberFactory.sh generates callnumberFactory.out

callnumberFactory.out contains rows for an HTML table.

The lines will look like this in a browser:

callNumberFactory.shCode walk-through pg 9 of 17



# Initialize flag to indicate if a row was printed.

# This flag determines whether to print a shaded row or not

rowprint=0;

#

max=`wc -l sortedfile.tab | cut -c1-3`

for ((i=1; i<=$max; i++))

do

#

# Grab one line of data from the file and save it temporarily in a file

#

awk NR==$i sortedfile.tab> htmlfactory.tmp

# Get contents of fields from the temporary file into table data elements

td1=`cat htmlfactory.tmp | cut -f1`







tab is the

default

delimiter

callNumberFactory.sh reads the fields in the SQL output file




# Set the background color of each row.

if [ "$rowprint" == "0" ]; then

rowcolor=FFFFFF; rowprint=1;

else

rowcolor=E4DFE6; rowprint=0;

fi

# Generate HTML tags around the retrieved data

NEWROW="<TR BGCOLOR=#"$rowcolor" valign=TOP><TD grp="$td2">

"

CALLNUMBER="<A HREF=\"http://"$myOPAC"/cgi-bin/Pwebrecon.cgi?BBID=

"$td3"\">"$td1"</A></TD>";

TITLE="<TD>"$td4"</TD>";

AUTHOR="<TD>"$td5" </TD>";

PUBLISHER="<TD>"$td6"</TD></TR>";

#

# Append each generated HTML line to the output file

echo $NEWROW $CALLNUMBER $TITLE $AUTHOR $PUBLISHER >> callnumberFactory.out

done


callNumberFactory.sh outputs HTML with field values




callNumberSectionFactory.sh

callNumberSectionFactory.sh prints headings and inserts relevant lines

from callnumberFactory.out after the heading.

callNumberSectionFactory.sh generates callnumbersectionFactory.out

Now the output will look like this:




lcguideFactory.sh

lcguideFactory.sh generates lcguideFactory.out

The RSS pages generated by this script are the .xml files in the rss directoryThe RSS icon displayed on the page is feed.png

rsspageFactory.sh and makeRSS.pl generate multiple .xml files




authorSectionFactory.sh & titleSectionFactory.sh

authorSectionFactory.sh generates authorsectionFactory.out

titleSectionFactory.sh generates titlesectionFactory.out




webPageMergeFactory.sh

WEB=webpagemergeFactory.out

echo ''> $WEB

# Static HTML text page header with date stamp

cat pageHeader.template >> $WEB

cat lcguideHeaderpt1.template >> $WEB

date >> $WEB

cat lcguideHeaderpt2.template >> $WEB

cat lcguideFactory.out >> $WEB

# call number section

cat callnumbersectionFactory.out >> $WEB

# Heading for the author section

cat authorsectionHeader.template >> $WEB

webPageMergeFactory.sh generates webpagemergeFactory.out

Continues

next page…




# Rows of titles by author

cat authorsectionFactory.out >> $WEB

#

# Heading for the title section

cat titlesectionHeader.template >> $WEB

#

# Rows of titles

cat titlesectionFactory.out >> $WEB

#

# Static HTML text footer

cat pagefooterFactory.out >> $WEB

#

cp $WEB $HOME/public_html/newlibrarybooks.html

#

webPageMergeFactory.sh

pageHeader.template



lcguideFactory.out

authorsectionHeader.template

callnumbersectionFactory.out

authorsectionFactory.out

titlesectionHeader.template

titlesectionFactory.out

pagefooterFactory.out

Sections of newlibrarybooks.html




The XML files for RSS pages are saved in the directory named “rss”.

There is one file per LC Class heading.

RSS files

newlibrarybooksB.xml

newlibrarybooksC.xml

newlibrarybooksD.xml

newlibrarybooksH.xml

.

.

.

newlibrarybooksR.xml

newlibrarybooksT.xml

/rss




The last steps are publishing the HTML and RSS files

1. Copy $HOME/public_html/newlibrarybooks.html to the directory

where your web server will display the page. (Copy rssnb.png too)

2. Copy the directory rss, and the .xml files it contains, to a directory

named rss located under the directory where you have placed

newlibrarybooks.html

• At GWU we perform the above steps with commands placed at

the end of the webpagemergeFactory.sh file

Publish the files



You may want to customize further

• Add steps to the end of webPageMergeFactory.sh

that copy the files to your web directory

• Customize the page header template with your

own library branding

• You may trigger the process automatically

with a cron task.

• Edit the SQL template for your own needs.

• If you prefer CSS formatting, convert the script from

HTML <table> to CSS <div> statements



QUESTIONS

In this presentation you learned how to:

• install SQL*Plus on Linux

• download GWU’s new books scripts

• edit GWU’s scripts

We hope you can adapt these scripts

for use at your own library.

Questions?

cummingsdceluna2012

Education

salt lake city

exlibris eluna

gwu voyager

libraryinstall sql

examplefrom voyager

utfrom voyager

mdbfrom voyager

web pageseluna