Managing Data Sources Within the Blackboard GUI

Mark Spurlock, Carl Hardin, C. LemarinierInnovative Technology Center

University of Tennessee Knoxville

Intended Audience

• Sysadmins at institutions using the snapshot and dsmtools who would like to automate and expand the functionality of each.

• Would-be building block developers who would like to see some working JSP code using the Bb API.

Helpful skills and knowledge: experience with Bb batch data integration, a little understanding of Perl and relational databases including Blackboard’s back end, JSP.

Team background and contact info:

• Mark Spurlock; Manager, Technical Systems Development; 10 years experience with Bb; spurlock@utk.edu

• Carl Hardin; Web Application Developer; 2 years experience with Bb and JSP; chardin@utk.edu

• Celine Marinier; Web Application Developer; experienced Java and JSP programmer; clemarin@utk.edu

• Web site: http://itc.utk.edu

University of Tennessee Profile

• Bb/CourseInfo school for 10+ years.• ASP hosted since 2002.• Approximately 30,000 active users during academic

year.• Approximately 56,000 accumulated course sites. (We

retain course materials for two years after the end of the course term.)

• 800 gigs of data in Blackboard.• 90 percent of students enrolled in at least one Bb site.

Problem description: Command-line snapshot and dsm

• Lots of switches and complex command lineExample:./snapshot_override.sh “-

Ddata.source.key=bb_group.dat” –f CRS_SNPSHT –t /usr/local/blackboard/apps/snapshot/data/bb_group.data –C /usr/local/blackboard/apps/snapshot/data/snapshot.properties –V utk.blackboard.com

• Isolates snapshot from GUI administration, requiring additional skill set (Linux command line).

• As is, requires lots of human intervention.

Typical means of control:

• Hand-editing of snapshot.properties• External “controller” script, written in some language

like Perl• Custom-designed controller perhaps provided by

Blackboard Global Services

Typical functions and process of snapshot controller

• Logging

• Transfer feed files (ftp) from SIS.

• Validate and prep files for the snapshot client.

• Call snapshot client for each file received.

• Cleanup of work files and archive.

• Email notice of completion along with errors.

Other “nice to have” functions:

• Quality-control checks (e.g. rollback, error tracking).

• Calendar-aware automation.

• System maintenance—implementation of institution’s data retention standards.

• Sysadmin control and correction through Bb APIs and GUI, rather than disjunction of the two.

How we’ve done it in the past

• Lots of HTML forms• Lots of Perl scripts• Lots of Cron jobs• Lots of human memory because lots of pieces were in

lots of places.

In some ways, this was necessary because we had to build up our knowledge and experience of what we needed the system to do for us. “A better way” had to evolve from experience.

Example: Course deletion

• Old policy: Keep everything forever or at least until faculty wanted to delete it (absolutely, positively):

1) Instructor requests via online form2) Request stored in database, email warning

sent to every instructor of course.3) Cron job processes request after two-week

grace period.4) Course reassigned to “TO_DELETE” data

source and disabled (hidden)5) Six months later—got to be sure!—course

actually deleted/purged.

Course deletion problems

• Not automated.• Not integrated (with business processes).• Faculty didn’t delete.• Customer service and policy frequently in conflict.• Process mysterious to everyone but me.

Worst feature: disjunction between customer face and backend

• Support (even GUI sysadmin) has little understanding of how things work.

• Limits customer service.• Process control not in the hands of those most

responsible for its accuracy, consequences.• Flip side, difficulty of ad hoc change not understood

by those requesting it.• Touching one piece can break something somewhere

else.• Little redundancy in knowledgeable personnel.

Problem Solution: Give more control to the GUI admin

• Command line becomes point and click.• GUI admin knows instantly what the snapshot will do

and when.• Though requiring technical expertise to set up,

requires less technical expertise to maintain and operate.

Existing data source building blocks

• Smart DSM – Limited to dsm.sh functionality.Located here:

http://www.blackboard.com/extend/b2.aspx?ExtensionID=10242

• bb-datasource-manager – Limited to dsm.shfunctionality. Could not get to run with Bb version 8.

Located here: http://code.google.com/p/bb-datasource-manager/

Smart DSM interface

Smart DSM interface, part 2

Functional correspondence of dsm.sh Smart DSM

• dsm.sh –f LIST –v host• dsm.sh –f CREATE –b datasource –v host• dsm.sh –f MODIFY –b datasource –v host• dsm.sh –f COUNTS(...) –b datasource –v host• dsm.sh –f DISABLE(...) –b datasource –v host• dsm.sh –f PURGE(...) –b datasource –v host• dsm.sh –f REMOVE –b datasource –v hostSee /usr/local/blackboard/apps/snapshot/bin/readme.txt

Shortcomings of Smart DSM

• dsm only—no snapshot functionality• Still manual and immediate• Not customizable (relies entirely on built-in

Blackboard data)

To address these, we knew we needed a database.

Design decision: How to implement the database

• Modify the existing Bb database or...• Build an extended data_source table in a separate

database.

Modifying the existing Bb database seemed bad because it might get destroyed by upgrades, not sure it was within our license, and could raise support issues.

MySQL database

• Building block requires extra fields.• Desire not to alter Bb database.• Already used MySQL database for other building

blocks (for example, student textbooks).• Means the snapshot controller and client can run from

any machine.• Con: Requires installation of MySQL driver on

application server. (Note: If you provide instructions to ASP, you TSM should be willing to do this.)

Alternatively, you can embed MySQL with the building block

• Download the latest Connector/ODBC - MySQLODBC driver from http://www.mysql.com/downloads/.

• Unzip the package.• Locate the driver/jar file (i.e. mysql-connector-java-

5.0.5-bin.jar).• Copy the jar file to the following folder in the building

block: WEB-INF/lib/.

Overview of utk-ds-tool

Building Block Sysadmin GUI

Bb OracleDatabase

CustomMySQL DB

Bb

AP

I

MyS

QL

Java

D

river

Perl D

BI SnapshotController

dsmsnapshot

utk-ds-tool interface – part 1

utk-ds-tool interface, part 2

utk-ds-tool interface, part 3

Overview of snapshot – controller integration

• Controller selects active records for processing.• Checks today’s date and sets values in feed files

appropriately (example: AVAILABLE_IND = ‘Y’ or AVAILABLE_IND=‘N’).

• Uses dsType field to know which snapshot function to call for each data source. Example:

snapshot_override.sh "-Ddata.source.key=$batchUid" -V $host -f ${dsType}__SNPSHT -t $dir/$batchUid -C $dir/snapshot.properties;

Snapshot controller and data retention

• GUI Sysadmin sets data retention and course availability based on university policies and academic calendar.

• Snapshot automatically cycles courses based on those calendar settings and values in the database.

• No manual intervention and everyone knows when things are going to happen.

Time for the nitty-gritty

That’s the 1,000 foot view.

Now for some coding details.

Overview of our development environment

• Blackboard 8• MyEclipse 6.0.1

Reference API'sbb-cms-admin.jarbb-platform.jarbb-taglibs.jar

• Tomcat 5.5• RHEL 3 (Red Hat Linux)• Oracle 10.4• MySQL 5• Perl 5+

Components of the UTK Data Source Tool (GUI)

• Link available only on the "System Admin“ tab.• view.jsp (home)

ConfirmDisable.jspDisableDataSource.jsp

ConfirmPurge.jspPurgeDataSource.jsp

modDataSource.jsp (modify form)processUpdate.jsp

addDataSource.jsp (insert form)processInsert.jsp

Database code – part 1 -- MySQL

Connecting MySQL within the JSP files

Since we knew we’d be using the MySQL database in several JSP files, we created an include file:

<%@ include file="../include/ConnMySQL.inc" %>

Contents of include file

<%Connection Conn = null ;try {

Conn = DriverManager.getConnection("jdbc:mysql://your.dbhost.ip:3306/databasename?user=********&password=********");

}catch (Exception E) {

out.println("Unable to load driver.");E.printStackTrace(new PrintWriter(out));

}Statement statement = null ;ResultSet rs = null ;

%>

Database code, part 2 – Oracle (virtual host)

<%// Begin: Initialize the BbServiceManager and set the contexttry {blackboard.platform.BbServiceManager.init("/usr/local/blackboard/config/service-config-snapshot-

jdbc.properties");// The virtual host is needed to establish the proper database context.VirtualInstallationManager vm =

(VirtualInstallationManager)BbServiceManager.lookupService(VirtualInstallationManager.class );String vhostUID = System.getProperty("dbhost","bb_bb60");VirtualHost vhost = vm.getVirtualHost(vhostUID);if (vhost == null ) {throw new Exception("Virtual Host '" + vhostUID + "' not found.");

}// Now that the vhost is set we can set the context based on that vhostContextManager cm =

(ContextManager)BbServiceManager.lookupService(ContextManager.class );Context context = cm.setContext(vhost);

} catch (Exception e) {System.out.println("Exception trying to init the BbPersistenceManager\n " + e.toString() +

"..exiting.\n");System.exit(0);

}// End: Initialize the BbServiceManager and set the context

%>

Code to load data from each database (view.jsp)

<%@page import="blackboard.base.BbList"%><% try {

// Loading all data sourcesBbList dsList = DataSourceLoader.Default.getInstance( ).loadAll();GenericFieldComparator compBatchUid = new

GenericFieldComparator(BaseComparator.ASCENDING, "getBatchUid", DataSource.class );GenericFieldComparator compDescription = new

GenericFieldComparator(BaseComparator.ASCENDING, "getDescription", DataSource.class );

%><bbUI:list collection="<%=dsList %>" collectionLabel="Available Data Sources"

objectId="eachDataSource" className="blackboard.admin.data.datasource.DataSource" resultsPerPage="-1">

<bbUI:listElement width = "10%" label= "Data Source (BatchUid)" name="BatchUid" comparator = "<%=compBatchUid%>">

<%=eachDataSource.getBatchUid() %></bbUI:listElement><bbUI:listElement width = "40%" label= "Description" name="Description" comparator =

"<%=compDescription%>"><%=eachDataSource.getDescription() %>

</bbUI:listElement>...

Code, continued (view.jsp)

<% String strType="";try {String s_batchuid = eachDataSource.getBatchUid();// Run select statement for MySQL (snapshot/data_source)// statement and rs are both declared in connMySQL.incstatement = Conn.createStatement();

// Clear the MySQL recordset rs for each record passed// rs = null;// MySQL query to check custom data_source tablestatement.executeQuery("SELECT batchUid, dsType FROM data_source WHERE batchUid =

'"+s_batchuid+"'");

rs = statement.getResultSet();

// Check for a matching record in data_source (MySQL)if (rs.next()) {

strType = rs.getString("dsType");// If a record is found in MySQL, verify that dsType is not nullif (strType != null ) {

%>

Code, continued (view.jsp)

<bbUI:listElement width = "5%" label= "" name="Modify" align="center"><A class="inlineAction"

HREF="modDataSource.jsp?BatchUid=<%=eachDataSource.getBatchUid() %>">Modify </A>

</bbUI:listElement><bbUI:listElement width = "5%" label= "" name="Disable" align="center"><A class="inlineAction"

HREF="confirmDisableRequest.jsp?BatchUid=<%=eachDataSource.getBatchUid() %>">Disable </A>

</bbUI:listElement><bbUI:listElement width = "5%" label= "" name="Purge" align="center"><A class="inlineAction"

HREF="confirmPurgeRequest.jsp?BatchUid=<%=eachDataSource.getBatchUid() %>">Purge </A>

</bbUI:listElement><%

Code, continued (view.jsp)

} //Closing if (rs.dsType != null)else {// We have a match in both databases, but dsType is null// Data sources with dsType equal to null (in MySQL) are not allowed to disable,

purge, or modify// The list elements are still inserted, but no buttons are made available in the

GUI%>

<bbUI:listElement width = "5%" label= "" name="Modify" align="center"></bbUI:listElement><bbUI:listElement width = "5%" label= "" name="Disable" align="center"></bbUI:listElement><bbUI:listElement width = "5%" label= "" name="Purge" align="center"></bbUI:listElement>

<%} // Closing else

} // Closing if (rs.next())

Code, continued (view.jsp)

else {// There was no matching record in MySQL// The list elements are still inserted, but no buttons are made available in the GUI

%><bbUI:listElement width = "5%" label= "" name="Modify" align="center"></bbUI:listElement><bbUI:listElement width = "5%" label= "" name="Disable" align="center"></bbUI:listElement><bbUI:listElement width = "5%" label= "" name="Purge" align="center"></bbUI:listElement>

<%} // Closing else (no matching record found in MySQL)

} catch (Exception exception) {exception.printStackTrace();throw new ServletException("Error : " + exception.getMessage());

}// Close database connectionsrs.close ();statement.close ();

%></bbUI:list>

Sample code for disabling data source

(DisableDataSource.jsp)

<%// Begin: Purging data source by BatchUid

try {DataSourcePersister dsPersister = blackboard.admin.persist.datasource.DataSourcePersister.Default.getInstance();

dsPersister.disableAllAdminObjects(strBatchUid, date); // Note: date has been set to null for our purpose

} catch (Exception e) {System.out.println("Exception trying to disable the data source by BatchUid\n " + e.toString() + "..exiting.\n");

System.exit(0);} // End: Disabling data source by BatchUid

%>

Sample code for purging data source

(DisableDataSource.jsp)

<%// Begin: Purging data source by BatchUidtry{DataSourcePersister dsPersister =

blackboard.admin.persist.datasource.DataSourcePersister.Default.getInstance();dsPersister.purgeAllAdminObjects(strBatchUid, date); // Note: date has been set to

null for our purpose}catch (Exception e){System.out.println("Exception trying to purge the data source by BatchUid\n " +

e.toString() + "..exiting.\n");System.exit(0);} // End: Purging data source by BatchUid

%>

Sample code for updating a data source(processUpdate.jsp)• This code shows how to update the MySQL database.

if(request.getParameter("ds_isActive")!=null){isActive=request.getParameter("ds_isActive");

}String s_query="UPDATE data_source SET

term='"+request.getParameter("ds_term")+"', active='"+isActive+"', dsType='"+request.getParameter("ds_type")+"', makeAvailDate=DATE('"+request.getParameter("ds_avail")+"'),

makeUnavailDate=DATE('"+request.getParameter("ds_unavail")+"'), disableDate=DATE('"+request.getParameter("ds_disable")+"'), purgeDate=DATE('"+request.getParameter("ds_purge")+"') WHERE batchUid='"+s_batchuid+"'";

Code, continued (processUpdate.jsp)

<%@ include file="../include/ConnMySQL.inc" %>

<%try {

statement = Conn.createStatement();int res=statement.executeUpdate(s_query);if (res==1) conf_message = "ok";else conf_message = "ko";statement.close ();

}…%>

Code, continued (processUpdate.jsp)

• This code shows how to update the Oracle database.

<%try {DataSource ds_selected = DataSourceLoader.Default.getInstance().loadByBatchUid(s_batchuid);

ds_selected.setDescription(s_description);

DataSourcePersister dsPersister = blackboard.admin.persist.datasource.DataSourcePersister.Default.getInstance();

dsPersister.modify(ds_selected);} …

Sample code for inserting a new data source

(processInsert.jsp)• This code shows how to update the MySQL database.

String s_dtmodified=java.util.Calendar.getInstance().toString();String s_description=request.getParameter("ds_description");

String mysql_query="INSERT into data_source (batchUid, term, active, dsType, makeAvailDate, makeUnavailDate, disableDate, purgeDate) values("

+ "'" + s_batchuid+ "', '" + s_term+ "', '" + s_isactive+ "', '" + s_type+ "', DATE('" + s_avail+ "'), DATE('" + s_unavail+ "'), DATE('" + s_disable+ "'), DATE('" + s_purge+ "'))";

Code, continued (processInsert.jsp)

<%@ include file="../include/ConnMySQL.inc" %><%

statement = Conn.createStatement();boolean res=statement.execute(mysql_query);if (res==true) conf_message = "ok";else conf_message = "ko";statement.close ();

%>

Code, continued (processInsert.jsp)

• This code shows how to insert new data sources to the Oracle database.

<%try {

DataSource new_ds = new DataSource();new_ds.setBatchUid(s_batchuid);new_ds.setDescription(s_description);new_ds.setDataSourceId(Id.newId(DataSource.DATA_TYPE));DataSourcePersister dsPersister =

blackboard.admin.persist.datasource.DataSourcePersister.Default.getInstance();

dsPersister.create(new_ds); }

Pitfalls encountered during development

• Reference API's in war files conflict with installation.• Blackboard drops the MySQL database driver

connection when Bb services are restarted.• Javascript form validations need special attention.

Reference API’s in MyEclipse (solution)

• The Blackboard API’s are extremely helpful while you are developing JSP files within MyEclipse.

• Exporting these API’s in your war file will cause your building block to fail when it is installed in Blackboard.

• Simply remove these reference API’s from your build path before creating your war file with MyEclipse:

bb-cms-admin.jarbb-platform.jarbb-taglibs.jar

Blackboard drops MySQL connection (solution)

• MySQL work around to get a new instance of the database driver when Bb services are restarted.

<%// registering the MySQL JDBC drivertry {

// The newInstance() call is a work around for some// broken Java implementationsClass.forName("com.mysql.jdbc.Driver").newInstance( );}

catch (Exception E) {out.println("Unable to load driver.");E.printStackTrace(new PrintWriter(out));}

%>

Javascript form validations (solution)

<%@page import="blackboard.platform.plugin.PlugInUtil"%><%PlugInUtil.authorizeForSystemAdmin(request, response);%><script language="Javascript"><!--function ValidateForm() {

if (document.update_form.ds_type.value == '' || document.update_form.ds_type.value==null) {alert('Please enter a type.');document.update_form.ds_type.focus();return false;

}if (document.update_form.ds_term.value == '' || document.update_form.ds_term.value==null) {

alert('Please enter a term.');document.update_form.ds_term.focus();return false;

}...}

--></script><form name="update_form" action="<%=PlugInUtil.getUri("utk", "utk_dsm","module/processUpdate.jsp")%>"

onSubmit="return ValidateForm();">...</form>

Future enhancements

• Choose database location and driver type in building block properties.

• Add vhost field to database table: many installations could run one snapshot client.

• Building of snapshot.properties file through retrieval of stored database values (for example, specification of which fields are snapshot controlled).

Conclusion and questions?