132 IEEE TRANSACTIONS ON EDUCATION, VOL. 43, NO. 2, MAY 2000

Data Warehousing: A Tool for the OutcomesAssessment Process

Joanne Ingham

AbstractTo meet EC-2000 assessment criteria, informationfrom all university constituents needs to be routinely collectedand tracked longitudinally. Typically, one finds that even whenthe information exits, it is located across several decentralizeddata bases run by different units. Additionally, many universitiesare either running older mainframe legacy systems or are inthe process of making the transition to newer systems. Theseconditions make it difficult to collect information regardingstudents and their performance throughout their undergraduatecareer and after. Data warehousing is a system of organizinginstitutional data that can support the assessment process. Thispaper describes one universitys experience with the developmentand management of a data warehouse through the support of theNational Science Foundation Gateway Coalition.

Index TermsAssessment, outcomes assessment.

I. INTRODUCTION

POLYTECHNIC University is a small, private institutionand a member of the National Science Foundation (NSF)Gateway Coalition. Programs are offered in engineering (chem-ical, civil, computer, electrical, mechanical), applied sciences(chemistry, environmental science, information management,math, physics, computer science), management, liberal studies,humanities, and social science. Three campuses serve approxi-mately 1700 undergraduates and 1300 graduate students.

The university has been using the Integrated Student In-formation System (ISIS) as a main source of data, as well asthe record-keeping system. Departmental records are typicallymaintained in local data bases, spreadsheets, and computerand paper files. Additional information is stored on universityUNIX servers, including e-mail, course information, and courseand departmental distribution lists. Currently, the universityis converting its system into an integrated enterprise system(Peoplesoft). Given the diversity of the existing systems andthe relative difficulty of retrieving information to respond tothe anticipated expectations of EC-2000 and the outcomesassessment process, establishing a data warehouse was crucial.Table I describes the comparative benefits of data warehousing.

A data warehouse literally warehouses information about anorganization in a secured computing systems environment toallow multiple users to extract meaningful, consistent, and ac-curate data for decision making [1]. A well-designed data ware-house supports an assessment environment in which human en-ergy is focused on the continuous improvement process and notthe mechanical collection of data.

Manuscript received August 1999; revised November 29, 1999.The author is with Polytechnic University, Brooklyn, NY 11201 USA (e-mail:

jingham@poly.edu).Publisher Item Identifier S 0018-9359(00)04310-7.

A review of the literature reveals that data warehousing is be-coming an increasingly popular way to store and retrieve data,primarily in business settings, and more recently in collegesand universities [2]. Information about this new application hasbeen reported in conference proceedings [3][5] and in journals[6][8]. In addition, several data warehouse organizations pro-vide information and white papers on their Web sites.1

This case study will provide a description of the process usedto build and maintain a data warehouse within the Office of In-stitutional Assessment. As regional and professional accredita-tion bodies and state education departments adopt requirementsfor documentation of continuous improvement, the benefits ofthe data warehouse will become evident over time.

II. THE DEVELOPMENT PROCESSCYCLE 1With support from the NSF Gateway Coalition, the process of

building the data warehouse began in the summer of 1997. Thegoal of this project was to store institutional data from all appro-priate areas of the university in a relational data-base format toenable timely, accurate analysis for tracking and analyzing pat-terns of change. The first cycle of development entailed buildingthe data warehouse The data were drawn from the universitylegacy system, departmental data bases, and external universitydata, as depicted in Fig. 1. There were four distinct steps, which,when completed, resulted in the birth of the data warehouse.

Step 1-Recognition and Support for a CentralizedProcess: In response to the assessment expectations ofthe NSF Gateway Coalition, as well as EC-2000, the effortwas made to establish the Office of Institutional Assessmentand to support the creation of a longitudinal tracking systemto follow students academic careers. In August 1997, adirector and project leader were hired. All necessary com-puter equipment and software (Access, SQL, Visual Basic,ASP, COBOL, C-shell Script) were purchased, installed, andconfigured by October. In this instance, the data warehousewas developed and operational within a six-month period.The warehouse was populated with ten years of institutionaldata from the legacy system. For larger institutions or thosewhose systems complexities differ, the time required for thisphase may vary.

Step 2Identify Available University Data: The secondstep involved determining which other institutional data ex-isted, who managed it, and how the data were formatted. Toaccomplish this, meetings were held with each departmentdirector to identify the nature and format of data they routinely

1See the Data Warehousing Knowledge Center at http://www.dataware-housing.org and The Business Intelligence and Data Warehousing Glossary athttp://www.sdgcomputing.com/glossary.htm.

00189359/00$10.00 2000 IEEE

INGHAM: DATA WAREHOUSING 133

TABLE ICOMPARATIVE BENEFITS OF USING A DATA WAREHOUSE VERSUS A LEGACY SYSTEM

Fig. 1. Building a data warehouse.

collected and stored. While the array of data available weresubstantial, it seemed that each department stored data indifferent formats. Integrating the potpourri of data files intoa consistent format presented a real challenge for the projectleader. For example, separate relational data-base managementsystems existed for offices whose format was determined byoutside funding agencies, the alumni functions and data wereoutsourced, and the academic departments used a variety ofsoftware and, often, paper reports. Data from state reports andthe Cooperative Institutional Research Program and the resultsof national assessments were processed by researchers at otherinstitutions.

As a result of requesting data from each department, a numberof critical issues became apparent. These issues arose, in part,due to a lack of awareness about the revised accreditation expec-tations. There was a recognized need to make public, and insti-tutionalize, the outcomes assessment process at the institution.The benefits of a university statement on assessment were rec-

ognized. Such a statement was prepared and is now included inthe new university catalog. Assessment activities had to becomeformally recognized as a routine part of university practice.

Second, several directors were reluctant to share their fileswith others. They expressed concern about maintaining the con-fidentiality of information collected for federally funded pro-grams. To address this concern, the vice president and dean ofengineering and applied sciences sent formal release letters clar-ifying the fundamental purpose of the request and insuring thatonly aggregate data would be reported. Last, meetings were con-ducted with the directors and data-base managers to determinethe scope of the current data and, most important, to identify thequality or purity of the data available. Unless meticulous care istaken to import accurate and clean data, the fundamental valueof the warehouse is compromised.

Step 3Data Integration: After all available data had beenidentified and examined, they then had to be integrated into thewarehouse. For research purposes and longitudinal tracking forthe Gateway Coalition, a decision was made to include ten yearsof history data. The history data were migrated from the univer-sity legacy system. These data included the 21-day census datafiles, the freshmen cohort files, graduation files, and registrarfiles containing grades. Files for the alumni were requested byprogram and years out as needed.

The actual collection, examination, and storage of informa-tion in the warehouse involved several activities. All files hadto be checked for data quality and to verify that the data werevalid. All duplicate and/or dead data needed to be eliminatedbefore migration could take place. In addition, data consistencywas checked repeatedly by cross-referencing data from differentsources.

Step 4Developing a Process to Feed the Data Ware-house: Ideally, the data warehouse should be built with thecapacity to accommodate assessment data as they are collected.A decision was made to move, when possible, away frompaper and pencil data collection strategies and move towardelectronic, Web-based methods. Testing the usefulness ofseveral different strategies was accomplished by piloting threeGateway Coalition assessment tools on a small scale. Table IIdescribes the comparative benefits of a Web-based process.

134 IEEE TRANSACTIONS ON EDUCATION, VOL. 43, NO. 2, MAY 2000

TABLE IICOMPARATIVE BENEFITS OF PAPER VERSUS WEB-BASED ASSESSMENT

Among the first Gateway assessment surveys prepared was afaculty survey to assess the degree to which faculty participationin the coalitions activities fostered instructional innovation inthe classroom. The completed paper forms were collected andthe data were entered manually. This process was very laborintensive and took three weeks to distribute, collect, and enter.

The first pilot effort with a Web-based evaluation involvedthe freshman design course. The survey data were collected andanalyzed electronically. Given that the evaluation form existed,populating the survey was the most time-consuming part. TheWeb evaluation form was available to students over a period ofthree weeks, and the results were available immediately.

The next pilot involved the mechanical engineering alumnisurvey. This survey was prepared and made available as both aWeb-based and paper version A letter with the Web address anda paper survey were mailed home, and the alumni could selectwhich method to use. Both approaches were employed to betterdetermine the relative response rates with an alumni population.

A Web-based civil engineering course-level assessment toolwas tested with nine sophomore- through senior-level courses.We were, at this point, very confident in the mechanics and se-curity of the technology. Based on further review of the resultsof the pilot experiences, an institutional decision was made togo ahead with a Web-based assessment approach and to pilotseveral more assessment tools.

III. EVALUATION OF CYCLE 1

After several months, a number of substantial accomplish-ments were achieved. An Office of Institutional Assessment wasestablished and the staff was selected. The space was preparedand the equipment was set up and functional. Institutional datafor a ten-year period were collected, cleaned, and integrated intothe warehouse. Having run several tests to confirm the accu-racy of information generated and the integrity of the system,the technology was considered operational. The first reports

on cohort persistence and graduation rates for gender and eth-nicity were run. Last, a series of beta-tests confirmed that theWeb-based survey process was functional and secure.

The startup phase required personnel and equipment. Cycle1 activities were completed over the course of six months byone full-time professional working exclusively on this project.A total of 960 hours were involved. As Cycle 2 was planned,a decision was made to add a student assistant at 20 hours perweek to handle the Web-based survey activities. As the ware-house became functional, requests for data were, as expected,numerous. The total spent on new computers, software and pe-ripherals, and eventually a dedicated server came to $15 000.An additional $35 000 in salaries brought the project costs to$50 000.

Along with these accomplishments, however, several prob-lems were encountered that necessitated modifications duringCycle 2. Specifically, as the data warehouse grew, we foundthat the data-base software performed slowly. In fact, the func-tionality and security of the software products were inadequatefor our expanding needs. As Web-based assessment procedureswere applied, the Web connection and speed were not suffi-cient for our needs. Parallel to this, the university Web teammade a decision to return to a UNIX system utilizing Oraclefor data-base activities.

IV. CYCLE 2SOLVING PROBLEMS

Experience with the warehouse capabilities and a full appre-ciation of the scope of the project prompted the decision to moveinto a UNIX environment for speed, security, and availabilityof institutional support and consistency. Having also made thedecision to adopt a Web-based assessment process, additionalissues had to be considered.

First, additional hardware was purchased. To support the Webactivities, a dedicated server was added to our system. A SunSolaris 26 Ultra 10 server and Sun DDS2 backup tape system

INGHAM: DATA WAREHOUSING 135

were purchased The software selected included Apache WebServer, Perl 5m and MySQL relational data base with built insecurity features. The MySQL data base will allow for a tableup to 6 GB, which we estimate to be adequate for a few decadesof institutional data. Concerns about sufficient room would notbe an issue for some time to come.

With this setup, the Apache could handle up to 50 concur-rent users. This feature is more than adequate for an organiza-tion with an anticipated enrollment of 5000 graduate and under-graduate students. Based on the first cycle results, new featuressuch as data integrity checking were also introduced. Resultsare now processed before they are submitted or inserted into thedata warehouse. As the warehouse continued to grow, other fea-tures were introduced to allow for searches by two, three, or fourletters of the last name to minimize typing.

V. WEB-BASED PROCESS MODIFICATIONS

When the preparation began for university-wide Web-basedcourse evaluations, a number of critical steps were taken. It wasat this juncture that the Gateway Coalition assessment activitiesbecame integral to the technology.

Working with the dean, department heads, faculty and theStrategic Planning Team, the universitys mission statement,program- and course-level goals, and objectives were prepared.For each undergraduate course, the goals and objectives werelinked with the competencies identified by the Gateway Coali-tion and EC-2000. A course evaluation form was developed in-corporating two component parts. The first part, developed bythe Gateway Coalition, was used to assess specific competen-cies and skills linked to EC-2000 Criterion 3. The second partwas a locally prepared course evaluation form developed by thefaculty.

Based on lessons learned through beta-tests, a course eval-uation process was established that allows each undergraduatestudent to complete course evaluations on a Web site2 only forcourses they were registered for that semester. We learned thatthe Web site should be available approximately three weeksprior to the final exam period. Second, the Web address shouldbe easy to remember and accessible from off-campus locations.

The design of the system insured that students were limitedto evaluating only courses they were registered in, and onlyone time. The evaluations were easy to complete and could becompleted at one sitting or several, as desired. Students werealso able to submit written comments. The abundance andrichness of written commentary provided in pilots were de-lightfully surprising. A few sentences, as well as paragraphs ofthoughtful commentary, were submitted. The comments rangedfrom positive to negative. The feedback was often filled withpraise for specific aspects of the course and instructor. Criticalcomments were, for the most part, constructive. The studentresponses were analyzed and prepared for dissemination usingAccess. This feedback then became part of the data warehouse.The final report was made available to the campus communityon the university intranet.

2See www2.poly.edu.

VI. WAREHOUSE AS A RESOURCE

As faculty and administrators learned more about the newexpectations of the NSF Gateway Coalition, ABET, CSAB, andMiddle States and the importance of outcomes assessment, thevolume of requests increased. Additionally, as information onretention and graduation rates was disseminated, the value of theassessment process and capabilities of the longitudinal trackingsystem were slowly recognized.

The benefits to the faculty, department heads, and adminis-trators were several. Access to timely, consistent, and thoroughdata as needed was faster and more convenient. The administra-tion of alumni surveys by the assessment office allowed the fac-ulty to look at the process and results and not spend a lot of en-ergy on the administration and collection of information. The es-tablishment of routine procedures for user-friendly course-levelassessment meant that the course evaluation process would behandled on an institutional rather than department level. Thewarehouse is also a resource to support the development and im-plementation of assessment plans for grants. Last, the universitynow has the ability to support institutional research using thelongitudinal tracking system.

VII. CONCLUSIONThe data warehouse currently manages 100 000 student

records covering a ten-year span, as well as 30 000 courserecords over a similar period of time. Data on freshman cohortretention across any number of variables, as well as similargraduation history, are available from 1988. Additional datahave been integrated into the system including alumni data,student placement test scores, and co-op information. Further,compared to statically processed information, the new datawarehouse has the capability of on-line dynamic results reportgeneration.

As benchmark data are gathered and the outcomes assess-ment process is refined, the data warehouse will expand expo-nentially. The end result is the availability of superior means ofdocumenting and analyzing continuous improvement on an in-stitutional, program, and course level.

ACKNOWLEDGMENT

The author wishes to thank the members of the NSF GatewayCoalition for their support with this project. She also wishes tothank Polytechnic University, Dr. W. R. McShane, Dr. R. Roess,and A. Polevoy.

REFERENCES[1] R. Bargain and H. Edelstein, Building, Using, and Managing the Data

Warehouse. Upper Saddle River, NJ: Prentice-Hall, 1997.[2] J. Frost, M. Dalyrmple, and M. Wang, Focus for institutional re-

searchers: Developing and using a student decision support system, inProc. 1998 AIR Annu. Forum, Minneapolis, MN, 1998, pp. 115.

[3] L. J. Mignerey, A data warehouseThe best buy for the money, inProc. CAUSE Annu. Conf., Orlando, FL, 1994, p. VI-5-1.

[4] J. D. Porter and J. J. Rome, A data warehouse: Two years later . . .lessons learned, in Proc. CAUSE Annu. Conf., Orlando, FL, 1994, p.VI-6-1.

[5] A. Polevoy and J. Ingham, Data warehousing: A tool for facilitatingassessment, in Proc. 29th ASEE/IEEE Frontiers in Education Conf.,San Juan, Puerto Rico, 1999, p. 11b1-7.

136 IEEE TRANSACTIONS ON EDUCATION, VOL. 43, NO. 2, MAY 2000

[6] C. R. Thomas, Information architecture: The data warehouse founda-tion, CAUSE/EFFECT, vol. 60, no. 2, p. 3133/3840, Summer 1997.

[7] M. Singleton, Developing and marketing a client/server-based datawarehouse, CAUSE/EFFECT, vol. 16, no. 3, pp. 4752, Fall 1993.

[8] M. Bosworth, Rolling out a data warehouse at UMass: A simple startto a complex task, CAUSE/EFFECT, vol. 18, no. 1, pp. 4045, Spring1995.

Joanne M. Ingham was born in Rome, NY, on January 20, 1952. She receivedthe B.S. degree in biology, secondary education, from the State University ofNew York, Oswego, the M.S. degree in counseling from Long Island University,Brookville Center, NY, and the Ed.D. degree in curriculum and instruction fromSt. Johns University, Jamaica, NY.

She has been an Administrator at Polytechnic University, Brooklyn, NY, since1994 and is currently the Director of Institutional Assessment and Retention inthe Office of Academic Affairs. She has been active in research in the fieldsof outcomes assessment and learning styles. She recently completed a researchproject with a Fulbright scholar from Mexico comparing the learning styles andmeasures of creative performance of American and Mexican engineering stu-dents. She taught undergraduate and graduate courses in education at St. JohnsUniversity; Queens College, Queens, NY; and Adelphi University, Garden City,NY. She is an international Consultant in learning styles.