Processing, Storing, and Viewing Use Data

Processing, Storing, and Viewing Use Data

Tansy MatthewsElectronic Resources & Libraries 2010Austin, TXJanuary 31, 2010

Welcome1AgendaBackground informationStoring use dataProcessing use data for storageXMLCSV/ExcelPresenting use over timeIm a Virgo and I was a cataloger, so I like to have everything laid out2BackgroundVirtual Library of Virginia is the consortium of the nonprofit academic libraries within the Commonwealth of Virginia. (www.vivalib.org)

VIVA's missionis to provide, in an equitable, cooperative and cost-effective manner, enhanced access to library and information resources for the Commonwealth of Virginia's nonprofit academic libraries serving the higher education community.

BackgroundComprises 39 state-assisted colleges and universities (at 55 campuses)6 doctoral institutions and 9 4-year comprehensive colleges and universities24 community and 2-year branch colleges(40 campuses)33 of Virginia'sindependent(private, not-for profit) colleges and universitiesOther educational and research institutions

BackgroundChallenge: Develop an efficient means of processing use data that would:Continue providing monthly use data updates for our members, on an institutional and product levelFacilitate long term storage of dataPermit the display of data to track trends and to demonstrate the consortiums contribution to higher education in Virginia

Restrictions: Build in-house, use only software on hand, no programming expertise.5What we realizedKeys to manipulation AND storageConsistent formattingColumnar dataConcatenate

Hey, Tansy!What do you mean by columnar data?BackgroundChallenge revised: Develop an efficient means of processing use data that would:Combine all of our disparate data reports into consistently formatted, columnar data.8Project COUNTERProject COUNTER helped by creating a standard format for use data reporting.http://www.projectcounter.org/What we came up withReceive data as CSV, Excel, or XML filesImport into ExcelXMLdirect import and formattingCSV/Excelcombine files using SobolSofts Import Multiple Text Files tool ($29.99)Develop macros to reformatStore in an Access database

10What this lets us do

AgendaBackground informationStoring use dataProcessing use data for storageXMLCSV/ExcelPresenting use over timeI was a cataloger, so I like to have everything laid out12What it looks like

Storing dataAccess/MySQLSpaceFlexibility

Easy import/export

AgendaBackground informationStoring use dataProcessing use data for storageXMLCSV/ExcelPresenting use over timeI was a cataloger, so I like to have everything laid out15XML 101Extensible Markup Language similar to HTMLXML data tags label the structure (not the appearance) of the documentHuman-readable markup languageXML documents must be translated by a separate program or application (e.g. Excel) - otherwise, theyll appear as marked up text.In English pleaseHTML

Hey! Thats me!In English please

Am not!XML

In English please

XML

Much better.In English pleaseXML

Taught him everything he knows.What does it look like?Example file: http://www.vivalib.org/xml_sample.txtWhy XML?XML facilitates the importation of data into Excel.Allows the user to determine where the data lands when its imported imperative for maintaining a consistent data format over time.One step import/formatting.22How do I get it into Excel?Open ExcelCreate a new workbookSelect Developer>XML>ImportExample: Download http://www.vivalib.org/sjp_07.xmlImport into ExcelUsing XML in ExcelSelect data to be importedAssign data positions (map elements)Change field namesAdding new dataData>XML>XML Map PropertiesFor example:Overwrite (http://www.vivalib.org/sjp_07_march.xml)Append (http://www.vivalib.org/dup_07.xml)Great, but what if theres no XML?For COUNTER compliant data:Compile COUNTER reports into a single Excel workbook (SobolSoft tool)COUNTER processing macros to format: http://www.vivalib.org/stats_training/counter_processing.xlsmSobolSoft tool

Lets try itOpen counter_processing.xlsmOpen http://www.vivalib.org/jr1_compiled.xlsxSelect Developer>MacrosSelect compile_counter_journal_reportsClick Run

Lets try itSelect month.Paste month in first empty column.

Data Collection Over TimeConsistent format allows data to be easily concatenated.Charts can be generated directly from PivotTables.Example: http://www.vivalib.org/extended_report_example.xlsAgendaBackground informationStoring use dataProcessing use data for storageXMLCSV/ExcelPresenting use over timeIm a Virgo and I was a cataloger, so I like to have everything laid out33Creating PivotTablesExcelData>PivotTable and PivotChart ReportChoose source of data & data rangeSelect a static rangeImportPlace data headingsExample file: http://www.vivalib.org/journals_example_sourcedata.xlsCreating PivotChartsClick on your PivotTableGo to PivotTable Tools>PivotChartCreate your chart.To detach your chart from the PivotTable, delete the PivotTableHelp!Some resources:lib-stats listserv: "Lib-Stats hosts information useful for the analysis of usage statistics in academic (and other) libraries. Current issues are debated on the lib-stats mailing list." To sign up for the mailing list, send an email to lib-stats-request@ncl.ac.uk.microsoft.public.excel: From Google groups. A general discussion group for Excel users. More specialized groups are also listed in Google groups. A good source for VBA macros. OzGrid Forum: A good source for Excel tips and VBA macros. Additional Informationhttp://www.vivalib.org/stats_training/stats_resources.htmlStats resourcesExample files

Also potentially useful:PivotTable tutorialhttp://www.vivalib.org/pivot_table_examples/pivot_table_tutorial.htmlQuestions?www.vivalib.orgtmatthe6@gmu.eduSheet1InstitutionProductRequestsDateInst_typeData_typeVendorFYJetson Space InstituteCyborg Growth & Design3Sep-06PrivateFull-TextSuperhero Journals PressFY07Jetson Space InstituteCyborg Growth & Design1Nov-06PrivateFull-TextSuperhero Journals PressFY07Jetson Space InstituteCyborg Growth & Design1Feb-07PrivateFull-TextSuperhero Journals PressFY07John Q. Public CollegeCyborg Growth & Design9Jul-06Public 2-YearFull-TextSuperhero Journals PressFY07Oscar Mayer UCyborg Growth & Design9Jul-06Public 4-YearFull-TextSuperhero Journals PressFY07Oscar Mayer UCyborg Growth & Design6Nov-06Public 4-YearFull-TextSuperhero Journals PressFY07Oscar Mayer UCyborg Growth & Design4Jan-07Public 4-YearFull-TextSuperhero Journals PressFY07Oscar Mayer UCyborg Growth & Design2Feb-07Public 4-YearFull-TextSuperhero Journals PressFY07Whatsamatta UCyborg Growth & Design12Jul-06DoctoralFull-TextSuperhero Journals PressFY07Whatsamatta UCyborg Growth & Design20Aug-06DoctoralFull-TextSuperhero Journals PressFY07Whatsamatta UCyborg Growth & Design6Sep-06DoctoralFull-TextSuperhero Journals PressFY07Whatsamatta UCyborg Growth & Design29Oct-06DoctoralFull-TextSuperhero Journals PressFY07Whatsamatta UCyborg Growth & Design24Nov-06DoctoralFull-TextSuperhero Journals PressFY07Whatsamatta UCyborg Growth & Design22Dec-06DoctoralFull-TextSuperhero Journals PressFY07Whatsamatta UCyborg Growth & Design9Jan-07DoctoralFull-TextSuperhero Journals PressFY07Whatsamatta UCyborg Growth & Design41Feb-07DoctoralFull-TextSuperhero Journals PressFY07Jetson Space InstituteCyborg Growth & Design1Oct-06PrivateFull-TextSuperhero Journals PressFY07Jetson Space InstituteCyborg Growth & Design31Nov-06PrivateFull-TextSuperhero Journals PressFY07Jetson Space InstituteCyborg Growth & Design3Jan-07PrivateFull-TextSuperhero Journals PressFY07John Q. Public CollegeCyborg Growth & Design8Jul-06Public 2-YearFull-TextSuperhero Journals PressFY07John Q. Public CollegeCyborg Growth & Design2Aug-06Public 2-YearFull-TextSuperhero Journals PressFY07John Q. Public CollegeCyborg Growth & Design2Sep-06Public 2-YearFull-TextSuperhero Journals PressFY07John Q. Public CollegeJournal of Cartoon Physics1Feb-07Public 2-YearFull-TextSuperhero Journals PressFY07Oscar Mayer UJournal of Cartoon Physics5Jul-06Public 4-YearFull-TextSuperhero Journals PressFY07

Chart19497279872197595255965889786965374176845555768243370248256728048

Institution Jetson Space Institute PrivateInst_type John Q. Public College Public 2-YearInst_type Oscar Mayer U Public 4-YearInst_type Whatsamatta U Doctoral

Chart29497279872197595255965889786965374176845555768243370248256728048

Institution Jetson Space Institute PrivateInst_type John Q. Public College Public 2-YearInst_type Oscar Mayer U Public 4-YearInst_type Whatsamatta U Doctoral

extended_reportNOTE: Data_type MUST be selected for Totals to be meaningful!Consortial UsageInstitutionInst_typeJetson Space InstituteJohn Q. Public CollegeOscar Mayer UWhatsamatta UGrand TotalData_typeFYPrivatePublic 2-YearPublic 4-YearDoctoralFull-TextFY049497279872199038FY057595255965887958FY069786965374179117FY076845555768248120Grand Total337024825672804834233

raw_dataInstitutionInst_typeVendorData_typeFYTotalJetson Space InstitutePrivateCitation Databases, Inc.SearchesFY07137Jetson Space InstitutePrivateDogford Univ PressFull-TextFY07344Jetson Space InstitutePrivateSuperhero Journals PressFull-TextFY07340John Q. Public CollegePublic 2-YearCitation Databases, Inc.SearchesFY0787John Q. Public CollegePublic 2-YearDogford Univ PressFull-TextFY0717John Q. Public CollegePublic 2-YearSuperhero Journals PressFull-TextFY0738Oscar Mayer UPublic 4-YearCitation Databases, Inc.SearchesFY07168Oscar Mayer UPublic 4-YearDogford Univ PressFull-TextFY07440Oscar Mayer UPublic 4-YearSuperhero Journals PressFull-TextFY07117Whatsamatta UDoctoralCitation Databases, Inc.SearchesFY0798Whatsamatta UDoctoralDogford Univ PressFull-TextFY075326Whatsamatta UDoctoralSuperhero Journals PressFull-TextFY071498Jetson Space InstitutePrivateCitation Databases, Inc.SearchesFY06160Jetson Space InstitutePrivateDogford Univ PressFull-TextFY06393Jetson Space InstitutePrivateSuperhero Journals PressFull-TextFY06585John Q. Public CollegePublic 2-YearCitation Databases, Inc.SearchesFY0697John Q. Public CollegePublic 2-YearDogford Univ PressFull-TextFY0614John Q. Public CollegePublic 2-YearSuperhero Journals PressFull-TextFY0655Oscar Mayer UPublic 4-YearCitation Databases, Inc.SearchesFY06183Oscar Mayer UPublic 4-YearDogford Univ PressFull-TextFY06475Oscar Mayer UPublic 4-YearSuperhero Journals PressFull-TextFY06178Whatsamatta UDoctoralCitation Databases, Inc.SearchesFY06118Whatsamatta UDoctoralDogford Univ PressFull-TextFY064676Whatsamatta UDoctoralSuperhero Journals PressFull-TextFY062741Jetson Space InstitutePrivateCitation Databases, Inc.SearchesFY05352Jetson Space InstitutePrivateDogford Univ PressFull-TextFY05345Jetson Space InstitutePrivateSuperhero Journals PressFull-TextFY05414John Q. Public CollegePublic 2-YearCitation Databases, Inc.SearchesFY05183John Q. Public CollegePublic 2-YearDogford Univ PressFull-TextFY0514John Q. Public CollegePublic 2-YearSuperhero Journals PressFull-TextFY0538Oscar Mayer UPublic 4-YearCitation Databases, Inc.SearchesFY05395Oscar Mayer UPublic 4-YearDogford Univ PressFull-TextFY05431Oscar Mayer UPublic 4-YearSuperhero Journals PressFull-TextFY05128Whatsamatta UDoctoralCitation Databases, Inc.SearchesFY05393Whatsamatta UDoctoralDogford Univ PressFull-TextFY054534Whatsamatta UDoctoralSuperhero Journals PressFull-TextFY052054Jetson Space InstitutePrivateCitation Databases, Inc.SearchesFY04270Jetson Space InstitutePrivateDogford Univ PressFull-TextFY04469Jetson Space InstitutePrivateSuperhero Journals PressFull-TextFY04480John Q. Public CollegePublic 2-YearCitation Databases, Inc.SearchesFY04205John Q. Public CollegePublic 2-YearDogford Univ PressFull-TextFY0418John Q. Public CollegePublic 2-YearSuperhero Journals PressFull-TextFY0454Oscar Mayer UPublic 4-YearCitation Databases, Inc.SearchesFY04293Oscar Mayer UPublic 4-YearDogford Univ PressFull-TextFY04571Oscar Mayer UPublic 4-YearSuperhero Journals PressFull-TextFY04227Whatsamatta UDoctoralCitation Databases, Inc.SearchesFY04412Whatsamatta UDoctoralDogford Univ PressFull-TextFY044910Whatsamatta UDoctoralSuperhero Journals PressFull-TextFY042309