wheres the data? conducting a first-year data audit karen paulson asheville, north carolina – july...
TRANSCRIPT
![Page 1: Wheres the Data? Conducting a First-Year Data Audit Karen Paulson Asheville, North Carolina – July 2002](https://reader035.vdocuments.mx/reader035/viewer/2022081518/551667d5550346b2068b6164/html5/thumbnails/1.jpg)
Where’s the Data?Conducting a First-Year Data Audit
Karen Paulson
Asheville, North Carolina – July 2002
![Page 2: Wheres the Data? Conducting a First-Year Data Audit Karen Paulson Asheville, North Carolina – July 2002](https://reader035.vdocuments.mx/reader035/viewer/2022081518/551667d5550346b2068b6164/html5/thumbnails/2.jpg)
The Data Audit Toolkit
Developed Through a Partnership
> The Policy Center on the First Year of College
• John Gardner, Betsy Barefoot, Randy Swing, Mike Siegel, and Marc Cutright
> NCHEMS
• Karen Paulson and Peter Ewell
Generously Supported by:
> The Pew Charitable Trusts
> The Atlantic Philanthropies
![Page 3: Wheres the Data? Conducting a First-Year Data Audit Karen Paulson Asheville, North Carolina – July 2002](https://reader035.vdocuments.mx/reader035/viewer/2022081518/551667d5550346b2068b6164/html5/thumbnails/3.jpg)
Pilot Institutions
Augustana College (IL) Blue Ridge Community College (VA) Lynchburg College (VA) Northeast State Technical Community College (TN) Ohio University Santa Fe Community College (FL) The University of Texas at El Paso University of Cincinnati University of Minnesota Duluth Washington State University
![Page 4: Wheres the Data? Conducting a First-Year Data Audit Karen Paulson Asheville, North Carolina – July 2002](https://reader035.vdocuments.mx/reader035/viewer/2022081518/551667d5550346b2068b6164/html5/thumbnails/4.jpg)
Data Audit
“The process of identifying data resources
and uses wherever they may be within an
institution and gathering them into a useable
information system.”
NCHEMS Administrative Rationale manual, page 15
![Page 5: Wheres the Data? Conducting a First-Year Data Audit Karen Paulson Asheville, North Carolina – July 2002](https://reader035.vdocuments.mx/reader035/viewer/2022081518/551667d5550346b2068b6164/html5/thumbnails/5.jpg)
Basic Premise I
Most of the Data Needed for Meaningful Analysis Already Reside in Institutional Data Systems
> Admissions
> Registration/Transcripts
> Student Surveys
> Assessment Databases
> Individual “Service Office” Records
![Page 6: Wheres the Data? Conducting a First-Year Data Audit Karen Paulson Asheville, North Carolina – July 2002](https://reader035.vdocuments.mx/reader035/viewer/2022081518/551667d5550346b2068b6164/html5/thumbnails/6.jpg)
Basic Premise II
The Major Task Is:
> To Organize These Data in Useful Ways
> To Make Access and Analysis as Flexible and Straightforward as Possible
![Page 7: Wheres the Data? Conducting a First-Year Data Audit Karen Paulson Asheville, North Carolina – July 2002](https://reader035.vdocuments.mx/reader035/viewer/2022081518/551667d5550346b2068b6164/html5/thumbnails/7.jpg)
The Data Audit Process
Is NOT Itself an End
Is a Means for Understanding and Improving the Use of Data and Information on Campus
Key to Improving the First Year of College
![Page 8: Wheres the Data? Conducting a First-Year Data Audit Karen Paulson Asheville, North Carolina – July 2002](https://reader035.vdocuments.mx/reader035/viewer/2022081518/551667d5550346b2068b6164/html5/thumbnails/8.jpg)
“…we tend to assume that all first-year
programs were implemented as planned
and that the experiences of all students
were uniform.”
![Page 9: Wheres the Data? Conducting a First-Year Data Audit Karen Paulson Asheville, North Carolina – July 2002](https://reader035.vdocuments.mx/reader035/viewer/2022081518/551667d5550346b2068b6164/html5/thumbnails/9.jpg)
In Reality There Are Three First-Year Experiences*
The “Official” One
(Designed and Published Plans)
The “Delivered” One
(Actual Institutional Actions)
The “Experienced” One
(Reality Including Student Choice)
* Adapted from Joan Stark
![Page 10: Wheres the Data? Conducting a First-Year Data Audit Karen Paulson Asheville, North Carolina – July 2002](https://reader035.vdocuments.mx/reader035/viewer/2022081518/551667d5550346b2068b6164/html5/thumbnails/10.jpg)
The Main Question….
How can your institution analytically disentangle
the many elements of the first college year and
provide evidence about the effectiveness, or lack
thereof, of related first-year programs, policies,
and procedures?
![Page 11: Wheres the Data? Conducting a First-Year Data Audit Karen Paulson Asheville, North Carolina – July 2002](https://reader035.vdocuments.mx/reader035/viewer/2022081518/551667d5550346b2068b6164/html5/thumbnails/11.jpg)
Some Preliminary Questions
What Is Unique About the First Year of College at Your Institution?
What Is it Like to Walk in a Student’s Shoes at Your Institution?
How Many Surveys, Tests, or Assessments Does a Student Really Experience During the Year?
Does Any Information from These Data Collections Get Communicated Back to the Students? How?
How Are These First-Year Data Used by Faculty and Administrators?
![Page 12: Wheres the Data? Conducting a First-Year Data Audit Karen Paulson Asheville, North Carolina – July 2002](https://reader035.vdocuments.mx/reader035/viewer/2022081518/551667d5550346b2068b6164/html5/thumbnails/12.jpg)
A Quick Self-Assessment
What Are the Primary Sources of Data About Students, Curricula, and Programs—Specifically Related to the First Year of College—Currently in Place at Your Institution?
Are Any of These Underutilized? Why?
What’s Missing About Which You Would Like Information?
![Page 13: Wheres the Data? Conducting a First-Year Data Audit Karen Paulson Asheville, North Carolina – July 2002](https://reader035.vdocuments.mx/reader035/viewer/2022081518/551667d5550346b2068b6164/html5/thumbnails/13.jpg)
Why a Data Audit?
Identify and Inventory Data Sources
Identify and Inventory Data Needs
Support Assessment of...
> What Happened
> What Mattered
Foster a Culture of Data Use
![Page 14: Wheres the Data? Conducting a First-Year Data Audit Karen Paulson Asheville, North Carolina – July 2002](https://reader035.vdocuments.mx/reader035/viewer/2022081518/551667d5550346b2068b6164/html5/thumbnails/14.jpg)
Primary Data Audit Activities
The Supply Side
Conducting a Campus-Wide Examination of Existing Data Sources
The Demand Side
Determining Which Data Are Most Needed for Evaluation, Assessment, and Decisionmaking
![Page 15: Wheres the Data? Conducting a First-Year Data Audit Karen Paulson Asheville, North Carolina – July 2002](https://reader035.vdocuments.mx/reader035/viewer/2022081518/551667d5550346b2068b6164/html5/thumbnails/15.jpg)
Key Components of a Data Audit
Identifying Sources of Data
Inventorying These Data Sources
Compiling Information About Data Sources
Identifying Gaps in Data Sources
Assessing What Data Users Need and Want to Know
Determining Which Analyses (Existing or Suggested) Have Utility for Stakeholders of Your Institution or Unit
![Page 16: Wheres the Data? Conducting a First-Year Data Audit Karen Paulson Asheville, North Carolina – July 2002](https://reader035.vdocuments.mx/reader035/viewer/2022081518/551667d5550346b2068b6164/html5/thumbnails/16.jpg)
Organizing a Data Audit
Institution-Wide Committee
> IR, Student Affairs, Academic Affairs, Advisors, Orientation Staff, Faculty, Registrar Staff, Students, etc.
Team Members
> People Directly Involved with the Institution
> People Directly Involved with Data
> First-Year Professionals not Normally “Data Connected”
> Leader(s) with Broad Campus Support and Respect
The Data Audit Leadership Team
![Page 17: Wheres the Data? Conducting a First-Year Data Audit Karen Paulson Asheville, North Carolina – July 2002](https://reader035.vdocuments.mx/reader035/viewer/2022081518/551667d5550346b2068b6164/html5/thumbnails/17.jpg)
The Right Attitude
Fresh Perspective
Open Attitude
Collaborative Approach
Not a “Gotcha” Mentality
![Page 18: Wheres the Data? Conducting a First-Year Data Audit Karen Paulson Asheville, North Carolina – July 2002](https://reader035.vdocuments.mx/reader035/viewer/2022081518/551667d5550346b2068b6164/html5/thumbnails/18.jpg)
Visiting Potential Data Sites
Reasons for Physical Visits
> Honors Unit Personnel on Their Turf
> Builds Relationships
> Allows You to Read Reactions
> Allow You to Do Immediate Follow-Up and Collect Artifacts
> Discovers Hidden Databases
![Page 19: Wheres the Data? Conducting a First-Year Data Audit Karen Paulson Asheville, North Carolina – July 2002](https://reader035.vdocuments.mx/reader035/viewer/2022081518/551667d5550346b2068b6164/html5/thumbnails/19.jpg)
Types of Data
Official Data
> Registrar Systems
> Admissions
> Institutional Research Offices
> Assessment Offices
“Unofficial” (Guerrilla) Databases
> Unit Records Kept in Local Computers
> Viewed as “Single” Purpose/Disposable Records
![Page 20: Wheres the Data? Conducting a First-Year Data Audit Karen Paulson Asheville, North Carolina – July 2002](https://reader035.vdocuments.mx/reader035/viewer/2022081518/551667d5550346b2068b6164/html5/thumbnails/20.jpg)
Supply Side Questions
Types of Data?
Who Collects Data and Why?
How Complete Are Data?
Where Do Data Go?
“Walking the Process”
![Page 21: Wheres the Data? Conducting a First-Year Data Audit Karen Paulson Asheville, North Carolina – July 2002](https://reader035.vdocuments.mx/reader035/viewer/2022081518/551667d5550346b2068b6164/html5/thumbnails/21.jpg)
Supply Side Questions to Ask
What Kinds of Records or Data Do You Keep on First Year Students?
How Complete Are Data Collected?
How Are Data Entered?
What Schedules Govern Your Data Collection?
How Are Data Updated?
![Page 22: Wheres the Data? Conducting a First-Year Data Audit Karen Paulson Asheville, North Carolina – July 2002](https://reader035.vdocuments.mx/reader035/viewer/2022081518/551667d5550346b2068b6164/html5/thumbnails/22.jpg)
Finding Existing Data
Follow Student “Footprints”
> Time-Based Investigation
Who Collects Data?
> Structures-Based Investigation
Why Are Data Collected?
> Functions-Based Investigation
![Page 23: Wheres the Data? Conducting a First-Year Data Audit Karen Paulson Asheville, North Carolina – July 2002](https://reader035.vdocuments.mx/reader035/viewer/2022081518/551667d5550346b2068b6164/html5/thumbnails/23.jpg)
Supply Side: Items to Collect
Actual Forms and Questionnaires
Data Element Dictionaries
Data Element Definitions
Database Structures and File Formats
![Page 24: Wheres the Data? Conducting a First-Year Data Audit Karen Paulson Asheville, North Carolina – July 2002](https://reader035.vdocuments.mx/reader035/viewer/2022081518/551667d5550346b2068b6164/html5/thumbnails/24.jpg)
Demand Side Questions
Who Are the Key Constituencies?
What Are the Existing Reports and Requirements?
To Whom Are Data and Information Reported?
What, and When, Are the Decision Cycles?
How Current and Accurate Does the Information Need to Be?
What Are the Gaps in Existing Data?
![Page 25: Wheres the Data? Conducting a First-Year Data Audit Karen Paulson Asheville, North Carolina – July 2002](https://reader035.vdocuments.mx/reader035/viewer/2022081518/551667d5550346b2068b6164/html5/thumbnails/25.jpg)
Demand Side Questions to Ask
To Whom Do You Report Data and Information?
What Information Do You Need or Wish You Had?
What Are the Gaps in Data?
How Are Data Used By Others?
![Page 26: Wheres the Data? Conducting a First-Year Data Audit Karen Paulson Asheville, North Carolina – July 2002](https://reader035.vdocuments.mx/reader035/viewer/2022081518/551667d5550346b2068b6164/html5/thumbnails/26.jpg)
Demand Side: Items to Collect
Representative Reports to Constituents
Copies of External Data Reporting Requirements
![Page 27: Wheres the Data? Conducting a First-Year Data Audit Karen Paulson Asheville, North Carolina – July 2002](https://reader035.vdocuments.mx/reader035/viewer/2022081518/551667d5550346b2068b6164/html5/thumbnails/27.jpg)
Looking for...
Completeness of Data Gathered
Availability of Data
Integrity of Data
Consistency of Data Definitions
Who Coordinates Data Collection?
Who Coordinates Use of Data?
Who Controls Data and Data Processes?
![Page 28: Wheres the Data? Conducting a First-Year Data Audit Karen Paulson Asheville, North Carolina – July 2002](https://reader035.vdocuments.mx/reader035/viewer/2022081518/551667d5550346b2068b6164/html5/thumbnails/28.jpg)
Data Audit Output
Synthesized and Coherent Picture of Existing Data
> Data Element Lists
> Data Definitions
> Data Locations and Locus of Responsibility
> Data Collection Timetables
> Plan for Common Data Collection/Sharing
> Identification of Unmet Data Needs
![Page 29: Wheres the Data? Conducting a First-Year Data Audit Karen Paulson Asheville, North Carolina – July 2002](https://reader035.vdocuments.mx/reader035/viewer/2022081518/551667d5550346b2068b6164/html5/thumbnails/29.jpg)
Design of Recommended Data Structure
Define Core Data Elements
Responsibilities and Data Flows
Core Indicators and Calculational Routines
![Page 30: Wheres the Data? Conducting a First-Year Data Audit Karen Paulson Asheville, North Carolina – July 2002](https://reader035.vdocuments.mx/reader035/viewer/2022081518/551667d5550346b2068b6164/html5/thumbnails/30.jpg)
Data Leverage Points
Problem Identification> Persistence Rate> Violations of Prerequisite Sequencing> Student/Advisor Ratios
Context Setting> Achievement by Gender or Race> Patterns of Student Performance> Entering Student Characteristics
To Inform Discussion> Overall Patterns Rather than Anecdotal or Single
Selling Decisions> Creating Buy-In for Action
![Page 31: Wheres the Data? Conducting a First-Year Data Audit Karen Paulson Asheville, North Carolina – July 2002](https://reader035.vdocuments.mx/reader035/viewer/2022081518/551667d5550346b2068b6164/html5/thumbnails/31.jpg)
Using Evidence to Stimulate and Manage Change
Start with Obvious Discrepancies> Between:
• Perceptions and Reality• Designs and Delivery• Expectations and Results
> Among Different Constituencies and Stakeholders
Avoid “Perfect Data Fallacy”> Recognize There Will Always Be “Errors”> Use Triangulation and Multiple Indicators> Know How Good Is Good Enough
Use Information in “Layers”> Avoid Excessive Complexity> Package Data Around Problems> Disaggregate as Needed in Response to Questions and Opportunities
Use Information to Start Discussions, not “Give Answers”> Develop Multiple Interpretations> Establish “Data Dialogue”
![Page 32: Wheres the Data? Conducting a First-Year Data Audit Karen Paulson Asheville, North Carolina – July 2002](https://reader035.vdocuments.mx/reader035/viewer/2022081518/551667d5550346b2068b6164/html5/thumbnails/32.jpg)
The Data Audit Toolkit(coming fall 2002)
Administrative Rationale
> A non-technical overview and rationale for a data audit.
Technical Manual
> A detailed guide, listing data elements and data, as well as step-by-step plans for a data audit.
![Page 33: Wheres the Data? Conducting a First-Year Data Audit Karen Paulson Asheville, North Carolina – July 2002](https://reader035.vdocuments.mx/reader035/viewer/2022081518/551667d5550346b2068b6164/html5/thumbnails/33.jpg)
Thank You!
Karen Paulson, 303.497.0354, [email protected]
Mike Siegel, 828.877.6009, [email protected]
If you have questions or want to discuss the data audit
and analysis project further, please contact: