› 2018 › 07 › 25141859 › 2018... · responding to open-ended polls1 using your smartphone...
TRANSCRIPT
![Page 1: › 2018 › 07 › 25141859 › 2018... · Responding to Open-Ended Polls1 Using your smartphone or computer, go to Menti.com and enter the 6 digit code for the polling question](https://reader031.vdocuments.mx/reader031/viewer/2022041818/5e5c17987f190b798d5665c1/html5/thumbnails/1.jpg)
1
Using your smartphone or computer, go to Menti.com and enter the 6 digit code for the polling question. Each polling question will have a new code.
Responding to Open-Ended Polls
![Page 2: › 2018 › 07 › 25141859 › 2018... · Responding to Open-Ended Polls1 Using your smartphone or computer, go to Menti.com and enter the 6 digit code for the polling question](https://reader031.vdocuments.mx/reader031/viewer/2022041818/5e5c17987f190b798d5665c1/html5/thumbnails/2.jpg)
Addressing the Needs of Assessment Professionals: Practical Recommendations for Higher Ed• Laura Ariovich• Conna Bral• Patricia Gregg• Matthew Gulliford• Jennifer Ann Morrow
![Page 3: › 2018 › 07 › 25141859 › 2018... · Responding to Open-Ended Polls1 Using your smartphone or computer, go to Menti.com and enter the 6 digit code for the polling question](https://reader031.vdocuments.mx/reader031/viewer/2022041818/5e5c17987f190b798d5665c1/html5/thumbnails/3.jpg)
3
Report Authors
Laura AriovichPrince George’s Community College
Conna BralMidwestern State University
Patricia GreggGeorgia State University
Matthew GullifordWatermark
Jennifer Ann MorrowUniversity of Tennessee
![Page 4: › 2018 › 07 › 25141859 › 2018... · Responding to Open-Ended Polls1 Using your smartphone or computer, go to Menti.com and enter the 6 digit code for the polling question](https://reader031.vdocuments.mx/reader031/viewer/2022041818/5e5c17987f190b798d5665c1/html5/thumbnails/4.jpg)
4
Project Purpose
and Resources
![Page 5: › 2018 › 07 › 25141859 › 2018... · Responding to Open-Ended Polls1 Using your smartphone or computer, go to Menti.com and enter the 6 digit code for the polling question](https://reader031.vdocuments.mx/reader031/viewer/2022041818/5e5c17987f190b798d5665c1/html5/thumbnails/5.jpg)
5
To analyze:
• practitioner and administrator
perceptions of assessment
• the roles/positions of the people in
higher education whose primary job
relates to learning outcomes assessment
• assessment activities on campus and the
level of assessment on campus, and
• professional development needs of
assessment practitioners in higher
education.
Purpose of Project
![Page 6: › 2018 › 07 › 25141859 › 2018... · Responding to Open-Ended Polls1 Using your smartphone or computer, go to Menti.com and enter the 6 digit code for the polling question](https://reader031.vdocuments.mx/reader031/viewer/2022041818/5e5c17987f190b798d5665c1/html5/thumbnails/6.jpg)
6
Watermark Survey (2016)
• practitioner and administrator perceptions of assessment
• purpose was to collect data to gain national insights into assessment and accreditation efforts in the
US.
Description of Datasets
Sample: 1074 respondents• 33% Faculty• 22% Department Chair/Coordinator• 21% Director• 9% Dean/Assistant Dean• 7% University Administrators (Pres, VP, Provost, etc.)• 6% Other Roles• 3% Analyst/Researcher
![Page 7: › 2018 › 07 › 25141859 › 2018... · Responding to Open-Ended Polls1 Using your smartphone or computer, go to Menti.com and enter the 6 digit code for the polling question](https://reader031.vdocuments.mx/reader031/viewer/2022041818/5e5c17987f190b798d5665c1/html5/thumbnails/7.jpg)
7
UKY Survey (2015)
• practitioner and administrator perceptions of assessment
• purpose was to examine institutional effectiveness professionals
assessment perceptions, activities, and job satisfaction.
Sample: 377 Assessment Professionals
○ 74% were highest level assessment person at their institution§ 52% Executive Director/Director§ 12% Asst/Assoc Vice President/Provost
Description of Datasets
![Page 8: › 2018 › 07 › 25141859 › 2018... · Responding to Open-Ended Polls1 Using your smartphone or computer, go to Menti.com and enter the 6 digit code for the polling question](https://reader031.vdocuments.mx/reader031/viewer/2022041818/5e5c17987f190b798d5665c1/html5/thumbnails/8.jpg)
8
Perceptions of
Assessment Professionals
![Page 9: › 2018 › 07 › 25141859 › 2018... · Responding to Open-Ended Polls1 Using your smartphone or computer, go to Menti.com and enter the 6 digit code for the polling question](https://reader031.vdocuments.mx/reader031/viewer/2022041818/5e5c17987f190b798d5665c1/html5/thumbnails/9.jpg)
9
Watermark(2016) participants were asked to describe how important assessment was
to their institution. Three primary themes emerged from the data:
Importance of Assessment
ImprovementAccountability
Implementation Concerns
![Page 10: › 2018 › 07 › 25141859 › 2018... · Responding to Open-Ended Polls1 Using your smartphone or computer, go to Menti.com and enter the 6 digit code for the polling question](https://reader031.vdocuments.mx/reader031/viewer/2022041818/5e5c17987f190b798d5665c1/html5/thumbnails/10.jpg)
10
Improvement
“How can we know what we do well and not so well without assessment? Assessment should guide our efforts to improve our outcomes.”
“There are so many legitimate but varying opinions on how to teach, what is going well and where improvement needs to occur. Assessments helps provide a common point of conversation.”
“Without sound assessment practices, we are only operating on "guess-work“ and convenience seekers who "like things the way they are.””
![Page 11: › 2018 › 07 › 25141859 › 2018... · Responding to Open-Ended Polls1 Using your smartphone or computer, go to Menti.com and enter the 6 digit code for the polling question](https://reader031.vdocuments.mx/reader031/viewer/2022041818/5e5c17987f190b798d5665c1/html5/thumbnails/11.jpg)
11
Accountability
“Assessment allows institutions to tell their story. It provides us a way to justify what we are doing is important to students, the community, and the state at-large. It allows us to show that we are making a difference.”
“We hold a duty of responsibility to those who entrust us with their future, as well as to companies who would employ them, to provide them with skills and abilities they can rely on throughout their life.”
![Page 12: › 2018 › 07 › 25141859 › 2018... · Responding to Open-Ended Polls1 Using your smartphone or computer, go to Menti.com and enter the 6 digit code for the polling question](https://reader031.vdocuments.mx/reader031/viewer/2022041818/5e5c17987f190b798d5665c1/html5/thumbnails/12.jpg)
12
Implementation Concerns
“Useful, well-designed assessments can guide an institution in worthwhile directions, but gathering numbers for the sake of just gathering numbers is a waste of valuable faculty time that ought to be spent teaching, working with students, or doing research.”
“It's needed, but it is not the purpose of higher education. That purpose is being eclipsed by "Assessment Madness.””
![Page 13: › 2018 › 07 › 25141859 › 2018... · Responding to Open-Ended Polls1 Using your smartphone or computer, go to Menti.com and enter the 6 digit code for the polling question](https://reader031.vdocuments.mx/reader031/viewer/2022041818/5e5c17987f190b798d5665c1/html5/thumbnails/13.jpg)
What do you like most about assessment?
Go to Menti.com and enter the code 556866.Type in up to three responses.
![Page 14: › 2018 › 07 › 25141859 › 2018... · Responding to Open-Ended Polls1 Using your smartphone or computer, go to Menti.com and enter the 6 digit code for the polling question](https://reader031.vdocuments.mx/reader031/viewer/2022041818/5e5c17987f190b798d5665c1/html5/thumbnails/14.jpg)
14
Watermark(2016) participants were asked what they liked the most about assessment.
Three primary themes emerged from the data:
What Assessment Professionals Like
Enjoyment of methodological aspectsEnjoyment of collaborative aspects
Appreciation for the impact/utility of assessment
![Page 15: › 2018 › 07 › 25141859 › 2018... · Responding to Open-Ended Polls1 Using your smartphone or computer, go to Menti.com and enter the 6 digit code for the polling question](https://reader031.vdocuments.mx/reader031/viewer/2022041818/5e5c17987f190b798d5665c1/html5/thumbnails/15.jpg)
15
Enjoyment of Methodological Aspects
“I love developing models to determine student engagement and achievement and then analyzing those models with data.”
“Finding a story within the numbers.”
“I like designing assessments because it is interesting to see if one can actually create something to measure something (like learning) that isn't readily observed.”
![Page 16: › 2018 › 07 › 25141859 › 2018... · Responding to Open-Ended Polls1 Using your smartphone or computer, go to Menti.com and enter the 6 digit code for the polling question](https://reader031.vdocuments.mx/reader031/viewer/2022041818/5e5c17987f190b798d5665c1/html5/thumbnails/16.jpg)
16
Enjoyment of Collaborative Aspects
“Providing faculty and administrators with support (e.g. data processing/analysis, assignment design suggestions, event coordination) and encouraging collaborative discussion to drive continuous improvement of process, instruction and student learning.”
“I love dialoguing with faculty about the assessment data they are collecting and how they can use the information they collect to continuously improve student learning.”
![Page 17: › 2018 › 07 › 25141859 › 2018... · Responding to Open-Ended Polls1 Using your smartphone or computer, go to Menti.com and enter the 6 digit code for the polling question](https://reader031.vdocuments.mx/reader031/viewer/2022041818/5e5c17987f190b798d5665c1/html5/thumbnails/17.jpg)
17
Impact/Utility of Assessment
“The opportunity to 'prove' that we are doing important work and contributing to student growth/abilities.”
“I enjoy the opportunity to see programs develop assessment plans that produce useful information, and to see that they are making changes with the information. This is rewarding because I know, ultimately, students are benefiting from these efforts.”
“It allows you to chart progress; understand strengths and weaknesses; provides guidance for decision making.”
![Page 18: › 2018 › 07 › 25141859 › 2018... · Responding to Open-Ended Polls1 Using your smartphone or computer, go to Menti.com and enter the 6 digit code for the polling question](https://reader031.vdocuments.mx/reader031/viewer/2022041818/5e5c17987f190b798d5665c1/html5/thumbnails/18.jpg)
18
What do you like least about assessment?
Go to Menti.com and click on “vote on another question.”
Enter the code 186360.
Type in up to three responses.
![Page 19: › 2018 › 07 › 25141859 › 2018... · Responding to Open-Ended Polls1 Using your smartphone or computer, go to Menti.com and enter the 6 digit code for the polling question](https://reader031.vdocuments.mx/reader031/viewer/2022041818/5e5c17987f190b798d5665c1/html5/thumbnails/19.jpg)
19
Watermark(2016) participants were asked what they disliked the most about assessment.
Three primary themes emerged from the data:
Issues with the methodological aspectsIssues with the need to persuade/teach others about assessment
Questions about the value of assessment itself
What Assessment Professionals Like Least
![Page 20: › 2018 › 07 › 25141859 › 2018... · Responding to Open-Ended Polls1 Using your smartphone or computer, go to Menti.com and enter the 6 digit code for the polling question](https://reader031.vdocuments.mx/reader031/viewer/2022041818/5e5c17987f190b798d5665c1/html5/thumbnails/20.jpg)
20
Issues with Methodology
“Sometimes assessment feels cumbersome, especially if we're starting from scratch or analyzing large data sets.”
“Dealing with ignorant and naive assessment boosters. I'm constantly amazed at how far folks with very little fundamental understanding of measurement, data analysis, organizations, and teaching get in this field.”
![Page 21: › 2018 › 07 › 25141859 › 2018... · Responding to Open-Ended Polls1 Using your smartphone or computer, go to Menti.com and enter the 6 digit code for the polling question](https://reader031.vdocuments.mx/reader031/viewer/2022041818/5e5c17987f190b798d5665c1/html5/thumbnails/21.jpg)
21
Need to Teach/Persuade Others
“Convincing other stakeholders involved in data gathering of the value of investing both time and money into the process.”
“My biggest frustrations are with assessment creators who don't understand how to build a good assessment, don't appreciate the information that can be derived from a well-constructed assessment and don't care to learn.”
![Page 22: › 2018 › 07 › 25141859 › 2018... · Responding to Open-Ended Polls1 Using your smartphone or computer, go to Menti.com and enter the 6 digit code for the polling question](https://reader031.vdocuments.mx/reader031/viewer/2022041818/5e5c17987f190b798d5665c1/html5/thumbnails/22.jpg)
22
Value of Assessment
“It is time-consuming and we don't have enough personnel to do it well all the time.”
“Assessment often turns into an additional layer of work that reveals only what one already knows about a program.”
“I least enjoy the pressure to quantify things that can't be quantified — the most important elements of any education — or the way assessment puts at the forefront things that can be measured as more important than things that cannot.”
![Page 23: › 2018 › 07 › 25141859 › 2018... · Responding to Open-Ended Polls1 Using your smartphone or computer, go to Menti.com and enter the 6 digit code for the polling question](https://reader031.vdocuments.mx/reader031/viewer/2022041818/5e5c17987f190b798d5665c1/html5/thumbnails/23.jpg)
23
Assessment Roles
and Activities
![Page 24: › 2018 › 07 › 25141859 › 2018... · Responding to Open-Ended Polls1 Using your smartphone or computer, go to Menti.com and enter the 6 digit code for the polling question](https://reader031.vdocuments.mx/reader031/viewer/2022041818/5e5c17987f190b798d5665c1/html5/thumbnails/24.jpg)
24
Assessment professionals who completed the University of Kentucky survey
(2015) were asked how much time they spent on assessment related work:
• 100% of their time (26%)
• 75-99% of their time (22%)
• 50-74% of their time (22%)
• 25-49% of their time (21%)
• <25% of their time (9%)
Assessment Professionals’ Activities
![Page 25: › 2018 › 07 › 25141859 › 2018... · Responding to Open-Ended Polls1 Using your smartphone or computer, go to Menti.com and enter the 6 digit code for the polling question](https://reader031.vdocuments.mx/reader031/viewer/2022041818/5e5c17987f190b798d5665c1/html5/thumbnails/25.jpg)
25
Assessment professionals who completed the University of Kentucky survey
(2015) were asked how much time they spent on assessment related work:
• 100% of their time (26%)
• 75-99% of their time (22%)
• 50-74% of their time (22%)
• 25-49% of their time (21%)
• <25% of their time (9%)
Assessment Professionals’ Activities
![Page 26: › 2018 › 07 › 25141859 › 2018... · Responding to Open-Ended Polls1 Using your smartphone or computer, go to Menti.com and enter the 6 digit code for the polling question](https://reader031.vdocuments.mx/reader031/viewer/2022041818/5e5c17987f190b798d5665c1/html5/thumbnails/26.jpg)
26
Assessment Professionals’ Activities
![Page 27: › 2018 › 07 › 25141859 › 2018... · Responding to Open-Ended Polls1 Using your smartphone or computer, go to Menti.com and enter the 6 digit code for the polling question](https://reader031.vdocuments.mx/reader031/viewer/2022041818/5e5c17987f190b798d5665c1/html5/thumbnails/27.jpg)
27
Assessment Professionals’ Roles
Assessment Professional
Assessment / Method Expert
Project Manager
Narrator/ Translator
Facilitator/ Guide <-> Political
Navigator
Change Agent
Jankowski & Slotnick (2015) Framework Watermark & AALHE Revised Framework
Project Manager
![Page 28: › 2018 › 07 › 25141859 › 2018... · Responding to Open-Ended Polls1 Using your smartphone or computer, go to Menti.com and enter the 6 digit code for the polling question](https://reader031.vdocuments.mx/reader031/viewer/2022041818/5e5c17987f190b798d5665c1/html5/thumbnails/28.jpg)
28
Assessment Professionals’ Roles
Assessment/Method Expert
• authentic assessments• curriculum mapping• data analysis• data cleaning• designing scoring protocols• determining the best indicators• qualitative analysis• quantitative analysis• reliability and validity issues• survey and assessment development• writing/developing rubrics
![Page 29: › 2018 › 07 › 25141859 › 2018... · Responding to Open-Ended Polls1 Using your smartphone or computer, go to Menti.com and enter the 6 digit code for the polling question](https://reader031.vdocuments.mx/reader031/viewer/2022041818/5e5c17987f190b798d5665c1/html5/thumbnails/29.jpg)
29
Assessment Professionals’ Roles
Narrator/Translator
• finding meaning within the data and communicating results• presenting data in visually-appealing or reader-friendly formats• providing a “big picture” perspective to institutional colleagues• support for evidence-based decision-making
![Page 30: › 2018 › 07 › 25141859 › 2018... · Responding to Open-Ended Polls1 Using your smartphone or computer, go to Menti.com and enter the 6 digit code for the polling question](https://reader031.vdocuments.mx/reader031/viewer/2022041818/5e5c17987f190b798d5665c1/html5/thumbnails/30.jpg)
30
Assessment Professionals’ Roles
Facilitator/Guide Political Navigator
• assisting faculty with improvement efforts• helping and mentoring others• interdisciplinary collaboration • managing accreditation• obtaining buy-in • overcoming resistance and obstruction
![Page 31: › 2018 › 07 › 25141859 › 2018... · Responding to Open-Ended Polls1 Using your smartphone or computer, go to Menti.com and enter the 6 digit code for the polling question](https://reader031.vdocuments.mx/reader031/viewer/2022041818/5e5c17987f190b798d5665c1/html5/thumbnails/31.jpg)
31
Assessment Professionals’ Roles
Change Agent
• active participation (not only provider but also end-user of data) • closing the loop • designing change• making a difference• redesign• reflection• responding to identified weaknesses• utilizing results
![Page 32: › 2018 › 07 › 25141859 › 2018... · Responding to Open-Ended Polls1 Using your smartphone or computer, go to Menti.com and enter the 6 digit code for the polling question](https://reader031.vdocuments.mx/reader031/viewer/2022041818/5e5c17987f190b798d5665c1/html5/thumbnails/32.jpg)
32
Assessment Professionals’ Roles
Project Manager
• overseeing the collecting, entry, and management of data• designing data gathering techniques• designing protocols• enforcing deadlines• finalizing analysis plans• follow-ups/reminders• managing scoring processes• organizing and archiving data• running reports• working with software
![Page 33: › 2018 › 07 › 25141859 › 2018... · Responding to Open-Ended Polls1 Using your smartphone or computer, go to Menti.com and enter the 6 digit code for the polling question](https://reader031.vdocuments.mx/reader031/viewer/2022041818/5e5c17987f190b798d5665c1/html5/thumbnails/33.jpg)
33
Professional
Development Needs
![Page 34: › 2018 › 07 › 25141859 › 2018... · Responding to Open-Ended Polls1 Using your smartphone or computer, go to Menti.com and enter the 6 digit code for the polling question](https://reader031.vdocuments.mx/reader031/viewer/2022041818/5e5c17987f190b798d5665c1/html5/thumbnails/34.jpg)
34
What areas/topics do you needprofessional development in?
Go to Menti.com and click on “vote on another question.” Enter the code 865232.
Type in up to three responses.
![Page 35: › 2018 › 07 › 25141859 › 2018... · Responding to Open-Ended Polls1 Using your smartphone or computer, go to Menti.com and enter the 6 digit code for the polling question](https://reader031.vdocuments.mx/reader031/viewer/2022041818/5e5c17987f190b798d5665c1/html5/thumbnails/35.jpg)
35
PD Method Preference (UKY Survey)
![Page 36: › 2018 › 07 › 25141859 › 2018... · Responding to Open-Ended Polls1 Using your smartphone or computer, go to Menti.com and enter the 6 digit code for the polling question](https://reader031.vdocuments.mx/reader031/viewer/2022041818/5e5c17987f190b798d5665c1/html5/thumbnails/36.jpg)
36
UKY (2015) survey were asked what professional development topics they would like to see offered:
staffing and resources
“managing a one-person office”
best practices in assessment
“research on what constitutes effective and meaningful practice in assessment.”
learning outcomes and co-curricular assessment
student affairs, general education, program evaluation
Professional Development Needs
![Page 37: › 2018 › 07 › 25141859 › 2018... · Responding to Open-Ended Polls1 Using your smartphone or computer, go to Menti.com and enter the 6 digit code for the polling question](https://reader031.vdocuments.mx/reader031/viewer/2022041818/5e5c17987f190b798d5665c1/html5/thumbnails/37.jpg)
37
UKY (2015) survey were asked what professional development topics they would like to see offered:
assessment tools
“adapting and writing rubrics”
collaboration
“more sharing of techniques and tips between institutions”
using data for change and/or strategic planning
“Now [that] we have the data...how do we enable change?”
Professional Development Needs
![Page 38: › 2018 › 07 › 25141859 › 2018... · Responding to Open-Ended Polls1 Using your smartphone or computer, go to Menti.com and enter the 6 digit code for the polling question](https://reader031.vdocuments.mx/reader031/viewer/2022041818/5e5c17987f190b798d5665c1/html5/thumbnails/38.jpg)
38
Respondents to the Watermark survey were asked what professional development topics they were interested in from a menu of nine topics:
• Respondents favored professional development topics that were practical in nature and/or those that would help them advance the use of assessment data in their institutions.
• Analyzing and interpreting data
• Documenting and reporting assessment results
Professional Development Needs
![Page 39: › 2018 › 07 › 25141859 › 2018... · Responding to Open-Ended Polls1 Using your smartphone or computer, go to Menti.com and enter the 6 digit code for the polling question](https://reader031.vdocuments.mx/reader031/viewer/2022041818/5e5c17987f190b798d5665c1/html5/thumbnails/39.jpg)
39
Conclusion and Recommendations
• Roles of assessment professionals in higher education vary widely and are evolving.
• What began as data collection for accreditation is evolving to change agent for organizational
improvement.
• Need flexible and effective approaches to professional development.
• Focus should be on equipping assessment professionals to improve teaching and learning.
![Page 40: › 2018 › 07 › 25141859 › 2018... · Responding to Open-Ended Polls1 Using your smartphone or computer, go to Menti.com and enter the 6 digit code for the polling question](https://reader031.vdocuments.mx/reader031/viewer/2022041818/5e5c17987f190b798d5665c1/html5/thumbnails/40.jpg)
40
Make sure to read the full report and a new infographic!
Download at watermarkinsights.com/aalhe-report