a utilization-focused approach to developing, mapping, and ... · based on utilization-focused...
TRANSCRIPT
©Yukiko Watanabe, John Davis, and John Norris Foreign Language Program Evaluation Project National Foreign Language Resource Center,
University of Hawaii at Manoa, 2012
A utilization-focused approach to developing, mapping, and assessing student learning outcomes in college
foreign language programs
1 of 22
Module V
Reflec&ng on assessment
Implemen&ng
U&lizing Planning
Situa&ng
Reflec&ng
Mapping SLOs
Developing SLOs
2 of 22
About Module V The goal of outcomes assessment is to enable academic programs to seek out, understand, and provide optimal conditions for student learning. A continual cycle of assessment inquiry and action calls for college faculty to take collective responsibility for educational effectiveness as part and parcel of what they do. To maximize the contributions of assessment, it is therefore essential to reflect on and improve associated practices as they are designed, implemented, and acted upon. Module V takes you through the steps of reflecting on and evaluating the assessment process. Note that the reflection phase is depicted at the center of the assessment cycle—although “reflection” is described here at the end of the modules, is not intended to be only a summative evaluation activity, but also a formative process that informs every step along the way towards useful assessment.
3 of 22
How to use the module Because outcomes assessment should be a collective endeavor, we encourage you to go through this module together with your colleagues. The module includes tasks and resources that will help facilitate your outcomes assessment planning.
à Work on the questions with your colleagues.
à If you see this icon, access the resources for further information and examples by clicking on the link provided.
4 of 22
The goal of reflection is to improve our practices by examining the ways in which assessment is conducted. Recall the Standards of Practice we outlined in Module II (i.e., Utility, Accuracy, Feasibility, Propriety, and Accountability). These standards provide a benchmark for sound assessment practice, and should guide the decisions, execution, and evaluation of assessment. Going back to the basic premise of utilization-focused outcomes assessment, the assessment quality in our framework emphasizes actual uses of assessment, that is, whether the plan and process supported intended use and whether the assessment was actually used as intended.
Reflec&ng Reflec&ng on the assessment path
5 of 22
The next few slides list some key characteristics of utilization-focused assessment for reflection. The statements are formed based on utilization-focused evaluation premises (Patton, 2012), a standards-based meta-evaluation checklist created by Stufflebeam (1999), and our experiences facilitating assessment in various college foreign language programs.
Reflec&ng on the assessment path
Reflec&ng
6 of 22
Reflecting: Situating & Planning
Situa-ng assessment Not at all
A liBle
Some-‐what
Verymuch
N/A
There was adequate representa&on of primary intended users and/or key stakeholders in the assessment working group. The assessment schedule was realis&c (mee&ngs, deadlines, etc.). Time commitment of those involved was respected.
Assessment working group members were clear about their responsibili&es. Assessment working group members followed through with their responsibili&es. Different posi&ons and percep&ons of different interest groups were respected. There was a consensus on the assessment approach and procedures. Opportuni&es to learn about outcomes assessment were provided throughout the process for those who needed them.
Judge to what extent the following were accomplished.
Reflec&ng
7 of 22
Reflecting: Situating & Planning U&lizing
Situa-ng assessment Not at all
A liBle
Some-‐what
Very much
N/A
Key stakeholders were involved in iden&fying assessment needs. Factors that may hinder assessment use were iden&fied from the beginning. The assessment commiBee strategized to overcome assessment barriers. Planning assessment Not
at all A
liBle Some-‐what
Very much
N/A
Assessment planning was transparent. Faculty were kept informed about assessment updates throughout the process. Op&ons for assessment ques&ons were solicited from various users (and were considered!). Priority was given to the most important assessment ques&ons. Assessment ques&ons were linked to intended purposes and uses. Assessment methods elicited informa&on that answered the priority assessment ques&ons.
8 of 22
Reflecting: Planning U&lizing
Planning assessment (con-nued) Not at all
A liBle
Some-‐what
Very much
N/A
Selected methods were chosen on the basis of considering strength and weaknesses of various methods. The selec&on of methods factored in the &me and resource constrains of the program context. Instruments went through pilot-‐tes&ng or careful review to eliminate any poten&al biases, errors, and irrelevances. Analysis and interpreta&on procedures assured credibility. Poten&al findings were an&cipated to improve assessment design.
9 of 22
Reflecting: Implementing U&lizing
Implemen-ng assessment Not at all
A liBle
Some-‐what
Very much
N/A
The assessment design was adapted to the changing condi&ons of the context and needs. Sufficient informa&on was obtained to address the assessment ques&on. There was a person in charge of data management. Data was managed in a secure manner to ensure data quality and ethics. Data collec&on ensured par&cipants’ privacy rights. The primary intended users were involved in interpre&ng findings. Procedures for data analysis and interpreta&on were monitored and recorded. Factors affec&ng data quality, analysis, and interpreta&on were monitored and recorded. Limita&ons to data interpreta&ons (due to design or unexpected factors affec&ng data gathering) were acknowledged. Consistency of scoring, categoriza&on, and coding was assured.
10 of 22
Reflecting: Use U&lizing
U-lizing assessment Not at all
A liBle
Some-‐what
Very much
N/A
The primary intended users were ac&vely involved in making judgments based on assessment findings. The primary intended users were ac&vely involved in genera&ng recommenda&ons. Nega&ve findings were turned into construc&ve feedback. Recommenda&ons considered contextual constraints. Reports were delivered in a &mely manner to be u&lized for decision-‐making. Repor&ng was clear and understandable to the audience(s). The format of repor&ng was appropriate and maximized findings use. Ac&on-‐planning was feasible to those who are involved in improving and developing the program. Overall, assessment findings were actually used as intended.
11 of 22
Reflecting: Use U&lizing
Consider whether there were any unintended uses or even misuses of assessment. If so, how would you avoid these and their potential negative consequences next time?
Unintended uses? ��� Misuses? Strategies for avoiding unintended uses and negative consequences?
12 of 22
Conclusion
The rest of the slides provide additional useful resources on outcomes assessment. Module V suggested a list of questions to examine during and at the end of the assessment cycle to improve future assessment practices. Examples of good and reflective assessment practices are much needed in the field of foreign language education, and we hope you will consider documenting and disseminating what worked and what didn’t work in your context. ���
Reflecting: Useful assessment Reflec&ng
13 of 22
We welcome any feedback from you on our Outcomes Assessment Modules. Please leave us some feedback about this module by accessing a
short online questionnaire here: ���[INSERT URL]
14 of 22
Look for examples: Duke University, Evansville University,
Georgetown University
Key Resources
15 of 22
• Brindley, G. (2001). Outcomes-based assessment in practice: Some examples and emerging insights. Language Testing, 18(4), 393-407.
• Liskin-Gasparro, J. E. (1995). Practical approaches to outcomes assessment: The undergraduate major in foreign languages and literatures. ADFL Bulletin, 26(2), 21-27.
• Brown, J. D., & Hudson, T. (1998). The alternatives in language assessment. TESOL Quarterly, 32(4), 653-675.
• Byrnes, H. (2002). The role of task and task-based assessment in a content-oriented collegiate foreign language curriculum. Language Testing, 19(4), 419-437.
• Dassier, J., & Powell, W. (2001). Formative FL program evaluation: Dare to find out how good you really are. In Dimension 2001: The odyssey continues. Selected proceedings of the 2001 conference on language teaching (pp. 15–30). Birmingham, AL: Southeast Council on Language Teaching.
Resources: SLO assessment in language education
16 of 22
• Delett, J. S., Barnhardt, S., & Kevorkian, J. A. (2001). A framework for portfolio assessment in the foreign language classroom. Foreign Language Annals, 34(6), 559-568.
• Mathews, T., J., & Hansen, C., M. (2004). Ongoing assessment of a university foreign language program. Foreign Language Annals, 37(4), 630-640.
• Morris, M. (2006). Addressing the challenges of program evaluation: One department’s experience after two years. Modern Language Journal, 90(4), 599-602.
• Norris, J. M. (2006). The why (and how) of student learning outcomes assessment in college FL education. Modern Language Journal, 90(4), 590-597.
• Norris, J. M., & Pfeiffer, P. (2003). Exploring the use and usefulness of ACTFL Guidelines oral proficiency ratings in college foreign language departments. Foreign Language Annals, 36(4), 572-581.
• Phillips, J. (2006). Assessment now and into the future. In A. Heining-Boynton (Ed.), 2005-2015: Realizing our vision of languages for all (pp. 75-103). Upper Saddle River, NJ: Pearson.
Resources: SLO assessment in language education
17 of 22
• Allen, M. J. (2004). Assessing academic programs in higher education. Bolton, MA: Anker.
• Bresciani, M. J., Zelna, C. L., & Anderson, J. A. (2004). Techniques for assessing student learning and development: A handbook for practitioners. Washington, DC: NASPA.
• Heiland, D., & Laura, J. R. (Eds.). Literary study, measurement, and the sublime: Disciplinary assessment. New York: The Teagle Foundation.
• Hernon, P., & Dugan, R. E. (eds.) (2004). Outcomes assessment in higher education: Views and perspectives. Westport, CT: Libraries Unlimited.
• Knight, P. (Ed.). (1995). Assessment for learning in higher education. London: Kogan Page.
• Maki, P. L. (2004). Assessing for learning: Building a sustainable commitment across the institution. Sterling, VA: Stylus.
• Suskie, L. (2009). Assessing student learning: A common sense guide (2nd ed.). San Francisco, CA: Jossey-Bass.
• Walvoord, B. A. (2004). Assessment clear and simple: A practical guide for institutions, departments and general education. San Francisco, CA: Jossey-Bass.
• Wiggins, G. (1998). Educative assessment: Designing assessments to inform and improve student performance. San Francisco: Jossey-Bass.
Resources: SLO Assessment in tertiary education
18 of 22
References • American Association for Higher Education (1992). Nine principles of good practice
for assessing student learning. Sterling, VA: Stylus. • Angelo, T. A. (1995). Reassessing (and Defining) Assessment. American Association
for Higher Education (AAHE) Bulletin, 48(2), 7-9. • Banta, T. W. (2002). Characteristics of effective outcomes assessment. In T. W. Banta
(Ed.), Building a scholarship of assessment (pp. 261-283). San Francisco, CA: Jossey-Bass.
• Grau Sempere, A., Mohn, M. C., & Pieroni, R. (2009). Improving educational effectiveness and promoting internal and external information-sharing through student learning outcomes assessment. In J. M. Norris, J. McE. Davis, C. Sinicrope, & Y. Watanabe (eds.), Toward useful program evaluation in college foreign language education (pp. 139-162). Honolulu: University of Hawaii, National Foreign Language Resource Center.
• Joint Committee on Standards for Educational Evaluation. (1994). The program evaluation standards (2nd ed.). Newbury Park, CA: Sage.
19 of 22
References • Mackay, R., Wellesley, S., Tasman, D., & Bazergan, E. (1998). Using institutional self-
evaluation to promote the quality of language and communication training programmes. In P. Rea-Dickens & K. P. Germaine (Eds.), Managing evaluation and innovation in language teaching: Building bridges (pp. 111-131). London: Longman.
• Norris, J. M. (2006). The why (and how) of student learning outcomes assessment in college FL education. Modern Language Journal, 90(4), 590-597.
• Patton, M. Q. (2008). Utilization-focused evaluation (4th ed). Thousand Oaks, CA: Sage. • Patton, M. Q. (2012). Essentials of utilization-focused evaluation. Thousand Oaks, CA:
Sage. • Stufflebeam, D. L. (1999). Program evaluations metaevaluation checklist. Retrieved
from http://www.wmich.edu/evalctr/archive_checklists/program_metaeval_10point.pdf
• Suskie, L. (2004). Assessing student learning. Bolton, MA: Anker.
20 of 22
References • Walther, I. (2009). Developing and implementing an evaluation of the foreign
language requirement at Duke University. In J. M. Norris, J. McE. Davis, C. Sinicrope, & Y. Watanabe (eds.), Toward useful program evaluation in college foreign language education (pp. 117-138). Honolulu, HI: University of Hawaii, National Foreign Language Resource Center.
• Watanabe, Y., Norris, J. M., & González-Lloret, M. (2009). Identifying and responding to evaluation needs in college foreign language programs. In J. M. Norris, J. McE. Davis, C. Sinicrope, & Y. Watanabe (Eds.), Toward useful program evaluation in college foreign language education (pp. 5–56). Honolulu: University of Hawai‘i, National Foreign Language Resource Center.
• Wright, B. D. (2006). Learning languages and the language of learning. Modern Language Journal, 90(4), 593-596.
21 of 22
http://www.nflrc.hawaii.edu/evaluation
Visit our website
22 of 22