cultivating collaborative leaders with a facilitative approach to … · 2019. 9. 17. ·...

Assessment Office Crawford Hall 230 & 231 2550 Campus Road, Honolulu, HI 96822

(808) 956-4283 (808) 956-6669

manoa.hawaii.edu/assessment

Cultivating Collaborative Leaders with a Facilitative Approach to Assessment Pre-conference workshop at the Assessment Institute, October 21, 2018, Indianapolis, IN. Facilitators Yao Hill, University of Hawai‘i at Mānoa, [email protected] Monica Stitt-Bergh, University of Hawai‘i at Mānoa, [email protected] Assessment Office website: http://www.manoa.hawaii.edu/assessment/ Assessment Leadership Institute website: http://www.manoa.hawaii.edu/assessment/institute/ Packet for workshop attendees

1. PowerPoint presentation slides and speaker notes (pp. 2-53)

2. Handouts (pp. 54-70)

a. Agenda, key points, activity worksheets

b. Facilitation resources and sources

c. Agenda: 2018 Assessment Leadership Institute

d. Tips: Facilitating meetings

e. Curriculum map example and role-play activity

f. Poster exhibit program (list of presenters; exhibit features)

3. Notes taken during the workshop (p. 71)

4. Photos from the workshop (p. 73)

5. Workshop evaluation form (p. 75)

mailto:[email protected]

http://www.manoa.hawaii.edu/assessment/

http://www.manoa.hawaii.edu/assessment/institute/


(808) 956-4283 (808) 956-6669


You may use these materials only for nonprofit educational purposes. Please give credit/cite appropriately. Hill, Y. Z. & Stitt-Bergh, M. (2018). Cultivating collaborative leaders with a facilitative approach to assessment. Assessment Institute, Indianapolis, IN.

Cultivating Collaborative Leaders with a Facilitative Approach to Assessment Pre-conference workshop at the Assessment Institute, Sunday, October 21, 2018, Indianapolis, IN.

We describe a model to cultivate collaborative assessment leaders who use facilitative approaches to the assessment process. The model consists of (a) multi-day assessment training for faculty/staff that includes building collaborative and facilitative skills, (b) long-term follow-up support, and (c) a campus showcase of projects. A key aspect is active learning techniques: faculty/staff learn and practice collaboration and facilitation skills. Participants will learn theoretical foundations, practice assessment facilitation activities, and adapt strategies for their own use. They will plan an assessment leadership development project for their campus that includes a facilitative approach to the assessment process. [Sponsored by AALHE]

Facilitators: Yao Zhang Hill and Monica Stitt-Bergh

Slide Presentations

Slide 1

Facilitation and CollaborationWhat, Why, and How

University of Hawai‘i at Mānoa Crawford 230 & 231 Assessment Office manoa.hawaii.edu/assessment


Slide 2

This session is for you to be able to:

Explain the importance of facilitation & collaboration skills for assessment leaders

Slide 3

What

Let’s start by defining what are facilitation and collaboration.



Slide 4

Group facilitation in assessment

It is interesting that several publications on the topic of facilitation acknowledges the difficulty to give “facilitation” a one single definition because facilitation skills encompass a constellation of different elements. Here, I present a definition of group facilitation in assessment that corresponds to my experience. Group facilitation in assessment are interactive activities that Elicit and harness the intellectual capital and goodwill in members of a group Guide the group in reaching a shared understanding of the program expectations, quality, and coherence; and Navigate the group to reach collaborative programmatic decisions



Slide 5

Group facilitation in assessmentElicit & harness intellect & goodwill

Slide 6

Group facilitation in assessmentGuide shared understanding/vision



Slide 7

Group facilitation in assessmentNavigate to group decisions

Slide 8

Collaboration

Build individual relationships

Move the group forward

Collaboration skills are the skills that bring people together. It has two dimensions, working together with individuals and guiding individuals to work together in groups to move the group forward.



Slide 9

Why

Why are facilitation and collaborative skills important? Slide 10

Facilitation & collaborative skills engage faculty in collaborative inquiry and action that moves assessment forward



Slide 11

Program learning assessment

A set of tools for faculty to collaboratively reflect on program quality and coherence to act on learning evidence for program improvement

This is our view of program learning assessment. Slide 12

Assessment is a process of collective inquirythat involves nested discussions, decisions, and action

Image fromhttps://images-na.ssl-images-amazon.com/images/I/41U-E4LsXlL._SX384_BO1,204,203,200_.jpg

Peggy Maki and quite a few of other thought-leaders in learning assessment acknowledge that assessment is a process of collective inquiry.



Slide 13

Facilitator/guide is one of the essential roles of an assessment professional

Natasha Jankowski Ruth Slotnick

Natasha and Ruth examined the job descriptions of assessment professionals. They interviewed thought-leaders in learning assessment. Their research lead to their conclusion that facilitator/guide is one of the five essential roles of an assessment professional. Natasha’s image is from: https://cdn-e.education.illinois.edu/faculty/portraits/njankow2.jpg Ruth’s image is from: https://media.licdn.com/dms/image/C5603AQEK_r-eB64OQw/profile-displayphoto-shrink_800_800/0?e=1545868800&v=beta&t=EKs3qJM1ngSEyGDqvmYsZmEXgjhLVkjebKegMtG7mZI



Slide 14

The broader field of evaluation recognizes the importance of the facilitation skills

The broader field of evaluation has a more established understanding of the importance of facilitation skills in evaluation work. In 2016, New Direction for Evaluation dedicated one issue on evaluation and facilitation. Slide 15

Jean King and her colleague in this Interactive Evaluation Practice book described concrete facilitation techniques and when they can be used in pages 101-147. Image source: https://images-na.ssl-images-amazon.com/images/I/51zNRwnFIFL._SX402_BO1,204,203,200_.jpg



Slide 16

Assessment is a device for change.Change moves at the speed of trust.

George KuhFounding director of NILOA

Facilitation can move the group forward. However, if there is no trust, people may not come to a decision, or worse, even if there is a decision, no one will carry it out. Therefore, building trust would be a pre-requisite for successful facilitation and for people to follow up with action plans generated during the facilitation session. George Kuh said in 2016 Assessment Institute that assessment is a device for change. Change moves at the speed of trust.



Slide 17

Active listening and relationship building

Group management

Build trust and make change through

To build trust, assessment leaders can use collaboration skills in terms of building personal relationships, using strategies such as active listening, understanding group dynamics and use strategies to move the group forward. Slide 18

How

Now that we explained what facilitation and collaboration skills are and why they are important, I will introduce some concrete strategies that we have used to train collaborative assessment leaders.



Slide 19

Assessment Leadership Institute (ALI) at UH Manoa

At UH Mānoa, the Assessment Office hosts the Assessment Leadership Institute every year since 2013. Participants go through a three-day training. Slide 20

Assessment Facilitation

Two foci of ALI

The training has two foci: assessment knowledge and skills and facilitation skills.



Slide 21

Learning Outcomes

Learning Opportunities

(Curri.Map)

Collection & Analysis of Evidence

Assessment Results

Improvement Plan

We organize our training topics around the steps in the assessment cycle and we provide opportunities for the participants to facilitate the assessment process in many of the steps.



Slide 22

SLO Development (Snow card)

In one ALI, we asked the participants to create program learning outcomes using a variation of the “snow card” technique. Snow card facilitation technique elicits participants’ input in a sheet of paper or half a sheet of paper. Participants post the sheets on the wall for all the participants to see. When there are many cards on the wall, participants can collaboratively classify the cards into different categories and give each category a label. In this picture, ALI participants were acting as the faculty members in the psychology department. They are developing their program specific learning outcomes that address critical thinking. Monica is the note-taker in this activity.



Slide 23

Curriculum Map Development (Sticky-notes

voting)

In the curriculum map facilitation activity, participants act as the psychology department faculty members, each teaching one or more courses. They use the “sticky-notes voting” technique to indicate how their course(s) address one or more of the program learning outcomes.

Slide 24

This picture shows how two ALI alumni used this technique in facilitating curriculum mapping for a newly merged undergraduate program.



Slide 25

Rubric Adaptation

(Score-Discuss-Rescore)

To facilitate rubric adaptation, we use the process of score-discuss-rescore. Faculty first read and score student work, discuss the rubric criteria or other criteria that influence their decision. After the discussion has saturated, faculty re-score the student papers based on the shared understanding of the rubric. Slide 26

Student 4 3 2 1

A 8 9 2 1 0 0B

This table shows that faculty are scoring two student papers: Paper A and Paper B. The column headers are rubric scores: 4, 3, 2, and 1. At the first round of scoring, 8 faculty members gave Paper A a score of 4. In the second round, 9 faculty scored the paper as 4.



Slide 27

Standard Setting (Bar graphing)

In the standard setting activity, we ask participants to write down their choice of score that indicate minimal competency. Participants put it on the chart as a bar graph. Faculty discuss their reasons and do another voting in Round 2. Slide 28

This picture shows Round 2 scores. The pink note: 2.4 shows the average score, which will serve as the score that represents the standard for minimum competency, or minimal satisfactory performance.



Slide 29

Creating program vision (making metaphor)

In the last facilitation activity in our institute, we ask participants to pick a picture to complete a metaphor sentence, such as: My vision of the program is like ___________, because _____________________________________________. The ideal student is like ___________, because _____________________________________________. Slide 30

Each participant then takes a turn to describe the picture that resembles their vision.



Slide 31

Collaboration skills

Just now, I described several facilitation tasks that we have participants carry out in the ALI. Collaboration skills is another important skill that we train the leaders in the ALI. We gave a lecture on the strategies that our colleagues tried and worked. Then we ask them to brainstorm strategies that have worked for them.



Slide 32

Build Personal Relationships• Knock on the door• Show genuine interest in faculty as a person• Use individualized short email with plain

language (cc someone important too)• Allow faculty to describe their frustration• Listen actively

We are all social beings and often desire to build relationship around us. To build that relationship, we can: • Knock on the door

• An assessment leader visited and talked to each of the senior faculty members before she brings the program merger idea to the whole faculty.

• Another leader visited individual faculty members’ office, summarizing their conversation notes, and remind the faculty what they have promised to do in a later email.

• In fact, knocking on individual’s doors is a strategy that almost every ALI participants have used in the process to carry out their assessment projects.

2. Show genuine interest in them as a person and someone that you respect. Small talks help. It’s okay to talk about their children, dogs, recent publication, awards, tenure, and so on. Learning about faculty’s expertise can be a rewarding experience. People love to offer advice on the area of their expertise. For example, a geophysicist advised me that it’s okay to visit Big Island when there are active volcano explosions going on. 3. Use individualized email and write short emails. It is a tendency for assessment leaders in ALI to write long emails to faculty member immediately after they complete the training. They want their faculty to know everything that they have learned in one email. We have to reminder them to avoid that temptation—only include one topic in one email. 4. Allow faculty to let out their frustration. The challenge that many faculty face in assessment is real: lack of time, lack of knowledge, frustration with colleagues., and so on. Let it out. Empty the emotional baggage before getting into the business. 5. The most important skill is active listening.



Slide 33

Active listening

Non-verbal expressions

I hear you

Paraphrase/Restatement

Focus future work on how it support students.

1. Use non-verbal posture to show that you are interested: leaning forward, keeping eye contact, taking notes, nodding. 2. Paraphrase/Restate to validate what they have said. Acknowledge and empathize with their feelings. 3. Leverage faculty existing work. Focus future work on how it supports students.

Slide 34

Validating others’ inputWhat words to use?• I hear you• I hear what you are saying• You must have felt• That shows you have a deep understanding of…• I can see that you care about…• It is not a surprise that you feel that way• That is stressful/frustrating/discouraging



Slide 35

Engage individual faculty membersMoving the group forward

After we described and brainstormed ways to engage individual faculty members, we provide the strategies to move the group forward. Slide 36

Work with willing and ready first

Yeah, let’s do

this!

If I understand it and believes that it is truly meaningful, I

will do it.Nah

For example, work with willing and ready first. Work with those who already function as the change agents and move those in the middle forward.



Slide 37

Use committee with clearly divided roles

Small steps each with a product

Finish a (draft)

product at each meeting

Slide 38

Build scholarship in a team

At a research university like ours, faculty greatly value scholarship. Inviting faculty to co-present and co-publish is a very effective way to get key players involved. This picture shows a senior faculty engaged all junior faculty members in an assessment project. She helps them present their work in the campus Assessment Poster exhibit. Later, she also led the group to publish an article on their ePortfolio system.



Slide 39

To scaffold and reinforce

1. Design facilitation simulation tasks2. Develop facilitation scripts3. Individual training beforehand4. During simulation: explain the task + reflection

afterward5. Reinforce techniques throughout6. Model techniques7. Use a variety of techniques

Just now, I described the major facilitation and collaboration strategies that we train our assessment leaders in the ALI. However, facilitation is not easy. We use these strategies to scaffold leaders in enhancing their facilitation skills • We designed 16 facilitation/notetaking tasks. So, each participant will have one or two

opportunities to practice facilitation in the ALI. • We developed the scripts for the leaders to rehearse their facilitation process. • I met with each leader for at least 30 minutes before the ALI to go over the task that they

are assigned • When it’s time for a leader to facilitate their assigned task, we first go over the task with all

the participants, a reminder to the practicing facilitator what he/she needs to go through. After each facilitation session, we ask the participants to reflect what have gone well.

• Before each task, we also repeat the facilitation techniques, such as preparing an agenda, having ground rules, making contribution visible, and so on.

• Monica and I do a role play at the very beginning of the institute to model validating and re-directing, which we are going to demonstrate shortly.

• We use different facilitation techniques so that participants have a taste and sense of how different facilitation may go.



Slide 40

Expand & practice facilitation

• Seek training• Check resources on the handout

Those are the scaffolding strategies for the assessment leaders that you train. What about enhancing facilitation strategies yourself. First is to seek training. Monica and I went through multiple facilitation training. For you, check out whether there is a facilitation/mediation course in the department of communication. Ask around on campus. Check out your local evaluation associations and their events and training. The handout lists resources and describes what they are. Slide 41

Meeting facilitation tipsDemonstration + Handout

Handout Page 4-5



Slide 42

Planning an Assessment Leadership Building Project

Monica’s and my approach to building assessment leaders is through capacity-building and professional development of faculty members who can lead collaborative inquiry and action in assessment work



Slide 43

Effective professional development practices• Significant amount• Engage faculty in active learning• Product oriented• Provide follow-up support• Provide opportunities to collaborate

and peer feedback• Meaningful incentives

These effective strategies were evidenced in the literature and in the projects that we observe. • Significant amount: In a meta-analysis of over 1000 studies that investigated the effect of

faculty professional development on student learning, the authors found that any training less than 14 hours had no impact on student learning. This guided our decision to design our ALI as 3-day training.

• Lectures and theories do not shape practice unless practitioners are actively applying learning in practice in the training. The training should allow the participants to apply the strategies immediately.

• Product oriented: Ask participants to produce a product at the end of the training, an assessment plan, a draft report, and so on. We ask our participants to give a five-minute presentation of their assessment project at the end. We dedicate 2.5 hours for the project presentations.

• Leaders when they go back to implement their projects, they often face issues specific to their context. They need guidance to apply what they learned or modify the strategies to fit their context. This requires follow-up support. We follow up with our leaders for at least one semester in support groups that has three or four leaders.

• This also give the opportunity for the leaders to learn from each other. Members of the support groups share institutional knowledge and politics, work flow strategies, reaffirming the good work that their peers are doing, thus strengthening the community of practice.

• Meaningful incentives help to transfer the external motivations to internal motivations once the leaders started practicing assessment and facilitation strategies in their own unit.



Slide 44

Professional Development Program Features

Link: http://manoa.hawaii.edu/assessment/ilo/pdmodels.html

I collected several models of effective professional development at the University of Hawai‘i at Mānoa and I found that they share these five characteristics. Slide 45

2012 and earlier PD:

One-time Workshops Individual Consultations Website/examples

Before I was hired 2012, the major form of professional development at the office is one-time workshops. It played its role in increasing assessment awareness and activities. As we re-examined our practice in light of the PD literature, we launched our Assessment Leadership Institute in 2013.



Slide 46

Assessment Leadership Institute Born in 2013

Slide 47

Components

IntensiveTraining

Follow-up Support

Scholarship Opportunities

It has three components.



Slide 48

Intentional recruitment of participantsThose who can lead collaborative inquiry and action in assessment

We were intentional in recruiting participants. We need the participants who can go back and lead faculty collaborative inquiry. This means that the person must be in power to make a change or can influence the curriculum. We are looking for someone who are: • In charge of assessment • Familiar with and have impact on curriculum, such as curriculum committee members • Demonstrate the ability to collaborate and lead faculty work Slide 49

IntensiveTraining

Support Group Mtgs


Focus on facilitation strategy training

Our training focuses on facilitation strategies as described before.



Slide 50

2017 Assessment Leadership Summer Institute

Facilitation Strategies

1. Prepare an agenda and a script

Agenda

1. Discuss the rubric2. Score Sample A3. Score Sample B4. Summary + next steps

We reinforce facilitation strategies throughout the ALI. Here you can see a review of facilitation strategies before the rubric adaption activity. Slide 51



2. Have a desired outcome and process

To revise the rubric to align with PLO through discussion and evaluation of student papers



Slide 52



3. Make contributions visible

FlipchartProjector

Slide 53


Strategies

4. Decide how to decide

Consensus Majority



Slide 54


Understand the role of the facilitator

• Create safe space• Neutral & provides guidance for the procedure• Sets the ground rules• Prepares the materials for the participants• Have someone else take notes• Summarizes input and disseminate• Signal when switching to a participant role

Slide 55

Product at the end

At the end of the Institute, each person or team present an assessment project plan.



Slide 56

Some can be interesting… Slide 57

Components

IntensiveTraining

Follow-up Support




Slide 58

3 Follow-up support group meetings

Participants choose a buddy or two to meet in the fall to report progress and gather feedback on the implementation/modification of the plan. The buddies work out a schedule to meet. The buddy system put the responsibility of finding the partner on the participants so that they would want to pay attention to what the other participants are doing. This is how our professional learning community forms. They often come together because they are in similar disciplines, like nursing and dental hygiene, or they have similar projects, like assessing Ph.D. programs.



Slide 59

Meetings are facilitated and supported

Learning communities work best when facilitated. I am the main facilitator of these meanings. I provide logistical support, like finding a room, bringing snacks, and so on. I facilitate the meeting, take notes, keep time, encourage peers to give feedback or share institutional knowledge & personal strategies. Follow up support allows me to provide just-in-time support. I provide further resources (e.g., rubrics, interview question examples , data presenting templates, use of results discussion questions) at these meetings. Participants engage in collaborative and innovative problem-solving with colleagues. After the meeting, I provide minutes and share with the whole group



Slide 60

Components

IntensiveTraining

Follow-up Support


We put a great focus on nurturing assessment scholarship among assessment leaders. Slide 61

Present at Poster Exhibit

ALI participants are required to present at the campus Assessment Poster Exhibit that we organize.



Slide 62

http://manoa.hawaii.edu/assessment/poster/index.htm

We showcase their work to leaders. We invite campus leaders to give a welcome talk. Slide 63

Campus leaders give out poster awards.



Slide 64

We record a 2-3 minutes short presentations of some posters and make them public on our website and our YouTube channel. We send these videos to the deans. University of Hawai‘i Assessment Office YouTube channel: https://www.youtube.com/channel/UCbLQVA_wN1LFD9Ie_2QmkXA We publish the posters on our website and on our university’s Scholar Space so that they are easily searchable by Google scholar. Link: https://scholarspace.manoa.hawaii.edu/handle/10125/40813



Slide 65

Foster scholarship and in-depth collaboration

As more leaders carry out their assessment projects, I identify similar projects and encourage leaders from different disciplines to collaborate in presenting and publishing.

Slide 66

This picture shows that faculty from a science undergraduate program and a humanities graduate program presented on their program merger process, facilitated by assessment tools. Now, I am “forcing” them to write it up to publish.



Slide 67

Ph.D. Rubric Inventory Project

I reached out to five leaders who worked on Ph.D. program assessment and collected the rubrics that they developed for assessment Ph.D. signature works such as dissertation proposals and dissertation defenses. I compiled these rubrics and plan to publish them on our website as UH Mānoa’s Ph.D. Rubric Inventory Project. Slide 68

Meaningful incentives

We provide meaningful incentives to our participants.



Slide 69

Food

At least 50% of the participants reported that having food helped them commit to the Institute. We provide quality lunch with morning and afternoon snacks. Slide 70

Certificates & Leadership Support

Our Institute completion certificates are signed by the chancellor and vice chancellor to show leadership support.



Slide 71

Certificates & Leadership Support

The campus leaders like Vice Chancellors for Academic Affairs would come and give out the certificates. Slide 72

iPads or equivalent

We provide iPads in the past to our leaders. Some really appreciate it and use the device in their assessment work. Some would commit to assessment regardless. However, when they received the iPad, after a semester of implementing the project and meeting with the support group buddies, they commented that it (the tablet) is a good fuel to help carry on. Currently, we are giving people a choice of iPad or up to $400 to support them going to a conference of their choice.



Slide 73

Paid training & resources

membership

We secured an additional $2500 to support assessment leaders to seek advanced facilitation trainings and present at the conferences. Slide 74

Peer network

As we train more assessment leaders, we started building a network of faculty members who then collaborate in different ways and use the facilitation and assessment strategies to aid their projects. Just this month, I reviewed a new program proposal of the Public Policy BA program. The program resembles interdisciplinary collaboration between Public Administration, Urban and Regional Planning, as well as Peace Institute. Because the two leaders in this new program



both participated in the ALI, developing a curriculum map and drafting an assessment plan only took us half an hour because we share the common language and assessment philosophy.



Slide 75

PROJECT EFFECT

We rigorously assessed our project effect Slide 76

Self-ReportSurvey

PlanContent Analysis

Observation Documentation

Using multiple data sources with both direct and indirect measures.



Slide 77

Achievement of ALI

• 4 past cohorts and 45 alumni• 100% achieved learning outcomes• 100% developed and implemented

assessment projects• 96% presented at Assessment Poster Exhibit• 9 are going to or have presented at

local/national conferences

Here is a quick summary of our achievement made by ALI leaders in the first four cohorts. Slide 78

Moved52+

ProgramsForward

45 ALI leaders from the first four cohorts moved more than 52 programs forward in assessment work.



Slide 79

96% engaged program faculty

Slide 80

Plan your project

1. Form groups of 32. Use 8 min to design your project of assessment

leader training on your campus3. 3 min share + 3 min feedback for each member in

the group4. Feedback on:

• 1-2 strengths/interesting aspects• 1 thing to consider with possible alternatives



Slide 81

8 minDesign your

project Handout p. 6

Slide 82

6 minMember 1

share + feedback



Slide 83

6 minMember 2

share + feedback

Slide 84

6 minMember 3

share + feedback



Slide 85

Share


(808) 956-4283 (808) 956-6669


Page 1 of 6 You may use these materials only for nonprofit educational purposes. Please give credit/cite appropriately. Hill, Y. Z. & Stitt-Bergh, M. (2018). Cultivating collaborative leaders with a facilitative approach to assessment. Assessment Institute, Indianapolis, IN.

Cultivating Collaborative Leaders with a Facilitative Approach to Assessment Facilitators Yao Hill, University of Hawai‘i at Mānoa, [email protected] Monica Stitt-Bergh, University of Hawai‘i at Mānoa, [email protected] Assessment Office website: http://www.manoa.hawaii.edu/assessment/ Assessment Leadership Institute website: http://www.manoa.hawaii.edu/assessment/institute/

Welcome! Agenda 1:30-1:45 Introduction; overview; ground rules; outcomes (lecture)

Activity 1: taking stock Activity 2: goal(s) for today

1:45-2:15 Facilitation and collaboration: What, why, and how (lecture and demonstration) 2:15-3:20 Our 3-day training schedule (lecture)

Activity 3: how to lead a mapping activity (role-play) 3:20-3:30 Break 3:30-3:50 Effective professional development practices (lecture) 3:50-4:20 Activity 4: planning a project 4:20-4:30 Wrap up

Activity 5: most important takeaway Activity 6: workshop evaluation

4:30 Workshop ends

Ground Rules 1. Participate actively 2. Encourage others to contribute 3. Listen with an open mind 4. Disagree with respect 5. Stay on point and on time

Workshop Outcomes List collaboration/facilitation skills Explain the importance of collaboration/facilitation skills Design training to enhance assessment leaders’ facilitative skills Plan a project that integrates training and follow-up support

http://www.manoa.hawaii.edu/assessment/

http://www.manoa.hawaii.edu/assessment/institute/

of 6 You may use these materials only for nonprofit educational purposes. Please give credit/cite appropriately. Hill, Y. Z. & Stitt-Bergh, M. (2018). Cultivating collaborative leaders with a facilitative approach to assessment. Assessment Institute, Indianapolis, IN.

Activity #1: Taking Stock A. List 1-3 things you’ve done in terms of assessment-related training for faculty (and/or staff,

administrators). [If none, write the training that you’d like to see happen on your campus.]

B. For each, indicate the focus or purpose of the training (e.g., increase participants’ assessment knowledge; assist or help participants complete an assessment report; help participants be able to use assessment management software; increase participants’ skills in leading assessment, etc.)

Assessment-related Training Focus or Purpose of Training 1.

2.

3.

Activity #2: Your Goal(s) for Today Given our workshop description, our session outcomes, and the assessment-related training you listed above, what is a word or short phrase that captures what you’d most like to leave with today? Be ready to share.


Facilitation and Collaboration Skills: What, Why, and How

Definition Group facilitation in assessment work: Use interactive strategies to

• elicit and harness the intellectual capital and goodwill in members of a group • guide the group in reaching shared understanding of the program expectations, quality, and

coherence; and • navigate the group to reach collaborative programmatic decisions

Collaboration: work together with individuals and guide individuals to work together in assessment.

Why important: Facilitation & collaborative skills engage faculty in collaborative inquiry and action that moves assessment forward

• Our view of program learning assessment: a set of tools for faculty to collaboratively reflect on program quality and coherence and to act on learning evidence for program improvement.

• Assessment process is a process of collective inquiry that involves nested discussions, decisions, and actions (Maki, 2004).

• Being a facilitator/guide is one of the five essential roles of an assessment professional (Jankowski & Slotnick, 2015)

• The broader field of evaluation has recognized the importance of facilitation and many provided examples and guidance (e.g., Fierro et al., 2016, King and Stevahn, 2013, pp. 101-147)

• Assessment is a device for change. Change moves at the speed of trust. (George Kuh) • Collaboration skills such as active listening and managing group work are essential to build

individual relationships and move the group forward.

Introduction to the UH Mānoa Assessment Leadership Institute

• Two foci in 3-day training: Assessment and facilitation • Topics and facilitation around assessment cycle

o Creating SLOs: Snow Card o Curriculum mapping: Stick-Notes voting o Developing rubrics: Score-discuss-rescore o Standard-setting: Bar-graphing o Program vision: Making metaphor

• Collaborative skills training o Strategies to build personal relationships o Model & practice active listening o Work with willing and ready first o Use committee with clearly divided roles o Progress with small steps with clear products for each step o Use meeting time to draft products (rather than just discussion) o Build scholarship in a team o Use disciplinary expertise to be creative (e.g., invisible theater)


Strategies to enhance facilitation skills in assessment-leadership training

• Design facilitation simulation tasks for participants to practice • Develop facilitation scripts • Provide one-on-one instruction beforehand • During simulations: explain the task first and end with reflection on good strategies • Introduce and reinforce general facilitation techniques throughout the training • Modeling meeting facilitation techniques • Use a variety of facilitation techniques throughout the training

Purposefully expand and practice facilitation skills and techniques • Seek training • Use resources (see handout)

Meeting Facilitation Demonstration Our facilitator’s notes on meeting facilitation (Day 1 Agenda from 2018 ALI)

• BINDER – 2 pages of tips • Role playing with Monica & Yao

Use redirection (after validation)

a. Bring up previous decision or idea

Monica: SCENARIO – faculty discussing collecting assignments to learn more about student achievement. I say, “What assignments are currently used in the capstone course?”

Yao: “What about an alumni survey? It’s so much easier.”

Monica: “Yao, you’re bringing up the idea of an alumni survey. The committee already decided not to do an alumni survey in fall so we will not discuss that anymore. Do you have another suggestion?”

b. Too much detail

Monica: SCENARIO: faculty discussing collecting assignments from the capstone course.

Yao: “I can create an upload system, maybe using Google Drive. We could have different folders for each person. I could create a Google form or a Google sheet to monitor and record who has submitted for which student. I could upload a list of our majors and include the demographic information to include later, gender, ethnicity, transfer credits, resident or non-resident . . . .

Monica: Yao, sorry to interrupt, but “It seems like we’re focusing on too many details right now. Can we move back to the larger question at hand?”


c. Unconnected idea

Monica: SCENARIO: faculty discussing what assignments to collect and from which courses

Monica: “Okay, we’re in agreement that we’ll collect from X and Y and we’ll collect the final research papers. Is there any other course that we can collect from?

Yao: “How are we going to distribute the results?”

Monica: “Yao, I hear you saying that we need to figure out how to distribute the results. Can you help me understand how what you’re saying is connected to what we are talking about—collecting seniors' research papers?”

d. Tangential idea

Monica: SCENARIO: faculty discussing what assignments to collect and from which courses

Monica: We’ve added one more course to collect from. Any other courses?

Yao: “Are we collecting from all students or a sample?”

Monica: “Sampling seems very important and tangential to this meeting’s focus. Can we put that in the minutes as something to discuss at a future meeting?”

Activity #3: Facilitation Activity—Curriculum Map Role Play See handout.

Planning an Assessment Leadership Project Effective professional development practices (Blank & De Las Alas, 2009; DeMonte, 2013; Garet et al., 2001; Gulamhussein, 2013; Hunzicker, 2010; Yoon et al., 2007)

• Significant amount (> 14 hours) • Engage faculty in active learning • Product Oriented • Provide follow-up support • Provide opportunities to collaborate and peer feedback • Meaningful incentives

Example PD programs at UH Mānoa: http://manoa.hawaii.edu/assessment/ilo/pdmodels.html Features of UH Mānoa ALI

• Intentional recruitment of participants • 3-day intensive training (with a focus on facilitation skills and strategies) • Project presentation on the 3rd day of training • One semester follow-up individually and in support groups • Support in poster presentation at the campus assessment poster exhibit • Fostering scholarship and in-depth collaboration • Meaningful incentives: food, iPads, support for conference attendance

http://manoa.hawaii.edu/assessment/ilo/pdmodels.html


Activity #4: Plan Your Project Who will be the participants? How would you select them? What knowledge, skills, and values (mind-set) do you want your participants to gain in the training? Ability to use facilitation techniques to guide faculty collaborative work in program assessment

What strategies would you use to train facilitation and collaboration skills? Design facilitation simulation tasks for participants to practice Develop facilitation scripts Provide one-on-one instruction/guidance beforehand During simulations: explain the task first and end with reflection on good strategies Introduce and reinforce general facilitation techniques throughout the training Model meeting facilitation techniques Use a variety of facilitation techniques throughout the training Other:

How would you plan to provide follow-up support and opportunities for peer feedback? What product do you expect from your participants after they complete the training? Assessment project presentation Assessment report draft Poster presentation A report on achievement Reflection on learning essay Other:

Would you provide incentives? What would be some options? Where would you get the funding? Food Grant/monetary award Book/equipment (iPad)/supplies Opportunity to attend conferences/training Reduced faculty workload Other:

Activity #5: Most Important Takeaway Write down a word or short phrase that captures the most important thing you learned today—your biggest takeaway. Be ready to share.

FACILITATION RESOURCES

1. Library of facilitation techniques by SessionLab: https://www.sessionlab.com/library

a. Searchable facilitation technique library. Allow you to filter techniques by number of participants, duration, and topics

b. Free version allows you plan for 10 sessions. Free access to public library of facilitation techniques

2. Engagement Streams Framework by the National Coalition for Dialogue & Deliberation (NCDD): http://www.ncdd.org/files/NCDD2010_Engagement_Streams.pdf

a. Describes different engagement streams (i.e., exploration, conflict transformation, decision-making, collaborative action)

b. Provides facilitation techniques for each stream with brief description of the technique and links to detailed description

3. Patton, M. Q. (2018). Facilitating evaluation. Thousand Oaks, CA: Sage Publications. 4. King, J. A., & Stevahn, L. (2013). Interactive evaluation practice: Mastering the interpersonal dynamics of

program evaluation. Thousand Oaks, CA: Sage Publications.

5. Kaner, S., Lind, L., Toldi, C., Fish, S., & Berger, D. (2007). Facilitator’s guide to participatory decision-making. San Francisco, CA: John Wiley & Sons/Jossey-Bass.

6. Donna Ching’s Facilitative Skills book and DVD package: https://www.pacificcollaboration.com/books/

7. The International Association of Facilitators (IAF) Core Competencies: https://www.iaf-world.org/site/professional/core-competencies

8. Monica and Yao’s workshops on assessment facilitation

a. Hill, Y. Z. (2017, April). Facilitating use of program assessment results for program improvement. Paper presented at the WASC Academic Resource Conference, San Diego, CA. Retrieved from http://hdl.handle.net/10125/48529

b. Hill, Y. Z. (2014, April). Collaborative rubric adaptation: Engaging multiple perspectives to define quality. Workshop conducted at the WASC Academic Resource Conference, Los Angeles, CA. Retrieved from http://hdl.handle.net/10125/48530

c. Stitt-Bergh, M. (2014). Activities to promote use of assessment results in higher education. Roundtable session presented at the American Evaluation Association Annual Conference, Denver, CO. Retrieved from https://www.academia.edu/7880640/Activities_to_Promote_Use_of_Assessment_Results_in_Higher_Education_roundtable_

d. Stitt-Bergh, M. (2015). Facilitation skills: A key to successful program assessment. AALHE Intersection. Retrieved from https://www.academia.edu/14695016/Facilitation_Skills_A_Key_to_Successful_Program_Assessment

PAPER ON OUR MODEL Hill, Y. Z. (2017). Building grassroots leaders for a sustainable assessment culture. In Proceedings of the Association

for the Assessment of Learning in Higher Education Conference 2017. Available in https://c.ymcdn.com/sites/www.aalhe.org/resource/resmgr/files/2017_CONFERENCE_PROCEEDINGS.pdf

https://www.sessionlab.com/library

http://www.ncdd.org/files/NCDD2010_Engagement_Streams.pdf

https://www.iaf-world.org/site/professional/core-competencies

https://www.iaf-world.org/site/professional/core-competencies

http://hdl.handle.net/10125/48529

http://hdl.handle.net/10125/48530

https://www.academia.edu/7880640/Activities_to_Promote_Use_of_Assessment_Results_in_Higher_Education_roundtable_

https://www.academia.edu/7880640/Activities_to_Promote_Use_of_Assessment_Results_in_Higher_Education_roundtable_

https://www.academia.edu/14695016/Facilitation_Skills_A_Key_to_Successful_Program_Assessment

https://www.academia.edu/14695016/Facilitation_Skills_A_Key_to_Successful_Program_Assessment

https://c.ymcdn.com/sites/www.aalhe.org/resource/resmgr/files/2017_CONFERENCE_PROCEEDINGS.pdf

SOURCES CITED Blank, R. K., & de las Alas, N. (2009). The Effects of Teacher Professional Development on Gains in Student

Achievement: How Meta Analysis Provides Scientific Evidence Useful to Education Leaders. Washington, DC: The Council of Chief State School Officers. Retrieved from http://files.eric.ed.gov/fulltext/ED544700.pdf

DeMonte, J. (2013). High-Quality professional development for teachers: Supporting teacher training to improve student learning. Washington, DC: Center for American Progress. Retrieved from http://files.eric.ed.gov/fulltext/ED561095.pdf

Garet, M. S., Porter, A. C., Desimone, L., Birman, B. F., & Yoon, K. S. (2001). What makes professional development effective? Results from a national sample of teachers. American Educational Research Journal, 38(4), 915–945.

Gulamhussein, A. (2013). Teaching the teachers: Effective professional development in an era of high stakes accountability. National School Board Association—Center for Public Education. Retrieved from http://www.centerforpubliceducation.org/Main-Menu/Staffingstudents/Teaching-the-Teachers-Effective-Professional-Development-in-an-Era-of-High-Stakes-Accountability/Teaching-the-Teachers-Full-Report.pdf

Fierro, R. S., Schwartz, A., Smart, D. H., & Brandon, P. R. (Eds.). (2016). Evaluation and Facilitation [Special Issue]. New Directions for Evaluation, 149.

Hunzicker, J. (2010). Characteristics of effective professional development: A checklist. Peoria, IL: Bradley University. Retrieved from http://files.eric.ed.gov/fulltext/ED510366.pdf

Jankowski, N. A., & Slotnick, R. C. (2015). The five essential roles of assessment practitioners. Journal of Assessment and Institutional Effectiveness, 5(1), 78–100. Retrieved from https://doi.org/10.5325/jasseinsteffe.5.1.0078

Maki, P. (2004). Assessing for learning: Building a sustainable commitment across the institution (1st ed). Sterling, VA: Stylus.

Yoon, K. S., Duncan, T., Lee, S. W.-Y., Scarloss, B., & Shapley, K. L. (2007). Reviewing the evidence on how teacher professional development affects student achievement: Issues & answers. (REL 2007-No. 033). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Education Laboratory Southwest. Retrieved from http://files.eric.ed.gov/fulltext/ED498548.pdf

2018 Assessment Leadership Institute Agenda

Begin End Duration Topic Helpful Readings

Mon. May 21

Kuy 106

8:30 9:30 60 Welcome and Introductions Self-introduction guide 9:30 9:40 10 Break 9:40 10:10 30 Program Assessment Introduction

10:10 10:40 30 Your Program’s Assessment Status & Your Project Plan

WASC Rubric

10:40 10:50 10 Break 10:50 11:20 30 Introduction to Student Learning

Outcomes Develop outcomes

11:20 11:30 10 Guest Speaker Presentation (Julie Walsh) 11:30 12:15 45 Lunch with poster presenter 12:15 12:30 15 Meeting Facilitation Tips 12:30 13:40 70 Curriculum Map (Facilitation Activity) Create a Curriculum Map 13:40 13:50 10 Break 13:50 15:20 90 Direct Assessment, Capstone and

Signature Assignment (Facilitation Activity)

Choose assessment method Capstone Experiences

15:20 15:30 10 Break 15:30 16:15 45 Work on Your Assessment Plan, Gallery

Walk, and Institute Reflection

Tues May 22

Kuy 106

8:30 9:00 30 Gallery Walk (cont.) and Peer Review 9:00 10:10 70 Rubric Adaptation (Facilitation Activity) Create Rubrics

10:10 10:20 10 Break 10:20 11:35 75 Standard Setting (Facilitation Activity) Standard Setting 11:35 11:45 10 Poster Presentation (Mee-Jeong Park) 11:45 12:45 60 Lunch with poster presenter 12:45 13:50 65 Data Analysis & Reporting 13:50 14:00 10 Break 14:00 14:30 30 Meeting Facilitation: Plan a Meeting 14:30 15:00 30 Assessment Resources & Scholarship 15:00 15:15 15 Institute Reflection 15:15 15:25 10 Break 15:25 16:00 35 Work on Your Plan & Presentation;

Get/Give Feedback; Schedule presentations

Wed. May 23

Kuy 106

8:30 9:00 30 Work on Your Presentation 9:00 10:15 75 Conduct Actionable Assessment – Use of

Results

10:15 10:30 15 Break 10:30 11:45 75 Faculty Engagement & Mission/Vision

Planning (Facilitation Activity)

11:45 12:00 15 Schedule Small Group Meetings (Fall) 12:00 13:00 60 Lunch (and upload presentation) 13:00 13:50 50 Attendee Presentations (group 1) Presentation template 13:50 14:00 10 Break 14:00 14:50 50 Attendee Presentations (group 2) Presentation template 14:50 15:30 40 Certificate Ceremony, Photos, & Institute

Reflection and Evaluation

Updated 05/14/2018

http://manoa.hawaii.edu/assessment/institute/materials/Self_Introduction_Guidelines.docx

http://manoa.hawaii.edu/assessment/resources/rubricoutcomesplanning.htm

http://manoa.hawaii.edu/assessment/howto/outcomes.htm

http://manoa.hawaii.edu/assessment/howto/mapping.htm

http://manoa.hawaii.edu/assessment/howto/methods.htm

http://manoa.hawaii.edu/assessment/howto/capstone.htm

http://manoa.hawaii.edu/assessment/howto/rubrics.htm

http://manoa.hawaii.edu/assessment/institute/materials/standard_setting.pdf

Assessment Leadership Institute Crawford Hall 230 & 231 Assessment Office, University of Hawai‘i at Mānoa manoa.hawaii.edu/assessment

Page 1 of 2 Stitt-Bergh, M.

Facilitating Meetings & Program Assessment Decision-making Monica’s Top Tips

1. Prepare an agenda to share and a script for yourself 2. Create an action-oriented agenda with a desired product (outcome) and process. Examples:

Desired Product/Outcome Process final student learning outcomes list by evaluating draft student learning outcomes narrow a list of commercial tests to best option by evaluating alternatives and dot voting understand curriculum coherence and identify gaps by creating a curriculum map ways to use assessment results by brainstorming a list of possible actions improve the program using assessment results by prioritizing a list of actions

3. Use redirection (after validation) a. Bring up previous decision or idea

“Melanie, you’re bringing up the idea of an alumni survey. The committee already decided not to do an alumni survey in fall so we will not discuss that anymore. Do you have another suggestion?” “Jonathan, here [point to statement] on the notes for today we’ve recorded your suggestion to change admissions standards. Is this recorded correctly?”

b. Too much detail “It seems like we’re focusing on too many details right now—like how student names will be redacted from the samples. Can we move back to the larger question at hand?”

c. Unconnected idea “Ryan, I hear you saying that the rubric needs to be changed. Can you help me understand how what you’re saying is connected to what we are talking about—the distribution of results?”

d. Tangential idea “Christie, sampling seems very important and tangential to this meeting’s focus. Can we put that in the minutes as something to discuss at a future meeting?”

4. Make contributions visible: Record ideas using markers + flip chart paper or computer + wall projector.

5. Decide how to decide. A conversation about how decisions will be made is often overlooked and when disagreement later occurs, a single voice can derail progress. I recommend the consensus method because it requires a participatory process that usually produces a superior decision with widespread acceptance and support for implementation. Consensus has been reached when everyone agrees the deliberation process has been fair, transparent, everyone feels heard, good information was used to make the final decision, and everyone is willing to support—but not necessarily agree with—the final decision. The goal is unity, not 100% agreement.

Other decision-making options include a) 85/15 rule—85% agreement is enough to pass; b) super majority—2/3 vote needed to pass; c) simple majority— 51% agreement is enough to pass. A simple majority is usually not a good

option because a 51/49 vote typically hinders implementation. 6. Reserve the last 5-10 minutes to summarize, communicate praise for accomplishments, and state

commitments/actions/next steps.

of 2 Stitt-Bergh, M.

More Facilitation Tips 1. Know desired outcomes upfront; have a product in mind before starting

2. Get clarity about expectations by making sure everyone is comfortable with the agenda

3. If you are a facilitator and part of the group, make it clear when you are facilitating and when you are speaking as part of the group. One method: physically re-position your body so your role is clearer. When speaking as a group member, preface your comment with, “As a member of this department.”

4. Ask only those questions you want the group to focus on

5. Role model the behavior you want to see

6. Use active listening techniques to listen as an ally, legitimize immediately and often, and validate every voice: nod, open posture, don’t interrupt, paraphrase what was said (record paraphrase on visible minutes) and get confirmation you understood

7. Provide a safe space for everyone’s contributions so that participants can see their self-interests in the context of others’ interests

8. Get everyone to contribute to the final outcome

9. Be comfortable in chaos

10. Redirect inappropriate responses (especially premature solutions)

11. Invite differing opinions

12. Maintain eye contact with everyone

13. Differentiate content/outcome from process

14. Hold people to speaking for themselves. Encourage “I” messages rather than “we” or “they” messages. For example, when someone says, “Some people say. . .”, simply state “Please let us know what you think of the proposal” and not record the “some people” idea in the notes.

15. Attend to volume/loudness. Loudness typically signals conflict emerging and people not feeling heard. Simply stating, “I notice that we are speaking loudly to each other” is usually sufficient to create awareness and reduce the volume. Don’t diagnose, just state the observation.

16. Attend to the number of interruptions. Calling attention to the issue usually settles it. “We are interrupting each other and our meeting ground rules state that one person at a time should talk.” Don’t diagnose, just state the observation.

17. Embrace one minute of silence. If things are moving too fast, not fast enough, if volume or interruptions are increasing, ask people to spend one minute silently thinking about what is being discussed (or one minute jotting down ideas).

Sources consulted: Brilhard, J.K. & Galanes, G.J. (1989). Effective Group Discussion. 6th Ed. Dubuque, IA: Brown. Ching, D.R. (2010). Facilitative Skills for Collaborative Leaders. Honolulu: Pacific Center for Collaboration. Sanaghan, P. & Gabriel, P.A. (2011). Collaborative Leadership in Action: A Field Guide for Creating

Meetings That Make a Difference. Amherst: HRD Press.


Page 1 of 2 Stitt-Bergh, M.

Curriculum Map Example: Excerpt from Biology

Requirements

Student Learning Outcomes

Apply the scientific method

Demonstrate laboratory techniques

Form strong biological

arguments supported by

evidence

Communicate using oral and

written reports

Awareness of careers & job

opportunities in biological sciences

BIOL 101 X X X

BIOL 202 X X X

BIOL 303 X X** X

BIOL 404 X** X

BIOL 4x6 senior seminar X** X** X**

Exit interview X**

** = collect evidence for program-level decision making from the students

What is a curriculum map? Why do it?

Curriculum mapping is a method to align instruction with desired goals and program outcomes. It can also be used to explore what is taught and how. The map or matrix

• Documents what is taught and when; • Reveals gaps in the curriculum; and • Helps design an assessment plan.

Benefits:

• Improves communication among faculty • Improves program coherence • Increases the likelihood that students achieve program-level outcomes • Encourages reflective practice

of 2 Stitt-Bergh, M.

Curriculum Map Activity: Psychology Student Learning Outcomes 1. Compare and contrast major concepts and themes in psychology 2. Interpret, design, and conduct basic psychological research 3. Apply ethical standards to evaluate psychological science and practice 4. (a) Craft clear and concise written communication to address specific audiences (lay, peer,

professional) and (b) deliver complex oral presentations and answer questions about psychological content

5. Interact effectively with others by deploying psychological concepts to facilitate effective interactions with people of diverse backgrounds

Psychology Degree Requirements

Psych 101:Intro to Psychology Gateway course that introduces students to the

major perspectives in psychology. Emphasis on the broad scope of modern-day

scientific psychology Evaluation = multiple-choice exams

Psych 209: Fundamentals of Psych Research Psychological research methodology and

techniques. Ethical issues in psychological research Evaluation = multiple-choice exams

Psych 303: Theories of Personality Survey of major perspectives, scientific issues,

applications, and research finding in the area of personality Emphasis on hands-on research Evaluation = multiple-choice exams; research

report in APA style

Psych 345: Experimental Psychology Focus on experimental research designs used in

psychological research Topics include research design and statistical

analysis Emphasis on appropriate designs for different

research questions Evaluation = multiple-choice exams

Psych 356: Psychobiology Behavior from a natural science viewpoint Topics include evolution, behavior genetics,

neural mechanisms, drugs and behaviors, neural bases of memory and cognition Evaluation = multiple-choice exams; analysis

papers

Psych 367: Social Psychology Cognitive, behavioral, and emotional effects of

people Topics include interpersonal relations,

attribution, attitudes, group behavior, stereotypes, social roles Evaluation = multiple-choice exams

Psych 414: Cognitive Development Overview of theoretical perspectives

concerning development of children’s thinking from birth through the early school years. Emphasis on how researchers design

experiments to assess cognition in children Evaluation = analysis papers; research proposal

in APA style

Psych 489: Senior Seminar in Psychology Capstone experience Students participates in supervised research or

fieldwork Evaluation = research proposal; research report

in APA style; oral presentation

Activity: Collaboratively create a curriculum map using the information provided.


Page 1 of 3

Curriculum Map Facilitation Activity Process and outcome. Faculty use sticky notes to create a curriculum map in order to help the program

visualize program coherence, identify any gaps in the curriculum, and suggest changes. Needed 1. Blank, over-sized curriculum map grid with degree requirements and (abbreviated) student learning

outcomes. “Pick one of many” degree requirements appear together on the grid—to aid analysis, draw a colored box around the grouping or print the grouping on same-colored paper.

2. Example of a completed curriculum map and list of guiding questions for discussion 3. List of program student learning outcomes 4. Sticky notes 5. Markers 6. Tape 7. Large flip chart paper (or computer+projector) 8. Faculty group, facilitator, note-taker

Time:

:03 Welcome participants and describe the activity.

“I’m glad to see you here today for the department meeting. Thank you for coming. The department approved the curriculum committee’s proposed student learning outcomes at the last meeting. The next step is to see where in the curriculum that students are significantly exposed to the outcomes and determine if we provide enough learning opportunities. We’ll do that using what’s called a curriculum map which is a simple matrix with degree requirements in rows in the first column and student learning outcomes across the top.

Here’s an example of a curriculum map from Biology. A filled-in cell indicates that the course significantly emphasizes an outcome. A blank cell means no significant emphasis. The content or skill may exist in the course, but it’s not directly emphasized. On this map, the asterisks indicate that student projects are collected for program-level assessment purposes.

The activity today involves you using sticky-notes to indicate whether the courses you teach directly address the SLOs and devote time to help students achieve the SLO. For those of you who teach the same course, please discuss and decide on what’s appropriate.

This map will represent our curriculum NOW. After we’ve finished, we’ll analyze it and discuss changes that might be considered.

There are sticky note and markers on your table for you to use. Place an “X” on the note and add to the map. Use an asterisk if student projects or tests can be collected for program assessment from the course. Any questions before we get started?”

ALTERNATIVE #1: Faculty put “1,” “2,” or “3” on the sticky-note to indicate some emphasis (1), moderate emphasis (2), significant emphasis (3), or no emphasis (blank).

ALTERNATIVE #2: Faculty put “I,” “R,” “M,” and “A,” on the sticky-notes to indicate Introduce, Reinforce, Master, and Assess for program-level assessment (“M & A” is allowed on a single sticky-note).

of 3 Assessment Leadership Institute

Facilitation tip. Dealing with an objection to already-made decisions (e.g., the SLOs):

Jon: “The student learning outcomes don’t include social justice. That should be added.”

Facilitator: “Okay, Jon, the curriculum committee used a collaborative process to develop the SLOs and the department approved the SLOs at the last department meeting, so we’re not going to take time to discuss adding outcomes at this time. Or, do you have a different concern?”

:10 The faculty members stand up and add their sticky-notes to the giant curriculum map. They discuss, as needed, among themselves to reach consensus on classes that several people teach.

:30 to :45

The facilitator leads a discussion about program coherence and ways to improve coherence. Below are guiding questions the facilitator and faculty colleagues can use to analyze the curriculum map.

Another person (or the facilitator if no one can help) records the gist of the ideas and suggestions using a marker+flip chart paper or with a computer+wall projector.

Facilitation tip. When using a flip chart and markers, alternate between two different colored markers such as blue and green for each idea. Do not use red except as a highlight color.

Response 1 in blue Response 2 in green Response 3 in blue

Coherence Questions

A. Does each course (and required experience) contribute to the program SLOs? [pause for answer, record answers, ask for suggestions]

Jenn: “CRS 310 has no Xs.”

Facilitator: “Okay, so CRS 310 doesn’t seem to contribute, does anyone have information about that course?”

Jon: “We haven’t offered that course since Mary retired. Maybe we should remove it from the curriculum or revise it.”

Facilitator: “Okay, remove or revise. Other ideas?”

B. Do we offer students enough learning opportunities for each outcome?

Mel: “Students have learning opportunities for outcomes 5 and 6 in the 100-level class and not again until their 400-level project course. Let’s add assignments for #5 and #6 to the 300-level courses.”

C. Does any course try to do too much? Is it possible for a single course to help students make significant improvement on all of the SLOs? What’s appropriate?

Chris: “The intro research methods course includes learning about major theories. I’m not sure it can cover that plus teach design, proposal writing, giving a presentation, and ethics.”

D. For programs that allow students to choose courses from a group of program electives: Is it possible for students to choose certain program electives and then not be exposed sufficiently to an outcome? If yes, what suggestions do you have?

of 3 Assessment Leadership Institute

Assessment Planning Questions

E. For undergraduate programs: Which senior-level courses already use assignments that align with program outcomes? Can we gather evidence of student learning from those courses? [mark these courses on the curriculum map with an “A” or asterisks]

F. For graduate programs: Where in the curriculum can we evaluate for mastery of the outcomes? What does evidence already exists? [E.g., theses, oral defense.]

For each question about coherence and planning, the facilitator allows the conversation to continue until saturation and then asks the group to consider a decision,

“It sounds like we’ve listed the possible ideas. Unless someone has a new idea, let’s see if we can select one or more of these actions for the curriculum committee to help implement.”

Voting options: dot voting, show of hands, secret ballot, email poll after meeting ends, etc. Continue until the curriculum has been analyzed, suggestions made, and courses or locations in the curriculum have been selected from which to collect evidence.

Always leave 5 minutes for wrap up.

:05 Facilitator wraps up the discussion: summarizes, praises accomplishments, and gives next steps.

“I’ve learned a lot about our curriculum today and while we do have more to discuss, we’re out of time today. We have two items that we agreed upon. We agreed to remove 310 and to add writing case studies in the 250 course and in all 300-level electives. I’ll type these notes and email them along with the curriculum map. I’ll put the issue of collecting evidence from 414 and 489 on our next meeting agenda. Thanks again for your work and thoughts today.”

ASSESSMENT FOR CURRICULAR IMPROVEMENT POSTER EXHIBIT FRIDAY, MARCH 24, 2017

SINCLAIR HERITAGE READING ROOM

Sponsored by UHM Assessment Office · Contact: Yao Z. Hill · [email protected] · (808) 956-4283

Thank you for joining us as we celebrate assessment successes and highlights from Mānoa faculty

and staff. We invite you to share assessment strategies, contribute ideas to program and

institutional learning assessment, and network with others.

SCHEDULE

11:30 am – 12:30 pm Presenters-only session

11:35 am Welcome from Associate Vice Chancellor for Academic Affairs Debora Halbert

12:30 – 2:00 pm Exhibit open to public

12:45 pm Welcome from Interim Chancellor David Lassner

1:50 pm Poster Awards presented by Interim Vice Chancellor for Academic Affairs Michael Bruno

POSTER WEBSITE https://manoa.hawaii.edu/assessment/poster/index.htm

All posters will be uploaded to the website following the event.

Keep them as a resource and share them with colleagues!

MAHALO NUI LOA! Poster presenters and co-presenters, Poster reviewers and volunteers, Invited speakers: Interim Chancellor David Lassner, Interim Vice Chancellor for Academic Affairs Michael

Bruno, Associate Vice Chancellor for Academic Affairs Debora Halbert, UHM Sinclair Library, UHM Department of Information and Computer Sciences, UHM Center for Instructional Support, UHM Facilities Management.

Let us know how the event went!

Complete our evaluation

http://www.surveyshare.com/t/PosterEval2017

WIN A DOOR PRIZE!1. Pick up a poster checklist sheet/map at the

registration table.

2. On the sheet, check next to each poster you

visit. 5 checkmarks = a chance to win a

prize!

3. Write your name and email at the top right.

4. Return the sheet to the registration table

(look for on map). Winners will be notified

shortly after the event through email.

BEST POSTER AWARDSNominate a poster in each of the following

categories:

1. Best poster design

2. Best faculty engagement strategies

3. Best use of results

Submit your nominations before

1:40 pm at the voting station (see map).

Winners will be announced at 1:50 pm.

https://manoa.hawaii.edu/assessment/poster/index.htm

http://www.surveyshare.com/t/PosterEval2017

2017 ASSESSMENT FOR CURRICULAR IMPROVEMENT POSTERS

HIGHLIGHTED FEATURES

SLO

s (

SLO

)

Cu

rric

ulu

m

Map

(C

M)

Me

tho

ds

(M)

Ru

bri

c (R

)

Use

of

Res

ult

s (U

SE)

Facu

lty

Enga

gem

en

t

(FE)

Form

ativ

e

Ass

ess

me

nt

(FA

)

COLLEGE

1 Program Completion and Mentor Surveys as Indirect Evidence of Learning: From Development to Use Jessica L. Miranda, College of Education, Dean's Office

M USE

2 Working Across Professions to Develop the Interprofessional Education Curriculum Pathway Jacqueline Ng-Osorio, Lorrie Wong, Carolyn Ma, Robin Arndt, Al Katz, Kamal Masaki, & Kelley Withy, Hawaii Interprofessional Education Wkgp

CM USE FA

GRAD

3 An Assessment-Informed Collaborative Initiative: Curriculum Mapping for PhD Program Improvement Priyam Das, Urban and Regional Planning

SLO CM FE FA

4 A Collaborative Assessment Process for Sustained Curriculum Improvement in Natural Resources and Environmental Management Susan E. Crow, Creighton M. Litton, Kirsten Oleson & Mehana Vaughan, Natural Resources and Environmental Management

SLO FE FA

5 Use of a Weighted Curriculum Map for Programmatic Improvements in Communication Sciences and Disorders Amy E. Lower, Communication Sciences and Disorders

SLO CM

6 The Anthropology PhD Curriculum Mapping & Dissertation Defense Rubric: Enhancing Student Learning Ann M. Sakaguchi & Christian E. Peterson, Anthropology

SLO CM R USE FE

7 Merging Graduate Degree Programs with the Program Assessment Tool Set Mee-Jeong Park, East Asian Languages and Literatures

SLO CM

UNDERGRAD

8 Implementation of Integrated Performance Assessments (IPA) in Beginning Level Chinese Language Classes Song Jiang, Jing Wu, Reed Riggs & Yijun Ding, Chinese – East Asian Languages and Literatures

SLO CM M R USE FA

9 New Grading Rubrics for Signature Assignments: Tropical Agriculture and the Environment Scot C. Nelson & Christine Nakahara, Tropical Plant and Soil Sciences

R

10 Assessment in Paradise: Using Data to Drive Undergraduate Geoscience Initiatives and Programmatic Changes Michael Guidry, Oceanography & Tiffany Tsang, UH Office of STEM Education

USE

11 Engaging Science Faculty in Program Assessment – Planting Seeds and Cultivating Growth Geoffrey S. Mathews, Physics & Astronomy

SLO CM R FE

12 Department of Mathematics Program Assessment via Exam Heiner Dovermann, Mathematics

M

13 From Revising Program SLOs to a Culture of Open Communication in Dental Hygiene Kristine M. Osada & Penny Ching, Dental Hygiene

SLO

14 Course Series Alignment: Examining a Three Course Sequence Lysandra Cook, Special Education

M R

15 Antidote to Support Underprepared Students – Summer Bridge Jim Gillespie & Katie Tuisaloo, Student-Athlete Academic Services

SLO CM FE

16 Assessing the Come Back to Mānoa Program: Why Seniors Leave and How to Help Them Graduate Shannon L. Johnson, Come Back to Mānoa – Outreach College

M

CAMPUS

17 Adopting a New Assessment Approach: Using SALG to Evaluate General Education Learning Outcomes Wendi V. Vincent, General Education Office/OVCAA

SLO M USE

18 The Institutional Learning Objectives and Undergraduate Assessment Jenifer Winter et al., Institutional Learning Objectives Implementation Committee

R USE FE


Notes taken during the workshop: Cultivating Collaborative Leaders with a Facilitative Approach to Assessment

Learning goals

• Rubric workshop • Meaningful facilitation • Shift mindset and practices • Build my own facilitation skills • Train the trainers

• Improve own collaborative skills • Build authentic process • Improve facilitation skills • Overcome complacency • Encourage faculty participation

Curriculum Mapping Activity Reflection

As a facilitator and a course faculty, I need to keep things going when I have to think about my own course. Be prepared before coming to meeting about one’s own answers so that to focus on facilitation steps. The feeling of overwhelming is normal.

As facilitator, I need to a lot of homework (which is required courses, course offering frequencies) If you are facilitating with a department, you maybe from that department, we are internal facilitator. Even for those who don’t have confidence doing it themselves, they ask for help. Appreciate the meta-cognitive conversation on alternatives, and worst case scenario Good job: Check where we are and whether we want to move forward or put the items on the agenda

to discuss later Like: how things work, can do in 90-minutes. People’s assumption is that it’s going to take a long time.

This is a very efficient way of getting the product done in a relative short period of time. Efficient as trainers to teach what curriculum map is.

Follow-up: take a picture of the map. Transfer to a word doc and share with the department. Question: how to engage indifferent faculty

Usually faculty care about the courses and are willing to contribute Make it fun and rewarding

How to lead? How to facilitate with a neutral position?

Publish an agenda (solicit agenda items before the meeting to solicit people’s input) Note-taking: if any at time that our notes do not reflect your thoughts, ask us to stop to record the

correct answers. Call out the silent ones. Use “role” hat Ask group whether okay to move forward with group decisions.

How to convince faculty to change their courses to do authentic assessment?

Small steps. “One small thing that you can do?” How to help students succeed conversation versus how do you want to change your course? Do a draft (rubric, assignment) for them.

Takeaways

The feasible approach Progress Facilitation techniques Creating a product (2) Keeping moving forward Focus on effective facilitation Make it fun Facilitative training equals assessment

success 1st draft an actual plan We will build capacity This can be done Consensus building

Modeling and practice Appreciation for the method Facilitation strategies More confident Importance of developing assessment

leaders



Photos Curriculum Map Activity



Goals

Our Group Photos


(808) 956-4283 (808) 956-6669


Thank you for your feedback. We will use it to improve what we do.

Workshop Evaluation

1. As a result of this workshop, to what extent can you do the following:

a. Explain the importance of collaboration/facilitation skills

Very Large Extent Large Extent Small Extent Very Small Extent Not Sure

b. Design training to enhance assessment leaders’ facilitative skills


c. Plan a project that integrates training and follow-up support


2. List one collaboration/facilitation skill:

3. Please rate the overall usefulness of this workshop. Very Useful Useful Of Little Use Not Useful At All No Opinion

4. To what extent was this workshop effective in increasing your understanding of the topic?

Very Effective Effective Somewhat Effective Not Very Effective Not Sure

5. Any constructive comments?