2016 Oral Communication Assessment Report

Oral Communications Working Group Members Aaron Bunker Rich Crow Valerie Hennings Jeremy Schnieder Mark Heisted assisted with rubric development Outcome Assessed Based on the 2015 NSSE data, our students report having numerous opportunities to speak in their classes ­­ even more than reported by students at the NSSE­2015 comparative schools. In the summer of 2016, at the recommendation of the Institutional Student Learning Committee, a group of faculty members developed an oral communications rubric to assist in assessing the quality of student oral presentations at Morningside College. After development, the rubric was used against a sampling of recorded student oral presentations solicited from faculty across campus, representing upper and lower division classes. These results will serve as a baseline for future assessments and continued development of the institutional assessment process for oral communications at Morningside. Process The process occurred over a period of four days. The group began by holding an open discussion on what constitutes a quality oral presentation. Included in the discussion was an intentional focus on components that are universal to all presentations and not content area specific in nature. This involved viewing numerous recorded speeches and discussing the impact, and mechanics, of each.

1

Next, the group determined which elements of an oral presentation were important to focus on when assessing the quality of an oral presentation. This involved grouping like presentation characteristics together to define key elements, as shown in the photo below.

The key elements identified were: Message, Verbal Communications, Non­verbal Communications, Situational Adaptation, and Overall Impact (see the Rubric Section below for more details on each of these elements). Once the elements were identified, the next step was to develop a functional rubric that could be used to efficiently assess oral presentations from a wide range of subject areas. See Appendix 1 for a copy of the rubric developed. The rubric was then tested against a small subset of student presentations.The original rubric was modified slightly to account for variances found among the presentation styles and content areas to produce the final product below. Finally, in groups of two evaluators, the participants assessed a larger set of student speeches pulled from a variety of classes across campus. A discussion arose out of the rubric modifications on how to deal with group speeches, as the group did not feel the original rubric accurately captured all of the essential dynamics of a group presentation. Work began to modify the rubric to more accurately direct assessment of group presentations; however, due to time constraints in the workshop, more group presentations were not assessed. A copy of the draft rubric is included below (Appendix 2). Individual Oral Communication Rubric

The sample of artifacts was assessed on a scale of one to five in terms of five elements of oral communication (Appendix 1):

1. Message Clarity/organization Evidence/support for thesis

2

Rhetorical features (i.e. introduction, conclusion) Overall effectiveness of message

2. Verbal Communications Cadence/pauses Inflection Volume/articulation Natural sounding/ease Overall effectiveness of verbal communications

3. Non­verbal Communications Gestures/movements Posture Audience connections Distracting behaviors Overall effectiveness of non­verbal communications

4. Situational Adaptation Framing of presentation for the audience Language usage Management of space/environment Use of visual aids Overall effectiveness of situational adaptation

All artifacts were assessed across these four categories except those that did not include a video component (i.e. podcasts); in those instances, audio­only artifacts were reviewed in terms of message, verbal communications, situational adaptation, and overall effectiveness. For artifacts that included group work, a second rubric based on similar elements, was developed (Appendix 2).

3

Results and Recommendations

Fig. 1 Number of artifacts scored 1 through 5 with 1 being the lowest and 5 being the highest for overall effectiveness of speaker oral communication. Y­axis represents frequency of responses for that score. CiC N=26, Upper level N=28

Fig. 2 Number of artifacts scored 1 through 5 with 1 being the lowest and 5 being the highest for speaker ability to deliver message. Y­axis represents frequency of responses for that score. CiC N=26, Upper level N=28

4

Fig. 3 Number of artifacts scored 1 through 5 with 1 being the lowest and 5 being the highest for speaker verbal communication.Y­axis represents frequency of responses for that score. CiC N=26, Upper level N=28

Fig. 4 Number of artifacts scored 1 through 5 with 1 being the lowest and 5 being the highest for speaker non­verbal communication. Y­axis represents frequency of responses for that score. CiC N=26, Upper level N=28

5

Fig. 5 Number of artifacts scored 1 through 5 with 1 being the lowest and 5 being the highest for speaker situational awareness. Y­axis represents frequency of responses for that score. CiC N=26, Upper level N=28

Fig. 6 Number of upper level artifacts scored 1 through 5 with 1 being the lowest and 5 being the highest for overall effectiveness of speaker oral communication per type of course. Y­axis represents frequency of responses for that score. Adv. Pub. Spk. N=12, Other N=16

6

Fig 7. Mean score comparisons between CiC and Upper Level students for each rubric category. Y­axis represents mean score for each category. NonV = non­verbal; Sit A = situational adaptation. Mean scores were on a scale of 1­5. CiC N=26, Upper level N=28

Message Verbal Non­verbal* Situational

Awareness

Overall

Effectiveness

Upper

Level

2.82 3.07 3.04 2.85 2.93

C&C 2.00 2.29 2.21 2.25 2.32

All

Students

2.42 2.68 2.65 2.54 2.62

*Upper Level N=24 due to podcasts

Table 1. Mean score comparisons between CiC, Upper level, and All students for each rubric category. Mean scores were on a scale of 1­5. CiC N=26, Upper level N=28

Results Summary

Contrary to what was expected, overall upper level student scores were not skewed more towards the higher end of the scale. Upper level students in Advanced Public Speaking tended to score higher than students with speeches from all other upper level classes ­­ most likely as a result of direct instruction regarding oral communication within the course.

7

CiC student scores skewed more towards the lower end of the scale as expected; however, in some areas they scored as high, or even higher than upper level students. In the non­verbal element, upper level students outperformed CiC students significantly. This can possibly be explained by student maturity, in that Morningside upper level students have had multiple experiences speaking in a class setting (as identified in the 2015 NSSE data) and as a result are more comfortable in this role. Recommendations regarding Oral Communication Faculty development opportunities on how to model, discuss, and assess the mechanics of good public speaking. Faculty incorporate discussion and emphasis (points) of public speaking when assigning presentations. Distribute oral communication rubrics to faculty for consideration when building presentation grading rubrics. Tracking of current CiC cohort and students into their upper level courses to examine changes in oral communication proficiency. Recommendations for Assessment Process and Rubric

Revise rubrics to better encompass a variety of communication “aids” (visual, audio, etc.) Revise rubrics to allow for choosing type of speech (podcast, panel discussion, etc.) before beginning to fill out rubric. Better technology be made available for grading oral communication in class (microphones, video cameras, etc.). Better context provided of speeches by faculty who submit work. Recommendations for Assessment of Group Work and Rubric

Continue to experiment with and revise group work rubric (Appendix 2)

8

Appendix 1 Oral Communication Rubric

9

10

11

12

Appendix 2 Group Presentation Rubric

Group Presentations: This rubric is meant to be used for the presentation as a whole. Thus, characteristics

should be assessed proportionally.

Artifact type Panel Presentation, Audio Presentation, Creative Presentation, Other

Message

Message is unclear and/or unorganized. Message use clarity and organization to enhance presentation.

Speakers uses ineffective or irrelevant support.

Speakers provide sound and skillfully used support.

Presentation is missing rhetorical features such as a conclusion.

Rhetorical features are sound and used well.

Presentation lacks transitions between ideas. Presentation skillfully transitions between ideas.

Overall Effectiveness

Verbal

Flow of the presentation is unnatural. Presentation flows naturally.

Speakers sound disinterested. Speakers sound engaged.

Speakers inappropriately use volume and lack articulation.

Speakers use articulation and volume to intensify presentation.

Speakers sound uncomfortable (uses fillers). Speakers sound at ease and natural.

Overall effectiveness

Nonverbal Communication

Speakers use unnatural or exaggerated gestures/ movements.

Speakers’ gestures and movements enhance the impact.

Speakers' postures conflict with tone and appear uncomfortable.

Speakers’ postures emulate verbal cues and tone.

Speakers lack connection with audience (appropriate eye contact).

Speakers demonstrate a connection and comfort with audience.

13

Speakers exhibit distracting behaviors. Distracting behaviors are not present.

Overall Effectiveness

Situational Adaptation

Presentation is not framed for the audience. Presentation is skillfully framed for the audience.

Language used does not fit situation or audience.

Speaker uses language that enhances impact.

Speakers do not effectively manage space or environment.

Speakers skillfully manage space and environment.

The use or nonuse of the communication aids detracts from impact.

The communication aids are integrated to enhance the effect.

Overall Effectiveness

Overall Effectiveness of Presentation

Balance: With the previous categories in mind

Single participant dominates communicative acts

Communicative acts spread appropriately

14