CAPSTONE MATTERS

Checkpointing for Success:Experience in Meteorology

Michael Richman

School of Meteorology

March 29, 2019

Task: To Develop Writers Who Can Communicate Science

� Capstone classes are a requirement in many universities

� At OU, every school/department has their own version

� Since The School of Meteorology (SoM) has a science and research curriculum, developing inquisitive scientists who can communicate science is our goal

� Two semester sequence of about 60 – 70 students

Two Semester Sequence

� Semester 1: Metr 4911� Learn the components of a specific research cycle� Select a topic and set research goals� Find a mentor� Develop and refine research ideas� Obtain data necessary to accomplish research

� Semester 2: Metr 4922� Use of develop appropriate technology/computational literacy

skills to accomplish their research goals. � Perhaps adjust research goals.� Communicate their research motivation both in writing and

verbally� Review and challenge the work of others� Create a research poster to present to their peers and present

to all faculty and interested scientists at the National Weather Center.

Mapping SoM Goals into Outcomes

� Goals from last slide need to be formalized to quantifiable entities

� Upon completion of the degree program, students should be able to: Work in groups to develop skills to conduct independent research at a professional level and convey their findings to their peers.

� Specific learning outcomes include:� Communicate scientific topics, methods, and results with

others (both verbal and written)� Review and challenge the work of others based on sound

arguments and evidence� Analyzing data and the significance of their research

� Develop a rubric for each outcome

How to Form and Measure Specific Goals

� During the fall semester, the goal is to work to a formal proposal of a research topic and methods

� Create checkpoints to gather data to assess�#1. Create a letter of intent to develop a

specific project idea �#2. Literature review of their research topic

to evaluate the state of the science �#3. Create an elevator talk to apply their

knowledge�#4. Construct a formal proposal

How to Form and Measure Specific Goals

� During the spring semester, the goal is to work to undertake an original research project, become proficient at presenting findings and field questions.

� Create checkpoints to gather data to assess� #5. Peer review of the proposal to examine the viability

of the proposed research � #6. An oral presentation with questions and answers to

explain the research and discuss its relevance� #7. A poster presentation open to all faculty members to

illustrate the key aspects of the and test their deeper understanding of the project

� #8. Compose a final Capstone paper written in a format for professional meetings of the American Meteorological Society.

Feedback

� We have learned that peer and faculty feedback is necessary for a successful Capstone experience

� A minimum of two professors and all Capstone class members outside of the group being graded provide feedback� Peer feedback: letter of intent, elevator talk,

proposal� Faculty feedback: oral presentation, poster,

written paper� Students respond to feedback: letter of intent,

elevator talk, poster and written paper.

Rubrics for Checkpoints

� Letter of intent is evaluated by suitability of research for a semester project.

� For elevator talk, the timing and how well the importance of the research is evaluated, as well as some idea of how the group will address the research

� The letter of intent formalizes the elevator talk and the feedback, so the expectation is some feedback will be folded into the research plan.

� All measures are direct.

� Scale: meets little of the expectation, meets part of the expectation, meets expectation, exceeds expectation, for this an all checkpoints à mapped to numeric grades for analysis.

Performance Targets

� Emphasis is on improvement as the semesters progress, so our target is 100% improvement. Also, running longitudinal analysis to gauge year-to-year changes.

� We examine the percentage of students who improve from one checkpoint to the next and our goal is a significant improvement, unless they start with the exceeds expectation

� The checkpoints are tested by aggregating all students to establish a confidence interval about the average grade

� Only improvements that exceed this confidence interval are deemed a success (and those that decrease below the level, a problem).

Results Semester 1

� Keyed to checkpoints

� Differences between checkpoints that tested the same skill were tabulated and counts of the number of students falling into each category were calculated

� For example, comparison of the final proposal (checkpoint 4) to the initial letter of intent (checkpoint 1) gave the following table

Grade Change

-13 -10 -4 0 3 4 10

# Students

3 3 3 5 9 3 10

Results continued

� Since the same students are evaluated, we have pairs of observations that lead to each change.

� Such a configuration is testable via a “paired t-test”

� Test is that the Null Hypothesis is no expectation of change. Alternative Hypothesis is a significant change in grade at ∝ = 0.05.

� t = "#$%&&' ($%&&

)$%&&*

, where "#$%&& is the mean change in grade,

($%&& = expected change in grade (0), )$%&& = standard deviation of the mean change in grades and n is the sample size (number of students).

Results continued

� Result for previous table was t= 1.315 for 36 students (35 d.f.) with a 95% CI of -0.876 to +4.098. The p-value was 0.197 (not significant change is testing at ∝ = 0.05 or ∝ = 0.10).

� Interpretation: Although the grades improved an average of 1.6 points from checkpoint 1 to 4, the change is not statistically significant.

� Similarly, same approach was used for comparing � Checkpoint 4 was compared to checkpoint 2, t=-

0.997� Checkpoint 4 was compared to checkpoint 3, t=-

0.548

What Did We Do With These Findings?

� Faculty buy-in is the key. The results are quantitative and should be used to diagnose issues (as opposed to criticizing the faculty member)

� In the present situation, discussion with the professor of record about why checkpoint 1 was statistically tied in score with checkpoint 4, as was checkpoints 2 and 4 and checkpoints 3 and 4. This was not the desired result but a deeper analysis revealed that checkpoint 1 (the letter of intent) was considered relatively easy by the students as they were motivated by the topic and it was near the beginning of the semester (prior to the heavy work load)

� Proposed changes: better explain expectations for elevator talk and provide students with some links to taped elevator talks considered to exceed expectation, to form a class lecture.