Download - Katherine Perez Jacqueline Peña Office of Planning and Institutional Effectiveness January 2009
Katherine PerezJacqueline Peña
Office of Planning and Institutional EffectivenessJanuary 2009
SLO matrix SLOs and Corresponding Assessment Methods
Workshop: Revising SLOs and Assessment Methods
Reporting Results and Use of Results Workshop: Writing the Results and Use of Results
PO matrix POs and Corresponding Assessment Methods Reporting Results and Use of Results
Workshop: Revising POs and Assessment Methods + Writing Results and Use of Results
Additional individualized assistance
SACS Accreditation Key SACS Deadlines:
September 10, 2009—Report due March 8-12, 2010—Onsite Review
Key IE Deadlines: February 1, 2009—Fall 2008 SLO matrices due June 1, 2009—Spring 2009 SLO and PO matrices
due
Continuous improvement Institutional Effectiveness Sound, research-based and assessment-
based decisions
Student Learning Outcome(Stated in Measurable Terms)
Assessment Method Results (Data Summary and Analysis)
Use of Results for Improving Student Learning
Link to the Mission:
Criteria: Can be observed and measured Relates to student learning towards the end of
the program Reflects an important concept
Formula: Who + Action Verb + What Students will identify and apply the key
components of a learning system. Students will identify and apply the five
key components of a learning system.
Principle 3.4.12 The institution’s use of technology enhances
student learning and is appropriate for meeting the objectives of its programs. Students have access to and training in the use of technology. (Technology use)
Required for all undergraduate programs Optional for 2007-2008 SLO matrices Required for 2008-2009 SLO matrices
General (not appropriate for SACS): Students will use information technology to gather and disseminate
information.
More Specific: Graduates of the program will demonstrate the ability to use
appropriate computer technology, software, and the Internet to complete their capstone research papers.
Much more Specific (appropriate for SACS): Students will be able to effectively demonstrate information
technology skills by locating and retrieving information on economic topics and issues, published research in Economics and related fields, and/or by finding information about the generation, construction, and meaning of economic data for their final research projects.
Students will write and present a capstone project that requires the use of Word, Excel, PowerPoint, and information technology.
Action Verbs [Who + Action Verb + What]
Knowledge
Comprehension
Application
Analysis Synthesis Evaluation
List Summarize Apply Analyze Combine Assess
Define Describe Demonstrate
Separate Integrate Rank
Describe Interpret Illustrate Explain Modify Test
Identify Predict Solve Connect Create Measure
Show Distinguish Examine Classify Design Judge
Examine Estimate Modify Arrange Compose Critique
State Paraphrase Classify Compare Prepare Discriminate
Find Compare Calculate Contrast Write Support
Label Contrast Compute Infer Formulate Hypothesize
Infer Construct Order Construct
Implement
Note: This table reflects the original Bloom’s Taxonomy Table. The modified version places evaluation as the fifth column and synthesis (creating) as the sixth column.
1. Artifact2. Collection of the artifact
Where, when, and how Census versus sampling
3. Criteria Minimum standards on a rubric/scale or
the percentage of correct items
4. Evaluation of the artifact Faculty panel or external evaluators
(reliability) Rubrics, embedded questions, etc.
Common Direct Measures
Common Indirect Measures
•Standardized exams•Exit examinations•Portfolios•Pre-tests and post-tests•Locally developed exams•Papers•Oral presentations•Behavioral observations•Thesis/dissertation
•Surveys or questionnaires•Student perception•Alumni perception•Employer perception•Focus groups•Interviews•Student records
Student Learning Outcome(Stated in Measurable Terms)
Assessment Method Results (Data Summary and Analysis)
Doctoral students will identify, define, and apply key literary theories (including formalism, historicism, mimesis, post-colonialism, structuralism, post- structuralism, psychoanalysis, modernism, and postmodernism) to their specialization areas in the field of English literature.
After completing all their courses, English literature doctoral students will take the theory comprehensive exam, which will be divided into five key literary theories (one theory per exam question) that the students learn throughout their coursework.
All comprehensive exams will be evaluated by a faculty panel of 3 members using a department rubric.
For each of the five literary theories, students will achieve a 3 or better on a 4-point rubric:1 = poor2 = acceptable3 = good4 = excellent
Use of Results for Improving Student Learning
Summary of Results Direct measures
Test items Performance as determined by rubrics
Indirect measures Surveys and questionnaires Interview and focus group data
Format Narrative Tables or charts
Analysis of results If pertinent, explain results in a narrative form by
interpreting results or using qualitative analysis of the data.
Every student learning outcome must have at least: One set of results One student learning improvement strategy (use of results)
All the students passed the assessment. 75% of the students met the criteria for
success. Our students passed the dissertation
defense on the first attempt. 75% of the students (n=15) achieved a 3
or better on all the rubric categories for the capstone course research paper.
Frequency of Student Results for all Four Categories of the Research Paper (n=20)
1 NOVICE
2APPRENT
ICE
3PRACTITIO
NER
4EXPERT
PERCENTAGE MEETING CRITERIA
Grammar 2 (10%) 0 (10%) 8 (40%) 8 (40%) 80% (n=16) met the criteria.
Essay Structure
4 (20%) 0 (15%) 11 (55%) 2 (10%) 65% (n=13) met the criteria.
Coherence of Argument
2 (10%) 0 (35%) 10 (50%) 1 (5%) 55% (n=11) met the criteria.
Application of MLA Rules
3 (15%) 0 (25%) 12 (60%) 0 (0%) 60% (n=12) met the criteria.
Criteria: Students will achieve a 3 or better on a 4-point rubric on all five sections of the master’s level thesis.
Student Learning Outcome(Stated in Measurable Terms)
Assessment Method Results (Data Summary and Analysis)
Doctoral students will identify, define, and apply key literary theories (including formalism, historicism, mimesis, post-colonialism, structuralism, post- structuralism, psychoanalysis, modernism, and postmodernism) to their specialization areas in the field of English literature.
After completing all their courses, English literature doctoral students will take the theory comprehensive exam, which will be divided into five key literary theories (one theory per exam question) that the students learn throughout their coursework.
All comprehensive exams will be evaluated by a faculty panel of 3 members using a department rubric.
For each of the five literary theories, students will achieve a 3 or better on a 4-point rubric:1 = poor2 = acceptable3 = good4 = excellent
20 students took the theory comprehensive exam during the 2008-2009 academic year.
70% of the students achieved the minimum criterion (i.e. scoring a 3 or better on all five theory components).
20% of the students scored less than 3 on at least 1 theory component.
10% of the students (n=2) scored less than a 3 on all five theory components and had to be rescheduled to retake the test during the summer during the summer B session.
Use of Results for Improving Student Learning
DO DON’T•DO focus on making specific improvements based on faculty consensus.
•DON’T focus on simply planning for improvements or making improvements without faculty feedback.
•DO address specific program improvements that will impact student learning.
•DON’T address assessment improvement plans such as revising the rubric.
•DO use concrete ideas. •DON’T write vague ideas.
•DO state strategies that are sustainable and feasible.
•DON’T use strategies that are impossible to complete in one year considering your resources.
•DO use strategies that can improve the curriculum and help students learn outside of courses.
•DON’T focus on only improving the curriculum.
Target met. Will continue to monitor. The faculty will meet and revise the three
introductory courses. A larger sample will be obtained. We will revise the rubric and have a calibration
session with the faculty prior to evaluating the student papers each semester.
A capstone course will be created that emphasizes research and thesis writing methods.
A student resource center will be created with computer equipment, writing tutors, and statisticians who can assist our students with course- and thesis-related writing and research.
Student Learning Outcome(Stated in Measurable Terms)
Assessment Method Results (Data Summary and Analysis)
Doctoral students will identify, define, and apply key literary theories (including formalism, historicism, mimesis, post-colonialism, structuralism, post- structuralism, psychoanalysis, modernism, and postmodernism) to their specialization areas in the field of English literature.
After completing all their courses, English literature doctoral students will take the theory comprehensive exam, which will be divided into five key literary theories (one theory per exam question) that the students learn throughout their coursework.
All comprehensive exams will be evaluated by a faculty panel of 3 members using a department rubric.
For each of the five literary theories, students will achieve a 3 or better on a 4-point rubric:1 = poor2 = acceptable3 = good4 = excellent
20 students took the theory comprehensive exam during the 2008-2009 academic year.
70% of the students achieved the minimum criterion (i.e. scoring a 3 or better on all five theory components).
20% of the students scored less than 3 on at least 1 theory component.
10% of the students (n=2) scored less than a 3 on all five theory components and had to be rescheduled to retake the test during the summer during the summer B session.
Use of Results for Improving Student Learning1. The department will assist faculty in organizing student-led comprehensive exam study groups a year prior to the expected
examination deadline. 2. A theory unit will be added to each literature course so that students are constantly exposed to multiple literary theories throughout
their coursework.3. In addition, the department will create and distribute a study list that includes major topics covered on the comprehensive exams,
useful websites for tutorials and reviews, and a bibliography of texts that are critical for success with the five theoretical concepts.
Student Learning Outcome(Stated in Measurable Terms)
Assessment Methods Results (Data Summary and Analysis)
Undergraduate English students will communicate effectively in written format.
During their final semester, in their senior seminar course (PPP 4677), students will write a 10-page research paper on a critical topic as applied to an English, Caribbean, or American novel.
20% of the students’ papers will be randomly selected and evaluated by a faculty panel of 3 members using a rubric that measures:(1) Grammar, (2) Essay Structure, (3) Coherence of Argument, (4) Application of MLA rules.
Students will achieve a 3 or better on a 4-point rubric:1 = novice2 = apprentice3 = practitioner4 = expert
100 students took the seminar in 07-08.20 papers were sampled.
100% met the minimum criterion for success (i.e. scoring a 3 or better on all four components).
Use of Results for Improving Student Learning
1. Undergraduate English students will be required to visit the Center for Academic Excellence once each semester to work with a writing tutoring for at least one 60-minute session.
2. Undergraduate English students who have difficulty with grammar, essay structure, coherence of argument, or application of MLA rules at any point during their academic program will be referred to a writing tutor at the Center for Academic Excellence.
3. Every undergraduate English course will require at least one 5-page paper that requires the use of MLA rules and principles.
1 2 3 4
G 0 0 12 8
E 0 0 10 10
C 0 0 15 5
M 0 0 20 0
71% 29%
Overview and Comparisons: Give an overview or summary of all the outcomes
together Discuss trends that you have seen over the years
Explanations Provide qualitative explanations for poor results or
exceptionally high results Notes and documentation affecting results
Response rate (e.g. Only 50% of the students completed the project.)
Inter-rater reliability (e.g. 2 faculty members reviewed the artifacts and the inter-rater reliability was only 60%.)
Assessment Commentary Revise or create instrument (e.g. artifact, rubric) Modify assessment methods (e.g. data collection,
sampling, criteria, evaluation process)
Write or revise your Results and Use of Results.
Results: Clearly state the results of each PO assessment (table or
narrative).
Use of Results: Explicitly state how you will use the results to improve the
program during the following assessment cycle/year.
Summary Page:1. Overview of results and use of results (looking at the big
picture)2. Trends over the last year or several years3. Comparison to national or state trends or data4. Explanations for poor or unexpected results5. Notes or documentation that could explain the results6. Assessment improvement plans
END OF THE SLO PORTION
What Is Next?
10-minute overview of Program Outcomes
PO writing assistance Continued SLO writing assistance
Program-level outcomes Focus on student success (not student learning) Formula = Who + Action Verb + What Examples:Graduates seeking employment in the field will
find such employment within 6 months of graduation.
Graduates will be satisfied with advising and mentoring services.
Faculty will publish at least one journal article or book chapter every two years.
Program Outcome(Stated in Measurable Terms)
Assessment Method Results (Data Summary and Analysis)
Full-time students will graduate from the English doctoral program within 7 years of program admission.
The program administrative assistant will keep track of students’ progress in the program from date of admission, making updates for each student at the end of each academic semester.
10 students graduated from the English doctoral program during the 2008-2009 academic year.
The program assistant analyzed the completion timeframe for each individual student and discovered that all of the graduating students spent over half of their graduate time working on the dissertation (3-6 years).
Use of Results for Improving Student Learning1. Doctoral advisors and dissertation chairs will be required to meet with their students twice a semester to go over the program of study, the students’
progress, and the students future goals and timelines. 2. The college will support the graduate student organization in developing dissertation work/study groups, for all incoming students in the program.
The dissertation groups should meet once a month to share drafts, establish new goals, create completion timelines, and do any other activities that the group members find helpful for successful completion of the program.
3. Students will be mailed and emailed warning letters after 4 years in the program reminding them that they need to graduate within 7 years of program admission.
Time of Completion for the 10 Students
5 students 6 years
2 students 7 years
2 students 8 years
1 student 9 years
Take 15 minutes to review or write at least one program outcome with corresponding assessment methods, results, and use of results.
PO Formula:Who + Action Verb + What
Assessment Methods Formula:1. State the instrument/method/tool (e.g. survey, national exam
results).2. Explain the data collection steps (e.g. when, where, how).3. State the success criteria (minimum standards for success).
Results: Clearly state the results of each PO assessment (table or
narrative).
Use of Results: Explicitly state how you will use the results to improve the
program during the following assessment cycle/year.
Marta Perez Director [email protected] 305-348-2733
Maria Corrales Coordinator [email protected] 305-348-0459
Katherine Perez Coordinator [email protected] 305-348-1418
Jacqueline Peña Coordinator [email protected]. 305-348-1367
Mayelin Felipe Computer Specialist [email protected] 305-348-0115
Karla Felipe Computer Specialist [email protected] 305-348-0115
Amanda Berhaupt Graduate Assistant [email protected] 305-348-2731
Randhir Kaur Graduate Assistant [email protected] 305-348-2731