outcomes assessment 2 program assessment joseph a. shaeiwitz west virginia university...
TRANSCRIPT
Outcomes Assessment 2
Program Assessment
Joseph A. ShaeiwitzWest Virginia
u
Daina M. BriedisMichigan State
Outline
ABET and engineering criteria Program objectives Program outcomes Assessment
performance criteria assessment measures – direct and
indirect rubrics
Review
Outline
ABET and engineering criteria Program objectives Program outcomes Assessment
performance criteria assessment measures – direct and
indirect rubrics
Review
Initial Quiz
What do you know about ABET? Describe the engineering criteria? According to ABET’s definitions,
what is the difference between outcomes and objectives?
Identify four assessment methods. Classify each as direct or indirect.
Initial Quiz
What do you know about ABET? Describe the engineering criteria? According to ABET’s definitions,
what is the difference between outcomes and objectives?
Identify four assessment methods. Classify each as direct or indirect.
Initial Quiz
What do you know about ABET? Describe the engineering criteria? According to ABET’s definitions,
what is the difference between outcomes and objectives?
Identify four assessment methods. Classify each as direct or indirect.
Initial Quiz
What do you know about ABET? Describe the engineering criteria? According to ABET’s definitions,
what is the difference between outcomes and objectives?
Identify four assessment methods. Classify each as direct or indirect.
Initial Quiz
What do you know about ABET? Describe the engineering criteria? According to ABET’s definitions,
what is the difference between outcomes and objectives?
Identify four assessment methods. Classify each as direct or indirect.
ABET and Engineering Criteria
ABET = Accreditation Board for Engineering and Technology
Engineering criteria changed to assessment basis (TQI) around 2000
Must prove that students are achieving objectives and outcomes measure output, feedback model previously, feed forward model
input outputassumed
curriculum
Feed Forward Model
oneclass
oneclass
education process
one course
one course
enteringcollege
graduate alumnus
Feedback Model
The Two Loops of the Engineering Criteria
Establish Indicatorsthat Objectives are
Being Achieved
Determine HowOutcomes will be
Assessed
Determine HowOutcomes will be
Achieved
Determine OutcomesRequired to Achieve
Objectives
Determine educational objectives
Evaluate/Assess
Input fromConstituencies
Formal InstructionStudent Activities
ABET and Engineering Criteria 8 criteria
students program educational objectives program outcomes and assessment professional component faculty facilities institutional support and financial
resources program criteria
ABET and Engineering Criteria
Focus of this workshop program educational objectives
definition how to establish how to assess
program outcomes and assessment definition how to establish how to assess
Minute PaperClearest vs. Muddiest Point
What have you just learned about ABET and assessment?
What points are the clearest? What points are the “muddiest?”
Outline
ABET and engineering criteria Program objectives Program outcomes Assessment
performance criteria assessment measures – direct and
indirect rubrics
Review
Program Objectives*
“…broad statements that describe the career and professional accomplishments that the program is preparing the graduates to achieve.”*
“Our graduates will be successful …” must define “successful”
*Criteria for Accrediting Engineering Programs, 2007-2008 Cycle, ABET, Inc., Baltimore, MD, http://www.abet.org
Program Objectives*
must be detailed and published include constituencies/periodically evaluated educational program to achieve outcomes
(defined later) and to prepare graduates for accomplishments that achieve objectives
ongoing evaluation to determine extent objectives attained, use results for program improvement
*Criteria for Accrediting Engineering Programs, 2007-2008 Cycle, ABET, Inc., Baltimore, MD, http://www.abet.org
Exercise
New program in nanobiomolecular engineering
Define two program objectives.
Outline
ABET and engineering criteria Program objectives Program outcomes Assessment
performance criteria assessment measures – direct and
indirect rubrics
Review
Program Outcomes* “…statements of what students are expected to
know and be able to do by the time of their graduation.”
Outcomes must “…foster attainment of the program objectives…”
Process to produce outcomes Assessment process, with documented results
demonstrating measurement demonstrating degree of achievement
Evidence results used for program improvement
*Criteria for Accrediting Engineering Programs, 2007-2008 Cycle, ABET, Inc., Baltimore, MD, http://www.abet.org
Program Outcomes Minimum outcomes “a-k” plus
program criteria Opportunity to be unique, i.e., define
unique outcomes, not just repeat “a-k”
Helpful to map program-defined outcomes into a-k
Helpful to map outcomes into classes Necessary to map outcomes into
objectives
WVUABET
1chemicalprocess
2communicate
3 computers
4 learn
independently and group
work
5lab and data
analysis
6continuing education
7safety,
societal,environmental
8ethics
9broad
education
a. apply math, sci, engr b. expts - design, conduct, analyze, interpret data c. design system d. multidisciplinary teams e. identify, formulate, solve engineering problems
f. professional and ethics g. communication h. broad education - global impact i. life-long learning j. contemporary issues k. use techniques, skills, modern engineering tools
WVU outcomeWVU class
1chemicalprocess
2communicate
3 computers
4 learn
independently and group
work
5lab and data
analysis
6continuing education
7safety,
societal,environmenta
l
8ethics
9broad
education
CHE 201 CHE 202 CHE 230 CHE 310 CHE 311 CHE 312 CHE 315 CHE 320 CHE 325 CHE 450/451 CHE 455/456
Exercise
Define three unique program outcomes for the nanobiomolecular engineering program.
Outline
ABET and engineering criteria Program objectives Program outcomes Assessment
performance criteria assessment measures – direct and
indirect rubrics
Review
Performance Criteria
What will students do to demonstrate achievement of outcome
Example: “An ability to communicate effectively.” (3g in ABET list)
What are attributes of effective communication?
Performance Criteria for Effective Communication When making an oral presentation,
students will maintain eye contact …
A written report will follow a prescribed format demonstrate proper grammar and
punctuation adhere to commonly accepted word
usage
Education
al
Objective
Learning
Outcome
Contempo-raryIssues
Professionally responsible
1.Has knowledge of current technological issues related to XXX engineering and society 2. Is able to discuss major political and societal issues and their pertinence to XXX engineering
Performan
ce Criteria
Exercise
Define two or three performance criteria for one of the outcomes previously defined.
Outline
ABET and engineering criteria Program objectives Program outcomes Assessment
performance criteria assessment measures – direct and
indirect rubrics
Review
Indirect vs. Direct Assessment Indirect – based mostly on student self-
evaluation surveys interviews focus groups
Direct – by faculty or some other means of evaluation of student performance (advisory boards evaluate design projects)
Indirect vs. Direct Assessment Indirect
necessary, but not sufficient provides uncalibrated snapshot
self-assessment not necessarily reliable Terminology may be unfamiliar
Direct necessary for quality assessment plan not now for all, but new Program
Evaluators are being trained to look for this feature
relies on faculty experience, expertise, and judgment
How do we know if the students have the requisite outcomes?
When the students think they do, based on student survey results
With direct evidence from student workEvidence is the key to accreditation
Faculty evaluation of student work is the key to providing evidence
Primary assessment of student outcomes should be based on student work (direct) e.g., student portfolios, student projects,
assignments and exams, some employer surveys where skill is observed
Secondary evidence Senior exit surveys, alumni surveys, employer
surveys (qualitative evidence based on opinion), other
Combination of both methods – triangulation
Assessment Measures
Indirect Assessment Measures
Surveys Interviews Course satisfaction surveys
Direct Assessment Measures
End-of-Course Assessments Targeted Assignments Capstone Experiences Capstone Exams Portfolios
End-of-Course Assessments
Course should have objectives perhaps set by department committee
Related to program outcomes Assigned problems (assignments,
exams, projects) each related to course objectives
Evaluation/reflection by instructor
End-of-Course Assessments
Advantages quick and easy assessment can be done in parallel with
grading Disadvantage
not comprehensive no big picture
Opinion a component of assessment plan
Targeted Assignments/Problems
Key assignments that relate to specific program outcomes
Multiple assignments per outcome recommended
Integrate through curriculum can demonstrate progress toward
achievement of program outcome
Targeted Assignments/Problems
Advantages quick and easy assessment can be done in parallel
with grading Disadvantages
none really need consistent evaluation method
with reliable inter-rater reliability
Capstone Experiences
Can be design, laboratory, research All programs have them Where students are supposed to
demonstrate and synthesize what learned
Usually includes teamwork, communication
Capstone Experiences
Advantages already required in program assessment can be done in parallel
with grading Disadvantages
none really need consistent evaluation method
with reliable inter-rater reliability
Capstone Exams
FE Exam detailed, subject-related results
available Department-generated
Capstone Exams
Advantages FE is standardized the most direct measure possible
Disadvantages FE may not set bar as high as some want students may not take department-
generated exam seriously if results do not impact grades or graduation
Portfolios
Collection of student work demonstrating outcomes
Must also evaluate the portfolio Can also have students reflect on
work in portfolio
Portfolios
Advantage comprehensive
Disadvantages portfolio evaluation is additional work need consistent evaluation method
with reliable inter-rater reliability
Other Direct Assessment Methods
Journals Concept maps Oral presentations with follow-up
questions (like M.S./Ph.D. defense)
Exercise
Select an assessment method to be used for direct assessment of program outcomes previously defined.
Outline
ABET and engineering criteria Program objectives Program outcomes Assessment
performance criteria assessment measures – direct and
indirect rubrics
Review
Rubrics
A set of categories developed from the performance criteria that define extent to which the performance criteria are met (progression towards attaining).
Developing Rubrics
Define levels of performance for each performance criterion best first to define top and bottom
levels of performance – then fill in middle ground
3-5 levels of performance recommended
Developing Rubrics
Standardized method to ensure inter-rater reliability
Specific definitions of terms like excellent, understand, not acceptable, exceeds expectations
Initial effort and periodic review required
Developing Rubrics If have five levels of performance, can
make each level a grade if have three levels, can make each level
A/C/F, and interpolate for B/D Use rubric for assessment and grading Advantage: Students and faculty have
clearly defined criteria for grading of what appears to be subjective (lab and design reports, oral reports, etc.)
“Application of math & science”
Problem Assessment Form Connects physical model with math model Able to write unsteady state mass balance Makes appropriate substitution for flow terms Makes appropriate simplification for flow terms Converts differential equation into Laplace form
correctly Understands idea for output/input form of
transfer function Correctly combines Laplace transforms in series Correct answer
Rubrics – Assessment Scale“Application of math & science” (5=high)
Level 5 Formulates models
correctly Applies calculus or linear
algebra to solve problems Correct calculations Correct statistical analysis . . .
Level 3 Formulates models with
some trouble Some understanding of
calc/linear algebra applications
Minor calc errors Minor statistical errors . . .
Level 1 Not able to model Cannot apply
calc/linear algebra Incorrect calculations Does not apply
statistics . . .
Education
al
Objective
Learning
Outcome
Contempo-raryIssues
Professionally responsible
1.Has knowledge of current technological issues related to XXX engineering and society 2. Is able to discuss major political and societal issues and their pertinence to XXX engineering
Performan
ce Criteria
Outcome: A knowledge of contemporary issues Performance Criteria Scoring Rubrics
Rating Scale & Element
Needs Improvement 1 2 Met Expectations 3 4Exceeded
Expectations5 N/A
Has knowledge of current technological issues related to chemical engineering and society (global warming, resource depletion, waste proliferation, etc.)
Has minimal knowledge of technological issues and their relevance to chemical engineering; has weak connection between the issue and scientific principles for analysis and has trouble developing solutions.
Has reasonable knowledge of technological issues; some may not be directly relevant to chemical engineering; can apply scientific principles to analysis and suggest solutions when guided.
Has thorough knowledge of current technology issues related to chemical engineering and is able to analyze them and propose solutions using scientific principles
Has knowledge of and is able to discuss major societal and political issues and their pertinence to chemical engineering
Has minimal knowledge of societal and political issues; if given an issue, does not see its connection to engineering without instruction; is minimally effective in discussion and presentation of such issues
Has reasonable knowledge of societal and political issues; recognizes some connection to chemical engineering, but misses the details; is somewhat effective in discussing and presenting such issues when prompted
Has thorough knowledge of societal and political issues related to chemical engineering; recognizes the “big picture” and the details; presents strong discussion of such issues
Rubric for Oral Presentations
Attribute 1-Not proficient 2-Progressing to proficiency
3-Proficient 4-Superior proficiency
Score
Effective use of Visual Aids (VA) Clarity and readability not clear or
readable difficulty reading clear and readable superior clarity and
readability
Use of space on VA VA unreadable because too crowded
too little or too much information of VA
appropriate amount of information on VA
VAs very well laid out
Lettering readable font unreadable font too small font readable Color, over- or under-use (if used) colors too hard to
distinguish, colors do not project well
poor choice and use of colors
primary/easily distinguishable colors
use of color enhances clarity of presentation
Wording concise slides full of text slides too wordy slides appropriate Appropriate amount of information per VA
so much information per VA or so much missing information to make VA useless
too much information per VA, missing information such as size of total pie
appropriate level of information per slide
Presentation Organization Logical order of topics totally disjointed,
no organization some items presented out of order
organization as per guidelines
superior organization enhances communication
Appropriate use of time: Not too long /short
far too long or far too short
somewhat too long or too short
appropriate length
Complete "story" told story missing, no story told
story incomplete complete story told
Introduction: Problem stated problem not stated, problem poorly stated
problem clearly stated
problem clearly stated, good perspective on problem shown
Rubric for Written Presentations
Attribute 1-Not proficient 2-Progressing to proficiency
3-Proficient 4-Superior proficiency
Score
Report Mechanics Organization inappropriate
content of most sections of report
some content in inappropriate section of report
content appropriate to all sections of report
unique organization enhances readability and/or understandability of report
Complete Story Told no story told, very incomplete
aspects of complete story missing
complete story told additional material enhances quality of report
Aesthetics unacceptable – e.g., tables and figures cannot be read/understood, fonts difficult to read
some portions are sloppy and difficult to read
text, tables, figures readable and understandable
text, tables, figures so clear and understandable as to enhance report impact
Format so many format errors as to make report useless
some format errors format followed unique format aspects that enhance report impact
Spelling any spelling errors only spelling errors are different spellings for same pronunciation
no spelling errors
Grammar and Punctuation too many grammar and punctuation errors
grammar and punctuation errors
only a very few minor grammar or punctuation errors
no grammar or punctuation errors
Length far too long or too short
too long or too short
appropriate length
Rubric for Majors and Design Projects
Attribute 1-Not proficient 2-Progressing to proficiency
3-Proficient 4-Superior proficiency
Score
Design of equipment, Analysis of performance of existing equipment, Understand interrelationship between equipment in process
Design of individual equipment major errors in individual equipment design
some errors in equipment design
equipment designed correctly
unique aspects of equipment design enhance result
Understand interrelationship between equipment on flowsheet
no understanding of equipment interrelationship
minimum understanding of equipment interrelationship
clear understanding of equipment interrelationship
exploitation of equipment interrelationship to enhance result
Constraints/limitations of individual equipment and flowsheet understood
constraints/ limitations not understood
not all constraints/ limitations understood
constraints/ limitations clearly understood
exploitation of constraints/ limitations to enhance result
Response to questions indicates understanding of ChE principles
response to questions demonstrates lack of understanding
response to questions shows gaps in understanding
response to questions shows clear understanding
response to questions shows superior understanding
Significance of conclusions understood
lack of understanding, no explanations
gaps in understanding, few explanations
clear understanding and explanations
superior understanding with in-depth explanations
Apply chemistry, math, physics, life science, engineering science
Apply engineering science inability to apply principles
a few basic principles applied
most principles applied, demonstration of effect on design
all principles applied and interwoven with engineering to complete design
Rubrics
Additional rubrics athttp://www.che.cemr.wvu.edu/ugrad/outcomes/rubrics/index.php
Instructions on rubrics
http://webquest.sdsu.edu/rubrics/weblessons.htm
Exercise
For the two or three performance criteria for one of the outcomes previously defined, and assuming the assessment method previously defined, begin to develop an evaluation rubric.
Outline
ABET and engineering criteria Program objectives Program outcomes Assessment
performance criteria assessment measures – direct and
indirect rubrics
Review
Recommendation
Strongly recommend adopting direct assessment measures into assessment activities
Will be expected in the near future!
Exercise
What are the two most important things that you learned in this workshop?
What is still unclear to you about program assessment?
Questions
?