engineering education research center, washington state university capstone engineering design...
Post on 20-Dec-2015
217 views
TRANSCRIPT
Engineering Education Research Center, Washington State University http://eerc.wsu.edu
Capstone Engineering Design Learning and Assessment
Denny Davis, PhD, PE Washington State University
Engineering Education SeminarPurdue UniversitySeptember 18, 2008
Engineering Education Research Center, Washington State University http://eerc.wsu.edu
Acknowledgements
• Project Leadership Team– Denny Davis, Howard Davis, Michael Trevisan,
Shane Brown, Washington State University– Steven Beyerlein, Jay McCormack, University of Idaho– Phillip Thompson, Seattle University– Olakunle Harrison, Tuskegee University
• Project Consultants– Susannah Howe, Smith College– Patricia Brackin, Rose-Hulman Institute of Technology– Paul Leiffer, LeTourneau University– Durward Sobek, Montana State University– Jerine Pegg, University of Idaho
• Funding– NSF DUE 0717561: Capstone Engineering Design Assessment:
Development, Testing, and Adoption Research
Engineering Education Research Center, Washington State University http://eerc.wsu.edu
Project Goal and Objectives
• Goal – Develop an integrated system for effective,
sustainable assessment of capstone engineering design outcomes
• Objectives– Develop an assessment system suitable for broad
adoption in capstone engineering design courses– Document effectiveness of the assessment
system to measure student achievement– Investigate factors that affect assessment
adoption by the capstone engineering design community
Engineering Education Research Center, Washington State University http://eerc.wsu.edu
Guiding Research Questions
Research Questions: 1. To what extent can assessments
measure desired performances in learner and solution development?
2. How can assessments be integrated effectively into capstone design courses?
3. How can adoption of assessments be encouraged in capstone design courses?
Engineering Education Research Center, Washington State University http://eerc.wsu.edu
What Outcomes?
Engineering Education Research Center, Washington State University http://eerc.wsu.edu
Areas of Performance
DesignProcesses
SolutionAssets
SolutionSolutionDevelopmentDevelopment
LearnerLearnerDevelopmentDevelopment
Team-workProfessional
Development
2.2.InformationInformationGatheringGathering
4.4.IdeaIdea
GenerationGeneration
3. 3. ProblemProblem
DefinitionDefinition5.5.
IdeaIdeaEvaluationEvaluation
6.6.IdeaIdea
RefinementRefinement
7.7.ImplementationImplementation
1.1.RecognitionRecognitionof Challengeof Challenge
Iteration/reflection
Engineering Education Research Center, Washington State University http://eerc.wsu.edu
Learner Development
• Professional Development– Individuals performing and improving
individual skills and attributes essential to engineering design
• Teamwork– Teams developing and implementing
collective processes that support team productivity in design
Engineering Education Research Center, Washington State University http://eerc.wsu.edu
Solution Development
• Design Processes– Practices implemented that effectively
and efficiently facilitate the production of valuable project assets
• Solution Assets– Results from a design project that meet
needs and deliver satisfaction and value to key project stakeholders
Engineering Education Research Center, Washington State University http://eerc.wsu.edu
Assessment Framework
Engineering Education Research Center, Washington State University http://eerc.wsu.edu
Interpretation
Model
Observation
Profile ofProfessional
• Roles• Behaviors
PerformanceCriteria• Learners• Solutions
Course Context• Project Mix
• Professor Preparation • Infrastructure/Resources
• Role in Program
Sampling• Student Sample
• Knowledge Sample• Time of Sample
Measures• Outcomes
• Levels• Metrics
Tasks• Individual Tasks
• Team Tasks
Reporting• Learning• Grading
• Improvement
Expectations• Students• Faculty• Clients
• Administrators• Accreditors• Employers
Scoring• Training
• Reliability
Profile ofLearner
• Background• Skill Set
• Motivation
Capstone Course Assessment Framework
*NRC, Knowing What Students Know.
Assessment Triangle*
Engineering Education Research Center, Washington State University http://eerc.wsu.edu
Performance Criteria: Learner
• Professional Development– Individuals document professional development
aligned with their personal and project needs, professional behaviors, and ways of a reflective practitioner.
• Teamwork– Teams demonstrate high productivity, synergistic
individual and joint contributions, a supportive team climate, and well-developed team processes.
Engineering Education Research Center, Washington State University http://eerc.wsu.edu
Performance Criteria: Solution
• Design Processes– Designers resourcefully iterate among problem
scoping, concept generation, and solution realization activities to co-develop problem understanding and a responsive design solution.
• Solution Assets– Designers deliver and effectively defend
solutions that satisfy stakeholder needs for functionality, financial benefit, implementation feasibility, and impacts on society.
Engineering Education Research Center, Washington State University http://eerc.wsu.edu
Purposes of Assessment
• Measure Achievement– Guide changes in instruction– Gather data for grading– Document student achievement in course– Study learning processes
• Facilitate Learning– Guide learners’ effort to greater learning– Teach self- and peer-assessment skills– Establish reflective practitioner mindset
Engineering Education Research Center, Washington State University http://eerc.wsu.edu
Assessment Design
Engineering Education Research Center, Washington State University http://eerc.wsu.edu
Assessing Reflective Practice
• Instructional Activities– Assign reflections on performances– Assign similar reflections multiple times
• Evidence– Quality of reflections improve over time– Reflective assignments reveal gains from natural
reflective practice
Reflectionon Goals
Reflectionon Progress
Reflection onAchievement
Engineering Education Research Center, Washington State University http://eerc.wsu.edu
Capstone Course Context
Problem ScopingPhase
Concept GenerationPhase
Solution RealizationPhase
Project Timeline
Solution Development
Learner Development?? ?? ???? ?? ??
Engineering Education Research Center, Washington State University http://eerc.wsu.edu
Structure of Assignments
Self-Rating• Importance• Level
Set Target• Describe it• Action plan
Progress• A strength• An opportunity
Formative
Self-Rating• Importance• Level
Strengths• Describe them• Explain causes
Extension• Envision it• Define impacts
Summative
Engineering Education Research Center, Washington State University http://eerc.wsu.edu
Structure of Feedback
StudentResponse
Self-Rating
Narrative• Descriptions• Analysis• Extension
Factor 1:
Factor 2:
Level1
Level3
Level5
Level2
Level4
PerformanceMetric
X
X
Narrative Feedback
• Comments and suggestions
Engineering Education Research Center, Washington State University http://eerc.wsu.edu
Assessing Learner Development
Problem ScopingPhase
Concept GenerationPhase
Solution RealizationPhase
Professional Development Processes
Team Contract
[T]Feedback
Project Timeline
Team Member
Citizenship [I]
Feedback
Team Development Processes
Act
iviti
esTe
am-
focu
sIn
divi
dual
-fo
cus
Team Member
Citizenship [I]
Feedback
Teamwork Achieved [I] Assessment
Assessment Notation: [I] = individual assignment [T] = team assignment
Growth Planning
[I] Feedback
Team Processes
[I] Feedback
Growth Progress [I] Feedback
Growth Achieved [I] AssessmentProfessional
Practices [I] Feedback
Engineering Education Research Center, Washington State University http://eerc.wsu.edu
Professional Development Assessments
Technical Interpersonal Individual
□ Analyzing information □ Communicating □ Practicing self-growth
□ Solving problems □ Collaborating □ Being a high achiever
□ Designing products □ Relating inclusively □ Adapting to change
□ Researching questions □ Leading others □ Serving professionally
GrowthPlanning
ProfessionalPractices
GrowthProgress
GrowthAchieved
Engineering Education Research Center, Washington State University http://eerc.wsu.edu
Teamwork Assessments
Team Relationship Joint Achievement Member Contribution Team Information
□ Building an inclusive climate
□ Establishing shared team goals
□ Allocating responsibility
□ Achieving in-team communication
□ Gaining buy-in, interdependence
□ Managing tasks to achieve team goals
□ Achieving quality work from members
□ Stakeholder communication
□ Resolving conflicts
□ Producing consensus
□ Facilitating team member growth
□ Building shared knowledge assets
TeamContract
TeamProcesses
Team MemberCitizenship
TeamworkAchieved
Engineering Education Research Center, Washington State University http://eerc.wsu.edu
Assessing Solution Development
Problem ScopingPhase
Concept GenerationPhase
Solution RealizationPhase
implementationdesign
Solution Realization Processes
Problem Scoping
Processes [I]Feedback
Project Timeline
Defined Problem [T] /
Design Reflection [I] Assessments
Concept Generation Processes
[I] Feedback
Selected Concept [T] /
Design Reflection [I] Assessments
Problem Scoping Processes
Concept Generation Processes
Act
iviti
esFe
edba
ckEv
alua
tion
Solution Realization
Processes [I] Feedback
Proposed Solution [T] /
Design Reflection [I] Assessments
Solution Realization Processes
[I] Feedback
Proposed Solution [T] /
Design Reflection [I] Assessments
Assessment Notation: [I] = individual assignment [T] = team assignment
Engineering Education Research Center, Washington State University http://eerc.wsu.edu
Design Processes/Solution Assets
Design Phase– Problem scoping– Concept generation– Solution realization
[Design Phase]Processes
DesignReflection
[Asset]
[Design Phase]Processes
DesignReflection
[Asset]
Asset– Defined problem– Selected concept– Proposed solution
Engineering Education Research Center, Washington State University http://eerc.wsu.edu
Assessment Evaluation
Engineering Education Research Center, Washington State University http://eerc.wsu.edu
Assessment Validity and Reliability
• Scoring Reliability– Inter-rater reliability
• Content Validity– Instructor and practitioner content
analysis
• Value to Users– Value gained by students and instructors
Engineering Education Research Center, Washington State University http://eerc.wsu.edu
Web-Based Implementation
Engineering Education Research Center, Washington State University http://eerc.wsu.edu
Why Web-Based Implementation?
• Practicality– Reduce paperwork of assessment process– Enable automated processing of data– Facilitate testing of assessments
• Effectiveness– More rapid feedback to students (learning)– Better view of team member performances– Supports reflection on progress and learning
Engineering Education Research Center, Washington State University http://eerc.wsu.edu
Web-based System Concept
InstructorMakes Assignment
(what, when)
Students Complete Assignment
(ratings, explanations)
Instructor PreparesFeedback
(ratings, comments)
Student RetrievesFeedback
from peers (anonymous)from instructor
Responses Saved and Compiled
(secure database)
1
3
42
Instructor ReceivesSummary Data
(ratings, comparisons)
5
Engineering Education Research Center, Washington State University http://eerc.wsu.edu
Make Assignment
Engineering Education Research Center, Washington State University http://eerc.wsu.edu
Engineering Education Research Center, Washington State University http://eerc.wsu.edu
Student Coaching of Members
What makes it strong?How does it benefit the team?
Engineering Education Research Center, Washington State University http://eerc.wsu.edu
Scoring of Student Work
Comments or suggestions
Engineering Education Research Center, Washington State University http://eerc.wsu.edu
Summary
Measure desired performances – Individual and team development– Design process and solution development– Reflective practices
Integration into capstone design– Assignments for instruction (formative)– Assignments for assessment (summative)
Adoption of assessments– Web-based implementation– Reference-based performance scoring– Testing underway
Engineering Education Research Center, Washington State University http://eerc.wsu.edu
Questions ? ?
ContactDenny Davis, Washington State University
Engineering Education Research Center, Washington State University http://eerc.wsu.edu
Scoring: Executive Summary
1Novice
2Beginner
3Intern
4Competent
5Expert
Problem or Opportunity
Uninformed; too broad, narrow, or
off-target
Vague understanding; questionable
details
Fair understanding; some reputable
detail
Good understanding;
substantive good detail
Superb understanding;
extensive proven detail
Solution Envisioned
Very vague idea; no vision for applicability
General idea; simple vision for
usefulness
Good idea; reasonable vision
for usefulness
Feasible solution; good vision for
usefulness
Superb solution; clearly useful,
feasible
Benefits of Solution
Unlikely or very limited benefits
possible
Small benefits; very narrow beneficiaries
Moderate benefits; narrow
beneficiaries
Good benefits; multiple varied beneficiaries
Many varied benefits; many beneficiaries
Writing Quality
Many errors; not understandable
Several errors; unclear; not interesting
Few errors; clear; somewhat
interesting
Very few errors; clear; very interesting
Error-free; clear; highly attractive,
compelling
Engineering Education Research Center, Washington State University http://eerc.wsu.edu
Scoring: Solution Specifications1
Novice2
Beginner3
Intern4
Competent5
Expert
Quality
ClarityIll-defined; not measurable
Vague; general expectations
Ok description, not quantitative
Clear targets, some measurable
Specific measurable performance
targets
RefinementNo additions or improvements
Very minor tweaks; minor improvement
One or more notable
improvements
Several revisions cause
improvement
Several revisions; some
transformative
Com
pleteness
FunctionalityIgnores most
important needsAddresses few
important needs
Addresses several important
needs
Addresses most vital needs, some
others
Addresses all vital, many other
needs
FinancialIgnores financial
needs, opportunities
Vaguely mentions cost limitations
Sets cost limits for project budget
Sets project budget and ROI
target
Sets budget, ROI; targets
opportunities
FeasibilityIgnores most
important issuesAddresses few
important issues
Addresses several important
issues
Addresses most vital issues, some
others
Addresses all vital, many other
issues
Social ImpactIgnores major social & safety
issues
Vaguely mentions social, safety
issues
Defines some social, safety requirements
Cites important codes for
compliance
Embraces all relevant codes &
standards
Engineering Education Research Center, Washington State University http://eerc.wsu.edu
Scoring: Proposed Solution
Proof of Performance
Functions considered Very few, little breadth Several, good breadth in types
Many, insightful breadth & measures
AnalysisNo analysis; opinions
onlySome analysis;
simplistic methodsExtensive analysis; best
methods
Strength of evidenceNo evidence to support
claimsGood evidence for some
claimsDefensible evidence for
important claims
Proof of Profitability
Development costsMajor concern; not
definedMinor concern: need
clarificationAttractive; sound; fully
justified
Cost/benefit ratioMajor concern; not
definedMinor concern: need
clarificationAttractive; sound; fully
justified
Market potentialLacks potential to fit a
marketSome potential to reach
a marketLikely to serve a growing market
Proof of Feasibility
ProducibilityMajor concern; known
problemsMinor concern: possible
problemsAttractive; methods fully
documented
UsabilityMajor concern; known
problemsMinor concern: possible
problemsAttractive; usability fully
documented
ServiceabilityMajor concern; known
problemsMinor concern: possible
problemsAttractive; servicing fully
documented
Proof of Impact
EnvironmentPossible serious negative impacts
Possible minor negative impacts
Certainly only positive impacts
Legal/politicalPossible serious negative impacts
Possible minor negative impacts
Certainly only positive impacts
Health & safetyPossible serious negative impacts
Possible minor negative impacts
Certainly only positive impacts