preparing students to incorporate stakeholder requirements in aerospace vehicle design
DESCRIPTION
Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design. Alexandra E. Coso Dissertation Defense Presentation Friday, March 28 th , 2014 Advisor: Dr. Amy Pritchett - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/1.jpg)
Preparing Students to Incorporate Stakeholder Requirements in
Aerospace Vehicle DesignAlexandra E. Coso
Dissertation Defense Presentation Friday, March 28th, 2014
Advisor: Dr. Amy PritchettCommittee: Dr. Brian German,
Dr. Wendy Newstetter, Dr. Jerry Seitzman, Dr. Robin Adams, and Dr. Alan Jacobsen
![Page 2: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/2.jpg)
2Image references at end of presentation
![Page 3: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/3.jpg)
Coso - Ph.D. Thesis Defense - March 28th, 2014 3
Stakeholders in Aircraft Design
Stakeholders: All those affected by the system. They may be users or non-users. They need not be clients or decision-makers. Stakeholders may be major or minor, and the
ways in which they interact with a large-scale system are myriad.- Gibson, Scherer, Gibson 2007, p.315
Non-Users
Design Firms
AirlinesFlight Attendants & Passengers
Manufacturers & Maintainers
Ground Personnel
Pilots
![Page 4: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/4.jpg)
4
Stakeholders in Complex System Design
1Maguire, 2001; 2Green & Jordan, 1999; 3Hupe, 2011; 4Chua & Feigh, 2011; 5Pew, 2008; 6Landsburg et al., 2008
Stakeholders are critical to the successful design of products in many fields1,2
International Civil Aviation Organization (ICAO) has called for more integrated consideration of stakeholders, through social sustainability efforts3
Human-related design criteria are part of certification
requirements established by regulators
Additional efforts are needed to examine how to better incorporate stakeholder requirements consistently
within the design process4-6
AE industry members have explored a variety of tools and
resources for better incorporating stakeholders
![Page 5: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/5.jpg)
5
Engineering Design Education
Accreditation Board for Engineering and Technology (ABET) defines program
outcomes that include need to graduate engineers who understand impact of
design solutions on environment, economy, and society1
1ABET, 2011; 2ASEE, 2013; 3Hinkle, 2013; 4Dym et al., 2001; 5Ro et al., 2012; 6Jordan & Lande 2012; 7Zoltowski 2010; 8McMasters, 2003
Movement within some fields of engineering to design learning environments which place a higher value on the needs and limitations of stakeholders6-7
Recently, ASEE and NSF report based on a survey of industry representatives,
highlights a need to improve how we prepare students to synthesize
engineering, business, and societal perspectives in the design process2
Studies of design and engineering students’ pathways support the need to determine how to better prepare students to incorporate stakeholder considerations into design3-5
![Page 6: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/6.jpg)
6
Engineering Design Education
Accreditation Board for Engineering and Technology (ABET) defines program
outcomes that include need to graduate engineers who understand impact of
design solutions on environment, economy, and society1
1ABET, 2011; 2ASEE, 2013; 3Hinkle, 2013; 4Dym et al., 2001; 5Ro et al., 2012; 6Jordan & Lande 2012; 7Zoltowski 2010; 8McMasters, 2003
There is a movement within some fields of engineering to design learning environments which place a higher value on the needs and limitations of stakeholders.6-7
Recently, ASEE and NSF report based on a survey of industry representatives,
highlights a need to improve how we prepare students to synthesize
engineering, business and societal perspectives in the design process2
Studies of design and explorations of engineering students’ pathways in industry support the need to determine how to better prepare students to incorporate stakeholder
considerations into design.3-5
Research is limited in terms of how to improve students’ understanding of the broader context of design and ability to balance the performance
and stakeholder-related considerations within a vehicle design8
Research Goals and Contributions(1) Examine how stakeholder considerations are currently
integrated into aerospace vehicle systems design(2) Examine how stakeholder considerations are currently
integrated into aerospace engineering design education(3) Design empirically-informed and theoretically-grounded
educational interventions and assessment tools that support students’ integration of stakeholder considerations
(4) Isolate the characteristics of the interventions and learning environment that support students’ integration of stakeholder considerations
![Page 7: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/7.jpg)
Coso - Ph.D. Thesis Defense - March 28th, 2014 7
RQ#4 - What educational interventions can enhance students’ understanding of and ability to integrate
stakeholder considerations into the design of an aerospace vehicle?
Phase I: Examination of Current Practice in Industry and Education
Phase II: Design of Educational Interventions and Assessment Tool
RQ#1 – What conditions enhance or inhibit the integration of
stakeholder considerations into the practice of aerospace vehicle
systems design?
RQ#3 – To what extent do aerospace engineering
students understand and take into consideration the effects
of design decisions on stakeholders?
RQ#2 – To what extent and how does the aerospace engineering
design curricula take into account stakeholder considerations?
Research Overview
![Page 8: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/8.jpg)
8
Research & Presentation Overview
Phase I: Examination
of Current Practice in
Industry and Education
Phase II: Design of
Educational Interventions
and Assessment
Tool
Practice of Aerospace
Engineering Design
Contributions and Future
Work
Aerospace Engineering
Design Curricula
Aerospace Engineering
Students’ Perceptions
Educational Intervention
Specifications
![Page 9: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/9.jpg)
Examination of Practice of AE Vehicle Design
RQ#1 – What conditions enhance or inhibit the integration of stakeholder considerations into the practice of aerospace vehicle systems design?
9
Human factors & related researchAddressed need and provided recommendations for how to
integrate stakeholder considerations into complex system design
Aircraft design researchExamined stakeholder considerations
within multidisciplinary design, aerodynamic analyses, and studies of
the operational environment
However, this literature has not empirically studied how stakeholder considerations are integrated within aerospace vehicle design industry’s
multidisciplinary design teams
![Page 10: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/10.jpg)
Coso - Ph.D. Thesis Defense - March 28th, 2014 10
Units of Analysis
Functional Groups
Embedded Multiple Case Study
Disciplinary Backgrounds Individuals
Research Setting
Aerospace Vehicle Design FirmTime Frame: 3 Months
Data Sources
First Round InterviewsSemi-Structured Protocol
Second Round Interviews Remaining Questions
Topics included: 1. Overview of the design process at the firm2. The challenges affecting the integration of stakeholder
considerations3. Previously used interventions to improve integration
![Page 11: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/11.jpg)
Coso - Ph.D. Thesis Defense - March 28th, 2014 11
Results: Group/Team-Level
Contributing Factors
Group StructureThat supports addressing stakeholder
considerations
Common GoalThat addresses stakeholder considerations
Human Factors
SpecialistsDesign
Engineers
Pilot Compartment Group
In-House Pilots
Design Engineers
HF Specialists
Other Functional Groups
![Page 12: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/12.jpg)
Image References included at end of presentation 12
Results: Individual-Level & Interaction-Level
Contributing Factors
Differing Perspectives About collaborations with
individuals from other disciplines
Shared Language & Boundary Objects
To support the exchange of ideas
![Page 13: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/13.jpg)
Coso - Ph.D. Thesis Defense - March 28th, 2014 13
Educational Intervention Specifications
From Industry Case Study
Group/Team-LevelID1. Stakeholder-centric learning environment
Interaction-LevelID2. Language and vocabulary for discussing stakeholder considerationsID3. Tools and resources to support integration of stakeholder considerations
Individual-LevelID4. Cross-disciplinary problem-solving experiencesID5. Value of designs created from multiple perspectives
![Page 14: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/14.jpg)
14
Research & Presentation Overview
Phase I: Examination
of Current Practice in
Industry and Education
Phase II: Design of
Educational Interventions
and Assessment
Tool
Practice of Aerospace
Engineering Design
Contributions and Future
Work
Aerospace Engineering
Design Curricula
Aerospace Engineering
Students’ Perceptions
Educational Intervention
Specifications
![Page 15: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/15.jpg)
Coso - Ph.D. Thesis Defense - March 28th, 2014 15
Examination of AE Design Education
RQ#2 – To what extent and how does the aerospace engineering design curricula take into account stakeholder considerations?
A review of publically-available syllabi from top aerospace
engineering programs and relevant published work
Review of the design processes presented in classical aerospace design textbooks (including Roskam; Raymer; Nicolai & Carichner) commonly used
within aircraft design capstone course
![Page 16: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/16.jpg)
16
Components of AE Design Curricula
1Peet & Mulder, 2004
Elective course offerings & systems engineering coursesIsolated course may be insufficient in training students to integrate
these topics into in their capstone courses and in the future1
Isolated Courses
Design Textbooks
Design Projects
Quantitative approximations (i.e. surrogates) are used to model some stakeholder characteristics
Mention of customer needs and certification at beginning of text, but not emphasized within the design processes themselves
[with exception of Nicolai and Carichner, 2010]
Distinctive from product design due to life span, purpose, and size of traditional aerospace vehicles
Impacts extent to which students experience entire design cycleContact with clients or stakeholders is limited
![Page 17: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/17.jpg)
Coso - Ph.D. Thesis Defense - March 28th, 2014
Educational Intervention Specifications
From Industry Case Study
ED1. Awareness of contextual and stakeholder considerations
ED2. Importance of considering broad context and wide variety of stakeholders
ED3. Methods for incorporating novel or qualitative stakeholder considerations
ED4. Interactions with clients or other stakeholders
17
From Examination of Design Curricula and Students’ Prior Knowledge
Group/Team-LevelID1. Stakeholder-centric learning environment
Interaction-LevelID2. Language and vocabulary for discussing stakeholder considerationsID3. Tools and resources to support integration of stakeholder considerations
Individual-LevelID4. Cross-disciplinary problem-solving experiencesID5. Value of designs created from multiple perspectives
![Page 18: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/18.jpg)
18
Research & Presentation Overview
Phase I: Examination
of Current Practice in
Industry and Education
Phase II: Design of
Educational Interventions
and Assessment
Tool
Practice of Aerospace
Engineering Design
Contributions and Future
Work
Aerospace Engineering
Design Curricula
Aerospace Engineering
Students’ Perceptions
Educational Intervention
Specifications
![Page 19: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/19.jpg)
19
Engineering Students’ Prior KnowledgeRQ#3 – To what extent do aerospace engineering students understand and
take into consideration the effects of design decisions on stakeholders?
1Carberry et al., 2010; 2Adams & Fralick, 2010; 3Ro et al., 2012
Design Understanding and Self-EfficacySelf-Efficacy in Design Scale1
andDesign Ranking Test2
Understanding of Contextual ConsiderationsContextual Competence Scale3
andOpen-Ended Follow-Up Question
Perceptions of the Role of Stakeholders in Design Scenario-Based Design Task
![Page 20: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/20.jpg)
Coso - Ph.D. Thesis Defense - March 28th, 2014 20
Senior Aerospace Engineering StudentsSample: 80 senior AE students enrolled in an aircraft design capstone course
• 8 women (~10%)• 5 international students (~6%)•30 students (~38%) completed a previous design course •52 students (~65%) described having industry experience•50 students (~63%) took a problem-based learning course within the department
![Page 21: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/21.jpg)
Coso - Ph.D. Thesis Defense - March 28th, 2014 21
Design UnderstandingImportant Design
ActivitiesProblem-ScopingCommunication
Not Considered “Most Important”
Iteration Making Trade-offs
Understand the Problem
Identify Constraints
Communicate
Evaluate
Test
Goal Set
Make Decisions
Making Trade-offs
Brainstorming
Planning
Generating Alternatives
Using Creativity
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
Most Important Design Activities
AE Professionals (n = 20) VT Seniors (n = 53) Study Students (n = 80)
Portion of Sample
Desig
n Ac
tiviti
es
![Page 22: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/22.jpg)
Coso - Ph.D. Thesis Defense - March 28th, 2014 22
Design UnderstandingImportant Design
ActivitiesProblem-ScopingCommunication
Not Considered “Most Important”
Iteration Making Trade-offs
Sketching
Abstracting
Building
Imagining
Decomposing
Synthesizing
Iterating
Prototyping
Visualizing
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Least Important Design ActivitiesAE Professionals (n = 20) VT Seniors (n = 53) Study Students (n = 80)
Percentage of Sample
Desig
n Ac
tiviti
es
Consistent with aircraft design students from
other aerospace engineering departments
![Page 23: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/23.jpg)
Coso - Ph.D. Thesis Defense - March 28th, 2014 23
Understanding of Contextual & Stakeholder Considerations
Consistent with aircraft design students from other
aerospace engineering departments
Important Design ActivitiesProblem-ScopingCommunication
Not Considered “Most Important”
Iteration Making Trade-offs
Students consider…Technical and logistical
considerationsDesign’s effects on customer
Economic and mission constraints
Students do not see impact of…
Broader context (i.e. political, historical
considerations)Stakeholders other than
customer(i.e. pilots, maintainers,
passengers)
![Page 24: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/24.jpg)
Coso - Ph.D. Thesis Defense - March 28th, 2014 24
Educational Intervention Specifications
From Industry Case Study
ED1. Awareness of contextual and stakeholder considerations
ED2. Importance of considering broad context and wide variety of stakeholders
ED3. Methods for incorporating novel or qualitative stakeholder considerations
ED4. Interactions with clients or other stakeholders
ED5. Importance of iteration and trade-offs
From Examination of Design Curricula and Students’ Prior Knowledge
Group/Team-LevelID1. Stakeholder-centric learning environment
Interaction-LevelID2. Language and vocabulary for discussing stakeholder considerationsID3. Tools and resources to support integration of stakeholder considerations
Individual-LevelID4. Cross-disciplinary problem-solving experiencesID5. Value of designs created from multiple perspectives
![Page 25: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/25.jpg)
25
Phase I Contributions
Phase I: Examination of Current Practice in
Industry and Education
Practice of Aerospace
Engineering Design
Aerospace Engineering
Design Curricula
Aerospace Engineering
Students’ Perceptions
Educational Intervention
Specifications
Empirical evidence of conditions which enhance or inhibit the integration of human factors and stakeholder considerations into
the aircraft design process.
Evaluation method to explore engineering students’ perceptions of how stakeholder
considerations are integrated into a complex system design
![Page 26: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/26.jpg)
26
Research & Presentation Overview
Phase I: Examination of Current Practice in
Industry and Education
Phase II: Design of
Educational Interventions
and Assessment
Tool
Contributions and Future
Work
Educational Intervention
Specifications
Requirements Lab
Stakeholders in Design Labs
![Page 27: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/27.jpg)
Coso - Ph.D. Thesis Defense - March 28th, 2014 27
Requirements Lab
Situational Factors
Learning Objectives
Assessment & Feedback
Learning Theory & Activities
GOAL: Introduce students to requirements, their importance, and how and why stakeholder-requirements should be incorporated into the requirements definition phase of the design process
Design Process
![Page 28: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/28.jpg)
Coso - Ph.D. Thesis Defense - March 28th, 2014 28
Requirements Lab
Student Characteristics (from In-Class Evaluation)Wide spectrum of design knowledge and industry
experience
Nature of the SubjectDesign should be flexible for use within other aerospace
courses and programs
Context of Teaching/Learning Situation (Study-Specific)Two 50-minute lectures per week (80 students),
One three-hour lab session (40 students per lab session)
Situational Factors
Learning Objectives
Assessment & Feedback
Learning Theory & Activities
Design Process
![Page 29: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/29.jpg)
Coso - Ph.D. Thesis Defense - March 28th, 2014 29
Requirements Lab
LO#3 - Students will be able to identify different stakeholders affected by a design problem
Based on ID3 & ED3
LO#4 - Students will be able to discuss how requirements affect design and the design process
Based on ED1
LO#6 - Students will be able to breakdown a problem and write requirements for performance and stakeholder
considerations Based on ID1
Situational Factors
Learning Objectives
Assessment & Feedback
Learning Theory & Activities
Design Process
![Page 30: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/30.jpg)
Coso - Ph.D. Thesis Defense - March 28th, 2014 30
Requirements Lab
Opportunities for Direct & Immediate FeedbackClass-wide discussions about requirements and stakeholders
“Forward-looking” Assessment/Check for UnderstandingDesign case provides an open-ended, real-life context for
students to practice developing requirements and selecting verification and validation procedures
Further Opportunities to Assess Students’ UnderstandingAnonymous Student Feedback Forms
Situational Factors
Learning Objectives
Assessment & Feedback
Learning Theory & Activities
Design Process
![Page 31: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/31.jpg)
Requirements Lab
1Kolb & Kolb, 2009 31
Concrete Experience(doing/having an experience)• Reflection on previous
design experiences • Related experience activity• Group work• Interactions with a client
Reflective Observation(reflecting on the experience)• Class-wide discussion about
requirements and stakeholders
Abstract Conceptualisation
(learning from the experience)• Class-wide discussion to
define characteristics of well-written requirements
Active Experimentation(trying out what you have
learned)• Design case• Group work• Interactions with stakeholders
Adaptation of Kolb’s Learning Cycle1
Situational Factors
Learning Objectives
Assessment & Feedback
Learning Theory & Activities
Design Process
![Page 32: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/32.jpg)
Requirements Lab
1Kolb & Kolb, 2009 32
Concrete Experience(doing/having an experience)• Reflection on previous
design experiences • Related experience activity• Group work• Interactions with a client
Reflective Observation(reflecting on the experience)• Class-wide discussion about
requirements and stakeholders
Abstract Conceptualisation
(learning from the experience)• Class-wide discussion to
define characteristics of well-written requirements
Active Experimentation(trying out what you have
learned)• Design case• Group work• Interactions with stakeholders
Adaptation of Kolb’s Learning Cycle1
Designed such that students will create knowledge about stakeholders and requirements through activities that require them to experience and reflect on the relationships between stakeholders,
requirements, and design
Situational Factors
Learning Objectives
Assessment & Feedback
Learning Theory & Activities
Design Process
![Page 33: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/33.jpg)
33
Research & Presentation Overview
Phase I: Examination of Current Practice in
Industry and Education
Phase II: Design of
Educational Interventions
and Assessment
Tool
Contributions and Future
Work
Educational Intervention
Specifications
Requirements Lab
Stakeholders in Design Labs
![Page 34: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/34.jpg)
Coso - Ph.D. Thesis Defense - March 28th, 2014 34
Stakeholders in Design Labs GOALS: • Have students consider how stakeholder requirements and
concerns can be integrated throughout the aircraft design process
• Have students define how stakeholder requirements will be incorporated into the design process they will follow during their team design project
Situational Factors
Learning Objectives
Assessment & Feedback
Learning Theory & Activities
Design Process
![Page 35: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/35.jpg)
Coso - Ph.D. Thesis Defense - March 28th, 2014 35
Stakeholders in Design Labs
Student Characteristics (from In-Class Evaluation)Wide spectrum of design knowledge and industry
experience
Nature of the SubjectDesign should flexible for use within other aerospace
courses and programs
Context of Teaching/Learning Situation (Study-Specific)Two 50-minute lectures per week (80 students),
One three-hour lab session (40 students per lab session)
Implemented several weeks into the semester
Situational Factors
Learning Objectives
Assessment & Feedback
Learning Theory & Activities
Design Process
![Page 36: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/36.jpg)
Coso - Ph.D. Thesis Defense - March 28th, 2014 36
LO#1 - Students will be able to identify relevant stakeholders of a fixed wing design and explain how their
concerns affect (or affected) the design solution
Based on specifications ID1, ED1, & ED2
LO#2 - Students will be able to assess methods for integrating stakeholder considerations into the design
process
Based on ID1, ID3, & ED3
Stakeholders in Design Labs
Situational Factors
Learning Objectives
Assessment & Feedback
Learning Theory & Activities
Design Process
![Page 37: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/37.jpg)
Coso - Ph.D. Thesis Defense - March 28th, 2014 37
Stakeholders in Design Labs
Opportunities for Direct & Immediate FeedbackStudent-generated design processes
Case study evaluationsClass-wide discussions
Additional Checks for UnderstandingCase study evaluations
Student reflectionsAnonymous student feedback forms
Situational Factors
Learning Objectives
Assessment & Feedback
Learning Theory & Activities
Design Process
![Page 38: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/38.jpg)
38
Stakeholders in Design Labs
1Kim, 2001; 2Bucciarelli, 1988
Social Constructivist Theory1
Knowledge is constructed through social interaction
Based on ID4 and ID5
Design as Reflective Practice2
Design requires negotiation among disciplines and perspectives
Based on ED5
Situational Factors
Learning Objectives
Assessment & Feedback
Learning Theory & Activities
Design Process
![Page 39: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/39.jpg)
Coso - Ph.D. Thesis Defense - March 28th, 2014 39
Stakeholders in Design Labs
Lab #1
Framing the Lessons
Design Process Negotiation #1
• Pre-Lab Reflection • Individual Design Process
DrawingSituational
Factors
Learning Objectives
Assessment & Feedback
Learning Theory & Activities
Design Process
![Page 40: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/40.jpg)
Coso - Ph.D. Thesis Defense - March 28th, 2014 40
Stakeholders in Design Labs
1. Understand the Problem
3. Seek Information
PlanningEvaluating
Generate Alternatives
Making Trade-offs
Make Initial Decisions
Testing Building
2. Visualize &
Brainstorm
Identify Constraints
Communicate Throughout!
Iterating
Situational Factors
Learning Objectives
Assessment & Feedback
Learning Theory & Activities
Design Process
![Page 41: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/41.jpg)
Coso - Ph.D. Thesis Defense - March 28th, 2014 41
Stakeholders in Design Labs
Lab #1
Framing the Lessons
Design Process Negotiation #1
Introduction to Stakeholder Mapping
• Pre-Lab Reflection • Individual Design Process
Drawing
Tool for isolating stakeholder considerations that can affect a design
Situational Factors
Learning Objectives
Assessment & Feedback
Learning Theory & Activities
Design Process
![Page 42: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/42.jpg)
Coso - Ph.D. Thesis Defense - March 28th, 2014 42
Stakeholders in Design Labs
Government Regulators(e.g. FAA, EASA)
Passengers
Commercial Aircraft
Pilots
Passengers
Maintainers/ Ground Personnel
Flight Attendants
Non-Users (e.g. people who live near an airport)
Design Firm/ Manufacturers
Suppliers
Wider seat widths
Improved Passenger Experience
Larger fuselage provides greater headroom when walking up and down the
aisle
Airlines
Stakeholder Mapping Technique (adapted from a technique used in Civil and
Environmental Engineering)
Situational Factors
Learning Objectives
Assessment & Feedback
Learning Theory & Activities
Design Process
![Page 43: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/43.jpg)
Coso - Ph.D. Thesis Defense - March 28th, 2014 43
Stakeholders in Design Labs
Lab #1
Framing the Lessons
Design Process Negotiation #1
Introduction to Stakeholder Mapping
Introduction to Case Study Assessment
Case Study Assessment
Wrap-Up
• Pre-Lab Reflection • Individual Design Process
Drawing
Tool for isolating stakeholder considerations that can affect a design
Real-world examples of aircraft design process and impact of stakeholders on design
Situational Factors
Learning Objectives
Assessment & Feedback
Learning Theory & Activities
Design Process
![Page 44: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/44.jpg)
Coso - Ph.D. Thesis Defense - March 28th, 2014 44
Stakeholders in Design Labs
Lab #2
Conclusion
Design Process Negotiation #2
Human-Centered Design Methods
Case Study Presentations
Framing the Lessons
• Student groups deconstruct an aircraft case study
• Outcomes include a stakeholder mapping and a design process
• Introduction to 5 HCD methods• Discussion of advantages and
disadvantages of methods and applicability to aircraft design
• Modification of initial design processes based on lessons learned in labs
• Indications of where stakeholders would be accounted for in the process
• Final reflections• Anonymous Student Feedback
Situational Factors
Learning Objectives
Assessment & Feedback
Learning Theory & Activities
Design Process
![Page 45: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/45.jpg)
45
Contribution of Intervention Design
Phase I: Examination of Current Practice in
Industry and Education
Phase II: Design of
Educational Interventions
and Assessment
Tool
Requirements Lab
Stakeholders in Design Labs
Empirically-informed and theoretically-grounded
instructional designs that can be easily integrated into an
aircraft design capstone course or adapted for complex systems design course
![Page 46: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/46.jpg)
46
Research & Presentation Overview
Phase I: Examination of Current Practice in
Industry and Education
Phase II: Design of
Educational Interventions
and Assessment
Tool
Contributions and Future
Work
Requirements Lab
Stakeholders in Design Labs
Implementation and Evaluation of Educational Interventions
![Page 47: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/47.jpg)
Coso - Ph.D. Thesis Defense - March 28th, 2014 47
Evaluation of Educational Interventions: Research Questions
RQ#4 - What educational interventions can enhance students’ understanding of and ability to integrate stakeholder considerations into the design of an aerospace vehicle?
RQ4.1 - What characteristics of the educational interventions support students’ abilities to integrate stakeholder considerations into the design of an aerospace
vehicle?
RQ4.2 - To what extent can the interventions help students integrate stakeholder considerations into the design of an aerospace vehicle?
RQ4.3 - What characteristics of the engineering design learning experience and learning environment support or hinder students’ integration of stakeholder
considerations into the design of an aerospace vehicle?
![Page 48: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/48.jpg)
48
Research & Presentation Overview
Phase I: Examination of Current Practice in
Industry and Education
Phase II: Design of
Educational Interventions
and Assessment
Tool
Contributions and Future
Work
Requirements Lab
Stakeholders in Design Labs
Evaluation of Educational
Interventions
Evaluation of the Learning
Environment
Stakeholders in Design
Rubric
![Page 49: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/49.jpg)
491Adapted from Moon et al 2011 & Kaufman et al 1995
Resources & Processes
• Students’ satisfaction with interventions
Acquisition
• Achievement of learning objectives
Application
• Adoption of intervention content in class
Organization & Course
• Factors within the learning environment
Societal
• Impact after and outside the course
Evaluation of Educational Interventions:Evaluation Framework1
RQ4.1 - What characteristics of the educational interventions support students’ abilities to integrate stakeholder considerations into the design of an aerospace vehicle?
RQ4.2 - To what extent can the interventions help students integrate stakeholder considerations into the design of an aerospace vehicle?
![Page 50: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/50.jpg)
Coso - Ph.D. Thesis Defense - March 28th, 2014 50
Results Summary: Data from Interventions
Requirements Lab [data from design case & feedback forms]
• Students formulated well-constructed high-level requirements and many students were able to identify a diverse group of stakeholders
• Students considered both performance and stakeholder considerations and most groups included at least three different verification and validation procedures
• Stakeholder-centric focus of the lab was not clear to students
Stakeholders in Design Labs [data from all activities & feedback forms]
• Stakeholder-centric focus supported students’ integration of stakeholder considerations in a variety of different ways in final design processes
• Case studies illustrated challenges of cross-disciplinary design• Misconceptions about HCD methods may have also developed over the
course of these lab sessions
![Page 51: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/51.jpg)
Coso - Ph.D. Thesis Defense - March 28th, 2014 51
Results Summary: Post-Intervention Evaluation
Design Understanding at Post-Evaluation
• Iteration: 23.8% of the students in the pre-evaluation selected it, as compared with 69.6% of students in the post-evaluation
• Making Trade-offs: 33.8% of the students in the pre-evaluation selected it, as compared with 48.2% of students in the post-evaluation
Stakeholder Considerations
• Students reported significantly more stakeholder-related considerations in the post-evaluations as compared with the pre-evaluations
![Page 52: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/52.jpg)
52
Research & Presentation Overview
Phase I: Examination of Current Practice in
Industry and Education
Phase II: Design of
Educational Interventions
and Assessment
Tool
Contributions and Future
Work
Requirements Lab
Stakeholders in Design Labs
Evaluation of Educational
Interventions
Evaluation of the Learning
Environment
Stakeholders in Design
Rubric
![Page 53: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/53.jpg)
53
Stakeholders in Design Rubric
REFERENCES: Zoltowkski, 2010; Crismond and Adams, 2012
Stakeholder Integration by Design Stage Design Understanding Stakeholder Integration
Examines students’ integration of stakeholder considerations into 3 stages of design
Specifically, students’• intent, • application of
methods, and • overall success
Scores students based on their application of the design process
0pts: Lack of knowledge about the process 4pts: abstract the process and be a reflective designer
Scores students based on the extent to which they integrated stakeholder considerations
0pts: Lack of appreciation
4pts: Leveraging multiple perspectives for a more innovative solution
![Page 54: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/54.jpg)
Coso - Ph.D. Thesis Defense - March 28th, 2014 54
Examination of Students’ Team Design Projects
Problem Definition Stage
Technology Integration Stage
Overall Design
Most teams did not state an intention to incorporate stakeholder considerations [as noted by 67.6% of judging panels’ scores] Or apply a design process that supported integration of stakeholder considerations [as noted by 58.8% of judging panels’ scores]
Most teams did not apply a design process that supported the integration of stakeholder considerations [as noted by 60.6% of judging panels’ scores]
68.6% of judges’ scores suggested that students were somewhat successful at integrating stakeholder considerations into their final design solutions
![Page 55: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/55.jpg)
Coso - Ph.D. Thesis Defense - March 28th, 2014 55
Examination of Students’ Team Design Projects
Example of Variations in Scores Across TeamsTeam E received 0’s from all 3 judges
Team C received two 3’s and one 2 from their judges
Median: 2Applies design process as presented in course and demonstrates how design
solution improved through an iterative design process
Median: 2Identifies stakeholder considerations
at isolated points in the design process, but overall these
considerations are not the basis for design decisions and are not
addressed consistently through design
Design Understanding[0 to 4pt Scale]
Stakeholder Integration[0 to 4pt Scale]
0 1 2 3 402468
1012141618
Frequency of Scores for Students' Team Design Project Reports
Design Understanding Stakeholder Integration
Score
Scor
e Fr
eque
ncy
![Page 56: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/56.jpg)
56
Research & Presentation Overview
Phase I: Examination of Current Practice in
Industry and Education
Phase II: Design of
Educational Interventions
and Assessment
Tool
Contributions and Future
Work
Requirements Lab
Stakeholders in Design Labs
Evaluation of Educational
Interventions
Evaluation of the Learning
Environment
Stakeholders in Design
Rubric
![Page 57: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/57.jpg)
571Adapted from Moon et al 2011 & Kaufman et al 1995
Resources & Processes
• Students’ satisfaction with interventions
Acquisition
• Achievement of learning objectives
Application
• Adoption of intervention content in class
Organization & Course
• Factors within the learning environment
Societal
• Impact after and outside the course
Evaluation of Educational Interventions:Evaluation Framework1
RQ4.3 - What characteristics of the engineering design learning experience and learning environment support or hinder students’ integration of stakeholder
considerations into the design of an aerospace vehicle?
![Page 58: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/58.jpg)
Coso - Ph.D. Thesis Defense - March 28th, 2014 58
Examination of the Learning Environment
TASK
[Ames, 1992]
Team Design Project Requirements
Characteristics of Educational
Interventions
Reliance on Design Process Structure
Available Tools and Resources
![Page 59: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/59.jpg)
Coso - Ph.D. Thesis Defense - March 28th, 2014 59
Examination of the Learning Environment
TASK
[Ames, 1992]
EVALUATIONPRACTICES
Team Design Project Requirements
Design of an Assessment Rubric
Characteristics of Educational
Interventions
Reliance on Design Process Structure
Available Tools and Resources
Use of Project Requirements as
Evaluation Strategy
Evaluation through Interactions with
External Audience
![Page 60: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/60.jpg)
Coso - Ph.D. Thesis Defense - March 28th, 2014 60
Examination of the Learning Environment
TASK
[Ames, 1992]
EVALUATIONPRACTICES
DISTRIBUTION OF AUTHORITY
Team Design Project Requirements
Design of an Assessment Rubric
Characteristics of Educational
Interventions
Reliance on Design Process Structure
Available Tools and Resources
Use of Project Requirements as
Evaluation Strategy
Evaluation through Interactions with
External Audience
Transition from Instructor-centered,
Individual-based Learning
Environment to
Student-centered, Team-based
Learning Environment
![Page 61: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/61.jpg)
61
Additional Phase II Contributions
Phase I: Examination of Current Practice in
Industry and Education
Phase II: Design of
Educational Interventions
and Assessment
Tool
Requirements Lab
Stakeholders in Design Labs
Evaluation of Educational
Interventions
Evaluation of the Learning Environment
Stakeholders in Design
Rubric
Empirical evidence of conditions
within the learning environment that impact students’
integration of stakeholder
considerations into a complex systems
design process.
Assessment tool to support students’ understanding of how to design with stakeholders in mind, and to assess students’ abilities to consider the impact of complex system design
decisions on stakeholders.
![Page 62: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/62.jpg)
62
Research & Presentation Overview
Phase I: Examination of Current Practice in
Industry and Education
Phase II: Design of
Educational Interventions
and Assessment
Tool
Contributions and Future
Work
![Page 63: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/63.jpg)
Coso - Ph.D. Thesis Defense - March 28th, 2014 63
Research Contributions & Future WorkIndustry Case Study – empirical evidence of conditions that enhance or inhibit the
integration of human factors & stakeholder considerations into the aircraft design process
In-Class Evaluation & Stakeholders in Design Rubric – evaluation methods to (1) explore engineering students’ perceptions of how stakeholder considerations are integrated into the vehicle design process, (2) support students’ understanding of how to design with stakeholders in mind, and (3) assess students’ abilities to consider the impact of complex system design decisions on stakeholders
Requirements Lab & Stakeholders in Design Labs - empirically-informed and theoretically-grounded instructional designs that can be easily integrated into an aircraft design
capstone course or adapted for complex systems design course.
Examination of Learning Environment - empirical evidence of conditions within the learning environment that impact students’ integration of stakeholder considerations into
a complex systems design process
![Page 64: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/64.jpg)
Coso - Ph.D. Thesis Defense - March 28th, 2014 64
Research Contributions & Future WorkDiscipline-based education research that informs teaching and learning within
aerospace engineering and supports continued research into this interdisciplinary field
Starting point for future research into pedagogical techniques which support students’ integration of stakeholder considerations into the design of complex systems
![Page 65: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/65.jpg)
Questions?
Acknowledgments
NSF Graduate Research Fellowship under Grant No. DGE-0644493.
NSF under Grant #EEC-1059472 to the American Society for Engineering
Education.
Thank you to all of the interview participants, the instructors, and the students for their assistance with this
study.
Thank you to my amazing peer debriefers, Subject Matter Experts,
and the former and current members of the Cognitive Engineering Center
at Georgia Tech!
![Page 66: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/66.jpg)
Related PublicationsJournal Papers+ (Submitted) Coso, A. and Pritchett, A. (2013) “The Role of Design Teams in the
Integration of Stakeholder Considerations.” Journal of AircraftConference Proceedings+ (Accepted) Coso, A. and Pritchett, A. (2014) “The Development of a Rubric to
Evaluate and Promote Students’ Integration of Stakeholder Considerations into the Engineering Design Process.” American Society for Engineering Education: Proceedings of the 2014 ASEE Annual Conference and Exposition. Indianapolis, IN
+ Coso, A. and Pritchett, A. (2014) “Incorporating Stakeholder Considerations in the Aircraft Design Process: A Focus on Aircraft Design Education.” Proceedings of AIAA SciTech. National Harbor, MD.
+ Coso, A. and Pritchett, A. (2013) “The Integration of Stakeholder Requirements within Aerospace Engineering Design Education.” American Society for Engineering Education: Proceedings of the 2013 ASEE Annual Conference and Exposition. Atlanta, GA.
Coso - Ph.D. Thesis Defense - March 28th, 2014 66
![Page 67: Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design](https://reader034.vdocuments.mx/reader034/viewer/2022042822/568168ac550346895ddf55c8/html5/thumbnails/67.jpg)
Image Referenceshttp://claesjohnson.blogspot.com/2012_07_01_archive.htmlhttp://www.amazon.com/Airplane-Design-Part-Preliminary-Airplanes/dp/188488542Xhttp://www.amazon.com/Mission-Analysis-Edition-TECHNOLOGY-LIBRARY/dp/0792359011https://www.aiaa.org/PubDetail.aspx?id=1796http://www.aviationweek.com/Blogs.aspx?plckBlogId=Blog:27ec4a53-dcc8-42d0-bd3a-01329aef79a7&plckPostId=Blog:27ec4a53-dcc8-42d0-bd3a-01329aef79a7Post:6e334c9f-0810-4874-a0be-d7e023b58135http://orbiterchspacenews.blogspot.com/2010/11/nasa-selects-companies-for-advanced.htmlhttp://www.disinfo.com/tag/spacex/http://www.open.ac.uk/about/campus/jennie-lee-research-labs/files/jennieleeresearchlabs/imagecache/node_standard/HCI%202%20Lab-1.jpghttp://www.manvitas.com/images/Corporate.jpghttp://www.inc.com/uploaded_files/image/speed-up-your-prototyping-process-pop_9632.jpghttp://www.kelvinyoung.com/v6/work/storyboard_01.jpghttp://www.nngroup.com/reports/paper-prototyping-training-video/
For literature references, view dissertation text
Coso - Ph.D. Thesis Defense - March 28th, 2014 67