DesignWebs to Support Engineering Design Student Projects
Sharad V. Oberoi
• Motivation• Related research• Research testbed• Research questions• Research findings • Contributions• Future work
Overview
12/13/2011 Sharad V. Oberoi: PhD Final Public Oral Exam
2
Research Testbed
• Rapid Prototyping of Computer Systems Class 2008
• Kiva
Related Research
• Collaborative learning in design
• Organizational memory
• Computational linguistics
• Information structuring and visualization approaches
Research Contributions
• An automated mechanism to visualize engineering design project documentation
• A surrogate measure to track the emergence of the shared solution
• A research test bed to validate different prior research findings
Future Work
• Analysis of graphical and verbal communication
• Real-time monitoring of noun phrase usage
• Functional and operational issues
Motivation
• How do students learn in engineering design project teams?
• How can additional information be provided to instructors to assess student learning?
Research Findings
• Design documentation can be visualized as concept-map based graphs
• Noun phrases act as surrogate measures for tracking design team dynamics
• DesignWebs can act as research platform to validate prior research
Research Questions
• How can the state of the artifact be summarized and presented visually to students and instructors from the project documents and Kiva discussions?
• What objective measures can be used to assess the student design process and gain insights into student design collaboration?
• How can prior research in design learning be validated?
Motivation• How do students learn from each other in engineering
design project teams?• How can additional information be provided to
instructors to assess student learning?
12/13/2011 Sharad V. Oberoi: PhD Final Public Oral Exam
3
Students in Design Projects
• Create most of the knowledge outside the classroom• Have little experience in team-work or in design• Consider final product as the most important take-away• Have little or no support infrastructure for information
management
12/13/2011 Sharad V. Oberoi: PhD Final Public Oral Exam
4
Instructors of Design Projects
• Intend to assess student learning outcomes, but have more ready access to product outcomes
• Often have to rely on the self-reported functioning of the teams to assess their progress
• Cannot always detect instances when students deviate from objectives
• Need a mechanism to monitor student progress in real-time
12/13/2011 Sharad V. Oberoi: PhD Final Public Oral Exam
5
Research Issues
• Assessing evidence of student learning requires monitoring of student activities
• Unobtrusive approaches are necessary:– Students alter behavior in front of instructors– Embedded observations are tedious
• Evidence of student collaboration process exists, but is difficult to summarize and use
12/13/2011 Sharad V. Oberoi: PhD Final Public Oral Exam
6
• Motivation• Related research• Research testbed• Research questions• Research findings • Contributions• Future work
Overview
12/13/2011 Sharad V. Oberoi: PhD Final Public Oral Exam
7
Related Research
• Collaborative learning in design
• Organizational memory
• Computational linguistics
• Information structuring and visualization approaches
Collaborative Learning in Design
• Collaborative learning allows students to hone their communication and negotiation skills (Dillenbourg 1999)
• When students with different strengths work together, they solve problems that would be beyond their reach if they were working alone (Vygotsky 1978)
• The depth of student explanations is correlated with how much students learn (Webb et al. 2002)
12/13/2011 Sharad V. Oberoi: PhD Final Public Oral Exam
8
Sharad V. Oberoi: PhD Final Public Oral Exam
9
Organizational Memory
• Reuse of design information in organizations often requires designers to recreate the rationale (Subrahmanian et al. 1997)
• Easy access to archives can help future teams be successful with lower transactional costs (Roth et al. 1998)
• Organizational memory systems are usually domain specific and cannot be readily adpated in project classes
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
10
Automatic Topic Segmentation
1. Lexical cohesion Models: – Lexical co-occurrence of thematically-related terms
indicates continuity in topic – Introduction of new vocabulary refers to a new topic
2. Content-oriented Models: – Combine lexical cohesion with other indicators of topic
shift such as cue phrases using probabilistic models (Reynar 1998)
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
11
Latent Dirichlet Allocation (LDA)
• Widely used to cluster document fragments and automatically assign them topic labels (Blei et al. 2003)
• The analysis is done at the semantic level by considering probabilities for each of the words occurring in the corpus
• The context of the word can be examined, making word disambiguation possible
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
12
Co-word Analysis
• Co-word analysis reduces a space of descriptors to a set of network graphs with the strongest associations between descriptors (Coulter et al. 1998)
• Co-word graphs construct multiple networks that highlight associations between keywords (Coulter et al. 1998)
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
13
Information Structuring
• As student expertise increases, the corresponding concept interrelatedness in concept maps resembles tightly integrated structures (Royer et al. 1993)
• Scientometrics identifies emerging research areas and the evolution of research areas (Callon et al. 1991; van Raan 1992)
• Library sciences have done research on organizing information stored in books and journals and providing structures for locating relevant information (Bhattacharya et al. 1998)
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
14
Information Visualization
• Domain visualization “aims to reveal realms of scientific communication as reflected through scientific literature and citation paths interwoven by individual scientists in their publications” (Börner et al. 2003)
• Longitudinal maps have been used to detect the evolving nature of scientific fields by scientometrists (Garfield 1994)
12/13/2011
• Motivation• Related research• Research testbed• Research questions• Research findings • Contributions• Future work
Overview
12/13/2011 Sharad V. Oberoi: PhD Final Public Oral Exam
15
Research Testbed
• Rapid Prototyping of Computer Systems Class 2008
• Kiva
Sharad V. Oberoi: PhD Final Public Oral Exam
16
Research Testbed
• Rapid Prototyping of Computer Systems (RPCS) 2008• The intermediate and final deliverables and
conversations captured through the Kiva
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
17
RPCS 2008
• 25 students; 3 phases; 4 major teams• Taught by two instructors and one teaching assistant• Goal: To design a virtual coach called Guru for new
power wheelchair users• Clients are actively involved with specific teams,
providing relevant expertise, project advice and any clarifications that are needed.
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
18
Kiva
• Light-weight collaboration tool• Combines functions of e-mail and bboards• Widely accepted and liked by student teams; it feels likes
chat and meets their needs• Each year’s Kiva has hundreds of threads and thousands
of posts and files• The posts are publicly visible
12/13/2011
• Motivation• Related research• Research testbed• Research questions• Research findings • Contributions• Future work
Overview
12/13/2011 Sharad V. Oberoi: PhD Final Public Oral Exam
19
Research Questions
1. Can concept maps of a design artifact be automatically extracted from the group conversation and documents for ongoing and archived project corpora?
2. Can the noun phrases used by students in a design project be used as a surrogate measure for assessing design team dynamics?
3. Can DesignWebs be used as a research framework to facilitate the validation of prior research findings?
Sharad V. Oberoi: PhD Final Public Oral Exam
20
Research Questions
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
21
Research Questions
1. Can the concept maps of a design artifact be automatically extracted from the group conversation and documents for ongoing and archived project corpora?
2. Can the noun phrases used by students in a design project be used as a surrogate measure for assessing design team dynamics?
3. Can DesignWebs be used as a research framework to facilitate the validation of prior research findings about the functioning and assessment of design project teams?
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
22
DesignWebs
• This question addresses the problem of information management in engineering design projects
• Evidence of student design collaboration is mined through computational linguistics and machine learning
• A concept-map based graphical representation is created that can be used to navigate the project corpus
• DesignWebs assist as navigation aids in contextualizing and viewing project archives by structuring the design information as graphs of related entities
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
23
Implementation of DesignWebs
• Extract the LDA model and apply hierarchical clustering to extract a hierarchical topic structure
• Collapsing of nodes• Calculate topic-term connections• Vector space models for all documents in a corpus are
combined to form a word-by-document matrix• Retrieval through a combination of navigation and query
by the student
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
24
DesignWebs as Representation of Design Knowledge
• DesignWebs reflect the shared structure of the artifact from design documentation
• DesignWebs assimilate information from multiple sources that would be beyond the comprehension of any single member of the project
• DesignWebs have the potential to reveal missing links between concepts or show seemingly unexpected connections
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
25
Research Questions
1. Can the concept maps of a design artifact be automatically extracted from the group conversation and documents for ongoing and archived project corpora?
2. Can the noun phrases used by students in a design project be used as a surrogate measure for assessing design team dynamics?
3. Can DesignWebs be used as a research framework to facilitate the validation of prior research findings about the functioning and assessment of design project teams?
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
26
Research Question 2
• Presents noun phrases as a surrogate measure for design team dynamics
• Shows that tracking the noun phrases over time and across team boundaries can reveal insights about teams collaboration
• Shows that shared vocabulary mirrors the work-flow between teams
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
27
Research Questions
1. Can the concept maps of a design artifact be automatically extracted from the group conversation and documents for ongoing and archived project corpora?
2. Can the noun phrases used by students in a design project be used as a surrogate measure for assessing design team dynamics?
a) Can the noun phrase counts in the design vocabulary of student design teams at different project milestones be used to assess their design process?
b) Can tracking the noun phrases, as they are introduced into the design vocabulary and shared across team boundaries over time, reveal insights into design team collaboration?
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
28
Noun Phrase Trends over Time
• Noun phrases have been used as a predictor of design performance (Mabogunje 1997)
• Noun phrases were extracted using Stanford part-of-speech tagger
• Challenges with RPCS 2008 data:1. Issues with task-specific groups2. Shared authorship of documents3. Role of references in structuring project knowledge
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
29
First Analysis
12/13/2011
• Only the Kiva documentation posted in the Kiva groups of the four main teams in the project were used
Sharad V. Oberoi: PhD Final Public Oral Exam
30
Second Analysis
12/13/2011
• Individual team members were followed throughout RPCS 2008 Kiva
• All postings and files were attributed to the parent teams, regardless of the group in which they were posted
Sharad V. Oberoi: PhD Final Public Oral Exam
31
Third Analysis
12/13/2011
• Time-based trends of noun phrases in the external references used by students were compared with the documents and discussions they had with each other
Sharad V. Oberoi: PhD Final Public Oral Exam
32
Research Questions
1. Can the concept maps of a design artifact be automatically extracted from the group conversation and documents for ongoing and archived project corpora?
2. Can the noun phrases used by students in a design project be used as a surrogate measure for assessing design team dynamics?
a) Can the noun phrase counts in the design vocabulary of student design teams at different project milestones be used to assess their design process?
b) Can tracking noun phrases, as they are introduced into the design vocabulary and shared across team boundaries over time, reveal insights into design team collaboration?
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
33
Connectivity of Key Ideas across Teams
• Noun phrases, used as design concepts, can be classified as: 1. Self-looping concepts2. Shared design concepts3. Transient design concepts4. Design escape concepts
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
34
Self-looping Concepts
• Some design concepts are introduced by a team, used only by them alone over time and become a part of the final artifact design
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
35
Shared Design Concepts
• Some design concepts are introduced by one team, get picked up by other teams, and are adopted in the shared vocabulary.
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
36
Transient Design Concepts
• Some design concepts are introduced by a team but disappear from its vocabulary with time
• They either represent abandoned design approaches, or are transformed into a related concept in the final design
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
37
Design Escape Concepts
• Some design concepts are introduced by one team and are important for the tasks assigned to other teams, but fail to attain ownership
• Transient design concepts and design escape concepts cannot be automatically differentiated
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
38
Noun Phrases mapped on System Architecture Diagram
• When design vocabulary is mapped on the system architecture diagram, it can assist the instructor in assessing team activities
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
39
System Architecture Diagram
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
40
Reflection of the System Architecture in the Design Team Vocabulary
12/13/2011
Self-looping concepts
Shared design concepts
Sharad V. Oberoi: PhD Final Public Oral Exam
41
12/13/2011Sensors Team in Phase 1Sensors Team in Phase 2Sensors Team in Phase 3
Student Activities reflected in Time-based Transformation of DesignWebs
Sharad V. Oberoi: PhD Final Public Oral Exam
42
Review of Tracking Noun Phrases
• Tracking the design vocabulary reveals:– Expansion and contraction of vocabulary over time – Insights into team collaboration– Missed or abandoned concepts– Work-flow between teams– Current state of the design
• Instructors can observe the changes in a team’s design knowledge unobtrusively by monitoring its design vocabulary
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
43
Research Questions
1. Can the concept maps of a design artifact be automatically extracted from the group conversation and documents for ongoing and archived project corpora?
2. Can the noun phrases used by students in a design project be used as a surrogate measure for assessing design team dynamics?
3. Can DesignWebs be used as a research framework to facilitate the validation of prior research findings about the functioning and assessment of design project teams?
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
44
DesignWebs as a Research Testbed
• Three diverse research studies are examined:1. Can using the semantic coherence of student
communications in team-based project courses act as an indicator of the design process? (Song et al. 2003)
2. Can applying metrics on the “communication artifacts” generated by computer supported collaborative tools can provide insights into the design process that created them? (Dutoit 1996)
3. Can various individual and group work processes when made more explicit to instructors assist them in carrying out the assessment processes? (Gweon 2008)
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
45
DesignWebs as a Research Testbed
• Three diverse research studies are examined:1. Can using the semantic coherence of student
communications in team-based project courses act as an indicator of the design process? (Song et al. 2003)
2. Can applying metrics on the “communication artifacts” generated by computer supported collaborative tools can provide insights into the design process that created them? (Dutoit 1996)
3. Can various individual and group work processes when made more explicit to instructors assist them in carrying out the assessment processes? (Gweon 2008)
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
46
Semantic Coherence of Student Communications
• Semantic coherence measures the similarity with which texts in a project corpus represent the variation in voices referred to in forming the concept
• A text analysis module called AgoraParse was used to extract communication metrics
• A language analysis module (AgoraProbe) was used to extract the semantic coherence metrics using LSA
• A graph of the semantic coherence was created using MATLAB
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
47
Semantic Coherence of RPCS 2008
12/13/2011
Semantic coherence trend for RPCS 2008 (Kiva discussions and documents)Semantic coherence trend for RPCS 2008 (Kiva discussions only)
Sharad V. Oberoi: PhD Final Public Oral Exam
48
Research Question 3
• Three diverse research studies are examined:1. Can using the semantic coherence of student
communications in team-based project courses act as an indicator of the design process? (Song et al. 2003)
2. Can applying metrics on the “communication artifacts” generated by computer supported collaborative tools can provide insights into the design process that created them? (Dutoit 1996)
3. Can various individual and group work processes when made more explicit to instructors assist them in carrying out the assessment processes? (Gweon 2008)
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
49
Communication Metrics to provide Insights into the Design Process
• Bruegge et al. (1997) used quantitative measures of b-board traffic as surrogates for inter-and intra-team communication metrics
• A software product can be measured through a number of measures such as code length, code complexity
• For comparability of results across projects, a formative approach was used
• Two structural equation models were employed for testing hypotheses on multiple projects
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
50
Research Question 3
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
51
Research Question 3
• Three diverse research studies are examined to demonstrate the wide applicability of DesignWebs as a research platform:1. Can using the semantic coherence of student communications
in team-based project courses act as an indicator of the design process? (Song et al. 2003)
2. Can applying metrics on the “communication artifacts” generated by computer supported collaborative tools can provide insights into the design process that created them? (Dutoit 1996)
3. Can various individual and group work processes when made more explicit to instructors assist them in carrying out the assessment processes? (Gweon 2008)
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
52
• Gweon (2008) used the mixed methods approach to support instructors in identifying difficulties in project teams
• Interviewed instructors to identify the group processes that instructors observe and desire in evaluating group work
• Five process assessment categories were subsequently identified
Process Assessment of Design Teams
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
53
Personal Goal Setting
• Students in RPCS 2008 were required to submit a work-log every week
• Students answered the following questions: 1. What did you accomplish this week? 2. What problems did you encounter this week? 3. What do you plan to work on next week?
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
54
Personal Progress
• Estimated in RPCS 2008 from student work logs• Involves subjective evaluation
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
55
• Tracks whether students actively attended team meetings or regularly posted on the Kiva
• Cannot distinguish between diligent students and fakes
Participation
12/13/2011
• Motivation• Related research• Research testbed• Research questions• Research findings • Contributions• Future work
Research Contributions
• An automated mechanism to visualize engineering design project documentation
• A surrogate measure to track the emergence of the shared solution
• A research test bed to validate different prior research findings
Overview
12/13/2011 Sharad V. Oberoi: PhD Final Public Oral Exam
56
Sharad V. Oberoi: PhD Final Public Oral Exam
57
Research Contributions
• An automated mechanism to integrate, visualize and navigate the evolving content of documents created by teams in engineering design projects
• A visual interface that summarizes the team communications to track the emergence of the shared solution using noun phrases as a surrogate measure
• A research test bed to validate different prior research findings about design project teams
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
58
Future Work
• Analysis of graphical and verbal communication
• Real-time monitoring of noun phrase usage• Functional and operational issues– Version mapping– Summarization and aggregation– Allowing complex queries
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
59
Questions?
12/13/2011
Research Focus
1. How can the student activities, as reflected in student project documents and discussions, be monitored by instructors actively and unobtrusively?
2. How can students visualize the state of the artifact as reflected in the project documents?
3. What objective measures can be used to assess the student design process and gain insights into student design collaboration?
12/13/2011 Sharad V. Oberoi: PhD Final Public Oral Exam
60