designwebs to support engineering design student projects sharad v. oberoi

DesignWebs to Support Engineering Design Student Projects

Sharad V. Oberoi

• Motivation• Related research• Research testbed• Research questions• Research findings • Contributions• Future work

Overview

12/13/2011 Sharad V. Oberoi: PhD Final Public Oral Exam

2

Research Testbed

• Rapid Prototyping of Computer Systems Class 2008

• Kiva

Related Research

• Collaborative learning in design

• Organizational memory

• Computational linguistics

• Information structuring and visualization approaches

Research Contributions

• An automated mechanism to visualize engineering design project documentation

• A surrogate measure to track the emergence of the shared solution

• A research test bed to validate different prior research findings

Future Work

• Analysis of graphical and verbal communication

• Real-time monitoring of noun phrase usage

• Functional and operational issues

Motivation

• How do students learn in engineering design project teams?

• How can additional information be provided to instructors to assess student learning?

Research Findings

• Design documentation can be visualized as concept-map based graphs

• Noun phrases act as surrogate measures for tracking design team dynamics

• DesignWebs can act as research platform to validate prior research

Research Questions

• How can the state of the artifact be summarized and presented visually to students and instructors from the project documents and Kiva discussions?

• What objective measures can be used to assess the student design process and gain insights into student design collaboration?

• How can prior research in design learning be validated?

Motivation• How do students learn from each other in engineering

design project teams?• How can additional information be provided to

instructors to assess student learning?


3

Students in Design Projects

• Create most of the knowledge outside the classroom• Have little experience in team-work or in design• Consider final product as the most important take-away• Have little or no support infrastructure for information

management


4

Instructors of Design Projects

• Intend to assess student learning outcomes, but have more ready access to product outcomes

• Often have to rely on the self-reported functioning of the teams to assess their progress

• Cannot always detect instances when students deviate from objectives

• Need a mechanism to monitor student progress in real-time


5

Research Issues

• Assessing evidence of student learning requires monitoring of student activities

• Unobtrusive approaches are necessary:– Students alter behavior in front of instructors– Embedded observations are tedious

• Evidence of student collaboration process exists, but is difficult to summarize and use


6


Overview


7

Related Research

• Collaborative learning in design

• Organizational memory

• Computational linguistics

• Information structuring and visualization approaches

Collaborative Learning in Design

• Collaborative learning allows students to hone their communication and negotiation skills (Dillenbourg 1999)

• When students with different strengths work together, they solve problems that would be beyond their reach if they were working alone (Vygotsky 1978)

• The depth of student explanations is correlated with how much students learn (Webb et al. 2002)


8

Sharad V. Oberoi: PhD Final Public Oral Exam

9

Organizational Memory

• Reuse of design information in organizations often requires designers to recreate the rationale (Subrahmanian et al. 1997)

• Easy access to archives can help future teams be successful with lower transactional costs (Roth et al. 1998)

• Organizational memory systems are usually domain specific and cannot be readily adpated in project classes

12/13/2011


10

Automatic Topic Segmentation

1. Lexical cohesion Models: – Lexical co-occurrence of thematically-related terms

indicates continuity in topic – Introduction of new vocabulary refers to a new topic

2. Content-oriented Models: – Combine lexical cohesion with other indicators of topic

shift such as cue phrases using probabilistic models (Reynar 1998)

12/13/2011


11

Latent Dirichlet Allocation (LDA)

• Widely used to cluster document fragments and automatically assign them topic labels (Blei et al. 2003)

• The analysis is done at the semantic level by considering probabilities for each of the words occurring in the corpus

• The context of the word can be examined, making word disambiguation possible

12/13/2011


12

Co-word Analysis

• Co-word analysis reduces a space of descriptors to a set of network graphs with the strongest associations between descriptors (Coulter et al. 1998)

• Co-word graphs construct multiple networks that highlight associations between keywords (Coulter et al. 1998)

12/13/2011


13

Information Structuring

• As student expertise increases, the corresponding concept interrelatedness in concept maps resembles tightly integrated structures (Royer et al. 1993)

• Scientometrics identifies emerging research areas and the evolution of research areas (Callon et al. 1991; van Raan 1992)

• Library sciences have done research on organizing information stored in books and journals and providing structures for locating relevant information (Bhattacharya et al. 1998)

12/13/2011


14

Information Visualization

• Domain visualization “aims to reveal realms of scientific communication as reflected through scientific literature and citation paths interwoven by individual scientists in their publications” (Börner et al. 2003)

• Longitudinal maps have been used to detect the evolving nature of scientific fields by scientometrists (Garfield 1994)

12/13/2011


Overview


15

Research Testbed

• Rapid Prototyping of Computer Systems Class 2008

• Kiva


16

Research Testbed

• Rapid Prototyping of Computer Systems (RPCS) 2008• The intermediate and final deliverables and

conversations captured through the Kiva

12/13/2011


17

RPCS 2008

• 25 students; 3 phases; 4 major teams• Taught by two instructors and one teaching assistant• Goal: To design a virtual coach called Guru for new

power wheelchair users• Clients are actively involved with specific teams,

providing relevant expertise, project advice and any clarifications that are needed.

12/13/2011


18

Kiva

• Light-weight collaboration tool• Combines functions of e-mail and bboards• Widely accepted and liked by student teams; it feels likes

chat and meets their needs• Each year’s Kiva has hundreds of threads and thousands

of posts and files• The posts are publicly visible

12/13/2011


Overview


19

Research Questions

1. Can concept maps of a design artifact be automatically extracted from the group conversation and documents for ongoing and archived project corpora?

2. Can the noun phrases used by students in a design project be used as a surrogate measure for assessing design team dynamics?

3. Can DesignWebs be used as a research framework to facilitate the validation of prior research findings?


20

Research Questions

12/13/2011


21

Research Questions

1. Can the concept maps of a design artifact be automatically extracted from the group conversation and documents for ongoing and archived project corpora?


3. Can DesignWebs be used as a research framework to facilitate the validation of prior research findings about the functioning and assessment of design project teams?

12/13/2011


22

DesignWebs

• This question addresses the problem of information management in engineering design projects

• Evidence of student design collaboration is mined through computational linguistics and machine learning

• A concept-map based graphical representation is created that can be used to navigate the project corpus

• DesignWebs assist as navigation aids in contextualizing and viewing project archives by structuring the design information as graphs of related entities

12/13/2011


23

Implementation of DesignWebs

• Extract the LDA model and apply hierarchical clustering to extract a hierarchical topic structure

• Collapsing of nodes• Calculate topic-term connections• Vector space models for all documents in a corpus are

combined to form a word-by-document matrix• Retrieval through a combination of navigation and query

by the student

12/13/2011


24

DesignWebs as Representation of Design Knowledge

• DesignWebs reflect the shared structure of the artifact from design documentation

• DesignWebs assimilate information from multiple sources that would be beyond the comprehension of any single member of the project

• DesignWebs have the potential to reveal missing links between concepts or show seemingly unexpected connections

12/13/2011


25

Research Questions




12/13/2011


26

Research Question 2

• Presents noun phrases as a surrogate measure for design team dynamics

• Shows that tracking the noun phrases over time and across team boundaries can reveal insights about teams collaboration

• Shows that shared vocabulary mirrors the work-flow between teams

12/13/2011

sfinger

RQ 2a and 2b


27

Research Questions



a) Can the noun phrase counts in the design vocabulary of student design teams at different project milestones be used to assess their design process?

b) Can tracking the noun phrases, as they are introduced into the design vocabulary and shared across team boundaries over time, reveal insights into design team collaboration?

12/13/2011


28

Noun Phrase Trends over Time

• Noun phrases have been used as a predictor of design performance (Mabogunje 1997)

• Noun phrases were extracted using Stanford part-of-speech tagger

• Challenges with RPCS 2008 data:1. Issues with task-specific groups2. Shared authorship of documents3. Role of references in structuring project knowledge

12/13/2011


29

First Analysis

12/13/2011

• Only the Kiva documentation posted in the Kiva groups of the four main teams in the project were used


30

Second Analysis

12/13/2011

• Individual team members were followed throughout RPCS 2008 Kiva

• All postings and files were attributed to the parent teams, regardless of the group in which they were posted


31

Third Analysis

12/13/2011

• Time-based trends of noun phrases in the external references used by students were compared with the documents and discussions they had with each other


32

Research Questions



a) Can the noun phrase counts in the design vocabulary of student design teams at different project milestones be used to assess their design process?

b) Can tracking noun phrases, as they are introduced into the design vocabulary and shared across team boundaries over time, reveal insights into design team collaboration?

12/13/2011


33

Connectivity of Key Ideas across Teams

• Noun phrases, used as design concepts, can be classified as: 1. Self-looping concepts2. Shared design concepts3. Transient design concepts4. Design escape concepts

12/13/2011


34

Self-looping Concepts

• Some design concepts are introduced by a team, used only by them alone over time and become a part of the final artifact design

12/13/2011


35

Shared Design Concepts

• Some design concepts are introduced by one team, get picked up by other teams, and are adopted in the shared vocabulary.

12/13/2011


36

Transient Design Concepts

• Some design concepts are introduced by a team but disappear from its vocabulary with time

• They either represent abandoned design approaches, or are transformed into a related concept in the final design

12/13/2011


37

Design Escape Concepts

• Some design concepts are introduced by one team and are important for the tasks assigned to other teams, but fail to attain ownership

• Transient design concepts and design escape concepts cannot be automatically differentiated

12/13/2011


38

Noun Phrases mapped on System Architecture Diagram

• When design vocabulary is mapped on the system architecture diagram, it can assist the instructor in assessing team activities

12/13/2011


39

System Architecture Diagram

12/13/2011


40

Reflection of the System Architecture in the Design Team Vocabulary

12/13/2011

Self-looping concepts

Shared design concepts


41

12/13/2011Sensors Team in Phase 1Sensors Team in Phase 2Sensors Team in Phase 3

Student Activities reflected in Time-based Transformation of DesignWebs


42

Review of Tracking Noun Phrases

• Tracking the design vocabulary reveals:– Expansion and contraction of vocabulary over time – Insights into team collaboration– Missed or abandoned concepts– Work-flow between teams– Current state of the design

• Instructors can observe the changes in a team’s design knowledge unobtrusively by monitoring its design vocabulary

12/13/2011


43

Research Questions




12/13/2011


44

DesignWebs as a Research Testbed

• Three diverse research studies are examined:1. Can using the semantic coherence of student

communications in team-based project courses act as an indicator of the design process? (Song et al. 2003)

2. Can applying metrics on the “communication artifacts” generated by computer supported collaborative tools can provide insights into the design process that created them? (Dutoit 1996)

3. Can various individual and group work processes when made more explicit to instructors assist them in carrying out the assessment processes? (Gweon 2008)

12/13/2011


45

DesignWebs as a Research Testbed





12/13/2011


46

Semantic Coherence of Student Communications

• Semantic coherence measures the similarity with which texts in a project corpus represent the variation in voices referred to in forming the concept

• A text analysis module called AgoraParse was used to extract communication metrics

• A language analysis module (AgoraProbe) was used to extract the semantic coherence metrics using LSA

• A graph of the semantic coherence was created using MATLAB

12/13/2011


47

Semantic Coherence of RPCS 2008

12/13/2011

Semantic coherence trend for RPCS 2008 (Kiva discussions and documents)Semantic coherence trend for RPCS 2008 (Kiva discussions only)


48

Research Question 3





12/13/2011


49

Communication Metrics to provide Insights into the Design Process

• Bruegge et al. (1997) used quantitative measures of b-board traffic as surrogates for inter-and intra-team communication metrics

• A software product can be measured through a number of measures such as code length, code complexity

• For comparability of results across projects, a formative approach was used

• Two structural equation models were employed for testing hypotheses on multiple projects

12/13/2011


50

Research Question 3

12/13/2011


51

Research Question 3

• Three diverse research studies are examined to demonstrate the wide applicability of DesignWebs as a research platform:1. Can using the semantic coherence of student communications

in team-based project courses act as an indicator of the design process? (Song et al. 2003)



12/13/2011


52

• Gweon (2008) used the mixed methods approach to support instructors in identifying difficulties in project teams

• Interviewed instructors to identify the group processes that instructors observe and desire in evaluating group work

• Five process assessment categories were subsequently identified

Process Assessment of Design Teams

12/13/2011


53

Personal Goal Setting

• Students in RPCS 2008 were required to submit a work-log every week

• Students answered the following questions: 1. What did you accomplish this week? 2. What problems did you encounter this week? 3. What do you plan to work on next week?

12/13/2011


54

Personal Progress

• Estimated in RPCS 2008 from student work logs• Involves subjective evaluation

12/13/2011


55

• Tracks whether students actively attended team meetings or regularly posted on the Kiva

• Cannot distinguish between diligent students and fakes

Participation

12/13/2011



• An automated mechanism to visualize engineering design project documentation

• A surrogate measure to track the emergence of the shared solution

• A research test bed to validate different prior research findings

Overview


56


57


• An automated mechanism to integrate, visualize and navigate the evolving content of documents created by teams in engineering design projects

• A visual interface that summarizes the team communications to track the emergence of the shared solution using noun phrases as a surrogate measure

• A research test bed to validate different prior research findings about design project teams

12/13/2011


58

Future Work

• Analysis of graphical and verbal communication

• Real-time monitoring of noun phrase usage• Functional and operational issues– Version mapping– Summarization and aggregation– Allowing complex queries

12/13/2011


59

Questions?

12/13/2011

Research Focus

1. How can the student activities, as reflected in student project documents and discussions, be monitored by instructors actively and unobtrusively?

2. How can students visualize the state of the artifact as reflected in the project documents?

3. What objective measures can be used to assess the student design process and gain insights into student design collaboration?


60

designwebs to support engineering design student projects sharad v. oberoi

Documents