Sharad Oberoi

Carnegie Mellon University

DesignWebs: Learning in Engineering Project Teams

2

Outline

• Common issues in engg project teams• Need for DesignWebs• Background• Data source• Approach• Visionary scenario• Research Questions

3

Student Engineering Projects

• Characteristics– Students bring together knowledge from

different sources:• collaborate among themselves• share knowledge• negotiate with each other, faculty members and

the client, in order to create engineering artifacts

– Students reuse previous knowledge and create new knowledge within the context of the problem

4

Common Issues in Engg Project Teams

• Knowledge isolation• Knowledge messiness• Transient team membership• Lack of knowledge synthesis

5

Collaborative Learning In Engg Projects

• Goals– Develop an infrastructure that supports knowledge

management for student engineering project teams– Assist members of engineering project teams to build

and retain knowledge created through discussions, project documents and sharing of artifacts

– Find evidence for learning in team communications constructed automatically from the project corpus

6

Designwebs

• Knowledge is created in student interactions through multiple modalities: email, online discussion forums, collaborative documents, etc.

• We visualize a knowledge management framework that:– Is shared: summarizing the team communications to

track the emergence of the shared solution– Aids visualization & navigation: Enables the

visualization and navigation of the ideas and their connections

– Is searchable: Retrieves queries generated explicitly by the user or implicitly during browsing

7

8

Need for DesignWebs

• DesignWebs will enable:– Within team knowledge-sharing– Across team knowledge-sharing– Global information access and integration

9

Background

• Group memory systems• Scientometrics• Textual Analysis of Design Conversations

– LDA Modeling– Cluster analysis– Co-word analysis

10

Group Memory Systems

• Organizational knowledge – Semantic knowledge– Episodic knowledge

• Group memory systems – act as a common repository to share information

gathered by individuals or developed by the team – Issue Based Information Systems

• Principle: “design process for complex problems is fundamentally a conversation among the stakeholders in which they bring their respective expertise and viewpoints to the resolution of design issues”

11

Scientometrics

• Scientometrics – studies the cognitive as well as socio-organizational

structures within a scientific community by analyzing the documents produced by that community

– Uses bibliometrics to identify emerging research areas and the evolution of research areas

12

Latent Dirichlet Allocation (LDA)

• LDA views every document as a mixture of various topics; each word in turn is attributable to one of the document's topics

• A generative process that models each document as a mixture of topics, and models each topic as a multinomial distribution over words

• Useful to model how collections of documents evolve over time

13

Cluster Analysis

• Used to group text segments with the goal of maximizing intra-group similarity and minimizing inter-group similarity

• Dimensionality reduction to derive useful representations of high dimensional data

• Vector Space Model – suffers from the vocabulary mismatch problem – alternative techniques: LSA, Lexical Chaining and

automatic discovery of vocabulary and thesauri

14

Co-word Analysis

• Uses content analysis to map the strength of association between keywords in textual data

• Basic principle:– it reduces a “space of descriptors (or keywords) to a set

of network graphs”

• These graphs do not display data like other statistical graphs, but construct multiple networks that highlight associations between keywords where possible

15

Background Review

• Group memory systems: requirements

• Scientometrics: modeling on large scale

• Textual Analysis of Design Conversations – LDA Modeling: topic model– Cluster analysis: hierarchical clustering of

topics– Co-word analysis: connections between terms

16

Capturing in-process data

• We have 6 years of data from project classes that used the Kiva for team collaboration– Combines functions of e-mail, bboards and chat– Each year’s Kiva has hundreds of threads and

thousands of posts and files

17

Capturing in-process data

18

Approach

• DesignWebs– synthesize the knowledge from multiple

sources of information used by team members during a project

– enable users to see connections between the concepts used in an artifact at different levels

– can reveal the current state of the project and the underlying structure of the artifact

19

Approach

• Seed the DesignWeb with research papers• Extract the Latent Dirichlet Allocation topic model• Apply hierarchical clustering • Label clusters and extract phrases• Calculate connections between topics and terms• Co-word analysis• Make relevant documents accessible• Generate the web-based navigable DesignWeb

20

Visionary Scenario

• A class of engineering graduate students are about to embark on a project involving an analysis of hybrid cars

• At the start of such a class, students normally are asked to perform a literature review to understand the current state of the technology and the issues involved

• To simulate a typical knowledge base at the start of such a project class, students were asked to perform a literature review from a corpus of 250 articles from Google Scholar

21

Demo

22

Challenges

• Segment resolution• Levels of abstraction• Draft reports are often not well-structured• Alternate views for different users• Credibility of source • Identifying the structure of created knowledge,

especially for different versions of the same document

23

Research Questions

• Can the knowledge about an artifact be extracted and synthesized from multiple sources of information used and created by teams during an engineering project?

• Can we identify evidence of the success/failure of a group from the team vocabulary?

• Can we automatically detect inception and connectivity of key ideas in groups at various artifact milestones?

24

Research Question # 1

• Can student teams use the DesignWebs to navigate and search for artifact knowledge extracted and synthesized from multiple sources of information during an engineering project?– Perform user studies

25

Research Question # 2

• Can we identify evidence of the success/failure of a group from the team vocabulary?– Previous research has focused on convergence

/divergence of vocabulary as sign of group cohesion– Advantage of having student data throughout the

evolution of the design artifact – Churning vs. convergence– Assimilation of ideas within and across teams

26

Research Question # 3

• Can we automatically detect inception and connectivity of key ideas in groups at various artifact milestones?– Introduction of new ideas and propagation across

teams– Data flow from end-to-end– Vocabulary at team boundaries– Can we detect detail?

27

Progress to date

• Research Question # 1– The first prototype of DesignWebs tool has been

developed and some user studies carried out– The second prototype is currently being evaluated

• Research Questions # 2 and 3– Data already collected– Experiments will begin as soon as the prototype

revison is completed

28

Questions?

29

Group Memory Systems

• Drawbacks:– extra effort to document activities is perceived as

having no immediate benefit– documentation is usually after-the-task and so any

unsuccessful approaches are not documented– creators and eventual users may have different

backgrounds and mental models of the knowledge structures, leading to difficulties in learning

– emphasis in most group memory systems has been on the content aspect with little significance given to the process and context