recommending knowledgeable people in a work integrated learning environment
DESCRIPTION
According to studies into learning at work, interpersonal help seeking is the most important strategy of how people acquire knowledge at their workplaces. Finding knowledgeable persons, however, can often be difficult for several reasons. Expert finding systems can support the process of identifying knowledgeable colleagues thus facilitating communication and collaboration within an organization. In order to provide the expert finding functionality, an underlying user model is needed that represents the characteristics of each individual user. With these slides, we present the APOSDLE People Recommender Service which is based on an underlying domain model, and on the APOSDLE User Model.TRANSCRIPT
Recommending Knowledgeable People
in a Work-Integrated Learning System
(RecSysTEL Workshop at EC-TEL 2010)
Günter Beham, Barbara Kump, Tobias Ley, Stefanie Lindstaedt
October 20, 10 / 2 Executive Board Meeting, Graz
Organisa(ons try to transform workplaces into more effec(ve learning environments
[e.g., Billet, 2000]
Knowledge Workers seek for inter-personal help
I am filling out this new report form. Any idea what all these abbreviations mean?
Hmm, not really but maybe Paul could help here. He filled a similar report last week.
[Kooken et al., 2007]
Challenge: Finding knowledgeable people for a topic within a company
APOSDLE Vision
Enable learning directly at the workplace
Support people in sharing their knowledge
Reuse available resources as learning materials
The APOSDLE Approach: Connecting user activities with organisational models to recommend knowledgeable people
The APOSDLE People Recommendation Workflow
September 29, 2010 / 9
How APOSDLE looks like
October 20, 10 / 10
• Screenshots vom APOSDLE Prototypen: Suggests und Coopera;on Wizard
People
Company Resources
3-Tier Architecture of APOSDLE Services
September 29, 2010 / 11
Organisational Models
September 29, 2010 / 12
Maintaining the APOSDLE User Model September 29, 2010 / 13
Viewing a Resource Performing a Task
Being Contacted Sharing a Resource
...
0 20 40 60 80
100
Identifying Knowledge Levels
September 29, 2010 / 14
Beginner Advanced
Expert
0
20
40
60
80
100
Beginner
Advanced
Expert
Where APOSDLE Services come into play Detecting the learning need of a worker
Finding a knowledgeable person who can help
September 29, 2010 / 15
Testing and evaluating the APOSDLE User Model and Services
Simulation Study Comparison of different algorithms for maintaining the user model
Which algorithm can detect a user‘s knowledge level best?
Workplace Evaluations Deployment of APOSDLE in 2 real work environments
Comparison of knowledge level as diagnosed by APOSDLE with self-assessment
September 29, 2010 / 17
Simulation Study (Example Design)
Fixed Parameters Number of persons, number of user events, inference algorithm
Variable Parameters User behavior (Beginner, Advanced, Expert)
Level Advanced
Behavior 60% norm.
Inference Frequency
Level Beginner
Behavior 60% norm.
Inference Frequency
Level Expert
Behavior 60% norm.
Inference Frequency
September 29, 2010 / 18
Simulation Result (Example)
6 Persons 1 Topic 50 events/Behavior type Inference: Weighted Frequencies
Simulation Result (Example)
6 Persons 1 Topic 50 events/Behavior type Inference: Weighted Frequencies with windowing
Deploying APOSDLE in real workplaces
Real-world evaluation in 2 Organisations
Library of a Distance University
10 Users, only 5 Users willing to participate in the self-assessment
Used APOSDLE for 4,5 Months
Self-assessment (online questionnaire)
Innovation Management (ISN)
6 Users
Used APOSDLE for 3 Months
Self-assessment and peer-assessment (using cards)
September 29, 2010 / 22
How well does APOSDLE detect the workers‘ Work topics?
Knowledge levels?
Library of a Distance University
How well does APOSDLE detect the workers‘ work topics?
September 29, 2010 / 23
APOSDLE user model
Work Topic Non-Work Topic Total
self-assessment Work Topic 133 81 214
Non-Work Topic 4 12 16
Total 137 93 230
In many cases, APOSDLE did not „know“ that topics were a user‘s work topics
Library of a Distance University How well does APOSDLE detect the workers‘ knowledge
levels?
September 29, 2010 / 24
APOSDLE user model
Expert Advanced Beginner No Work Topic Total
Self-assessment
Expert 1 27 11 44 83
Advanced 3 39 20 31 93
Beginner 2 24 6 6 38
No Work Topic 0 4 0 12 16
Total 6 94 37 93 230
APOSDLE classified users mostly „advanced“ where they regarded themselves as „beginners“ or „experts“
Innovation Management
How well does APOSDLE detect the workers‘ work topics?
September 29, 2010 / 25
APOSDLE user model
Work Topic (%) Non-Work Topic (%) Total
self-assessment Work Topic 356 (41.7) 334 (39.0) 690 (80.7) Non-Work Topic 51 (6.0) 114 (13.3) 165 (19.3) Total 407 (47.7) 448 (52.3) 855 (100)
In many cases, APOSDLE did not „know“ that topics were a user‘s work topics
September 29, 2010 / 26
Number of user interactions with APOSDLE
The more interaction with APOSDLE, the more correct detections of work topics and non-work topics
Innovation Management How well does APOSDLE detect the workers‘ knowledge
levels?
APOSDLE user model
Expert Advanced Beginner No Work Topic Total
Self-assessment
Expert 27 130 29 162 348 Advanced 11 73 19 82 185 Beginner 7 49 11 90 157 No Work Topic 5 36 10 114 165
Total 50 288 69 448 855
APOSDLE classified users mostly „advanced“ where they regarded themselves as „beginners“ or „experts“.
September 29, 2010 / 27
Discussion of Outcomes
In many cases, APOSDLE was not able to identify a user‘s work topics Users NEVER dealt with this topic within APOSDLE
Evaluation period too short? Rather: not enough system usage during evaluation period
In many cases, APOSDLE erroneously diagnosed „advanced“ level Improve algorithms
Self-assessment may also be erroneous/biased Better „external measure“ for workplace evaluations??
September 29, 2010 / 28
Outlook
Improving algorithms for diagnosing user knowledge Cross-validation with existing data
Further evaluations of the user model in other organisations
Combination of different recommendation strategies
Evaluating People Recommendations Evaluation Setup?
Lab studies
Field studies
September 29, 2010 / 29
