![Page 1: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/1.jpg)
Crowdsourcing for Educational Evaluation ine-Learning
Raj Agrawal
M.Tech ProjectUnder the guidance of
Prof. Deepak B. Phatak
Computer Science & EngineeringIndian Institute of Technology, Bombay
June 23, 2014
![Page 2: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/2.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
1 IntroductionCrowdsourcingClassification of CrowdsourcingMotivation
2 Crowdsourcing in Educational EvaluationNeed of Crowdsourcing in Educational EvaluationPeer to Peer Evaluation System
3 Problem Scope
4 Proposed SolutionMeasuring Knowledge Level of the ParticipantsGroup Formation in Peer EvaluationAllocation Strategies of Projects for Evaluation Based on KnowledgeLevelCalculating Final Grade
5 Implementation Details
6 Conclusion
7 References
Outline Crowdsourcing for Educational Evaluation in e-Learning 2
![Page 3: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/3.jpg)
Introduction
![Page 4: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/4.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Introduction”Crowdsourcing”
• What is Crowdsourcing?
• Distributive Problem solving Approach
• Crowdsourcing Users• Requestor• Workers
• Classification:• Applications• Performance
Introduction Crowdsourcing for Educational Evaluation in e-Learning 4
![Page 5: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/5.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Introduction”Crowdsourcing”
• What is Crowdsourcing?
• Distributive Problem solving Approach
• Crowdsourcing Users• Requestor• Workers
• Classification:• Applications• Performance
Introduction Crowdsourcing for Educational Evaluation in e-Learning 4
![Page 6: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/6.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Introduction”Crowdsourcing”
• What is Crowdsourcing?
• Distributive Problem solving Approach
• Crowdsourcing Users• Requestor• Workers
• Classification:• Applications• Performance
Introduction Crowdsourcing for Educational Evaluation in e-Learning 4
![Page 7: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/7.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Introduction”Crowdsourcing”
• What is Crowdsourcing?
• Distributive Problem solving Approach
• Crowdsourcing Users• Requestor• Workers
• Classification:• Applications• Performance
Introduction Crowdsourcing for Educational Evaluation in e-Learning 4
![Page 8: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/8.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Classification of Crowdsourcing
Figure: A Classification of Crowdsourcing
Introduction Crowdsourcing for Educational Evaluation in e-Learning 5
![Page 9: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/9.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Introduction”Motivation”
• The main motivation behind crowdsourcing workers are:• Financial Incentives, e.g., Amazon Mechanical Turk
• Entertainment, e.g., the ESP Game• Create a community, e.g., Wikipedia and Stack Overflow.
Introduction Crowdsourcing for Educational Evaluation in e-Learning 6
![Page 10: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/10.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Introduction”Motivation”
• The main motivation behind crowdsourcing workers are:• Financial Incentives, e.g., Amazon Mechanical Turk• Entertainment, e.g., the ESP Game
• Create a community, e.g., Wikipedia and Stack Overflow.
Introduction Crowdsourcing for Educational Evaluation in e-Learning 6
![Page 11: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/11.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Introduction”Motivation”
• The main motivation behind crowdsourcing workers are:• Financial Incentives, e.g., Amazon Mechanical Turk• Entertainment, e.g., the ESP Game• Create a community, e.g., Wikipedia and Stack Overflow.
Introduction Crowdsourcing for Educational Evaluation in e-Learning 6
![Page 12: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/12.jpg)
Crowdsourcing in Educational Evaluation
![Page 13: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/13.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Need of Crowdsourcing in Educational Evaluation
• MOOC Courses• Flexible self paced method of learning• Attracts large number of participants
• Traditional approaches of evaluation
• Use the concept of Crowdsourcing• Motivational issues
• Peer to peer evaluation system
Crowdsourcing in Educational Evaluation Crowdsourcing for Educational Evaluation in e-Learning 8
![Page 14: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/14.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Need of Crowdsourcing in Educational Evaluation
• MOOC Courses• Flexible self paced method of learning• Attracts large number of participants
• Traditional approaches of evaluation
• Use the concept of Crowdsourcing• Motivational issues
• Peer to peer evaluation system
Crowdsourcing in Educational Evaluation Crowdsourcing for Educational Evaluation in e-Learning 8
![Page 15: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/15.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Need of Crowdsourcing in Educational Evaluation
• MOOC Courses• Flexible self paced method of learning• Attracts large number of participants
• Traditional approaches of evaluation
• Use the concept of Crowdsourcing• Motivational issues
• Peer to peer evaluation system
Crowdsourcing in Educational Evaluation Crowdsourcing for Educational Evaluation in e-Learning 8
![Page 16: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/16.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Need of Crowdsourcing in Educational Evaluation
• MOOC Courses• Flexible self paced method of learning• Attracts large number of participants
• Traditional approaches of evaluation
• Use the concept of Crowdsourcing• Motivational issues
• Peer to peer evaluation system
Crowdsourcing in Educational Evaluation Crowdsourcing for Educational Evaluation in e-Learning 8
![Page 17: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/17.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Peer to Peer Evaluation System
Peer Evaluation
Peer evaluation is the evaluation of work by one or morepeople of similar competence to the creator of the workcalled peer
• Evaluation can be completed in time
• Learning advantage
Crowdsourcing in Educational Evaluation Crowdsourcing for Educational Evaluation in e-Learning 9
![Page 18: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/18.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Peer to Peer Evaluation System
Peer Evaluation
Peer evaluation is the evaluation of work by one or morepeople of similar competence to the creator of the workcalled peer
• Evaluation can be completed in time
• Learning advantage
Crowdsourcing in Educational Evaluation Crowdsourcing for Educational Evaluation in e-Learning 9
![Page 19: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/19.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Peer to Peer Evaluation System
Peer Evaluation
Peer evaluation is the evaluation of work by one or morepeople of similar competence to the creator of the workcalled peer
• Evaluation can be completed in time
• Learning advantage
Crowdsourcing in Educational Evaluation Crowdsourcing for Educational Evaluation in e-Learning 9
![Page 20: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/20.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Challenges
• Diversity in Intelligence
• Reliability
• Poor judgment
• Time Consuming
• Lack of motivation
Crowdsourcing in Educational Evaluation Crowdsourcing for Educational Evaluation in e-Learning 10
![Page 21: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/21.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Challenges
• Diversity in Intelligence
• Reliability
• Poor judgment
• Time Consuming
• Lack of motivation
Crowdsourcing in Educational Evaluation Crowdsourcing for Educational Evaluation in e-Learning 10
![Page 22: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/22.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Challenges
• Diversity in Intelligence
• Reliability
• Poor judgment
• Time Consuming
• Lack of motivation
Crowdsourcing in Educational Evaluation Crowdsourcing for Educational Evaluation in e-Learning 10
![Page 23: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/23.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Challenges
• Diversity in Intelligence
• Reliability
• Poor judgment
• Time Consuming
• Lack of motivation
Crowdsourcing in Educational Evaluation Crowdsourcing for Educational Evaluation in e-Learning 10
![Page 24: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/24.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Challenges
• Diversity in Intelligence
• Reliability
• Poor judgment
• Time Consuming
• Lack of motivation
Crowdsourcing in Educational Evaluation Crowdsourcing for Educational Evaluation in e-Learning 10
![Page 25: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/25.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Problem Scope
Problem
Diversity in intelligence and domain knowledge, amongparticipants, creates an unavoidable problem of ensuringfairness in evaluation. Fairness here refers to the ability ofparticipants to evaluates their peers assignments
Proposed Solution
The problem of ensuring fairness in peer evaluation systemmay be resolved by exploiting the knowledge levels of theparticipants
Problem Scope Crowdsourcing for Educational Evaluation in e-Learning 11
![Page 26: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/26.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Problem Scope
Problem
Diversity in intelligence and domain knowledge, amongparticipants, creates an unavoidable problem of ensuringfairness in evaluation. Fairness here refers to the ability ofparticipants to evaluates their peers assignments
Proposed Solution
The problem of ensuring fairness in peer evaluation systemmay be resolved by exploiting the knowledge levels of theparticipants
Problem Scope Crowdsourcing for Educational Evaluation in e-Learning 11
![Page 27: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/27.jpg)
Proposed Solution to Ensure Fairness in Peer
Evaluation
![Page 28: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/28.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Proposed System
Figure: Proposed System Workflow
Proposed Solution Crowdsourcing for Educational Evaluation in e-Learning 13
![Page 29: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/29.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Measuring Knowledge Level of the Participants
• Use performance history of participants• Assumption: Prerecorded data is available for all participants in the
course.• In practical scenario this assumption may not always be correct .
• Take a quiz• Assumption: An expert is available who can set the quality quiz to
test knowledge• Delay in workflow• Performance of the system is highly dependent on the quality of the
quiz.
• Use a combination of above approaches.• Assumption: An expert is available to
• Ensure that performance data is available of all participants• To set the quality quiz to test participant knowledge
Proposed Solution Crowdsourcing for Educational Evaluation in e-Learning 14
![Page 30: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/30.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Measuring Knowledge Level of the Participants
• Use performance history of participants• Assumption: Prerecorded data is available for all participants in the
course.• In practical scenario this assumption may not always be correct .
• Take a quiz• Assumption: An expert is available who can set the quality quiz to
test knowledge• Delay in workflow• Performance of the system is highly dependent on the quality of the
quiz.
• Use a combination of above approaches.• Assumption: An expert is available to
• Ensure that performance data is available of all participants• To set the quality quiz to test participant knowledge
Proposed Solution Crowdsourcing for Educational Evaluation in e-Learning 14
![Page 31: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/31.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Measuring Knowledge Level of the Participants
• Use performance history of participants• Assumption: Prerecorded data is available for all participants in the
course.• In practical scenario this assumption may not always be correct .
• Take a quiz• Assumption: An expert is available who can set the quality quiz to
test knowledge• Delay in workflow• Performance of the system is highly dependent on the quality of the
quiz.
• Use a combination of above approaches.• Assumption: An expert is available to
• Ensure that performance data is available of all participants• To set the quality quiz to test participant knowledge
Proposed Solution Crowdsourcing for Educational Evaluation in e-Learning 14
![Page 32: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/32.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Group Formation
• Number of evaluations per assignment
• Number of groups
• Number of particiants in each group
Our Approach
If we want that each peer should be evaluated by M peers,then we should form M groups. If the total number ofparticipants are N, then we should have ( N
M ) participantsin each group, and each participant will evaluate Mparticipants.
Proposed Solution Crowdsourcing for Educational Evaluation in e-Learning 15
![Page 33: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/33.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Group Formation
• Number of evaluations per assignment
• Number of groups
• Number of particiants in each group
Our Approach
If we want that each peer should be evaluated by M peers,then we should form M groups. If the total number ofparticipants are N, then we should have ( N
M ) participantsin each group, and each participant will evaluate Mparticipants.
Proposed Solution Crowdsourcing for Educational Evaluation in e-Learning 15
![Page 34: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/34.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Allocation Strategies of Projects for Evaluation Based onKnowledge Level
Classification
Participants are classified in classes C1,C2, ...Cn based ontheir knowledge level, whereKnowledge level(Ci ) > Knowledge level(Ci+1)
• Allocate project of participant from class Ci to participant of classCi+1
• Limitation: Does not guarantee fair evaluation
• Allocate project of participant from class Ci to participant of classCi−1
• Limitation: Participants may not have any significant learningadvantage by the process of evaluation.
• Allocate projects for evaluation of participant Pi of class Ck ,to exactly 1 participant from each class C1,C2.....,Cn
Proposed Solution Crowdsourcing for Educational Evaluation in e-Learning 16
![Page 35: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/35.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Allocation Strategies of Projects for Evaluation Based onKnowledge Level
Classification
Participants are classified in classes C1,C2, ...Cn based ontheir knowledge level, whereKnowledge level(Ci ) > Knowledge level(Ci+1)
• Allocate project of participant from class Ci to participant of classCi+1
• Limitation: Does not guarantee fair evaluation
• Allocate project of participant from class Ci to participant of classCi−1
• Limitation: Participants may not have any significant learningadvantage by the process of evaluation.
• Allocate projects for evaluation of participant Pi of class Ck ,to exactly 1 participant from each class C1,C2.....,Cn
Proposed Solution Crowdsourcing for Educational Evaluation in e-Learning 16
![Page 36: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/36.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Allocation Strategies of Projects for Evaluation Based onKnowledge Level
Classification
Participants are classified in classes C1,C2, ...Cn based ontheir knowledge level, whereKnowledge level(Ci ) > Knowledge level(Ci+1)
• Allocate project of participant from class Ci to participant of classCi+1
• Limitation: Does not guarantee fair evaluation
• Allocate project of participant from class Ci to participant of classCi−1
• Limitation: Participants may not have any significant learningadvantage by the process of evaluation.
• Allocate projects for evaluation of participant Pi of class Ck ,to exactly 1 participant from each class C1,C2.....,Cn
Proposed Solution Crowdsourcing for Educational Evaluation in e-Learning 16
![Page 37: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/37.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Calculating Final Grade
• Normalizing the grades from each class• Normalizing factor of a class [normalizing factor(Class(Ci))]• Normalized score by participant Pi of class Ci =
normalized score(Pi ) = score(Pi ) ∗ normalizing factor(Class(Ci ))
• Calculating final grade: Average of normalized scores from allparticipants is the final grade of project.
Final grade(project(pj)) =
n∑i=0
normalized score(partcipantij )
n
Proposed Solution Crowdsourcing for Educational Evaluation in e-Learning 17
![Page 38: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/38.jpg)
Implementation Details
![Page 39: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/39.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
RequirementsParticipant’s Requirements
• He should be able to register themselves in a new course
• He should be able to submit any pending assignment
• He should be able to appear in any quiz available
• He should have an interface to download his peer’s submission forevaluation
• He should have an interface to submit the evaluation results
Implementation Details Crowdsourcing for Educational Evaluation in e-Learning 19
![Page 40: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/40.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
RequirementsParticipant’s Requirements
• He should be able to register themselves in a new course
• He should be able to submit any pending assignment
• He should be able to appear in any quiz available
• He should have an interface to download his peer’s submission forevaluation
• He should have an interface to submit the evaluation results
Implementation Details Crowdsourcing for Educational Evaluation in e-Learning 19
![Page 41: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/41.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
RequirementsParticipant’s Requirements
• He should be able to register themselves in a new course
• He should be able to submit any pending assignment
• He should be able to appear in any quiz available
• He should have an interface to download his peer’s submission forevaluation
• He should have an interface to submit the evaluation results
Implementation Details Crowdsourcing for Educational Evaluation in e-Learning 19
![Page 42: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/42.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
RequirementsParticipant’s Requirements
• He should be able to register themselves in a new course
• He should be able to submit any pending assignment
• He should be able to appear in any quiz available
• He should have an interface to download his peer’s submission forevaluation
• He should have an interface to submit the evaluation results
Implementation Details Crowdsourcing for Educational Evaluation in e-Learning 19
![Page 43: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/43.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
RequirementsParticipant’s Requirements
• He should be able to register themselves in a new course
• He should be able to submit any pending assignment
• He should be able to appear in any quiz available
• He should have an interface to download his peer’s submission forevaluation
• He should have an interface to submit the evaluation results
Implementation Details Crowdsourcing for Educational Evaluation in e-Learning 19
![Page 44: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/44.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
RequirementsInstructor’s Requirements
• He should be able to create a new assignment for a course• He should be able to add new evaluation criteria to the system, for
assignment evaluation
• He should be able to add questions to the pool• He must have an interface to add/update/delete questions
• He should be able to create quiz with available questions for aspecific assignment
• He should be able to add a new course to the system
Implementation Details Crowdsourcing for Educational Evaluation in e-Learning 20
![Page 45: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/45.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
RequirementsInstructor’s Requirements
• He should be able to create a new assignment for a course• He should be able to add new evaluation criteria to the system, for
assignment evaluation
• He should be able to add questions to the pool• He must have an interface to add/update/delete questions
• He should be able to create quiz with available questions for aspecific assignment
• He should be able to add a new course to the system
Implementation Details Crowdsourcing for Educational Evaluation in e-Learning 20
![Page 46: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/46.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
RequirementsInstructor’s Requirements
• He should be able to create a new assignment for a course• He should be able to add new evaluation criteria to the system, for
assignment evaluation
• He should be able to add questions to the pool• He must have an interface to add/update/delete questions
• He should be able to create quiz with available questions for aspecific assignment
• He should be able to add a new course to the system
Implementation Details Crowdsourcing for Educational Evaluation in e-Learning 20
![Page 47: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/47.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
RequirementsInstructor’s Requirements
• He should be able to create a new assignment for a course• He should be able to add new evaluation criteria to the system, for
assignment evaluation
• He should be able to add questions to the pool• He must have an interface to add/update/delete questions
• He should be able to create quiz with available questions for aspecific assignment
• He should be able to add a new course to the system
Implementation Details Crowdsourcing for Educational Evaluation in e-Learning 20
![Page 48: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/48.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Use Case Diagram
Implementation Details Crowdsourcing for Educational Evaluation in e-Learning 21
![Page 49: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/49.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Database Design
Implementation Details Crowdsourcing for Educational Evaluation in e-Learning 22
![Page 50: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/50.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Participant’s Side InterfaceView Courses
Implementation Details Crowdsourcing for Educational Evaluation in e-Learning 23
![Page 51: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/51.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Participant’s Side InterfaceSubmit Assignment
Implementation Details Crowdsourcing for Educational Evaluation in e-Learning 24
![Page 52: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/52.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Participant’s Side InterfaceAttempt Quiz
Implementation Details Crowdsourcing for Educational Evaluation in e-Learning 25
![Page 53: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/53.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Participant’s Side InterfaceEvaluate Assignments
Implementation Details Crowdsourcing for Educational Evaluation in e-Learning 26
![Page 54: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/54.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Participant’s Side InterfaceSubmit Evaluation Result
Implementation Details Crowdsourcing for Educational Evaluation in e-Learning 27
![Page 55: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/55.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Instructor’s Side InterfaceCourse Details
Implementation Details Crowdsourcing for Educational Evaluation in e-Learning 28
![Page 56: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/56.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Instructor’s Side InterfaceAdd Question
Implementation Details Crowdsourcing for Educational Evaluation in e-Learning 29
![Page 57: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/57.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Instructor’s Side InterfaceCreate Quiz
Implementation Details Crowdsourcing for Educational Evaluation in e-Learning 30
![Page 58: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/58.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Instructor’s Side InterfaceCreate Assignment
Implementation Details Crowdsourcing for Educational Evaluation in e-Learning 31
![Page 59: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/59.jpg)
Conclusion
![Page 60: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/60.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Conclusion
• Large number of participants in MOOC
• Traditional approach of evaluation is not suitable
• Use peer evaluation
• Challenge: Diversity in intelligence and knowledge among peers
• Exploit knowledge level information to distribute task of evaluationto ensure fairness
Conclusion Crowdsourcing for Educational Evaluation in e-Learning 33
![Page 61: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/61.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Conclusion
• Large number of participants in MOOC
• Traditional approach of evaluation is not suitable
• Use peer evaluation
• Challenge: Diversity in intelligence and knowledge among peers
• Exploit knowledge level information to distribute task of evaluationto ensure fairness
Conclusion Crowdsourcing for Educational Evaluation in e-Learning 33
![Page 62: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/62.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Conclusion
• Large number of participants in MOOC
• Traditional approach of evaluation is not suitable
• Use peer evaluation
• Challenge: Diversity in intelligence and knowledge among peers
• Exploit knowledge level information to distribute task of evaluationto ensure fairness
Conclusion Crowdsourcing for Educational Evaluation in e-Learning 33
![Page 63: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/63.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Conclusion
• Large number of participants in MOOC
• Traditional approach of evaluation is not suitable
• Use peer evaluation
• Challenge: Diversity in intelligence and knowledge among peers
• Exploit knowledge level information to distribute task of evaluationto ensure fairness
Conclusion Crowdsourcing for Educational Evaluation in e-Learning 33
![Page 64: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/64.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Conclusion
• Large number of participants in MOOC
• Traditional approach of evaluation is not suitable
• Use peer evaluation
• Challenge: Diversity in intelligence and knowledge among peers
• Exploit knowledge level information to distribute task of evaluationto ensure fairness
Conclusion Crowdsourcing for Educational Evaluation in e-Learning 33
![Page 65: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/65.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Future Work
• Implement other strategies, for measuring knowledge level
• Perform experiments with large number of participants
• Integration of developed system with edX platform
Conclusion Crowdsourcing for Educational Evaluation in e-Learning 34
![Page 66: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/66.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Future Work
• Implement other strategies, for measuring knowledge level
• Perform experiments with large number of participants
• Integration of developed system with edX platform
Conclusion Crowdsourcing for Educational Evaluation in e-Learning 34
![Page 67: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/67.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
Future Work
• Implement other strategies, for measuring knowledge level
• Perform experiments with large number of participants
• Integration of developed system with edX platform
Conclusion Crowdsourcing for Educational Evaluation in e-Learning 34
![Page 68: Crowdsourcing for Educational Evaluation in e-Learning · 2014-06-23 · Crowdsourcing for Educational Evaluation in e-Learning Raj Agrawal M.Tech Project Under the guidance of Prof](https://reader033.vdocuments.mx/reader033/viewer/2022050522/5fa5cfce48b3b60ce37d07af/html5/thumbnails/68.jpg)
Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References
References I
Roy Ballantyne, Karen Hughes, and Aliisa Mylonas.
Developing procedures for implementing peer assessment in large classes using an action research process.Assessment & Evaluation in Higher Education, 27(5):427–441, 2002.
Luis von Ahn and Laura Dabbish.
Labeling images with a computer game.In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’04, pages 319–326, New York, NY,USA, 2004. ACM.
M.E.D. van den Bogaard and G.N. Saunders-Smits.
Peer #x00026; self evaluations as means to improve the assessment of project based learning.In Frontiers In Education Conference - Global Engineering: Knowledge Without Borders, Opportunities Without Passports, 2007.FIE ’07. 37th Annual, pages S1G–12–S1G–18, 2007.
Daniel Weld, Eytan Adar, Lydia Chilton, Raphael Hoffmann, Eric Horvitz, Mitchell Koch, James Landay, Christopher Lin, and
Mausam Mausam.Personalized online education a crowdsourcing challenge.AAAI Workshops, 2012.
Man-Ching Yuen, I. King, and Kwong-Sak Leung.
A survey of crowdsourcing systems.In Privacy, security, risk and trust (passat), 2011 ieee third international conference on and 2011 ieee third internationalconference on social computing (socialcom), pages 766–773, 2011.
References Crowdsourcing for Educational Evaluation in e-Learning 35