TEMPLATE DESIGN © 2007

www.PosterPresentations.com

Aleahmad, T,. Aleven, V,. and Kraut, R. (in press) Open community authoring of worked example problems. In Proceedings of the International Conference of the Learning Sciences.

Aleahmad, T,. Aleven, V,. and Kraut, R. (under review) Open community authoring of targeted worked example problems . Proceedings of Intelligent Tutoring Systems 2008.

Anand, P. G., & Ross, S. M. (1987). Using Computer-Assisted Instruction to Personalize Arithmetic Materials for Elementary School Children. Journal of Educational Psychology, v79 n1 p72-78 Mar 1987.

Giles, J. (2005). Internet encyclopaedias go head to head, Nature, 438(7070), 900-901. doi: 10.1038/438900a.

Ku, H., & Sullivan, H. (2002). Student performance and attitudes using personalized mathematics instruction. Educational Technology Research and Development, 50(1), 21-34.

Murray, T. (1999). Authoring intelligent tutoring systems: An analysis of the state of the art. International Journal of Artificial Intelligence in Education, 10, pp. 98-129.

Schwonke, R., Wittwer, J., Aleven, V., Salden, R., Krieg, C., & Renkl, A. (2007). Can tutored problem solving benefit from faded worked-out examples. European Cognitive Science Conference, 23–27.

Important precondition for community-based authoring seems to be met: volunteers contribute high-quality math examples

Separating the wheat from the chaff required little effort

Both professional educators and amateurs contributed a large portion of useful materials.

Contributions from math teachers were not superior to those from others

Math teachers did write the best problem statements but amateurs wrote the best solutions

The student profile feature of the interface successfully drew out personalized resources

On every attribute the profile increased the likelihood of targeting it

Open community authoring of worked example problemsTuradg Aleahmad, Vincent Aleven, Robert Kraut

Human Computer Interaction Institute, Carnegie Mellon Universityturadg@cmu.edu

Abstract

Materials and Methods

Authoring Interface

Sample Submission

Quality of Open Access to Authoring

Literature Cited

Conclusions and Future Work

AcknowledgementsThanks to the ASSISTments project team.Thanks for Flickr user jenrock for photos used in personas.The research reported  here was supported in part by the Institute of Education Sciences, U.S. Department of Education, through Grant R305B040063 to Carnegie Mellon University.The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through "Effective Mathematics Education Research" program grant #R305K03140 to Carnegie Mellon University. The opinions expressed are those of the authors and do not represent views of the U.S. Department of Education.

Open collaborative authoring systems such as Wikipedia are growing in use and impact. How well does this model work for the development of educational resources? In particular, will sufficient numbers of volunteers participate and contribute materials of high quality? Further, will they create resources that are targeted to students’ specific learning needs and, in order to engage students more, their personal characteristics? We conducted an experiment to explore these questions through the use of a novel prototype tool for community authoring of worked examples for a specific math skill. Participants were professional teachers (math and non- math) and amateurs. Participants were randomly assigned to the standard tool, or to an enhanced version that prompts authors to create materials for a specific (fictitious) student. We find that while there are differences by teaching status (math teacher, other teacher or amateur), all three groups made contributions of worth (as judged by two independent teacher raters) and that targeting a specific student leads contributors to author materials with greater potential to engage students. The experiment suggests that community authoring of educational resources is a feasible model of development and can enable new levels of personalization.

Effect of Student Profiles on Authoring

Mixed methodology experiment drew upon volunteers over the web to create worked example problems

Contributions were rated by machine and by two experts Three components of worked example evaluated by the

criteria below: Statement, Work and Explanation (alpha=0.61 for statement and 0.78 for all)

Numerical value

Rating category

Definition

0 Useless No use in teaching and it would be easier to write a new one than improve this one.

1 Easy fix Has some faults, but they are obvious and can be fixed easily, in under 5 minutes.

2 Worthy Worthy of being given to a student who matches on the difficulty and subject matter. Assume that the system knows what's in the problem and what is appropriate for each student, based on their skills and interests.

3 Excellent Excellent example to provide to some student. Again, assume that the system knows what's in the problem and what is appropriate for each student, based on their skills and interests.

Attribute With generic

(G)

With profiles not mentioning

attribute (N)

With profiles mentioning

attribute (M)

F-test(G-M)

F-test(N-M)

Female pronoun 5% 4% 16% 9.68* 12.82**Male pronoun 19% 14% 19% 0.004 1.19Sports word 9% 9% 24% 18.01** 11.89**TV word 4% 4% 10% 8.36* 2.63†Music word 2% 2% 9% 6.92* 8.93**Home word 14% n/a 20% 3.60* n/a

Count of submissions in each quality classification

Verbal skill shown

Sign. diffs

Mean reading level of

contribution

Std Err General math skill

shown

Sign.diffs

Probability of using 3-4-5

triangle

Std Err

High A 3.78 0.24 High A 16% 0.05Medium A B 3.56 0.32 Medium A B 26% 0.05Low B 2.93 0.33 Low B 27% 0.04GENERIC B 3.20 0.16 GENERIC A B 21% 0.03

Correspondence of verbal and math skill levels with the authoring interface

Example profile

Explanation of work

Illustrating solution steps

Illustrating solution

Problem statement

Submissions analyzed across teacher status of participant: math teacher, other teacher, or amateur

Motivation Work to solution Open collaborative development can work as well as

centralized development by experts (Gilles 2005) Traditional model-based tutors cost much time from skilled

experts (Murray 1999) Worked examples complement tutoring systems (Schwonke

et al. 2007). Personalization improves student engagement and test scores

(Ku and Sullivan 2002 , Anand and Ross 1987)

Of the raw submissions made, half were incompliant but also trivial to filter by simple automated methods

Of the remaining, a novice and a veteran teacher were able to rate each of them on three attributes in less than a minute each

About 1/10th were ready to help students learn without needing any modification

Statements were the highest quality components and solutions were the most difficult parts to author well

Math teachers were best at authoring problem statements

Amateurs authored the best worked solutions

Probabilities of authoring matching an attribute(†p<.10 *p<.05 **p<.001 )

All features of the profile display were accounted for in the problems submitted.

Discussing a male very much more likely than a female

Female profile brings likelihood to on par with males

High and low reading levels differed by almost a grade level

Submissions for profiles with high general math skill level were one third less likely to make use of simple 3-4-5 triangle problems

Experimental manipulation in which the subjects were presented with a student profile and told to provide instruction for that student

Math teachers Other teachers AmateursRegistered 131 170 1126Contributed also 70 72 428Passed vetting also 26 35 220