This article was downloaded by: [University of Chicago Library]On: 05 October 2014, At: 04:43Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH,UK

Studies in Higher EducationPublication details, including instructions for authorsand subscription information:http://www.tandfonline.com/loi/cshe20

Engaging Text: Assessingparaphrase and understandingIvan A. UemlianinPublished online: 09 Sep 2010.

To cite this article: Ivan A. Uemlianin (2000) Engaging Text: Assessing paraphrase andunderstanding, Studies in Higher Education, 25:3, 347-358, DOI: 10.1080/713696160

To link to this article: http://dx.doi.org/10.1080/713696160

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all theinformation (the “Content”) contained in the publications on our platform.However, Taylor & Francis, our agents, and our licensors make norepresentations or warranties whatsoever as to the accuracy, completeness,or suitability for any purpose of the Content. Any opinions and viewsexpressed in this publication are the opinions and views of the authors, andare not the views of or endorsed by Taylor & Francis. The accuracy of theContent should not be relied upon and should be independently verified withprimary sources of information. Taylor and Francis shall not be liable for anylosses, actions, claims, proceedings, demands, costs, expenses, damages,and other liabilities whatsoever or howsoever caused arising directly orindirectly in connection with, in relation to or arising out of the use of theContent.

This article may be used for research, teaching, and private study purposes.Any substantial or systematic reproduction, redistribution, reselling, loan,sub-licensing, systematic supply, or distribution in any form to anyone is

expressly forbidden. Terms & Conditions of access and use can be found athttp://www.tandfonline.com/page/terms-and-conditions

Dow

nloa

ded

by [

Uni

vers

ity o

f C

hica

go L

ibra

ry]

at 0

4:43

05

Oct

ober

201

4

Studies in Higher Education Volume 25, No. 3, 2000

Engaging Text: assessingparaphrase and understandingIVAN A. UEMLIANIN

University of Wales Bangor, UK

ABSTRACT This article reports on an innovative exercise designed to assess students’ paraphrase of

a given text, the ability to paraphrase being an important component part of understanding. Students

were given a source text consisting of a short passage from an introductory psychology textbook. A

checklist was drawn up enumerating the points of information in the source text, and the paraphrases

were then marked in terms of how many points of information and structure were reproduced.

Analysis of the marks showed three things: that the task was more than trivial, and involved more

than a single simple `skill’ ; that information central to the theme of the passage was reproduced more

reliably than peripheral information, with some signi® cant aberrations; and that points of infor-

mation clustered together into `kinds’ . The discussion explores this notion of kinds of information, and

notes its relevance for student feedback.

Introduction

The ability to provide an adequate paraphrase of a text is a necessary component of the abilityto understand that text. Approaches to understanding from two rather different perspectivesprovide some justi® cation for this assertion.

In cognitive psychology, understanding is at least in part de® ned as translating another’sconceptual system into a form which can be ® tted into one’ s own broad scheme of things(Kellogg, 1995), in a process similar to translating a foreign language into one’ s own.Translation shares some common features with paraphrase: for example, we can see thesimilarity between replacing ad praesens ova cras pullis sunt meliora with `eggs today are betterthan chickens tomorrow,’ and replacing the latter with `a bird in the hand is worth two in thebush’ . Translation between languages often includes a kind of silent’ paraphraseÐ for somepurposes, `every cloud has a silver lining’ may be considered a better translation of post nubila

Phoebus than the literal `after clouds, Phoebus’ (Ehrlich, 1987). Quine’s (1960) work on theindeterminacy of translation further points up this similarity. If translation can be describedas paraphrase, then paraphrase is more than simply a test of or a result of understanding;paraphrase is part of what it is to understand. In practice, attempts to understand often takethe form of attempts to paraphrase: one attempts to understand something by trying toarticulate it in different words, or to explain it to someone else (a ® rmer challenge,demanding that one’s paraphrase is not idiosyncratic or incomplete).

Recent work in the philosophy of mind (Hunter, 1998) explores understanding as a kindof `propositional perception’ , separate from, and providing justi® cation for, beliefs about atext’s meaning. In this model, I understand [text] to mean [proposition]’ becomes `I perceive[proposition] in [text]’ . A paraphrase is then an account of the reader’ s perceptions, or anassertion of their beliefs about the text (p. 565). Following these two approaches, paraphraseis at least an essential part, and perhaps the whole, of certain kinds of conceptual under-

ISSN 0307-5079 print; ISSN 1470-174X online/00/030347-12 Ó 2000 Society for Research into Higher Education

Dow

nloa

ded

by [

Uni

vers

ity o

f C

hica

go L

ibra

ry]

at 0

4:43

05

Oct

ober

201

4

348 A. Uemlianin

standing, with the quality of one’ s understanding manifested in the quality of paraphrase onecan produce. However, in educational literature, paraphrase is often opposed to understand-ing. A `paraphrase’ is contrasted with an `honest, carefully considered response’ (Marshall &Rowland, 1993; p. 153), to indicate a super® cial level of understanding. A more adequatetextual response, indicating a higher level of understanding, is thought of as somethingqualitatively differentÐ either because the source text has been more thoroughly processed bythe reader, or because the new text includes information not explicit in the source.

Paraphrase can also be de® ned rather vaguely: `When you paraphrase, you re-expressanother person’ s thoughts in your own words’ (Marshall & Rowland, 1993; p. 199); to showyour grasp of the ideas you have been studying you have to express them for yourself, in your

own words ¼ what you must avoid doing is using other people’ s words,’ (Chambers &Northedge, 1997; pp. 137± 138; emphasis in original). Such de® nitions obscure the importantpractical problems of how a reader processes a text for understanding. Further, the authorsseem unaware of the dif® cult and perhaps irrelevant notions of self-expression, originality andlexical ownership which their de® nitions raise. With no other more principled criteria,assessments of student’s paraphrases and understanding can seem quite arbitrary andmysterious, almost aesthetic.

The vagueness of conceptions of paraphrase has an effect on ideas about quotation andplagiarism. Questions of when a quotation is justi® ed, or of when close paraphrase becomesplagiarism, seem to involve the expert reader’s subjective stylistic judgement: consideration ofsuch objective matters as the discourse function of the paraphraseÐ how the paraphraserelates to the source text, and how the purpose of the paraphrase is represented in its formand contentÐ and the discourse functions of quotation are often not made explicit. Studentsoften seem unaware that their work includes large sections of plagiarised text, or of overuseof quotation. For example, one student arranged to write an essay for me as a revisionexercise. We agreed together on a topic, and a small number of journal articles as sourcematerial, and the student submitted the essay a short time later. To my surprise, the essayconsisted entirely of large passages reproduced almost verbatim from the set readings. Asthere were no marks attached to this essay, I found it hard to understand the motivation forthis and, indeed, the student had been unaware that this was plagiarism.

Methods assessing paraphrase often carry over this vagueness of de® nition into theassessment and analysis. For example, Kirby & Pedwell (1991) analysed their source textinto four levels of importance: unimportant details (which are discounted), importantdetails, main ideas and themes’ (p. 300). Students’ paraphrases were given a single numericalscoreÐ one point for each `important detail’ , three points for each `main idea’ and ® ve pointsfor a theme’ (p. 301). Selinger (1995) used a similar scoring system, giving 10 points for thetext’ s thesis, 10 for each main idea, and six each for the `ideas that directly supported themain ideas’ (p. 16).

There are three problems with this method of assessment. First, the method is extremelyad hoc, and requires every source text to be coded by an `expert’ into main ideas, themes, andso on. It also seems to preclude discussion of the discourse function of a paraphrase: everypossible discourse function would require a separate coding of the source text by the expert.Second, the implication that a main idea, for example, is worth three important details, andthat therefore a paraphrase which reproduces 3n important details but omits the main ideasis just like one which reproduces n main ideas but omits the detail, is dubious. For example,it ignores the important principle that `main ideas’ depend on `important details’ . Third, andmost important for my purposes, these coding systems seem not to allow a closer analysis ofwhat information and perhaps what kinds of information (not necessarily in hierarchicalterms) are being reproduced in students’ paraphrases.

Dow

nloa

ded

by [

Uni

vers

ity o

f C

hica

go L

ibra

ry]

at 0

4:43

05

Oct

ober

201

4

Assessing Paraphrase and Understanding 349

In 1995± 96, at the School of Psychology, University of Wales, Bangor, I was involved insetting up a new ® rst-year moduleÐ `Psychology, Communication and Information Technol-ogy’ Ð as an introductory psychology and study skills’ module. Our aim was to use insightfrom psychology to develop students’ consciousness and control over their study and learningpractice. Teaching and assessment methods re¯ ected the module’ s investigative, dialogicapproach (Uemlianin, 1998; Uemlianin & Beverley 1998a, 1998b). Engagement with textand the manipulation of information were recurrent themes, and a more explicit de® nition ofparaphrase, and a discussion of its role in understanding became necessary. Assessment ofparaphrase developed by 1997± 98 into the exercise studied in this article.

Paraphrase was de® ned as the reproduction of the information content and structure ofsource text. Plagiarism was de® ned as unacknowledged paraphrase or quotation (side-stepping problems of authorial intent). Quotation was de® ned as licensed when reproductionof a form of words was necessary for adequate information reproduction: for example, whenthe source text coins a famous phrase, when the paraphrase intends to draw attention tocertain presuppositions of the source text, or when it is otherwise necessary to draw attentionto the surface form of the source text. In other words, the acceptable level of similaritybetween paraphrase and source is not an absolute, but depends on the function or purposeof the paraphrase.

The apparently narrow de® nition of paraphrase as information reproduction broadensthe scope of the term and invites one to consider the purpose of each particular act ofparaphrase: in one situation, a super® cial translation of sentences may be suf® cient (e.g.relating a recipe); in others, an explication of some of the presuppositions in the source textmay be necessary (e.g. writing a critique); in still others, it may be appropriate to drawattention to the source text’s explicit and implicit intertextuality (e.g. reviewing a piece ofresearch). All of these are paraphrases of a source text, with various focuses of attention, theiressential differences determined by the function of the paraphrase in the discourse of whichit becomes a part. The `main points’ of a source text, the `essential information’ , are relativeto this function.

In the exercise detailed in this article, students were given as homework a short text toread and paraphrase. For the assessment, each proposition in the source text was coded assuch and given a point in the paraphrase. As well as a single numerical score, a point-by-pointchecklist was returned to each student as feedback and used in further analysis. On ® rst sightthis scoring system seems much more primitive than those reviewed earlier. However, it doesovercome some of their weaknesses:

· the system is simple, requiring only that each statement is logged (including references toother texts), along with logical connections between statements (e.g. because, therefore);

· the scoring marks objective properties of the source text, the paraphrase, and the relationbetween them (i.e. which propositions from the source are present in the paraphrase);

· for any given text, the scoring is the same for a paraphrase of any purpose. In other words,the differentiation of paraphrases by function is a stage of analysis after this initial pointcount;

· the scoring system provides a ® rm basis for further analysis of paraphrases (see resultssection).

The exercise was used as an assessment of students’ `paraphrasing skills’ and, indirectly, oftheir understanding (though see Barrow, 1987 and Davis, 1998, for sceptical remarks in theseareas); it was alsoÐ primarily Ð used as a tool for promoting discussion of the issues involved.The exercise was in part an attempt to design `assessment and feedback procedures which inthemselves constitute episodes of learning’ (Stefani, 1998).

Dow

nloa

ded

by [

Uni

vers

ity o

f C

hica

go L

ibra

ry]

at 0

4:43

05

Oct

ober

201

4

350 A. Uemlianin

Paraphrase ExerciseSubmission Date 5 `Two weeks from today.’

Assignment

The text below is taken from Gleitman (1995), p. 296. The source text is the passage headed`Automatization’. Introduce the text and write a paraphrase of its content. Figures and diagrams are notrequired. Your passage should be referenced fully in A.P.A. format.

Reference

Gleitman, H. (1995). Psychology. 4th Edition. London: W.W. Norton.

{FOR COPYRIGHT REASONS THE ORIGINAL SOURCE TEXT IS NOT INCLUDED}

FIG. 1. Handout.

Method

A class of 163 ® rst-year undergraduate students taking an introductory module on psy-chology and `study skills’ was given a homework exercise as shown in Fig. 1 (for copyrightreasons the source text itself is not reproduced).The session at which the assignment wasdistributed discussed the role of paraphrase in learning for understanding in similar terms tothose in the introduction to this article. The session was also explicit about the method ofassessment of the exercise. i.e. the use of a point-by-point checklist.

The source text was an excerpt from an introductory textbook, Gleitman (1995, p. 296),on automatisation of reading and the Stroop effect. The text, approximately 300 words long,introduces the notion of automatisation, describes the Stroop effect and the classic exper-iment demonstrating it, provides an explanation of the results and proposes a way of avoidingthe effect (i.e. blurring one’s vision so that one can see colours without being able to read).

The checklist (shown in Fig. 2) lists every point of information in the source text:citations in text and references are listed as separate points; structural features, like inferentialrelations (e.g. because, therefore) are listed as part of the point they introduced (e.g. as partof the consequent). Altogether the checklist lists 25 points. For each point, a student’ sparaphrase was given 1 mark if that point was present in the paraphrase, 0 marks if the pointwas absent, and -1 mark if the point was misrepresented. Unnecessary or unacknowledgedquotation was penalised by the appropriate point being marked as absent. There were alsooverall penalty marks for grammar, spelling and quality ( 2 1 mark for each; applied only ifgrammar, spelling or quality obtruded on meaning). Apart from these penalties, the surfaceform of the paraphrase was not assessed, and attention concentrated on the representation inthe paraphrase of points of information from the source.

Paraphrases were distributed to several markers (inter-marker reliability was found to begreater than 98%). The marked checklists were returned to each student, along with theirparaphrase, as feedback, and the total mark became part of the module’ s assessment. Furtherstatistical analysis of distribution and covariance of marks between points was carried outusing SPSS 7.5. Results from this analysis were fed back to students in a later general revisionsession.

Results

Results can be reported of three analyses: total marks by student, total marks by point, andcovariance of point scores.

Dow

nloa

ded

by [

Uni

vers

ity o

f C

hica

go L

ibra

ry]

at 0

4:43

05

Oct

ober

201

4

Assessing Paraphrase and Understanding 351

Paraphrase Checklist

1 INTRODUCTION TO PASSAGE

2 Citation: Gleitman, 1995.

3 Automatisation, de® nition: routinisation of sub-components of skilled activities has costs as well asbene® ts.

4 Automatisation, bene® t: Performance can be `automatised’ therefore requiring less attention.

5 Cost: Once set in motion operations are dif® cult to turn off.

6 Words automatically trigger reading routines. We can’t see words without reading them.7 Ð Citation: La Berge, 1975.

8 Stroop effect.

9 Ð Citation: Stroop, 1935.

10 Task: Subjects are asked to name, as quickly as possible, colours in which groups of letters areprinted.

11 Condition 1: Groups of letters not words; no trouble.

12 Condition 2: Groups of letters are colour terms.

13 Condition 2: Colour terms do not correspond to colours in which they are printed.

14 Condition 2: Responses signi® cantly slower.

15 Hypothesis: subjects are reading words as well as attending to colours. `Violent response con¯ ict’.

16 Overcoming interference (a): blurring eyes reduces the effect.

17 Overcoming interference (b): subject can see colours but cannot read words.

18 Ð Citation: Jensen, 1965

19 Ð Citation: Reisberg et al., 1980.

20 REFERENCES SECTION

21 Gleitman, H. (1995). Psychology. Fourth Edition. London: W.W. Norton.

22 Jensen, A.R. (1965). Scoring the Stroop test. Acta Psychologica, 24, 398± 408.

23 La Berge, D. (1975). Acquisition of automatic processing in perceptual and associative learning. InRabbitt, P.M.A., & Dormic, S. (Eds). Attention and performance, Vol. 5. London: Academic Press.

24 Reisberg, D., Baron, J. & Kemler, D.G. (1980). Overcoming Stroop interference: the effects of practiceon distractor potency. Journal of experimental psychology: Human perception and performance, 6,140± 50.

25 Stroop, J.R. (1935). Studies of interference in serial verbal reactions. Journal of experimentalpsychology, 18, 643± 62.

FIG. 2. Checklist.

Total Marks by Student

Fig. 3 shows the frequency distribution of total marks by studentÐ roughly, a right-skewednormal distribution. With the average mark at 19/25, the task was clearly not dif® cult butneither was it completely trivial, with 10% of students getting less than half marks.

Total Marks by Point

Fig. 4 shows the distribution of marks by point. A chi-squared test showed that thisdistribution differs signi® cantly from chance (p , 0.01): in other words, some points wererepresented in signi® cantly more/fewer paraphrases than were others. However, we cannotconclude directly from this that certain points were more dif® cult than others, or involvedmore interpretation (it should be noted that every point was explicit in the source text; theexercise did not involve `reading between the lines’ , or explicating presuppositions).

Dow

nloa

ded

by [

Uni

vers

ity o

f C

hica

go L

ibra

ry]

at 0

4:43

05

Oct

ober

201

4

25

20

15

10

5

0

4.00

5.00

8.00

9.00

10.0

011

.00

12.0

013

.00

14.0

015

.00

16.0

017

.00

18.0

019

.00

20.0

021

.00

22.0

023

.00

24.0

025

.00

Total marks

Fre

quen

cy

160

Mar

ks

25232119171513119751

Point number

3

150

140

130

120

110

100

90

80

70

60

352 A. Uemlianin

FIG. 3. Total marks by student.

FIG. 4. Total marks by point.

Three questions we can ask of this graph are: which points received the most, and theleast, representation? How did representation of citations compare with representation of thematching references? How well represented was the central theme of the source textÐ theStroop experiment?

Points with most and least representation. Table I shows the points which received the mostrepresentation. Point 20, the references section, was stipulated in the assignment instructions

Dow

nloa

ded

by [

Uni

vers

ity o

f C

hica

go L

ibra

ry]

at 0

4:43

05

Oct

ober

201

4

Assessing Paraphrase and Understanding 353

TABLE I. Points receiving most representation

Point Marks Information

20 157 References section present9 153 In-text citation of Stroop, 19358 150 Stroop effect named

TABLE II. Points receiving least representation

Point Marks Information

24 79 Reference for Reisberg et al., 198015 91 Hypothesis explaining Stroop results

2 95 In-text citation of Gleitman, 1995

and relates to the form of the paraphrase rather than its content. Points 8 and 9 name andcite a reference for the main topic of the source text, the Stroop effect. The high scoringpoints seem to show students’ sensitivity to priority of information.

The low scoring points go some way to con® rming this. Table II shows the points whichreceived the least representation. Points 2 and 24 might be regarded as low priority’information: information which plays little role in developing the thesis of the passage. TheGleitman citation and reference were given in the assignment instructions and played no roleat all in the information content of the source text; their role was to locate the source text,and to identify the paraphrase as such rather than as original research by the student. TheReisberg et al. citation and reference played little role and had a low priority position in thesource text, a see also’ in the ® nal sentence.

However, point 15Ð the explanation of Stroop’ s resultsÐ is perhaps the most importantpoint in the whole passage. The fact that this point was one of the lowest scoringÐ in otherwords, that this information was absent or misrepresented in more students’ texts than almostany other piece of information monitored by the assessmentÐ represents a common failure tograsp a central point of the passage, and contradicts the apparent sensitivity to importanceshown by other high and low scoring points.

Citations and References. With the exception of the Gleitman reference (which is exceptionalhere, not being part of the source text), sources were more often cited than given references(Table III). It is plausible to suppose that, for some texts and for some purposes, citations aregiven a higher priority than full references. In some cases, a full reference might be regarded(rightly or wrongly) as super¯ uous: for example, when works cited are well-known `classics’ ,or when both author and reader have ready access to a comprehensive bibliography. In the

TABLE III. Citations and references

Text Citation point Marks Reference point Marks Difference

Gleitman 2 95 21 126 2 31Jensen 18 143 22 118 25La Berge 7 140 23 117 23Reisberg 19 103 24 79 24Stroop 9 153 25 120 33

Dow

nloa

ded

by [

Uni

vers

ity o

f C

hica

go L

ibra

ry]

at 0

4:43

05

Oct

ober

201

4

354 A. Uemlianin

TABLE IV. Representation of the central theme

Point Marks Information

10 125 Experiment: subjects are asked to name, as quickly as possible,colours in which groups of letters are printed.

11 128 Condition 1: groups of letters are not words; subjects have notrouble.

12 147 Condition 2: groups of letters are colour terms.13 140 BUT: colour terms do not correspond to colours in which they are

printed.14 104 EFFECT: subjects’ responses signi® cantly slower.15 91 Hypothesis: subjects are reading words as well as attending to

colours. There is a `violent response con¯ ict’ .

paraphrase exercise, citation marks a piece of information as found’ , and a full reference caneasily be located (e.g. by the marker, or by a revising student at a later date) in the giventextbook (Gleitman, 1995). Giving a full reference is low priority as it adds no informationnot easily obtainable in the identi® ed source.

The central theme. After points 8 and 9 name and provide a citation for the Stroop effect,points 10± 15 describe Stroop’s experiment (see Table IV). Apart from points 12 and 13(points which were repeated in a colour ® gure in the textbook), the points tail away quitemarkedly, with some important points (i.e. 10, 14 and 15) scoring very low indeed. Again,this does not ® t into the information priority explanation. Perhaps some points are in someway more dif® cult to locate, remember and reproduceÐ even though all points have anexplicit and simple surface form.

Covariance of Point Scores

Finally, I looked at how points’ representation varied across paraphrases, and whether groupsof points might vary together. For example, in a group of paraphrases with a given pointcorrectly represented (e.g. point 12), would there be a tendency for certain other points (e.g.points 13, 15) also to be correctly represented? For this I used factor analysis. Factor analysislooks at how variables co-vary in this way, clustering together variables which do so intofactors. Each factor can then be regarded as an abstract variable in its own right, accountingfor a dimension of variability in the data. For example, in a personality questionnaire, a setof questions which co-vary might be interpreted as revealing an aspect of personality.

The main four factors returned are summarised in Table V. As the table shows, pointsdid cluster together into interesting groups. Points relating to the references and the citationsin the text were grouped together in factor 1; points relating to the description of theexperiment in factor 2. Factor 3 is slightly more complex: while the four points co-variedstrongly, points 3 and 4 (de® ning points in the introductory part of the source) variedinversely to points 16 and 17 (on overcoming the Stroop effect in the experiment); in otherwords, in paraphrases where points 16 and 17 were present, points 3 and 4 tended to beabsent or misrepresented, and vice versa. This may represent an exclusive focus on eithertextbook de® nitions or research ® ndings, or on earlier or later parts of the source text. Factor4 grouped points explicitly stipulated in the assignment instructions (i.e. to introduce theparaphrase as such, and give the Gleitman citation and reference).

Dow

nloa

ded

by [

Uni

vers

ity o

f C

hica

go L

ibra

ry]

at 0

4:43

05

Oct

ober

201

4

Assessing Paraphrase and Understanding 355

TABLE V. Point clustering

(a) Factor 1: SourcesPoint Information

22 Reference: Jensen, A.R. (1965)23 Reference: La Berge, D. (1975)24 Reference: Reisberg, D., Baron, J. & Kemler, D.G. (1980)25 Reference: Stroop, J.R. (1935)7 Citation: La Berge, 1975

18 Citation: Jensen, 196519 Citation: Reisberg et al., 1980

(b) Factor 2: DescriptionPoint Information

11 Experiment, condition 1: groups of letters are not words12 Experiment, condition 2: groups of letters are colour terms13 Experiment, condition 2: colour terms do not correspond to the colours in

which terms are printed14 Experiment, condition 2: subject’s responses signi® cantly slower than in

condition 115 Experiment, explanation of result: subjects are automatically reading words

as well as attending to colours

(c) Factor 3: De® nitions or ® ndingsPoint corr. Information

3 2 ve Automatisation, de® nition: routinisation of subcomponents ofskilled activities

4 2 ve Automatisation, bene® t: performance requires less attention16 1 ve Overcoming interference (a): blurring eyes reduces the effect17 1 ve Overcoming interference (b): subject can see colours but cannot

read words

(d) Factor 4: InstructionsPoint Information

1 Introduce paraphrase2 Citation: Gleitman, 1995

21 Reference: Gleitman, H. (1995)

Nine factors were returned with an Eigenvalue . 1, accounting for 68% of the variance.Of these there were four factors with an Eigenvalue . 1.8, accounting for 42% of thevariance; factor 4 accounting for 7.4%. Loadings reported in the table were all above 0.44.The tables show the points in numerical order rather than order of loading strength.

Paraphrases tended to be good or bad at the level of the factor: one paraphrase mightrepresent the experiment well but be poorly referenced; another might be well referenced,while stressing de® nitions over interpretations; and so on. Once factors have been de® ned,paraphrases (or personality questionnaires, etc.) can be given a numerical score for eachfactor. Note that the factors are not mutually exclusive, so it is possible for a paraphrase toexcel at all four.

Dow

nloa

ded

by [

Uni

vers

ity o

f C

hica

go L

ibra

ry]

at 0

4:43

05

Oct

ober

201

4

356 A. Uemlianin

Discussion

This study used a population of 163 students, compared with much smaller numbers in otherrecent studies of paraphrase (n 5 35 in Kirby & Pedwell, 1991; n 5 58 in Selinger, 1995).However, 163 is still a small sample and this, together with the fact that the study looked atonly one text with one population, means that any conclusions drawn from these results canbe made only tentatively. To discount the possibility that these results are artefactual, itwould be necessary to replicate the study with a number of different kinds of source text anddifferent populations. Nonetheless, the results are suggestive in a number of ways.

First, the task seems to have been worthwhile, in pedagogical terms: the skewed normaldistribution of marks shows that the exercise was non-trivial, probably involving more thatthe application of a single skill’ (Blalock, 1979). Informal feedback from students on theassignment was generally positive, that the assignment had made them think about para-phrase in a new way, and that the assessment had shown them how easy it is to miss outimportant points.

Second, the point-by-point analysis shows that the exercise, simple as it is, can be quitesensitive to variations in salience between the points. Information which played little or norole in the text’s main themes, and was perhaps regarded as low-priority information byreaders, showed up in the results as low-scoring points.

Third, the facts that Stroop’s experiment was so patchily represented in the paraphrases,and that one of the central points in the text was also one of the most weakly represented,both point to a need for fuller discussion with students on reading for the perception,selection and re-presentation of information, and on reading for understanding, with anemphasis on how the latter depends on the former. Too often paraphrase is characterised asregurgitation’ and opposed to understanding. However, the writing of an adequate para-phrase presupposes an adequate understanding (just as regurgitation presupposes somedigestion). Not only is paraphrase an important measure of (i.e. behavioural evidence for)understanding, it is an important element in a wide range of discourses: legal, political andphilosophical, not to mention everyday conversation. In scienti® c practice, a researchercombines and paraphrases research from the literature to provide the questions to whichhis/her own research will respond; the researcher is assessed by peers, not only on the qualityof the research itself, but also on the quality of his/her apparent understanding, as demon-strated in the paraphrase, of the discipline questions. If text’ is de® ned broadly as `signifyingmatter’ (e.g. Bakhtin, 1986), then the whole scienti® c enterprise is based on a kind ofparaphrase.

Finally, it is interesting that points seem to cluster into factors along informational lines.Excepting factor 4, which relates to the assignment rather than to the text, the factors seemto represent perceived `kinds’ of information, clustering together points relating to intertex-tual sources, narrative and interpretation. These students, paraphrasing this text, seem tohave responded differentially to information of different kinds. Without claiming that thesekinds are objective and irreducible properties of propositions in texts, the idea that there area number of well-de® ned roles which information may play in a text is re¯ ected in theseresults. If we extend the technique to other texts, other functions may be identi® able in thisway. The simple informational functions identi® ed here may be supplemented by, forexample, rhetorical and logical functions relating to the construction of arguments (Vickers,1988), and dialogic functions relating to intertextuality and the involvement of the reader.

This kind of textual analysis can be applied to produce detailed, structured feedback.The information reviewed in the results section could be given in a form useful to the student:either to describe their performance, the performance of the population in general, or the

Dow

nloa

ded

by [

Uni

vers

ity o

f C

hica

go L

ibra

ry]

at 0

4:43

05

Oct

ober

201

4

Assessing Paraphrase and Understanding 357

source text itself. For example, a discussion of the factors returned by analysis could be usedto broaden appreciation of the text-in-hand, and to improve approaches to reading in general;a student’ s numerical factor scores could be translated into commentary like, `your para-phrase reproduced the experiment accurately, but was weak on the interpretation of theresults’.

Conclusion

A rigorous de® nition of paraphrase based on propositional perception (Hunter, 1998) andinformation processing (Kellogg, 1995), and which avoids contrasting paraphrase withunderstanding or with self-expression, provided the basis for a study of `paraphrasing skills’of a small sample of undergraduate psychology students (n 5 163). The ® ndings include thefollowing.

· Paraphrase is not a trivial task, in two senses.

1. It is not a simple automatic process, and competence cannot be taken for granted.2. It is not secondary to, or separate from, more cognitively complex aspects of text

processing (e.g. reading, understanding, writing).

· Failure to paraphrase (i.e. missing or misrepresenting points) is not random but seems tobe systematic.

1. Students seem to be sensitive to importance of information, missing points which theyregard as less importantÐ to the passage or to their own purposes: the difference inrepresentation between citations and references is a special case of this.

2. A high proportion of paraphrases can omit or misrepresent a central point of theirsource text (in this study 72/163), and can weakly represent the passage’s central theme.

· Students seemed to perceive kinds of information in the source.

These ® ndings have relevance beyond academic psychology, and carry implications for allundergraduate teaching: from promoting an awareness of the usefulness of paraphrase and itsrelationship to understanding, through to enabling detailed general and individual feedback,and encouraging open discussion of the information structure of texts. The study was simpleto prepare and analyse and can be used as part of module assessment.

The study was, however, a small one and many of its ® ndings are perhaps artefactual.Using the technique with a wide range of texts and populations is necessary to differentiatespurious results from those which reveal a pedagogical, psychological or linguistic reality. Thetechnique certainly encourages interesting discussion of reading and understanding coursetextsÐ so experimental validation can proceed hand-in-hand with teaching innovation. Thetechnique can also be easily varied to be used with texts which include irrelevant or dubiousmaterial (e.g. eye-witness reports), texts which include hidden assumptions or logical errors,and so on.

Apart from simple validation, there are, of course, deeper questions about the techniqueand its approach to paraphrase, both practical and conceptual. For instance, checklists areuseful for short texts with simple internal structure, but for longer or more complex texts adifferent mapping might be more appropriate; however, I think the use of a comprehensive,objective representation of the source text with which to compare paraphrases is the correctapproach. On a more conceptual level, this approach treats a paraphrase as a kind ofinformational subset of the source text, which could perhaps be developed to include texts`behind’ the source text, on which it might draw or rely.

Dow

nloa

ded

by [

Uni

vers

ity o

f C

hica

go L

ibra

ry]

at 0

4:43

05

Oct

ober

201

4

358 A. Uemlianin

These are questions for further research. The purpose of the present study was to invitestudents and teachers to look again at paraphrase, and perhaps to re-examine their approach.The technique was intended to provide an assessment method which was itself a learningepisode, and to allow teachers and students to embark on a common enterpriseÐ researchinto paraphrase.

Acknowledgement

The author would like to thank the editors (past and present) and two anonymous refereesfor making valuable comments as a previous version of this paper.

Correspondence: Ivan A. Uemlianin, 6 Tai Seion, Llanddeiniolen, Caernarfon, GwyneddLL55 3AF, UK; e-mail: i.a.uemlianin@celtic.co.uk

REFERENCES

BAKHTIN, M.M. (1986) Speech Genres and Other Late Essays, trans. V.W. MCGEE (Austin, TX, University ofTexas Press).

BARROW, R. (1987) Skill talk, Journal for the Philosophy of Education, 21, pp. 187± 197.BLALOCK, H.M. (1979) Social Statistics (Auckland, NJ, McGraw-Hill).CHAMBERS, E. & NORTHEDGE, A. (1997) The Arts Good Study Guide (Buckingham, Open University Press).DAVIS, A. (1998) The limits of educational assessment, Journal of the Philosophy of Education, 32, whole issue.EHRLICH, E. (1987) A Dictionary of Latin Tags and Phrases (London, Robert Hale).GLEITMAN, H. (1995) Psychology, 4th edn (London, Norton).HUNTER, D. (1998) Understanding and belief, Philosophy and Phenomenological Research, 58, pp. 559± 580.KELLOGG, R.T. (1995) The Psychology of Writing (New York, Oxford University Press).KIRBY, J.R. & PEDWELL, D. (1991) Students’ approaches to summarisation, Educational Psychology, 11,

pp. 297± 307.MARSHALL, L. & ROWLAND, F. (1993) A Guide to Learning Independentl y (Buckingham, Open University

Press).QUINE, W.V.O. (1960) Word and Object (Cambridge, MA, MIT Press).SELINGER, B.M. (1995) Summarizing text: developmental students demonstrate a successful method, Journal

of Developmental Education, 19, pp. 14± 19.STEFANI, L.A.J. (1998) Assessment in partnership with learners, Assessment and Evaluation in Higher Education,

23, pp. 339± 351.UEMLIANIN, I.A. (1998) `Entertainment and education’ : a case study of a newsgroup as an on-line tutorial,

Computers in Psychology 98 Conference Proceedings (York, CTI Centre for Psychology).UEMLIANIN, I.A. & BEVERLEY, M. (1998a) ASI & IT questionnaires and students’ performance, Computers in

Psychology 98 Conference Proceedings (York, CTI Centre for Psychology).UEMLIANIN, I.A. & BEVERLEY, M. (1998b) A positive role for OMR forms in assessment, Computers in

Psychology 98 Conference Proceedings (York, CTI Centre for Psychology)VICKERS, B. (1988) In Defence of Rhetoric (Oxford, Clarendon Press).

Dow

nloa

ded

by [

Uni

vers

ity o

f C

hica

go L

ibra

ry]

at 0

4:43

05

Oct

ober

201

4